tokf

tokf is a configuration-driven command-line interface (CLI) tool designed to optimize the output of various commands before it is processed by a language model (LLM). By reducing LLM context consumption by 60–90%, tokf enhances the efficiency of command outputs from tools like git, cargo, and docker. This tool intercepts verbose command outputs, applies filters based on TOML configuration files, and presents only essential information to the AI, ensuring a clearer and more concise communication.

Key Features

Streamlined Output Reduction

tokf drastically simplifies command outputs:

• Before and After: For example, running cargo test outputs 61 lines of verbose information, while tokf can reduce it to a single line:

  ✓ 47 passed (2.31s)
  

Command Filtering

Using tokf, any command can be filtered efficiently. Here are examples of its usage:

tokf run git push origin main
tokf run cargo test
tokf run docker build .

To test filters against specific outputs, use:

tokf test filters/git/push.toml tests/fixtures/git_push_success.txt --exit-code 0

Automatic Hook Installation

For users of AI coding tools, tokf offers the option to set up automatic filtering with a simple command:

tokf hook install --global

This feature eliminates the need to prepend commands with tokf each time.

Built-in Filter Library

tokf comes equipped with a library of built-in filters for common commands, such as git, cargo, and docker. Users can easily customize or create their own filters, adapting the tool to their specific needs.

Customization and Safety

The filter logic is flexible, allowing for straightforward customization through editing TOML files. Each filter can be independently tested and verified to ensure their expected behavior. Safety checks are also available to help identify issues like prompt or shell injection, enhancing security during use.

Conclusion

tokf is particularly beneficial for developers and teams leveraging AI technologies for coding assistance. It provides a powerful solution for decluttering command output, facilitating clearer communication with language models, and improving overall workflow efficiency.