ctxdiff: A Powerful Local Debugger for LLM Context Windows

ctxdiff serves as a local-first debugging tool designed for managing the context window of Large Language Model (LLM) agents. This unique tool provides a way to see precisely what your LLM saw during each interaction, making it easier to diagnose issues, optimize performance, and ensure stability.

Value Proposition

Every LLM call's context is meticulously recorded, allowing users to investigate their agents' behavior on a turn-by-turn basis. By leveraging this insight, users can ascertain what content affected their prompt, evaluate changes, and identify potential issues, such as what caused the agent to misbehave in the first place.

Key Features

• Context Engineering: ctxdiff helps clarify that context is critical to performance, capturing every detail of what the model sees, in what order, and at what cost.

• Cross-Platform Compatibility: Supports both Python and JavaScript/TypeScript, ensuring that the .ctrace format is uniform and accessible across different programming languages.

• Git-Style Turn Diffing: Analyze changes in the context window meticulously with features akin to git diff, allowing users to observe what was added, evicted, or modified character by character.

• Token Attribution: Provides clarity on where token budgets are allocated per turn, helping users discern how many tokens were spent on system prompts, user messages, and more. It even detects unused tools that incur additional costs without providing any benefit.

• Prompt-Cache Profiling: Monitors when cache prefixes break, detailing the specific modifications that triggered these changes and estimating the impact on token usage.

• Multi-Agent Support: Allows tracking of multiple LLM agents in a single run, attributing calls to individual agents for a comprehensive overview of interactions and support handoffs.

• Self-Contained HTML Dashboard: The CLI facilitates a one-file dashboard that visualizes agent interactions, complete with session breakdowns and contextual analysis tools, which is safe to share in debug tickets.

Use Cases

• Identify the exact changes that caused an agent to fail to respond correctly.
• Optimize usage by analyzing token spending per turn and identifying dead schemas that could be removed for efficiency.
• Make informed updates to prompt designs, minimizing unwanted behaviors in agent responses by refining what the model sees at each turn.

Quick Start Example

To initiate a trace and record context information, only a few lines of code are necessary:

from ctxdiff import trace  
from openai import OpenAI  
  
tracer = trace.init("customer-support-agent")  
client = tracer.wrap(OpenAI())  
client.chat.completions.create(  
    model="gpt-4o",  
    messages=[{"role": "user", "content": "What's your refund window?"}],  
)  
tracer.close()  

This captures context information into a .ctrace file, which can be further analyzed with ctxdiff view to visualize agent actions and performance.

For more comprehensive guides, features, and examples, please refer to the ctxdiff documentation.

By integrating ctxdiff, LLM developers and researchers can ensure their models operate within optimal parameters, thus enhancing utility while minimizing unintended responses.