tokencap

tokencap is a powerful Python library designed to enable effective token budget enforcement and usage tracking for AI agents. This tool simplifies the management of token consumption across various AI frameworks, offering seamless integration with providers such as Anthropic and OpenAI.

Key Features

• Comprehensive Budget Control: Enforce token limits on a per-session, per-tenant, or per-pipeline basis. When budgets are exceeded, tokencap proactively blocks further calls to prevent unexpected costs.

• Flexible Integration: tokencap can wrap the API clients directly or integrate with popular agent frameworks like LangChain and CrewAI using a simple patch method. Here’s how to use it:

  import tokencap
  import anthropic
  
  # Direct SDK use
  client = tokencap.wrap(anthropic.Anthropic(), limit=50_000)
  
  # For frameworks like LangChain
  tokencap.patch(limit=50_000)
  

• Real-time Monitoring: Keep track of token usage with straightforward status checking, allowing for immediate insight into consumption metrics and budget status.

  status = client.get_status()
  print(f"Used tokens: {status.dimensions['session'].used} / {status.dimensions['session'].limit} tokens")
  

• Actionable Policies: Customize budget enforcement through policies that trigger warnings, degrade model usage to less expensive alternatives, or block further API calls entirely once thresholds are met.

  tokencap.patch(policy=my_policy)
  

• No Infrastructure Required: Operate with zero infrastructure needed as the library runs directly in your process.

• OpenTelemetry Support: Integrate effortlessly with OpenTelemetry for detailed metrics emission, enhancing observability and monitoring capabilities.

The Problem Addressed

In the dynamic landscape of AI, runaway costs from unregulated agent behavior can lead to hefty bills. Traditional budgeting tools often rely on financial metrics that can fluctuate with provider pricing models, making manual tracking difficult. tokencap addresses this by focusing on token counts, providing a consistent and clear method for budget enforcement across varying models and pricing tiers.

Use Cases

• Control costs for multi-tenant applications by implementing strict token usage rules across all clients.
• Safeguard against unintended expenses in research environments where agents may inadvertently enter retry loops.
• Ensure compliance in production systems where project budgets require stringent adherence to spending limits.

In conclusion, tokencap provides a reliable solution for maintaining budget discipline within AI applications, offering essential tools for cost management and operational efficiency without the need for additional infrastructure.