state-harness

State Harness is a powerful runtime safety net designed specifically for Large Language Model (LLM) agents. It aims to prevent unnecessary token spirals and associated costs, enabling precise diagnostics of why tasks fail. The library integrates seamlessly with your Python environment, leveraging a Rust core for performance and utilizing the principles of Lyapunov stability theory to identify and report runaway behaviors effortlessly.

Key Features

• Cost Efficiency: Saves substantial operational costs by detecting unstable tasks before they escalate. When failures occur, it provides clear reasons for the failure and actionable suggestions for improvement without making additional LLM calls.

• Zero-Diagnosis Costs: Get comprehensive diagnostics about incidents without incurring extra computational costs. Each failure is classified with direct patterns, evidence, and cost savings insights.

• Pattern Recognition: Captures various failure scenarios including:

  • Context Spiral: Detects when token usage grows disproportionately compared to a defined baseline.
  • Retry Storm: Identifies repeated calls that have low variance, indicating potential issues such as tool failures.
  • Policy Drift: Detects when the agent deviates from its expected conversational path.
  • Early Explosion: Triggers on excessive token usage within the first few interactions, often due to oversized prompts.
  • Budget Exhaustion: Alerts when token usage cumulates to the set budget limit.

Example Usage

The library offers simple integration with Python, allowing developers to implement stability checks efficiently. Below is a basic example leveraging the GrowthRatioGuard:

from state_harness import GrowthRatioGuard, FailureReport

guard = GrowthRatioGuard(token_budget=50_000)

with guard:
    for turn in agent_loop:
        result = llm.invoke(turn.prompt)
        guard.record_step(tokens_used=result.usage.total_tokens)

# Analyze what went wrong after execution
report = FailureReport.from_guard(guard)
print(report)

Who Can Benefit from State Harness?

• Development Teams building search-tree agents or handling large numbers of LLM tasks can benefit from automated diagnostics and failure pattern classification, ultimately improving operational efficiency.
• Platform Teams that manage substantial agent workloads daily can implement this tool to minimize manual tracing efforts and enhance error detection.
• Researchers aiming to benchmark agent performance can leverage the diagnostics provided by State Harness to better understand task failures and refine their models.

Importance of Diagnostics

A budget cap may signal that a task was terminated, but State Harness offers in-depth diagnostics to pinpoint exactly why. Such insights into failures light the path toward improved agent design and reduced costs over time.

Getting Started

Installing State Harness is straightforward:

pip install state-harness

With its simple setup, anyone can benefit from enhanced efficiency and diagnostics in their LLM applications. State Harness caters well to high-demand environments where understanding task dynamics plays a vital role in performance outcomes.