Why agentrial exists

AI agents are non-deterministic. Running a test once tells you nothing. We ran 5 agent archetypes 400 times each:

Every agent fails at a specific step. The Flaky Coding agent: 71% execution failures, 29% planning failures. Knowing WHERE it fails changes WHAT you fix.

Key Features

• Multi-trial execution — run N times (default 10), get confidence intervals instead of pass/fail
• Wilson score intervals — accurate even with small samples (N=10)
• Step-level failure attribution — Fisher exact test + Benjamini-Hochberg correction
• Cost per success — the metric that matters in production, not cost per call
• 6 framework adapters — LangGraph, CrewAI, AutoGen, Pydantic AI, OpenAI Agents, smolagents
• CI/CD integration — GitHub Action blocks PRs when reliability drops below threshold
• Snapshot testing — statistical regression detection across versions
• LLM-as-judge — calibrated with Krippendorff's alpha, rule-based fallback included
• MCP security scanner — prompt injection, tool shadowing detection
• Production monitoring — CUSUM + Page-Hinkley drift detection

Quick start

pip install agentrial
agentrial init
agentrial run --trials 100

Who it's for

Developers building AI agents with LangGraph, CrewAI, AutoGen, or similar frameworks who need to know if their agent still works after a prompt change, model swap, or code update — before it hits production.

MIT licensed. Everything runs locally. 349 tests, zero external dependencies required.