promptcheckup

promptcheckup

Evaluate the Impact of Prompt Changes on AI Agents
Enhance the performance of AI agents by validating prompt changes with confidence. The promptcheckup tool provides a rigorous pass/fail assessment for prompt modifications, enabling users to determine if a change significantly improves an agent's performance.

Key Features

• Paired Replay Validation: Tests both old and new prompts on identical inputs to eliminate variability, providing reliable results about their comparative effectiveness.
• Confidence Interval Insights: Each assessment includes a confidence interval at 95%, indicating the reliability of the verdict—whether a prompt is statistically better, worse, or inconclusive.
• Integrates with CI/CD: Built to run seamlessly in continuous integration environments without any external dependencies.

Usage Example

To quickly see how promptcheckup operates, the following commands can be executed:

# Clone the repository and navigate to the directory
git clone https://github.com/gael55x/promptcheckup && cd promptcheckup

# Run an example evaluation
PYTHONPATH=src python3 examples/planted/planted.py

This execution demonstrates comparing prompts:

Naive comparison (different weeks): B - A = -1.56  -> picks A  
Paired replay (same days): mean B - A = +0.36, 95% CI [+0.17, +0.55] -> better

In this instance, despite the naive comparison suggesting that prompt A was superior, the paired replay indicates that prompt B indeed performs better when evaluated on the same input conditions.

Why Choose promptcheckup?

Why not simply compare scores?

• Control Against Variability: Many factors can influence AI performance; promptcheckup mitigates these by conducting evaluations under consistent conditions.
• Minimizes False Positives: Reliably detects improvements while maintaining a conservative false positive rate, ensuring that inaccurate claims of improvement are minimized.

For instance, historical data has shown that 41.3% of naive comparisons may select the less effective prompt. In contrast, paired replay achieves over 91% accuracy in correctly identifying prompts' effectiveness.

Reporting Outcomes

Upon execution, results will indicate if the new prompt is indeed superior, reinforcing the confidence one can have in the adjustments made to an AI agent:

B is better: 28 of 30 cases improved (95% CI +0.78 to +0.99).

Additional Features

• Dynamic Agent Integration: Works with any program capable of reading from and writing to JSON through standard input and output, accommodating diverse frameworks and languages.
• Extensive Command Functionality: Evaluate weekly performance, propose edits for improvement, or conduct custom checks on self-defined conditions.
• Robust Testing Framework: Facilitates the creation of unit tests ensuring expected performance while avoiding common pitfalls such as lookahead biases in decision-making.

Learn More and Get Involved

Explore detailed documentation for setup, commands, and advanced configurations on the official GitHub repository: promptcheckup on GitHub. Contributions, issue reports, and feedback are encouraged to foster community improvements.