muteval

muteval is a powerful mutation testing tool designed specifically for evaluating large language model (LLM) assessments. By intentionally degrading the system being tested, muteval probes whether your evaluation suites can detect regressions effectively.

Key Features

• Mutation Testing: Unlike conventional tools that focus solely on system performance, muteval addresses the efficacy of your evals. It systematically degrades the system under test and reruns existing eval suites against each degraded version, generating a mutation score that quantifies how many injected regressions were caught. This score highlights survivors—the regressions that went undetected, providing valuable insights into potential gaps in coverage.

• Configurable Configurations: Users can easily create a configuration file that outlines the evaluation setup. This includes specifying prompts, cases, and defining custom eval checks to ensure their existing pipelines are robust.

from muteval import MutEvalConfig, checks

config = MutEvalConfig(
    prompt="YOUR_SYSTEM_PROMPT",
    cases=[{"input": "...", "order_id": "A123"}],
    run=my_run_fn,
    evals=[
        checks.contains_case("order_id"),
        checks.grounded("context"),
    ],
)

• Robust Reporting: After running tests, muteval provides clear analytics including mutation scores and severity rankings of any detected survivors. This allows users to focus on gaps that genuinely require attention.

• Audit Capability: Beyond mutation scoring, muteval offers muteval probe which audits eval quality through metrics such as judge reliability and statistical adequacy, ensuring that assessments are not only effective but also trustworthy.

Why muteval Matters

Traditional regression tools may alert you to problems within a system's performance, but they often overlook whether the evaluation strategies utilized are sufficiently sensitive. muteval fills this gap by applying established mutation testing principles to LLM evaluations—testing the tests themselves.

How to Use

Starting with muteval is straightforward:

1. Initialize: Create a config placeholder using muteval init.
2. Check: Validate your setup with muteval check to ensure everything is wired correctly before running tests.
3. Run: Execute your tests with muteval run, monitoring for mutation scores and survivors.

Sample Outputs

After a typical run, users receive detailed outputs, indicating mutation scores and listing survivors. Each survivor is coupled with a suggested eval to address the regression, showcasing an intuitive approach to enhancing the robustness of evaluation suites.

• Example Output:

  Mutation score: 33%  [████████░░░░░░░░░░░░░░░░]  (2/6 mutants killed)
  2 SURVIVED  (output changed but evals didn't notice):
  - [HIGH] SURVIVED  [delete_sentences]
      ```
  

• Coverage Gap Tracking: With tools to assess and track mutant behavior, users can refine their evaluation tactics for more reliable assessments.

For more detailed instructions and a thorough understanding of how to implement muteval, please consult the official documentation at muteval documentation.

By adopting muteval, test suites evolve, leading to more reliable evaluations of large language models, ultimately ensuring that they meet expected performance standards reliably.