goldset

goldset is an innovative tool designed to transform production traffic into a versioned, deduplicated, and coverage-tracked regression test suite, enhancing the quality of testing for language models. Unlike traditional gold standard generation methods, goldset emphasizes a crucial human confirmation step to ensure that past model errors do not become part of the benchmark.

Key Features

• Curation Accelerator: Streamlines the process of creating a golden test suite from real production data, allowing for quick validation of known issues while maintaining a focus on accuracy.
• Candidate Management: Proposes candidates for testing based on actual model outputs and failures, but requires human verification to confirm expected outcomes before integration into the golden set.
• Built-In Coverage Tracking: Automatically tracks coverage against observed traffic and allows external taxonomy specifications to measure against established benchmarks.
• Robust Workflow: The tool offers a structured workflow for managing JSON workspaces, enabling easy resumption and detailed tracking of every step, from ingestion of logs to export of the golden suite.

Example Workflow

To utilize goldset, the following commands can be executed:

# Read logs and redact PII before storage
goldset ingest ./logs.jsonl --out ws.json
# Cluster inputs into candidate intents
goldset cluster --workspace ws.json
# Deduplicate and select candidates
goldset select --workspace ws.json
# Confirm expected outcomes for each case
goldset label --workspace ws.json
# Lock the developmental splits
goldset roles --workspace ws.json --lock
# Export the golden suite
goldset export --workspace ws.json --out suite.yaml

A case is deemed golden only after it has been confirmed by a human or trusted oracle and carries a set of assertions. These assertions can range from mentioning necessary terms to determining the state of the system.

Input Format

The input log must be in JSON format, typically containing an entry for each interaction which includes the input, model output, and various associated signals for error tracking. An example input format is as follows:

{"input": "how do I cancel my subscription?", "output": "Go to settings to cancel.", "signals": {"thumbs_down": true, "escalated": false}, "timestamp": 1723000000}

Use cases & validation

goldset’s capability to produce valid test suites has been validated using real-world datasets, such as the PII-safe Bitext customer-support corpus. The clustering and assessment results reaffirm the tool's effectiveness in deriving both intent and coverage against defined specifications.

For developers and researchers, goldset opens up pathways to enhance the integrity of LLM testing, ensuring that auto-accepted outputs do not compromise overall model performance. Check out the live demo report to see how well goldset captures and validates known issues and improvements.