BOOTH

BOOTH: A Lightweight Checkpoint Library for LLM Outputs
BOOTH acts as an intermediary checkpoint between applications and Large Language Model (LLM) calls, providing structured decision-making capabilities about the LLM's outputs. This library is dedicated to evaluating whether a particular output should be accepted, reconsidered, marked as ambiguous, subjected to custom validation, or flagged as uncertain.

Note: BOOTH does not claim to know the absolute truth; it verifies if an output meets predefined acceptance criteria.

The concept of its name derives from a ticket booth or toll booth, which serves to verify that necessary conditions are met before allowing passage.

Current Status

Version: 0.4.3
BOOTH offers a variety of functionalities designed to enhance the communication and interpretation of LLM responses, including:

• Ambiguity detection
• Self-reported confidence checking
• Reconsideration retries for low-confidence answers
• Retry management for unparseable model responses
• An optional validator for custom rules on pass/fail checks
• Support for synchronous and asynchronous checkpoint functions
• Evidence-agreement checking against application-supplied data
• Configurable thresholds for confidence and evidence
• Structured result objects with detailed fields to track processing
• Attempt history tracking
• Status indicators like VERIFIED, REPAIRED, AMBIGUOUS, UNCERTAIN, and BLOCKED

Importantly, BOOTH is designed to be provider-agnostic, requiring no specific LLM provider, retrieval system, or framework.

Core Functionality

Example Usage

A standard LLM call might resemble this:

answer = call_llm(prompt)  

With BOOTH, a checkpoint can be introduced as follows:

import booth  

result = booth.check(  
    call_llm,  
    "What is the capital of France?"  
)  

if result.ok:  
    print(result.answer)  
else:  
    print(f"BOOTH returned {result.status}")  

BOOTH prompts the model to provide structured data about its response, which may look like this:

{  
  "ambiguous": false,  
  "interpretations": [],  
  "chosen_interpretation": null,  
  "answer": "Paris",  
  "confidence": 0.95  
}  

Key Features

• Ambiguity Detection: BOOTH enables the detection of questions with multiple valid interpretations. For instance, a query about the "capital of Georgia" could refer to either the nation or the U.S. state.

• Confidence Checking: The model's self-reported confidence is utilized as a key acceptance signal and can be configured with a minimum threshold.

• Reconsideration Retries: In cases where the answer lacks sufficient confidence, BOOTH can prompt the model to reassess its response and potentially provide a more reliable answer.

• Custom Validation: An optional validator function can be employed to enforce additional rules on the provided answer, allowing for a tailored check before a response is accepted.

• Evidence Agreement Checking: A dedicated function allows BOOTH to compare an answer against previously retrieved evidence, ensuring alignment between them.

Result Object Structure

The results from a BOOTH operation encapsulate important details, including:

• answer: The model's validated answer.
• status: The outcome's status (e.g., VERIFIED, AMBIGUOUS).
• confidence: The model's reported confidence level.
• all_parse_failed: Indicates if all attempts failed to produce a parseable output.
• parsed: The unmodified raw response from the model.

Conclusion

In summary, BOOTH is a flexible and lightweight checkpoint solution that enhances the handling of LLM outputs by enabling detailed checks for ambiguity, confidence, and custom validation. It serves as a vital component for applications utilizing LLMs, ensuring that only reliable responses are acted upon while maintaining transparency and control over the decision-making process.