BOOTH

A few months ago, a person working an airport gate helped me find mine after I'd gotten lost, boarding pass in hand, signs everywhere, still at the wrong gate. She didn't know where I'd flown in from, how the aircraft was built, or anything about my itinerary beyond that one boarding pass. She didn't need to. She knew the airport, and she knew what to check.

That's the idea BOOTH is built around. The industry's default answer to "the model got it wrong" has mostly been to make the model know more: more context, more compute, bigger models, more tools bolted around it. And the problem still happens, because it isn't always a knowledge problem. Sometimes what's missing isn't more information going in. It's something whose only job is checking whether what came out actually meets the bar.

BOOTH doesn't try to know more than your model does. It just knows what to check.

A real example, not a hypothetical Same model, same question, asked with and without BOOTH. This is an actual comparison run, not a constructed demo.

Answer Without BOOTH "90 days" (fabricated, no such number exists in the actual policy) With booth.check_with_evidence() "45 days" (matches the real policy document exactly) No amount of self-reported confidence catches that first answer. Only checking it against something real does.

result = booth.check_with_evidence( answer=llm_answer, evidence=retrieved_docs, compare_fn=your_comparison_function, )

if result.status == booth.BLOCKED: print("The answer doesn't agree with what was actually retrieved.") (The specific fabricated number will vary from run to run, that's just uncalibrated sampling. The pattern is the reliable part.)