Retrieval Lab

Retrieval Lab is an advanced benchmarking tool designed for evaluating Retrieval-Augmented Generation (RAG) retrieval configurations on custom datasets. This tool not only identifies the best-performing configurations but also provides comprehensive insights into the reasons behind failed queries.

Key Features

Retrieval Lab enables detailed comparisons across various parameters:

• Chunking Strategies: Explore fixed-size, recursive, semantic, and parent-child chunking methods.
• Embedding Techniques: Utilize local models such as E5 and BGE, or a deterministic offline test embedder for reliable outcomes.
• Retrieval Methods: Assess performance using exact dense, BM25 sparse, or hybrid retrieval approaches with reciprocal rank fusion.
• Reranking Options: Choose from no reranking, lexical overlap reranking, or a cross-encoder based on sentence-transformers.
• Customizable Cutoffs: Adjust candidate counts, final top-k results, and context-token budget settings for precise control over retrieval efficacy.
• Indexing: Leverage both exact search and approximate nearest-neighbor search using HNSW for optimizing retrieval performance on large datasets.

Each execution of Retrieval Lab generates a self-contained HTML report that summarizes the results, including ranked configurations, confidence intervals, cost measurements, and detailed explanations of failure attribution through various stages:

representation → ann_index → candidate_generation → fusion → reranker_demotion → final_cutoff → budget_cutoff

These stages help clarify where evidence is lost in the retrieval pipeline, ensuring a nuanced understanding of model performance.

Example Benchmark Report

Users can refer to a live benchmark report based on a synthetic corpus containing 22 documents and 400 labeled queries: View Benchmark Report. This showcase illustrates the output of Retrieval Lab, empowering users to replicate results on their datasets.

Quick Start Instructions

To quickly get started with the tool, an offline demo can be used, requiring only:

pip install retrieval-lab
retrieval-lab demo

For more robust testing using actual models:

pip install "retrieval-lab[real-embed,rerank]"

retrieval-lab run \
  --corpus docs.jsonl \
  --queries queries.jsonl \
  --embed-models e5,bge \
  --chunkers fixed:200,fixed:400,recursive:400,parentchild:800x200 \
  --retrieval dense,sparse,hybrid \
  --rerank none,ce \
  --candidate-n 15 \
  --top-k 5 \
  --measure-latency \
  --json results.json \
  --html report.html

The generated report.html can be opened directly in any browser, with full accessibility regardless of network connectivity.

Input Data Requirements

Retrieval Lab requires two JSONL files for execution:

• docs.jsonl: Contains one source document per line, detailing the document ID and text.
• queries.jsonl: Comprises labeled queries with corresponding gold answers to evaluate model performance.

Automation and CI Integration

Retrieval Lab supports continuous integration by allowing users to set quality gates using the --fail-under option to ensure results meet specific thresholds:

retrieval-lab run \
  --corpus docs.jsonl \
  --queries queries.jsonl \
  --fail-under 0.80 \
  --json results.json

With its ability to elucidate the intricacies of retrieval failures and optimization strategies, Retrieval Lab stands as a focused solution for researchers and practitioners aiming to enhance RAG systems through meticulous benchmarking.