rag debugger — Enhance RAG retrieval with effective…

Pitch

Description

Comments

About this project

rag-debugger provides insight into the RAG retrieval pipeline, identifying issues that could affect user experience. By analyzing selections from your vector DB and detecting knowledge gaps before users encounter them, it ensures that retrieval is not only accurate but comprehensive, leading to more reliable applications.

rag-debugger is a powerful tool designed to enhance the performance of RAG (Retrieval-Augmented Generation) retrieval pipelines by providing comprehensive insights into the retrieval process. It allows developers to intercept, inspect, and rectify issues that often lie within the retrieval of documents rather than the actual code.

Overview

In the world of RAG, many bugs stem from incorrect selection of document chunks, leading to unaddressed knowledge gaps that may only come to light when users encounter problems. rag-debugger addresses these concerns by offering visibility into the responses of your vector database, enabling quick identification of what is returned, the reasoning behind those selections, and highlighting knowledge that is missing from your information base.

Key Features

Gap Detection: Proactively identify gaps in your knowledge base before they are reported by users. This feature analyzes queries and shows missing information with recommended fixes, such as:

from rag_debugger import GeminiClient, GapDetector

client = GeminiClient()  # set GEMINI_API_KEY env var
detector = GapDetector(client)

chunks = your_retriever.get_relevant_documents(query)
report = detector.analyze(query, [{"content": c.page_content} for c in chunks])
print(report)

Sub-Intent Decomposition: Enhance query analysis by breaking down complex queries into simpler sub-intents. This approach ensures that even crucial components such as refund policies are detected, even if primary chunks perform well in isolation.
Session Grouping: Organize multi-turn conversations into a single session for an aggregated view of retrieval quality. This capability ensures thread safety during concurrent requests in web applications, allowing for:
```
with rd.session(id="conv-123", user="user-42") as s:
    retriever.get_relevant_documents("first query")
    retriever.get_relevant_documents("follow-up query")
summary = s.summary()
print(summary)
```
Local Dashboard: Visualize and monitor retrieval events through a local dashboard that displays essential metrics including average scores, worst scores, and detailed gap flags, with the information refreshing every 10 seconds:
```
rd.dashboard()  # opens http://localhost:7842
```

Seamless Integrations: Works flawlessly with popular frameworks such as LangChain, LlamaIndex, and custom retrieval pipelines, simplifying your architecture while enhancing functionality:

retriever = rd.wrap_retriever(vectorstore.as_retriever(), label="docs")  # LangChain
retriever = rd.wrap_retriever(index.as_retriever())  # LlamaIndex
retriever = rd.wrap_retriever(my_retriever, method="fetch_docs")  # Custom object

Backend Support

By default, rag-debugger utilizes Google Gemini for both LLM and embeddings, ensuring high-quality performance while remaining accessible via the free tier of Google AI Studio.

This tool is essential for any developer aiming to improve the robustness of their RAG applications by highlighting what knowledge is absent and making informed decisions on necessary improvements.