CRoM-Context-Rot-Mitigation--EfficientLLM

CRoM (Context Rot Mitigation)-EfficientLLM is a robust Python toolkit crafted for optimizing the context provided to Large Language Models (LLMs). This innovative toolkit enables developers to effectively select, re-rank, and manage text chunks, ensuring compatibility with a model's context budget while maximizing relevance and minimizing performance drift.

Ideal for developers engaged in RAG (Retrieval-Augmented Generation) pipelines, CRoM enhances the usability of limited context windows, empowering users to achieve efficient and effective interactions with language models.

Key Features

• Budget Packer: Efficiently consolidates high-scoring text chunks within a defined token budget utilizing a stable sorting algorithm.
• Hybrid Reranker: Leverages both sparse (TF-IDF) and dense (Sentence-Transformers) retrieval scores, offering robust document reranking for superior quality.
• Drift Estimator: Tracks semantic drift between consecutive model responses using L2 or cosine distances coupled with Exponentially Weighted Moving Average (EWMA) smoothing.
• Observability: Integrates Prometheus metrics for monitoring token savings, alongside drift alerts in production environments.
• Extensible Plugins: Facilitates optional plugins for advanced reranking (FlashRank), compression (LLMLingua), and drift analysis (Evidently).
• Comprehensive Benchmarking: Features a command-line interface (CLI) for thorough end-to-end pipeline evaluation, budget sweeps, and quality-versus-optimal analysis.

Quickstart

Interactive Demo

Experience CRoM live without any installation:
🎯 Try CRoM Demo on Hugging Face Spaces
The interactive demo allows users to:

• Experiment with budget packing algorithms
• Visualize context drift detection
• Observe real-time performance metrics
• Adjust token budgets and witness immediate results

Example CLI Benchmarking

For evaluation, the package includes a powerful crom-bench CLI:

# Default End-to-End (Search→Rerank→Pack→Mock LLM)
crom-bench e2e --budget 0.3

# Optional: High-precision configuration with plugins
crom-bench e2e --budget 0.3 \
  --use-flashrank --flashrank-model ms-marco-TinyBERT-L-2-v2 \
  --use-llmlingua --compress-ratio=0.6 \
  --use-evidently

Freely extend the capabilities of LLMs with CRoM, facilitating more relevant responses within the constraints of context budgets.