Covenant

Covenant is a robust, self-hosted AI gateway designed to enhance the security of language model (LLM) interactions. By rigorously hardening each request sent to models and every response delivered back to the client, Covenant ensures a secure and reliable API experience.

Key Features:

• OpenAI-Compatible API: Easily integrates with OpenAI, Anthropic, or custom Ollama instances through a drop-in replacement that requires no code modifications in existing applications.

• Advanced Prompt Injection Detection: Utilizing a three-tier security system, Covenant effectively mitigates prompt injection risks. The detection pipeline includes:

  • Tier 1: A regex-based pattern guard that operates under 1ms.
  • Tier 2: An ML guard using the DeBERTa-v3 model for ambiguous prompts, providing around 100ms detection time.
  • Tier 3: An async LLM judge that increases recall to 63.5% without introducing latency.

• Semantic Response Caching: Features a semantic cache utilizing FAISS for efficient storage and retrieval based on cosine similarity, ensuring fast responses and improved performance.

• Behavioral Contracts: Implements behavioral rules on all requests and responses across nine contract types, enabling predefined checks to bolster content integrity and security.

• Provider Routing: Facilitates seamless switching between AI providers with unified request formatting. Choose between cost-efficiency, latency optimization, or explicit routing policies.

• Robust Rate Limiting: Built-in rate limiting mechanisms using NGINX and Redis prevent abuse and ensure fair resource distribution among API users.

• Comprehensive Observability: Integrates with Langfuse for distributed tracing and offers Prometheus metrics for monitoring system health and performance.

Architecture:

Each request flows through a ten-stage processing pipeline:

Client → NGINX → FastAPI Gateway
  1. Rate Limiter
  2. Pattern Guard
  3. Cache Lookup
  4. ML Guard
  5. Pre-call Contracts
  6. Provider Router
  7. Provider Adapter
  8. Post-call Contracts
  9. Cache Write
  10. Langfuse Trace

API:

Covenant provides an OpenAI-compatible API at the endpoint /v1/chat/completions, where you can also define an app_id to utilize behavioral contracts for added security. Other endpoints include health checks and contract monitoring functionalities.

For further insights into leveraging Covenant for enhanced language model interactions, refer to the official blog: covenant blog.

Covenant stands out as a pivotal solution for developers and organizations looking to fortify their AI applications while maintaining compatibility with existing interfaces.