ai_glue

ai_glue provides robust AI observability and governance for applications utilizing OpenAI and Anthropic models. It serves as a transparent layer that not only enhances the governance of AI interactions but also delivers valuable insights into usage and spending.

Key Features

• Comprehensive Logging: Every AI call is meticulously logged, capturing crucial information such as provider, model, tokens, cost, latency, project, and session.
• Cost Management: Gain visibility into spending across different projects, teams, and environments, facilitating better financial oversight.
• PII Detection: Implements per-call flag review workflows for personally identifiable information (PII) detection, ensuring compliance and data protection.
• Governance Controls: Enforce stringent governance rules that block unapproved models, enforce cost caps, and set rate limits to maintain control over AI usage.
• Unified Dashboard: Aggregate data from various instances (development, staging, production) into one user-friendly dashboard, providing tailored views for engineering teams or executive leadership.
• Seamless Integration: Add governance capabilities to existing applications with just a single change to an environment variable, eliminating the need for extensive code modifications.

Understanding ai_glue

ai_glue does not act as a replacement for your existing AI framework. Instead, it integrates with current applications and model providers to enhance governance, auditing, spending visibility, and policy enforcement. Essentially, it functions as a transparent proxy equipped with a comprehensive dashboard.

Deployment Options

• Self-hosted Evaluation: Users can clone the repository, configure their environment, and operate it independently.
• Team Deployment: An IT-managed instance allows individual developers to integrate governance without directly handling API keys, enhancing security and simplifying oversight.

Integration Modes

1. Proxy Mode: Existing applications can point to ai_glue as a proxy server, ensuring that all AI calls are logged and governance-checked without any alteration to the application code.
2. Wrapper Mode: For new applications, a simple wrapper class enables effortless integration while maintaining existing APIs for both Anthropic and OpenAI models.

Governance and Configuration

Users can easily modify governance rules using the governance.yaml file, which applies changes immediately without requiring a server restart. Key configurations include model allowlists, PII detection triggers, and individual project spend caps.

Data Management and Exporting

ai_glue not only logs interactions but makes it simple to export conversation logs in formats suitable for fine-tuning and analysis, ensuring that data continuity is maintained even if models are deprecated.

Example Export Paths

Export individual sessions or entire project conversations for analysis, enabling tighter control over training data gathered through these engagements.

Monitoring and Analytics

The dashboard provides various views to cater to different roles:

• Audit View: Engineers can drill down into the full call log, including daily metrics and flagged PII instances.
• Executive View: Leadership receives an overview of total AI spending, active projects, and potential risks, devoid of technical jargon.
• Aggregate View: Side-by-side comparisons of activity across development, staging, and production environments with detailed insights.

Security and Compliance

With a focus on security, governance configurations are treated with care, limiting access and ensuring sensitive information remains protected. The system's logging flexibility supports compliance needs by providing a detailed account of all interactions, while PII filtering safeguards against sensitive data exposure.