benni-inference-engine

benni-inference-engine (BIE) provides an efficient GPU+CPU hybrid inference engine for the Benni OS ecosystem, specifically designed to run advanced frontier Mixture-of-Experts (MoE) models seamlessly on consumer hardware. This sophisticated engine incorporates powerful features such as Skill 81 Ceiling Extraction, Quantized KV Cache, Speculative Decoding, and supports OpenAI-compatible serving modes, making it suitable for diverse applications.

Key Features:

• Hybrid Inference Runtime: Execute sophisticated AI models that require high-performance processing power available with a combination of both GPU and CPU resources.
• Skill 81 Ceiling Extraction: This feature addresses behavioral capability ceilings to ensure optimal model performance.
• Quantized KV Cache: Achieve over 70% VRAM savings, enabling more efficient model usage on limited hardware.
• Speculative Decoding: Enhanced efficiency in decoding speeds, crucial in high-memory demand scenarios.
• OpenAI-compatible API: Easily integrate with various existing frameworks and applications through a familiar API structure.

Architecture Overview:

The architecture of BIE is built on the premise of treating models not as black boxes but as cognitive systems that can be engineered. The core design principles include:

1. Dual-Model Sovereign Engine: Combines both primary and secondary models to achieve a unified processing capability.
2. Automatic Request Dispatch: Simplifies the user experience by eliminating manual configuration for different query types.
3. Multimodal Vision Engine: Enables visual understanding and processing directly without the need for cloud services.
4. Adaptive Memory Management: Implements dynamic layer allocation to cater to the usage needs according to the available hardware configuration.

Usage Example:

The benni-inference-engine can be used easily through CLI commands. Here’s a quick example of how to serve the model:

# Serve OpenAI-compatible API (default model)
bie serve --port 8080

Supported Models & Performance:

• The engine currently supports a range of models, including but not limited to Cyber-Tiel-Coder-35B and Gemma-4-26B, providing varying levels of performance based on VRAM and RAM usage profiles. Benchmarks show critical performance metrics that demonstrate the engine's ability to process tokens efficiently at higher speeds than traditional systems.

Project Structure:

The structure of BIE is organized to facilitate easy navigation and expansion. It includes scripts for core functionality, testing, models, and a user-friendly command line interface that allow advanced users to leverage the full capabilities of the engine.

Getting Involved:

As part of a broader Benni OS ecosystem, the benni-inference-engine is open for contributions and improvements. Explore the full range of modular components available to enhance your applications and make use of cutting-edge AI capabilities directly from your local environment.

For further information and to join the project, visit the GitHub repository and explore the documentation. Dive into a new era of hybrid inference with benni-inference-engine, designed for users looking to maximize the effectiveness of their AI models on consumer-grade hardware.