KVBoost

KVBoost is a powerful KV-cache toolkit designed for enhancing the efficiency of local inference with Hugging Face causal language models (LMs). It provides significant performance improvements through innovative features tailored specifically for multi-turn conversations and complex agent loops. By implementing chunk-level key-value (KV) cache reuse, KVBoost minimizes re-encoding of previously seen text, allowing for faster response times during inference without requiring changes to the underlying model or tokenizer.

Key Features

• Chunk-Level KV Cache Reuse: This feature allows for the reuse of content-addressed KV cache across multiple requests. When prompts share a chunk-aligned prefix, the re-encoding work is skipped, effectively speeding up inference times.
• Custom FlashAttention-2 CUDA Kernel: KVBoost includes an optimized softmax kernel that enhances performance on devices equipped with NVIDIA GPUs.
• AWQ Layer Streaming: It allows larger models to run on GPUs with limited VRAM by streaming model weights as needed while retaining the essential layers in memory.
• Speculative Decoding: This capability enhances throughput by proposing multiple tokens that can be verified in a single forward pass, efficiently utilizing processing power.
• OpenAI-Compatible HTTP Server: KVBoost supports a drop-in server compatible with existing OpenAI SDKs, enabling seamless integration into applications without complex modifications.
• Telemetry and Performance Tracking: The toolkit offers comprehensive telemetry outputs to monitor performance metrics such as time-to-first-token (TTFT), KV reuse ratios, and more, aiding in application observability and optimization.

Quick Usage Example

KVBoost can be easily integrated into Python applications that use Hugging Face models. Below is a sample code demonstrating its application in a multi-turn chat session:

from kvboost import KVBoost
from transformers import AutoTokenizer

MODEL_ID = "Qwen/Qwen2.5-3B-Instruct"
SYSTEM_PROMPT = "You are a senior Python engineer. Be concise and show working code."

class ChatSession:
    def __init__(self, model_id: str = MODEL_ID):
        self.engine = KVBoost.from_pretrained(model_id)
        self.tokenizer = AutoTokenizer.from_pretrained(model_id)
        self.engine.warm(SYSTEM_PROMPT)
        self.history: list[dict] = [{"role": "system", "content": SYSTEM_PROMPT}]

    def send(self, user_msg: str, max_new_tokens: int = 256) -> str:
        self.history.append({"role": "user", "content": user_msg})
        prompt = self.tokenizer.apply_chat_template(
            self.history, tokenize=False, add_generation_prompt=True,
        )
        reply = self.engine.generate(prompt, max_new_tokens=max_new_tokens).output_text
        self.history.append({"role": "assistant", "content": reply})
        return reply

This example showcases how KVBoost maintains efficiency with minimal latency increase while managing the conversation's context.

By leveraging features like AWQ layer streaming, KVBoost supports operations on models that exceed available VRAM, leading to broader applicability on different hardware setups.

KVBoost stands out because it works natively with the existing Hugging Face ecosystem and does not require any complex model adaptations. This makes it an ideal solution for developers looking to optimize LLM inference in their applications.