Mixture of Scars

Mixture of Scars (MoS) & SCache (\mathcal{O}(1)): A Breakthrough in Transformer Efficiency

Overview: Mixture of Scars (MoS) and SCache aim to address the dual memory limitations faced by autoregressive Transformers ranging from 0.5B to 72.71B parameters. This project presents an innovative solution to the problems of static parameter bloat and the increasing resource demands of traditional memory techniques.

Key Innovations:

1. Mixture of Scars (MoS): This technique combines canonical Block-64 INT4 weights with Activation-Aware SVD (ASVD) based low-rank scar experts, facilitating efficient training and inference. It utilizes streaming shards to enhance the speed and efficiency of GPU computations, achieving a significant memory reduction and speed improvement across multiple architectures, including Apple Silicon and NVIDIA CUDA Triton.

   • Performance Metrics: The MoS approach has demonstrated a dramatic reduction in memory usage (from 135.43 GiB to 38.12 GiB) at 72.71B, with reports indicating up to 195,720× reduction in outliers when compared to standard implementations, ensuring 100% top-5 agreement on accuracy.

2. SCache (Dynamic Cache): This feature acts as a drop-in solution for HuggingFace Transformers, bringing a constant (\mathcal{O}(1)) memory consumption model which leverages a unique FIFO mechanism and advanced memory management techniques. The SCache system efficiently manages memory by capturing context and utilizing fast weights without the traditional bloat associated with larger Transformers.

   • Cost Efficiency: For instance, in tests, SCache showed a 25.7× reduction in memory requirements while simultaneously improving processing speed compared to standard KV caches.

Implementation Examples:

A brief overview of how to utilize these systems effectively:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from scar_cache import SCacheDynamicCache

model_id = "Qwen/Qwen2.5-3B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")

# Establishing a constant O(1) cache
cache = SCacheDynamicCache(window_size=256, num_sink_tokens=4, num_slots=256)

inputs = tokenizer("Your 32k+ token context here...", return_tensors="pt").to(model.device)
out = model.generate(**inputs, past_key_values=cache, max_new_tokens=64)

Further Reading:

For detailed research, performance benchmarks, and theoretical foundations, the full documentation is available:

• Official Preprint: Read the full preprint (v1.3.0)
• Technical Details and Roadmap: Dive into the technical aspects within the documentation outlining the future developments and applications of MoS and SCache.

This project strives to enable more efficient and scalable AI solutions, empowering applications that were previously constrained by the size and complexity of language models.