Micro-Expert-Router is a Rust execution engine designed for Mixture-of-Experts models, optimizing performance by keeping routers in RAM and enabling on-demand expert loading from NVMe SSDs. This innovative approach leverages high-speed PCIe connections and quantisation techniques to significantly enhance model efficiency on affordable hardware.
The Micro-Expert-Router (MER) is a cutting-edge Rust execution engine designed specifically for Mixture-of-Experts (MoE) models. This innovative project leverages the power of PCIe-attached NVMe drives to enhance model performance by hot-swapping individual experts on demand, efficiently utilizing resources for optimal performance.
O_DIRECT positional reads to bypass the kernel’s page cache, offering a streamlined approach to data handling by exploiting the SSD’s high sequential read speeds (6-14 GB/s).The architecture employs a multi-layer cache system that ensures that only frequently used experts remain in memory:
Cache Management: It integrates a predictive caching mechanism using a blend of sparse Markov chains, a Locality Monitor, and a Neural Speculator to efficiently manage access to expert models. This setup guarantees that the most relevant experts are always ready for retrieval from storage.
Modular Design: The Rust crate organizes functional components into single-responsibility modules, which include the router for directing data flow, buffer management for handling I/O operations, and specialized modules for predictive analysis.
An example workflow for expert activation includes:
+------------+ +-------------+ +-----------+ +-------------------+
Token → | Router | → | Expert IDs | → | LRU Cache | → | SwiGLU FFN |
| LinearGate | | e.g. [3,7] | +-----+-----+ | per expert, |
| or Markov | +------+-------+ | Miss | gate-weighted sum |
+-----+------+ | ↓ +-------------------+
│ │ +------------------+
│ Hidden State │ | BufferPool slot | ←─────┐
↓ │ | (Aligned, Pre- | │
+------------------------+ │ | Allocated) | │
| Predictive Controller | │ +--------+---------+ │
| S = 2nd-order Markov | │ ↓ │
| L = LocalityMonitor | → │ +------------------+ │ On Arc drop
| M = NeuralSpeculator | │ | pread(2) read | │
| E = S ∪ L ∪ M | │ | O_DIRECT + (Opt)| │
+-----------+------------+ │ +--------+---------+ │
│ │ ↑ │
↓ ↓ | │
Non-evicting Prefetches NVMe SSD → DMA → RAM ──────┘
↓
Bytes reinterpreted as weights → matmul
This process optimally retrieves and manages the data necessary for running multiple expert models, ensuring that latency is minimized while processing throughput remains high.
The Micro-Expert-Router is a powerful solution for managing and executing Mixture-of-Experts models, providing substantial improvements in data handling and processing efficiency. By integrating refined storage utilization methods with innovative data management strategies, it empowers users to leverage larger and more complex models than ever before.
No comments yet.
Sign in to be the first to comment.