Narwhal

Narwhal is an adaptive, disaggregated inference framework for vLLM. It hot-swaps engines between prefill and decode roles as demand changes, without reloading model weights. It runs on NVIDIA and AMD GPUs and scales from a single GPU to multi-node fleets.

Key Features

• Role hot-swap: Reassigns prefill and decode roles across a fixed GPU fleet, with NIXL key-value (KV) transfer between them.
• Serving: Serves completion and chat requests, streamed or buffered, with latency-aware admission.
• Fault tolerance: Fails over to a warm-standby router and readmits engines against their live process generation.
• Measurement: Profiles each engine's prefill and decode cost model, runs ordered benchmark points, and keeps the evidence from each point.
• Observability: Exports router and engine metrics to Prometheus, with a provisioned Grafana dashboard.
• Operator tooling: Validates fleet files offline and collects private diagnostic bundles.
• Development mode: Runs two to eight engines on one NVIDIA CUDA GPU under Ubuntu or WSL2.

How Engines Change Roles

The role controller scores the current prefill/decode split and each adjacent split one engine move away. A split's score is its worst projected SLO ratio across time to first token (TTFT), time per output token (TPOT), and decode queueing. The controller moves only when an adjacent split improves the score by at least a configured margin. New requests follow the revised split, while resident requests finish on their assigned engines.

Evaluation

On Kimi-K3 with six TP8 engines, Narwhal was compared against Dynamo Planner and Ray Serve LLM on identical hardware, using AIPerf to replay a fixed schedule of rapidly changing workloads. With prefix caching on, Narwhal delivered 76.18% of chat/document requests within the latency SLO, against 56.76% for Dynamo Planner and 51.66% for Ray Serve LLM, and had the highest completion and SLO-qualified rates on both workloads. The main trade-off was an 11-second p95 TTFT tail on mixed traffic. Read the full evaluation.

Getting Started

Install on Linux with Python 3.11+:

Try it on one GPU with narwhal dev init and narwhal dev up, or follow the gated deployment guide to bring up a multi-node fleet.

Built on Arrow

Narwhal's scheduling algorithms derive from Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture by Wu et al. (2025).

Narwhal is open source under Apache-2.0. See the documentation and GitHub repo.