Optimized deployment stack for vLLM on NVIDIA Blackwell (RTX 5090) utilizing Linux Kernel 6.14. This repository offers a ready-to-use solution that resolves compatibility issues between the Blackwell architecture (SM_120) and the latest kernel, ensuring high-performance inference for Large Language Models (LLMs).

Key Features

• Enhanced Compatibility: This stack effectively addresses common Flash-Attention symbol errors encountered on NVIDIA Blackwell, providing a seamless experience.
• Optimized Performance: Achieves impressive throughput of 59.0 tokens/s using the DeepSeek-R1-32B-AWQ model configuration on dual RTX 5090 setups.
• Robust Memory Management: Implements advanced techniques to mitigate memory fragmentation and peer-to-peer (P2P) deadlocks which are prevalent in standard deployments, thus enhancing model stability.

Technical Insights

1. Integrating Linux Kernel 6.14 with Blackwell

This deployment fully embraces the NCCL_DMABUF_ENABLE=1 setting which utilizes the native Linux DMA-BUF subsystem for effective memory handling. This transition not only stabilizes P2P communications but also minimizes overhead associated with older modules like nvidia_peermem.

2. Innovative Attention Mechanics

The implementation of FlashInfer serves as a replacement for the deprecated Flash-Attention library. This switch effectively alleviates undefined symbol errors, allowing the optimized inference engine to operate smoothly on cutting-edge hardware.

3. Fine-Tuning for Maximized Throughput

The stack employs memory allocation configurations such as optimized PYTORCH_ALLOC_CONF, tailored for maximum performance on the new kernel, preventing VRAM fragmentation, and ensuring efficient resource utilization.

Performance Metrics

• Throughput: Approximately 59.0 tokens/s on DeepSeek-R1-32B.
• Prefix Cache Hit Rate: 44.4%, contributing to lower latencies in repetitive queries.
• KV Cache Utilization: Kept at 1.2%, offering substantial headroom for high-concurrency tasks with extended context requirements.

Engineering Excellence

Developed through a meticulous iterative process, this stack represents a culmination of successful strategies aimed at stabilizing Large Language Models on Blackwell architecture. It includes advanced thread management techniques suitable for multi-core environments, ensuring resilient and robust AI deployment.

Explore Further

• Documentation and Benchmarks: For a closer look at achieving sustained performance and detailed benchmarks, please refer to the included project documentation.

• Community Contribution: This project not only aims to simplify deployment for users of Blackwell hardware but encourages collaboration to improve future iterations.

• Dockerhub: https://hub.docker.com/r/malkaf/vllm-blackwell-optimizer