RL.cu: LLM Reinforcement Learning in Pure CUDA

Overview
RL.cu offers a comprehensive implementation of the reinforcement learning (RL) pipeline for large language models (LLMs) using pure CUDA, eliminating dependencies on frameworks like PyTorch. This project features hand-written CUDA kernels, a vLLM-style inference engine, and Group Relative Policy Optimization (GRPO) training—all designed to achieve high performance and efficiency in LLM training and inference.

Project Highlights

• Performance: RL.cu is reported to be 1.37x faster than traditional implementations utilizing TRL with vLLM, matching rewards while significantly reducing computation time.
• Complete Implementation: The project encompasses the full RL loop, enabling seamless integration of training and inference phases without the need for weight transfers between models.
• Advanced Kernels: Custom kernels including FlashAttention-2, RMSNorm, and AdamW optimizers are efficiently implemented for both forward and backward passes, enhancing computational performance and memory management.

Key Features

1. CUDA Kernels: Implements essential functions for LLMs including attention mechanisms, normalization, embedding, and sampling—optimized with FP16 input and FP32 accumulation.
2. Inference Engine: Designed with continuous batching and paged key-value caching, the inference engine accelerates response times and resource utilization during model serving.
3. Training Techniques: Supports both Supervised Fine-Tuning (SFT) and GRPO for comprehensive model training, utilizing gradient checkpointing to save memory during backward passes.
4. Benchmark Results: Demonstrates competitive performance metrics including throughput and wall-clock times in various scenarios against leading alternatives.

Architecture

The structure of RL.cu is meticulously organized for clarity and efficiency, incorporating individual directories for kernels, model definitions, engine logic, and training functionalities. Example of the architecture includes:

RL.cu
├── src/kernels/          # Hand-written CUDA kernels  
├── src/model/           # Model definitions and functionalities  
├── include/engine/       # Inference engine interfaces  
└── include/training/     # Training frameworks and optimizers  

Getting Started

For those interested in exploring or contributing to this project, detailed build and usage instructions are provided within the repository. Commands for running inference, training, and testing can be easily accessed, allowing users to benchmark and extend the capabilities of RL.cu effectively.

Future Development

The project welcomes contributions aimed at enhancing its capabilities, such as multi-GPU support, additional model architectures, and advanced decoding techniques.

For more information and access to the repository: [Visit RL.cu on GitHub](https://github.com/KJLdefeated/RL.