This repository, rlhf-from-scratch, provides both a theoretical and practical exploration of Reinforcement Learning with Human Feedback (RLHF), specifically as it applies to Large Language Models. The focus is on guiding users through the core steps of RLHF using concise and understandable code, rather than offering a complete production framework.

Key Components

• src/ppo/ppo_trainer.py: Implements a straightforward Proximal Policy Optimization (PPO) training loop for updating a language model policy.
• src/ppo/core_utils.py: Contains utility functions for computation including rollout processing, advantage and return calculations, and reward wrappers.
• src/ppo/parse_args.py: Provides command-line interface (CLI) functionality for argument parsing during training runs.
• tutorial.ipynb: This Jupyter notebook integrates the various components by combining theoretical insights, small experiments, and practical code examples.

Topics Explored in the Notebook

• Overview of the RLHF pipeline, encompassing preference data, the reward model, and policy optimization.
• Brief demonstrations of reward modeling techniques, fine-tuning using PPO, and comparative analyses.
• Practical insights along with runnable code snippets to replicate simple experiments.

Getting Started

The repository is designed to be interactive and user-friendly: simply open the tutorial.ipynb notebook in Jupyter to run the cells interactively and explore the content. Additionally, a review of the src/ppo/ directory offers insights into how the notebook corresponds with the training and utility code.

For a condensed version or specific examples such as a single script for a minimal DPO or PPO demo, feedback for enhancements is welcome.