What is RLHF (Reinforcement Learning from Human Feedback)?

RLHF is the technique used to align LLMs with human preferences. Phase 1: Supervised Fine-Tuning (SFT) — fine-tune on high-quality demonstrations. Phase 2: Reward Model Training — human annotators rank multiple model outputs; a reward model is trained to predict human preferences. Phase 3: RL Fine-Tuning — use Proximal Policy Optimization (PPO) to optimize the policy (the LLM) to maximize the reward model's score while penalizing deviation from the SFT model (KL divergence constraint). RLHF is central to ChatGPT, InstructGPT, and Claude's training.