RLHF

file 17 · 4:10 · subtitles burned in

Chapters

1. 0:00Introduction
2. 0:20Why preferences, not answers
3. 0:45Stage 1: start from an SFT model
4. 1:01Stage 2: comparisons train a reward model
5. 1:22Worked example: the reward model’s job
6. 1:43Stage 3: the PPO loop
7. 2:02The KL leash
8. 2:26What RLHF costs
9. 2:50The reinforcement family today
10. 3:14Run it: lesson 16 (toy) and 18
11. 3:37Recap

In this video Why models are trained on people’s comparisons, the three stages of RLHF (reinforcement learning from human feedback), and the leash that limits a model gaming its scorer.

What it explains: reward model, PPO, KL leash, reward hacking, GRPO/RFT

3 key points

1. 1

   RLHF learns from comparisons, for tasks with no single right answer.

   People struggle to write the perfect reply, but easily pick the better of two. A reward model turns those picks into scores: 2.1 for an apology with a fix, -0.4 for a curt reply, so about a 92% chance a person prefers the first.

2. 2

   Three stages: train on examples first (SFT), then a reward model, then a loop (PPO) that chases the score on a leash.

   Unleashed, the model games the scorer with flattery, padding and repeated phrases: reward hacking. The KL penalty charges it for drifting from the starting model, and β (beta) sets the leash length.

3. 3

   Powerful but heavy: 4 models in memory at once, against 1 for ordinary fine-tuning, and many settings to tune.

   The 4 are the model being trained, a frozen copy, the reward model and a value model. Relatives: GRPO (used for many reasoning models) compares a group of answers with each other and drops the value model; Fireworks’ RFT (reinforcement fine-tuning) scores each answer with a program, such as tests passing.

Used in the course

• Day 12 · Step 1 · Training toys, part 2

The title card in the video says “Video 17 of 18”: that is the file order. The course plays the videos in the order of its days.

Download mp4 (14.8 MB)