Training Techniques

file 06 · 3:14 · subtitles burned in

Chapters

1. 0:00Introduction
2. 0:09Start from a base model
3. 0:22Supervised fine-tuning (SFT)
4. 0:41LoRA: two small matrices
5. 1:00Preference tuning (DPO)
6. 1:15Reinforcement fine-tuning (RFT)
7. 1:34Which method to choose
8. 1:54Worked example: ticket triage
9. 2:17Pick the method from the failure
10. 2:39Recap

In this video The four standard ways to change how an existing model behaves, how to choose one from what is going wrong, and why LoRA (training a small add-on instead of the whole model) keeps it cheap.

What it explains: overview: SFT, LoRA, DPO, RFT and how to choose

3 key points

1. 1

   Pick the method from the kind of mistake: examples teach format, comparisons teach taste, an automatic score teaches correctness.

   Can you write the right answer? Supervised fine-tuning (SFT: training on examples of it). Can you only say which of two answers is better? Preference tuning (DPO, direct preference optimization). Can a program score an answer? Reinforcement fine-tuning (RFT). Missing facts need looking up (retrieval), not training.

2. 2

   LoRA (low-rank adaptation) keeps it cheap: the model stays frozen (unchanged) and only a small add-on is trained, so one running model can carry many add-ons.

   The video’s ticket example: 2,000 examples x 600 tokens (word pieces) x 2 passes = 2.4 million training tokens, about $1.20 at $0.50 per million. Accuracy rose from 81% to 89%, and the add-on is about 30 MB, a tiny fraction of the model.

3. 3

   Try a better prompt and retrieval first: they are free.

   Train only when those stop helping. Then look at what is still wrong: the video’s example stops at 89% and asks what the other 11% of mistakes look like, because that picks the next method.

Used in the course

• Day 7 · Step 1 · Eval harness

The title card in the video says “Video 6 of 18”: that is the file order. The course plays the videos in the order of its days.

Download mp4 (11.4 MB)