Full vs LoRA vs DoRA

course 20 of 22

lesson 17 · lessons/17-full-vs-peft-mlx

1 h 05 Free Mac

What you will do and why

You train one real model three ways on the same 800 support tickets: change every number, or train a small add-on (LoRA, and its variant DoRA). One table then shows what each cost, what it gained and what it broke.

Why it matters: Lesson example: the fully trained model saves a file of about 990 MB; the LoRA add-on saves about 9 MB, 110 times smaller, with accuracy close to full.

You are done when: compare_variants.py has printed one table with rows for the untrained model and for full, LoRA and DoRA, showing peak memory, file size, task accuracy and the general quiz.

Start this first

Run it from the 03-labs folder with the Python environment on (source .venv/bin/activate). It checks the data, downloads the model the first time (about 1 GB) and trains it three times: 15 to 45 minutes in all. Leave it running while you watch. For the other commands, open a second terminal window, go to 03-labs, run source .venv/bin/activate, then cd lessons/17-full-vs-peft-mlx.

cd lessons/17-full-vs-peft-mlx && bash run_variants.sh

Watch

Supervised Fine-Tuning · 4:02

Download mp4 (15.2 MB)

Chapters

In this video What training-by-example data looks like, why only the answer is graded, how much data you need, and how to read the training curves.

3 key points

Supervised fine-tuning (SFT) teaches by example: a prompt in, the ideal answer out, and only the answer is graded.
Each ticket example holds a system message (the standing instructions), the customer’s ticket and the ideal JSON answer. The trainer skips the first two when scoring (masking the prompt); run_variants.sh turns this on with --mask-prompt.
A few hundred clean examples beat many noisy ones.
Typical ranges for ticket triage with LoRA on a 7B model: 50 examples mostly fix the format, 500 add 8 to 12 points of accuracy, 2,000 add 12 to 16, then it flattens. 20,000 noisy ones often do worse than 2,000 clean. Today’s data: 800 training tickets.
Read the two loss curves together: training loss (how wrong it is on the examples it learns from) and validation loss (how wrong on examples held back).
Both falling: it is learning. Training falling while validation rises: it is memorising, so stop earlier or add data. Both flat and high: check the data, or raise the learning rate (the size of each nudge). The video’s usual starting values: about 0.0001 (written 1e-4) for LoRA, 10 times smaller for full.

In plain words

Today you train one small model three ways on the same 800 support tickets. Full fine-tuning changes every number in it. LoRA freezes it and trains a small add-on; DoRA is a LoRA variant. One table then shows what each cost (memory, time, file size), what it gained (accuracy) and what it broke (general knowledge).

Picture it

Full fine-tuning reprints the whole textbook with your changes, and while it works it needs about 8 times the book’s room for drafts and notes. LoRA sticks a pad of notes on the pages that matter: cheap, easy to swap, and the original text stays readable underneath. With LoRA almost all the weight you carry is the book itself.

With real numbersQwen2.5-0.5B-Instruct, lesson 17’s calculator and expected table

The model: 490 million numbers, stored at 2 bytes each (bf16, a 16-bit format): 0.98 GB, about 1 GB.
Full fine-tuning: 490 million x 16 bytes = 7.8 GB, 8 times the model. The working numbers for each batch of examples (activations) come on top: the lesson expects a peak of about 8 to 10 GB.
LoRA at rank 16 (the add-on’s size setting, in the calculator’s example): 2,162,688 trained numbers, 0.44% of the model. Memory: the 0.98 GB frozen model plus 2.16 million x 16 bytes (0.03 GB), about 1.0 GB. With activations, about 2 to 3 GB at peak.
Saved file, from the lesson’s expected table: about 990 MB for full (a whole new model) against about 9 MB for LoRA, 110 times smaller. Your own run’s sizes are the real figures.
The same sums for an 8B model: 128 GB to train fully, 16.2 GB with LoRA, 4.7 GB with QLoRA (LoRA on a 4-bit copy of the model).

Words to know

Full fine-tuning: Training that changes every number in the model. Example: 7.8 GB of training memory for the 0.5B model.
LoRA and DoRA: Freeze the model and train a small add-on beside it. DoRA also trains one extra number per row of each grid, setting how strong that row is; it often scores 1 to 2 accuracy points higher and trains a bit slower. Example: under 1% of the numbers trained.
Checkpoint: The file a training run saves: the whole changed model for full fine-tuning, only the add-on for LoRA. Example: about 990 MB against about 9 MB in the lesson’s expected table.
Forgetting: Getting better at the trained task and worse at everything else. Example: a drop in the 12-question general quiz.

From the lesson

lessons/17-full-vs-peft-mlx/README.md

What and why

Today’s three videos (Full Fine-Tuning, Supervised Fine-Tuning and Parameter-Efficient Fine-Tuning) make three claims. This lesson tests all three on a real model:

Full fine-tuning costs about 16 bytes per parameter to train, against about 2 to serve.
LoRA and DoRA train under 1% of the weights and land close to full fine-tuning on a narrow task.
Full fine-tuning risks more forgetting, meaning getting better at your task and worse at everything else.

You train Qwen2.5-0.5B-Instruct three ways on the same ticket data. Then you read cost (memory, minutes, checkpoint size), gain (task accuracy) and damage (a general quiz) from one table. The model is small enough that full fine-tuning fits on a 16 GB Mac.

The files, in order

step	file	runs on	does
0	`sft_data_check.py`	anywhere	checks the data: JSON, roles, ends-with-assistant, duplicates, test leakage, length, label balance
1	`peft_params.py`	anywhere	trainable parameters and training memory for full, LoRA, DoRA and QLoRA, from real model shapes
2	`run_variants.sh`	Mac	`mlx_lm.lora --fine-tune-type full / lora / dora`, same data and settings
3	`compare_variants.py`	Mac	one table: trainable %, peak memory, val loss, minutes, checkpoint MB, task accuracy, general quiz

This checker earned its place while the course was being built. On its first run it found that the synthetic dataset had 301 training tickets identical to test tickets, which would have inflated every fine-tune score. data/make_tickets.py now dedupes, keeps the splits disjoint and keeps one phrasing per category for the test set only.

Run

cd lessons/17-full-vs-peft-mlx
python sft_data_check.py
python peft_params.py --model 0.5b          # what you are about to train
python peft_params.py --model 8b --rank 16  # the same maths at a size customers use
python peft_params.py --model 70b --rank 64 --mlp

bash run_variants.sh                        # ~15–45 min for all three
python compare_variants.py

What each command does

python sft_data_check.py
Checks the 800 training tickets before any training. It looks for unreadable lines, unknown speakers (each message must be marked as the instructions, the customer, the model or a tool), examples that do not end with the answer, duplicates and over-long examples. It also catches any ticket that also sits in the 100-ticket test set (a leak). Look for rows 800 and no problems found.
python peft_params.py --model 0.5b
Works out, from the model’s real shape, how many numbers each method trains and how much memory training needs for today’s 0.5B model. Look for 7.8 GB for full against about 1.0 GB for LoRA, which trains 2,162,688 numbers (0.441%). Its LoRA file size says about 4 MB, not the lesson’s expected 9 MB: the calculator assumes rank 16 and 2 bytes per saved number, while run_variants.sh uses the trainer’s own defaults. Your own run’s files give the real size.
python peft_params.py --model 8b --rank 16
The same sums for Llama 3.1 8B, a size customers use. Look for 128.0 GB for full, 16.2 GB for LoRA and 4.7 GB for QLoRA (LoRA on a 4-bit copy of the model); the saved file is 16.0 GB for full against 27 MB for LoRA. The Full Fine-Tuning video’s rough figures are about 20 million numbers, 20 GB and 40 MB; the calculator counts rank 16 on the attention grids exactly.
python peft_params.py --model 70b --rank 64 --mlp
A 70B model at a higher rank, also adapting the MLP grids (the other big block in every layer). Look for 1129.6 GB for full, about 1.1 TB, against 154.5 GB for LoRA and 53.0 GB for QLoRA.
bash run_variants.sh
Checks the data again, then trains the model three ways, full, LoRA and DoRA, with the same tickets and settings: 300 steps of 4 examples (1,200 examples, 1.5 passes over the 800). Full uses a learning rate (the size of each nudge) 10 times smaller: 0.00001 against 0.0001, printed 1e-5 and 1e-4. Expect 5 to 15 minutes per method. If you started it before the video, let that run finish instead of starting a second one. Look for Val loss (how wrong it is on 100 held-back tickets) falling every 100 steps, and a Peak mem figure in each run.
python compare_variants.py
Reads the three training logs, then tests the untrained model and each trained one on two things: the category of 100 held-out tickets (30 in wordings never seen in training), and Day 4’s 12 quick general questions. Look for task accuracy jumping from about 40% to high for all three, LoRA and DoRA close to full, and any quiz drop in the full row.

Tight on memory? Run VARIANTS="lora dora" bash run_variants.sh and skip full. Short on time? Use ITERS=150.

What you should see (shape, not exact numbers)

How to read it

Each row is one method. trainable_% is how much of the model it trained; peak_mem_GB the most memory it used; minutes how long it took; val_loss how wrong it still was on held-back tickets. ckpt_MB adds up everything in the run’s folder, including the trainer’s in-between copies, so it can read several times the saved file. task_acc_% is the share of test tickets sorted correctly, and general_quiz the 12-question check for forgetting. The lesson’s table is the expected shape, not exact figures, and your trainer version may count a little differently. Yours lists the untrained model first, then dora, full, lora (alphabetical). Read across a row: what it cost, what it bought, what it broke.

variant              trainable_%  peak_mem_GB  val_loss  minutes  ckpt_MB  task_acc_%  general_quiz
base (no training)           0.0          nan       nan      0.0      0.0        ~40        ~9/12
full                       100.0        ~8–10     lowest     slowest   ~990       high     may drop
lora                        ~0.4         ~2–3      close      fast      ~9        high     ≈ base
dora                        ~0.45        ~3        close      a bit slower ~9     high     ≈ base

Peak memory: full is several times LoRA’s. That is claim 1.
Checkpoint: about 1 GB for full against single-digit MB. That is why multi-LoRA serving works.
Task accuracy: all three jump well above base, and LoRA and DoRA land close to full. That is claim 2. Look at the held-out phrasings too, since they test generalisation rather than memory.
General quiz: if any variant drops, it is usually full. That is claim 3. On 12 questions it is a smoke alarm, not proof.

Check yourself

3 questions. Say your answer out loud, then tap to check it.

Day 11’s real training run starts from a model stored at 16 bits per number (bf16). On Day 8 you trained an add-on for a 7-billion-number model stored at 4 bits. Why must this comparison of full fine-tuning and LoRA use a 16-bit model, not a 4-bit one?Show answerHide

In plain words

Full fine-tuning changes every stored number, and 4-bit numbers are too coarse to take the tiny nudges training makes. Day 8’s QLoRA worked only because it never changed the 4-bit numbers: it trained a separate add-on instead.

Picture it

A 4-bit number is like a ruler marked only in whole centimetres. Training wants to move each mark by a hair’s width, and on that ruler the nudge rounds away to nothing. QLoRA leaves the coarse ruler alone and writes the fine corrections on a separate note.

With real numberslesson 17, and Day 8’s lesson 11

4 bits can hold only 2 x 2 x 2 x 2 = 16 different values; 16 bits can hold 65,536.
Day 11’s model (Qwen2.5-0.5B-Instruct) at 16 bits: 490 million numbers x 2 bytes = 0.98 GB, small enough to train fully on a 16 GB Mac (about 8 to 10 GB at peak).
Day 8’s QLoRA: a 7B model frozen at 4 bits plus a small 16-bit add-on, about 6 to 10 GB to train. A full fine-tune of the same model needs about 120 GB (it has 7.6 billion numbers: 7.6 x 16 = 122 GB).

Words to know

bf16: A 16-bit number format used for training, 2 bytes per number. Example: Day 11’s model is 0.98 GB in bf16.
QLoRA: LoRA on a model stored in 4 bits: the model stays frozen, only the add-on trains. Example: about 6 to 10 GB for a 7B model on Day 8.
Frozen: Left unchanged during training. Example: the 4-bit model in QLoRA.
Gradient: The correction worked out for each trainable number at each training step. Example: full fine-tuning keeps one for every number.

Go deeper: the engineer version

The kit's question

Why must the base be bf16 here, and not the 4-bit model from lesson 11?

The kit's answer

Full fine-tuning updates the weights, and you can’t train 4-bit weights directly. QLoRA only works because the base stays frozen.

More detail: Quantized weights sit on a coarse grid, and rounding to that grid gives no useful gradient, so a training step cannot move them reliably. QLoRA keeps the 4-bit base frozen (only read, and turned back into 16-bit numbers for the math) and sends gradients only into the 16-bit adapter. run_variants.sh uses the bf16 base for all three runs so that full, LoRA and DoRA compare fairly.

Day 11’s memory calculator (peft_params.py --model 8b) says LoRA needs about 16 GB to train, but full fine-tuning needs 128 GB. Where do LoRA’s 16 GB go?Show answerHide

In plain words

Almost all of it is the model itself, frozen and stored at 2 bytes per number. The add-on and its training state are tiny. Store the frozen model in 4 bits instead (QLoRA) and it drops to about 5 GB.

Picture it

You hire a moving truck to deliver one small box: nearly all the cost is the truck, not the box. In LoRA the truck is the frozen model and the box is the add-on. QLoRA books a smaller truck: the same model squeezed into 4 bits.

With real numberspeft_params.py --model 8b --rank 16, lesson 17

Frozen model: 8 billion numbers x 2 bytes = 16 GB.
LoRA add-on at rank 16: 13.6 million numbers (0.17% of the model) x 16 bytes = 0.22 GB.
Total: 16 + 0.22 = 16.2 GB. Full: 8 billion x 16 bytes = 128 GB, about 8 times as much.
QLoRA: the frozen model at 4 bits is half a byte per number, plus a little for shared scale numbers: 0.5625 bytes. 8 billion x 0.5625 = 4.5 GB, plus 0.22 = 4.7 GB, about 5 GB.

Words to know

Frozen model: The original model, kept unchanged and only read during LoRA training. Example: 16 GB for an 8B model at 2 bytes per number.
Adapter (add-on): The small set of extra numbers LoRA trains and saves. Example: 13.6 million numbers for an 8B model at rank 16.
Optimizer state: The running averages the optimizer (Adam) keeps per trained number to size each nudge. Example: 8 of the 16 bytes per number.
QLoRA: LoRA on a model stored in 4 bits. Example: 4.7 GB for an 8B model instead of 16.2 GB.

Go deeper: the engineer version

The kit's question

peft_params.py --model 8b says LoRA needs about 16 GB, but full needs 128. Where does LoRA’s 16 GB go?

The kit's answer

Almost all of it is the frozen bf16 base, 2 bytes × 8B. The adapter’s optimizer state is tiny. Switch to QLoRA and it drops to about 5 GB.

More detail: peft_params.py counts LoRA as 2 B x base parameters (frozen bf16 weights, no gradients or optimizer state) + 16 B x adapter parameters (bf16 weight and gradient, fp32 Adam m and v, fp32 master copy). QLoRA’s 0.5625 B per base weight is 4 bits plus the shared scales of each small block. Activations come on top of every figure and grow with batch x sequence length; --grad-checkpoint trims them.

In today’s runs, full fine-tuning makes nudges 10 times smaller than LoRA’s (a learning rate of 0.00001 against 0.0001). Why?Show answerHide

In plain words

Full fine-tuning moves every number in the model, including the ones that hold its general skills. Big steps across all of them would damage what the model already knows.

Picture it

Restoring a painting: if you may touch every inch, you use a fine brush and light strokes, or you ruin the parts that were already right. Painting on a clear sheet laid over it (LoRA), bolder strokes are safe, because the original underneath is untouched.

With real numbersrun_variants.sh and the Supervised Fine-Tuning video

LoRA and DoRA: learning rate 1e-4 (0.0001). Full: 1e-5 (0.00001), 10 times smaller.
Full moves all 490 million numbers. LoRA at rank 16 (the calculator’s example) moves about 2.2 million (0.44%) and the rest stay frozen. Your training log’s Trainable parameters line gives today’s exact share.
The Supervised Fine-Tuning video gives the same pair: about 1e-4 for LoRA, about 10 times smaller for full.
The damage would show in the general quiz: the untrained model scores about 9 of 12, and a drop below that is usually in the full row.

Words to know

Learning rate: How big a nudge each training step gives each number. Example: 1e-4 (0.0001) for LoRA, 1e-5 (0.00001) for full.
Forgetting: Getting better at the trained task and worse at everything else. Example: a lower general quiz score.
General quiz: Day 4’s 12 quick questions (math, facts, format), reused here to spot forgetting. Example: about 9 of 12 for the untrained model.
Training step: One update: the model sees a few examples and adjusts. Example: 300 steps of 4 tickets.

Go deeper: the engineer version

The kit's question

Why is full fine-tuning’s learning rate 10× smaller?

The kit's answer

Every weight moves, so a big step damages what the model already knows.

More detail: An Adam update moves each weight by roughly the learning rate, so 1e-5 across every weight is still a large total change to the network. Adapter updates are confined to a low-rank correction that starts at zero (B = 0), so a larger rate is safe. Too large a rate in full fine-tuning shows up as a general-quiz drop, the forgetting that claim 3 of the lesson looks for.

Explain what you learned

Question Should we fine-tune the whole model or use LoRA?

One clear answer

I ran full, LoRA and DoRA on the same data. LoRA got within a couple of points of full at a fraction of the memory, with a 9 MB checkpoint instead of a gigabyte, and without denting general ability. For a narrow task, I start with LoRA and only go full when the eval gap is real.

What this means

“I ran full, LoRA and DoRA on the same data”: I trained the same model three ways on the same 800 tickets with the same settings, so the comparison is fair. Quote your own table’s numbers.
“LoRA got within a couple of points of full”: LoRA’s task accuracy landed within about 2 percentage points of full training. The Full Fine-Tuning video says the same: usually within 1 to 2 points on a narrow task.
“at a fraction of the memory”: About 2 to 3 GB at peak for LoRA against about 8 to 10 GB for full, on the 0.5B model. On an 8B model: 16.2 GB against 128 GB.
“with a 9 MB checkpoint instead of a gigabyte”: The lesson expects the saved LoRA add-on at about 9 MB, and the fully trained model as a whole new copy of about 990 MB. Say your own adapters.safetensors sizes from today’s run. Small files are why one server can hold many customers’ add-ons.
“and without denting general ability”: The 12-question general quiz stayed about where the untrained model was (about 9 of 12): no forgetting.
“For a narrow task, I start with LoRA”: For one well-defined job, like sorting tickets into 4 categories, LoRA is my first choice.
“and only go full when the eval gap is real”: I switch to full fine-tuning only if tests on the customer’s own examples show LoRA falling measurably short.

Your numbersSaved on this device and collected in the Day 11 wrap-up.

1. Peak memory while training: full / LoRA / DoRA (GB)

Hint: The peak_mem_GB column of compare_variants.py. The table lists dora first; write them as full / LoRA / DoRA, separated by slashes.

2. Saved file size: full / LoRA / DoRA (MB)

Hint: The size of each saved file: run ls -lh adapters/*/adapters.safetensors in the lesson folder (M means about a million bytes). The ckpt_MB column adds up the whole folder, including the trainer’s in-between copies, so it can read several times higher. Write them as full / LoRA / DoRA.

3. Task accuracy: untrained / full / LoRA / DoRA (%)

Hint: The task_acc_% column: tickets whose category the model got right, out of 100 held-out tickets. The table lists the untrained model, then dora, full, lora; write them in the label’s order.

4. General quiz: untrained / full / LoRA / DoRA (out of 12)

Hint: The general_quiz column of compare_variants.py. The table lists the untrained model, then dora, full, lora; write them in the label’s order.

Done when

compare_variants.py has printed one table with rows for the untrained model and for full, LoRA and DoRA, showing peak memory, file size, task accuracy and the general quiz.

NextFinish Day 11: wrap-up and drill (10 min)

Stuck?

The full run stops with an out-of-memory error, or the Mac slows to a crawl.: Full training peaks at about 8 to 10 GB. Close big apps and try again, or skip it. To skip it, first remove what the failed run left behind in the lesson folder, or compare_variants.py will try to test it: rm -rf logs/full.log adapters/full. Then run VARIANTS="lora dora" bash run_variants.sh. Record that full training did not fit on your Mac: it is a useful measured limit for your comparison.
Training takes too long.: Run ITERS=150 bash run_variants.sh: 150 steps instead of 300, about half the time. Note it next to your numbers.
run_variants.sh stops with fix the data first.: sft_data_check.py found a problem in the training file and printed its type with the first line number. If you edited data/tickets/, rebuild it from the 03-labs folder with python data/make_tickets.py.
compare_variants.py says No logs yet: run bash run_variants.sh first.: It reads the logs/ folder inside lessons/17-full-vs-peft-mlx. Run bash run_variants.sh there first, then python compare_variants.py from the same folder.
compare_variants.py says the model evaluation was skipped.: The accuracy and quiz columns need mlx-lm on an Apple-silicon Mac. Turn the environment on (source .venv/bin/activate from 03-labs) and run it again. The memory, time and file-size columns come from the logs either way.
mlx_lm.lora: command not found: mlx-lm is missing from the environment. From 03-labs, run source .venv/bin/activate, then pip install -r requirements.txt (Day 1’s setup), then bash run_variants.sh again.

Go deeper: the lab book's training lab for this step All fixes

Code in this step

compare_variants.py Python · 99 lines

1
"""
2
Lesson 17 · step 3 — one table: what each method cost, what it bought, what it broke.
3

4
From the training logs (works anywhere):
5
  trainable %, peak memory, final validation loss, wall time, checkpoint size on disk
6

7
From the models themselves (Apple silicon, needs mlx-lm):
8
  task accuracy     triage category on the 100 held-out tickets (30 in phrasings never trained on)
9
  general quiz      the 12 general questions from lesson 04, which checks for FORGETTING (video 14)
10

11
    python compare_variants.py               # logs + evaluation
12
    python compare_variants.py --logs-only   # skip the model evaluation
13
"""
14
import argparse
15
import importlib.util
16
import json
17
import re
18
from pathlib import Path
19

20
from felab import record, table
21
from felab.tickets import SCHEMA, SYSTEM, grade, load_test
22

23
HERE = Path(__file__).parent
24
MODEL = "mlx-community/Qwen2.5-0.5B-Instruct-bf16"
25

26

27
def parse_log(path: Path) -> dict:
28
    txt = path.read_text(errors="ignore")
29
    m = re.search(r"Trainable parameters:\s*([\d.]+)%\s*\(([\d.]+)M", txt)
30
    peaks = [float(x) for x in re.findall(r"Peak mem\s*([\d.]+)\s*GB", txt)]
31
    vals = re.findall(r"Val loss\s*([\d.]+)", txt)
32
    wall = re.search(r"wall_seconds\s+(\d+)", txt)
33
    return {
34
        "trainable_%": float(m.group(1)) if m else float("nan"),
35
        "trainable_M": float(m.group(2)) if m else float("nan"),
36
        "peak_mem_GB": max(peaks) if peaks else float("nan"),
37
        "val_loss": float(vals[-1]) if vals else float("nan"),
38
        "minutes": int(wall.group(1)) / 60 if wall else float("nan"),
39
    }
40

41

42
def dir_mb(p: Path) -> float:
43
    return sum(f.stat().st_size for f in p.rglob("*") if f.is_file()) / 1e6 if p.exists() else float("nan")
44

45

46
def evaluate(adapter: str | None) -> tuple[float, int]:
47
    """Greedy-decode the test tickets and the general quiz with mlx-lm."""
48
    from mlx_lm import generate, load
49
    model, tok = load(MODEL, adapter_path=adapter)
50

51
    def ask(messages, max_tokens):
52
        prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
53
        return generate(model, tok, prompt=prompt, max_tokens=max_tokens, verbose=False)
54

55
    rows = load_test()
56
    ok = sum(grade(ask([{"role": "system", "content": SYSTEM}, {"role": "user", "content": r["ticket"]}], 60), r)["category_ok"]
57
             for r in rows)
58
    spec = importlib.util.spec_from_file_location("q", HERE.parent / "04-quantization" / "quality_check.py")
59
    q = importlib.util.module_from_spec(spec)
60
    spec.loader.exec_module(q)          # safe: its main() only runs under __main__
61
    quiz = sum(want.replace(" ", "").lower() in ask([{"role": "user", "content": qq}], 40).replace(" ", "").lower()
62
               for qq, want in q.QA)
63
    return 100 * ok / len(rows), quiz
64

65

66
def main() -> None:
67
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
68
    ap.add_argument("--logs-only", action="store_true")
69
    a = ap.parse_args()
70

71
    variants = [p.stem for p in sorted((HERE / "logs").glob("*.log"))]
72
    if not variants:
73
        raise SystemExit("No logs yet: run bash run_variants.sh first.")
74
    rows = []
75
    can_eval = not a.logs_only and importlib.util.find_spec("mlx_lm") is not None
76
    if can_eval:
77
        print("evaluating base model …", flush=True)
78
        acc, quiz = evaluate(None)
79
        rows.append({"variant": "base (no training)", "trainable_%": 0.0, "peak_mem_GB": float("nan"),
80
                     "val_loss": float("nan"), "minutes": 0.0, "ckpt_MB": 0.0, "task_acc_%": acc, "general_quiz": f"{quiz}/12"})
81
    for v in variants:
82
        r = {"variant": v, **parse_log(HERE / "logs" / f"{v}.log"), "ckpt_MB": dir_mb(HERE / "adapters" / v)}
83
        r.pop("trainable_M")
84
        if can_eval:
85
            print(f"evaluating {v} …", flush=True)
86
            acc, quiz = evaluate(str(HERE / "adapters" / v))
87
            r.update({"task_acc_%": acc, "general_quiz": f"{quiz}/12"})
88
        rows.append(r)
89
        record("17-variants", r)
90
    print("\n" + table(rows))
91
    print("\nRead across a row: what it cost (memory, minutes, MB) → what it bought (task accuracy) →"
92
          "\nwhat it broke (general quiz; a drop means forgetting). On a narrow task, LoRA and DoRA"
93
          "\nshould land near full fine-tuning at a fraction of the memory and checkpoint size.")
94
    if not can_eval:
95
        print("\n(model evaluation skipped: needs mlx-lm on Apple silicon)")
96

97

98
if __name__ == "__main__":
99
    main()

peft_params.py Python · 63 lines

1
"""
2
Lesson 17 · step 1 — how many parameters does each method train, and how much memory does it need?
3

4
Counts come from the model's real shapes (config.json), not rules of thumb.
5
For a weight matrix of shape (in × out), a LoRA adapter of rank r adds  r·(in + out)  parameters.
6
DoRA adds one magnitude value per output channel on top.
7

8
Memory to TRAIN (weights + gradients + Adam state; activations excluded):
9
  full   16 B × all params                       (bf16 weights & grads, fp32 Adam m, v, master copy)
10
  LoRA    2 B × base  + 16 B × adapter params     (bf16 frozen base)
11
  QLoRA  ≈0.56 B × base + 16 B × adapter params   (4-bit frozen base)
12

13
    python peft_params.py --model 8b --rank 16
14
    python peft_params.py --model 70b --rank 64 --mlp
15
    python peft_params.py --model 0.5b          # what lesson 17 trains on your Mac
16
"""
17
import argparse
18

19
# hidden, layers, attention heads, kv heads, head_dim, mlp (intermediate) — from each config.json
20
MODELS = {
21
    "0.5b": ("Qwen2.5-0.5B", 896, 24, 14, 2, 64, 4864, 0.49e9),
22
    "7b": ("Qwen2.5-7B", 3584, 28, 28, 4, 128, 18944, 7.6e9),
23
    "8b": ("Llama-3.1-8B", 4096, 32, 32, 8, 128, 14336, 8.0e9),
24
    "70b": ("Llama-3.1-70B", 8192, 80, 64, 8, 128, 28672, 70.6e9),
25
}
26

27
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
28
ap.add_argument("--model", choices=MODELS, default="8b")
29
ap.add_argument("--rank", type=int, default=16)
30
ap.add_argument("--mlp", action="store_true", help="also adapt the MLP (gate, up, down)")
31
ap.add_argument("--layers", type=int, help="adapt only the last N layers (default: all)")
32
a = ap.parse_args()
33

34
name, d, L, H, KV, hd, ff, total = MODELS[a.model]
35
layers = a.layers or L
36
mats = {"q": (d, H * hd), "k": (d, KV * hd), "v": (d, KV * hd), "o": (H * hd, d)}
37
if a.mlp:
38
    mats.update({"gate": (d, ff), "up": (d, ff), "down": (ff, d)})
39

40
lora_per_layer = sum(a.rank * (i + o) for i, o in mats.values())
41
dora_per_layer = lora_per_layer + sum(o for _, o in mats.values())
42
lora = lora_per_layer * layers
43
dora = dora_per_layer * layers
44

45
print(f"{name}: {total / 1e9:.1f}B params · hidden {d} · {L} layers · GQA {H}q/{KV}kv heads")
46
print(f"adapting {', '.join(mats)} in {layers} layers at rank {a.rank}\n")
47
print("  per layer:")
48
for m, (i, o) in mats.items():
49
    print(f"    {m:5s} {i:>6} × {o:<6}  full {i * o / 1e6:7.2f} M   LoRA r·(in+out) = {a.rank * (i + o) / 1e3:7.1f} K")
50

51
GB = 1e9
52
rows = [
53
    ("full fine-tune", total, 16 * total),
54
    ("LoRA", lora, 2 * total + 16 * lora),
55
    ("DoRA", dora, 2 * total + 16 * dora),
56
    ("QLoRA (4-bit base)", lora, 0.5625 * total + 16 * lora),
57
]
58
print(f"\n  {'method':20s} {'trainable':>14s} {'share':>8s} {'train memory*':>14s} {'checkpoint':>12s}")
59
for m, t, mem in rows:
60
    ckpt = f"{2 * total / GB:.1f} GB" if m.startswith("full") else f"{2 * t / 1e6:.0f} MB"
61
    print(f"  {m:20s} {t:>14,.0f} {100 * t / total:7.3f}% {mem / GB:11.1f} GB {ckpt:>12s}")
62
print("\n  * weights + gradients + optimizer only. Activations add more and grow with batch × sequence length;"
63
      "\n    gradient checkpointing shrinks them. Compare with the 'Peak mem' mlx_lm.lora prints.")

run_variants.sh Bash · 40 lines

1
#!/usr/bin/env bash
2
# ─────────────────────────────────────────────────────────────────────────────
3
# Lesson 17 · step 2 — the same model, same data, three ways (videos 14, 15, 16).
4
#
5
#   full   every weight trains                       lr 1e-5 (full FT needs a smaller step)
6
#   lora   frozen base + low-rank B·A in every layer  lr 1e-4
7
#   dora   LoRA on direction + a magnitude vector     lr 1e-4
8
#
9
# Model: Qwen2.5-0.5B-Instruct in bf16. Small enough that FULL fine-tuning fits on a
10
# 16 GB Mac (≈0.5B × 16 bytes ≈ 8 GB + activations). QLoRA needs a quantized base; for
11
# full fine-tuning the base must NOT be quantized, so we use bf16 for all three.
12
#
13
#   --mask-prompt       loss on the assistant answer only (video 15)
14
#   --num-layers -1     all layers, so the three runs are comparable
15
#   --grad-checkpoint   recompute activations: less memory, a bit slower (video 14)
16
#
17
# ~5–15 min per variant on M-series. Logs → logs/<variant>.log, weights → adapters/<variant>/
18
# ─────────────────────────────────────────────────────────────────────────────
19
set -euo pipefail
20
cd "$(dirname "$0")"
21
MODEL=${MODEL:-mlx-community/Qwen2.5-0.5B-Instruct-bf16}
22
ITERS=${ITERS:-300}
23
VARIANTS=(${VARIANTS:-full lora dora})
24
DATA=../../data/tickets
25
mkdir -p logs adapters
26

27
python sft_data_check.py || { echo "fix the data first"; exit 1; }
28

29
for v in "${VARIANTS[@]}"; do
30
  lr=1e-4; [[ $v == full ]] && lr=1e-5
31
  echo "▸ $v  (lr $lr, $ITERS iters)"
32
  start=$SECONDS
33
  mlx_lm.lora --model "$MODEL" --train --data "$DATA" \
34
    --fine-tune-type "$v" --num-layers -1 --mask-prompt --grad-checkpoint \
35
    --batch-size 4 --iters "$ITERS" --learning-rate "$lr" \
36
    --steps-per-report 20 --steps-per-eval 100 \
37
    --adapter-path "adapters/$v" 2>&1 | tee "logs/$v.log"
38
  echo "wall_seconds $(( SECONDS - start ))" >> "logs/$v.log"
39
done
40
echo "▸ next: python compare_variants.py"

sft_data_check.py Python · 95 lines

1
"""
2
Lesson 17 · step 0 — check an SFT dataset BEFORE you spend compute on it (video 15).
3

4
Catches the mistakes that quietly ruin training runs:
5
  ✗ lines that are not valid JSON, or have no "messages"
6
  ✗ conversations that do not END with an assistant turn (nothing to learn)
7
  ✗ empty contents, unknown roles
8
  ✗ exact duplicates (the model over-learns them)
9
  ✗ test-set leakage: training examples whose user text appears in the held-out test set
10
  ✗ rows longer than your max sequence length (silently truncated = broken answers)
11
  ~ label balance (for the triage task: category counts parsed from the answers)
12

13
    python sft_data_check.py                                   # data/tickets/train.jsonl vs test.jsonl
14
    python sft_data_check.py my_train.jsonl --test my_test.jsonl --max-tokens 2048
15
Exit code 1 if any ✗ check fails, so it can gate a training script.
16
"""
17
import argparse
18
import collections
19
import json
20
import sys
21
from pathlib import Path
22

23
ROOT = Path(__file__).resolve().parents[2]
24
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
25
ap.add_argument("train", nargs="?", default=str(ROOT / "data/tickets/train.jsonl"))
26
ap.add_argument("--test", default=str(ROOT / "data/tickets/test.jsonl"))
27
ap.add_argument("--max-tokens", type=int, default=1024, help="trainer's max sequence length")
28
a = ap.parse_args()
29

30
rows, problems = [], collections.defaultdict(list)
31
for n, line in enumerate(Path(a.train).read_text().splitlines(), 1):
32
    if not line.strip():
33
        continue
34
    try:
35
        obj = json.loads(line)
36
    except json.JSONDecodeError:
37
        problems["not valid JSON"].append(n)
38
        continue
39
    msgs = obj.get("messages")
40
    if not isinstance(msgs, list) or not msgs:
41
        problems['missing "messages"'].append(n)
42
        continue
43
    roles = [m.get("role") for m in msgs]
44
    if any(r not in ("system", "user", "assistant", "tool") for r in roles):
45
        problems["unknown role"].append(n)
46
    if roles[-1] != "assistant":
47
        problems["does not end with an assistant turn"].append(n)
48
    if any(not str(m.get("content", "")).strip() for m in msgs):
49
        problems["empty content"].append(n)
50
    toks = sum(len(str(m.get("content", ""))) for m in msgs) // 4 + 8 * len(msgs)   # ~4 chars/token + template
51
    if toks > a.max_tokens:
52
        problems[f"longer than --max-tokens {a.max_tokens}"].append(n)
53
    rows.append((n, obj, toks))
54

55
# duplicates
56
seen = {}
57
for n, obj, _ in rows:
58
    key = json.dumps(obj["messages"], sort_keys=True)
59
    if key in seen:
60
        problems["exact duplicate"].append(n)
61
    seen.setdefault(key, n)
62

63
# test leakage: user text that also appears in the test set
64
test_texts = set()
65
if Path(a.test).exists():
66
    for line in Path(a.test).read_text().splitlines():
67
        if line.strip():
68
            t = json.loads(line)
69
            test_texts.add((t.get("ticket") or next((m["content"] for m in t.get("messages", []) if m["role"] == "user"), "")).strip())
70
for n, obj, _ in rows:
71
    user = next((m["content"] for m in obj["messages"] if m["role"] == "user"), "").strip()
72
    if user in test_texts:
73
        problems["user text also in the TEST set (leakage)"].append(n)
74

75
# label balance (triage-specific, skipped if answers are not JSON)
76
cats = collections.Counter()
77
for _, obj, _ in rows:
78
    try:
79
        cats[json.loads(obj["messages"][-1]["content"]).get("category", "?")] += 1
80
    except Exception:
81
        pass
82

83
toks = sorted(t for *_, t in rows)
84
print(f"{a.train}\n  rows {len(rows)}   tokens p50 ≈{toks[len(toks) // 2] if toks else 0}   max ≈{toks[-1] if toks else 0}")
85
if cats:
86
    total = sum(cats.values())
87
    print("  label balance  " + "  ".join(f"{c} {100 * k / total:.0f}%" for c, k in cats.most_common()))
88
blocking = {k: v for k, v in problems.items()}
89
if not blocking:
90
    print("  ✓ no problems found")
91
for k, v in blocking.items():
92
    print(f"  ✗ {k}: {len(v)} rows (first: line {v[0]})")
93
if "exact duplicate" in blocking:
94
    print("\nDuplicates make the model over-learn those rows; dedupe before training.")
95
sys.exit(1 if blocking else 0)