TTS and render scripts

Kokoro text-to-speech and the render loop.

scripts/add_training_series.py Python · 243 lines

1
"""One-off: add videos 14–18 (training deep dives) and repoint every end card at repo lessons + lab ids."""
2
import json
3
from pathlib import Path
4

5
P = Path(__file__).resolve().parents[1] / "src" / "narration.json"
6
d = json.loads(P.read_text())
7

8
def S(visual, text, **params):
9
    return {"visual": visual, "params": params, "text": text}
10

11
def title(n, headline, sub, learn, text, extra=""):
12
    return S("Title", text, kicker=f"Video {n} of 18 · training deep dive{extra}", headline=headline, sub=sub, learn=learn)
13

14
V = {}
15

16
# ───────────────────────────── 14 · Full fine-tuning ─────────────────────────────
17
V["full-finetuning"] = {"title": "Full Fine-Tuning", "subtitle": "Every weight moves, and what that costs", "accent": "teal", "scenes": [
18
    title(14, "Full fine-tuning", "Every weight moves, and what that costs",
19
          "what full fine-tuning is, why it needs about 16 bytes per parameter, and when it is worth it",
20
          "This is the first of five deep dives on training. We start with the most direct way to change a model: full fine-tuning, where every single weight is allowed to move. It is the most powerful option and the most expensive, and you should know exactly why."),
21
    S("TrainLoop", "Every kind of training runs the same loop. The model reads an example and predicts each next token. The loss measures how surprised it was by the right answer. Backpropagation works out, for every weight, which direction would have reduced that surprise. Then the optimizer nudges each weight a tiny step in that direction, and the loop repeats, thousands of times.",
22
      note="Full fine-tuning, LoRA, SFT and DPO all run this loop. They differ in which weights move and what the loss rewards."),
23
    S("WeightUpdate", "In full fine-tuning, that nudge applies to every weight. For a seven billion parameter model, that is seven billion numbers updated on every step. Nothing is frozen. It is like re-editing an entire encyclopedia, where LoRA would add a few sticky notes.",
24
      mode="full", note="A 7B model: 7,000,000,000 trainable numbers, every step."),
25
    S("MemoryLedger", "Here is where the cost comes from. Serving a model only needs the weights, about two bytes each. Training needs far more per parameter: the weights, a gradient of the same size, the Adam optimizer's two running averages kept in full precision, and a full precision master copy. That is roughly sixteen bytes per parameter before you count activations. For seven billion parameters, that is over a hundred gigabytes.",
26
      paramsB=7, totalText="112 GB + activations", note="Serving the same model needs about 14 GB. Training needs about 8× that."),
27
    S("Worked", "Put numbers on it. A one billion parameter model needs about sixteen gigabytes, which is one large GPU. An eight billion model needs around a hundred and thirty, so at least two eighty-gigabyte GPUs. A seventy billion model needs over a terabyte, which means sixteen H100s and a sharding framework like DeepSpeed ZeRO or FSDP to split the weights, gradients and optimizer state across them. LoRA on that same seventy billion model needs a small fraction of that.",
28
      heading="Worked example · what full fine-tuning needs", scenario="≈16 bytes per parameter, plus activations",
29
      rows=[["1B model", "≈ 16 GB · one big GPU"], ["8B model", "≈ 130 GB · 2× H100 80 GB"], ["70B model", "≈ 1.1 TB · 16× H100 + sharding"], ["LoRA on the same 70B", "≈ 160 GB · 2–4× H100"]],
30
      answer="Memory, not compute, decides whether you can full fine-tune"),
31
    S("CardGrid", "Teams make it fit with a standard toolbox. Mixed precision computes in bf16. ZeRO or FSDP shards everything across GPUs. Gradient checkpointing recomputes activations instead of storing them, trading time for memory. Eight-bit optimizers shrink the Adam state. Offloading parks state in system memory, at a speed cost. And often the best move is a smaller model.",
32
      heading="How teams make full fine-tuning fit", cols=3,
33
      cards=[["Mixed precision", "bf16 compute, fp32 master copy", "≈16 B/param"], ["ZeRO / FSDP", "shard weights, grads, optimizer across GPUs", "memory ÷ N GPUs"], ["Grad checkpointing", "recompute activations, don't store them", "less memory, ~30% slower"],
34
             ["8-bit optimizers", "Adam state in 8 bits", "16 → ≈10 B/param"], ["CPU offload", "optimizer state in system RAM", "fits more, runs slower"], ["Right-size the model", "a full 8B tune can beat a 70B LoRA", "try this first"]]),
35
    S("BeforeAfter", "Compare it with LoRA on the same eight billion model and the same data. Full fine-tuning trains eight billion parameters in around a hundred and thirty gigabytes. LoRA trains about twenty million in about twenty. The full checkpoint is sixteen gigabytes; the adapter is about forty megabytes. On a narrow task, LoRA usually lands within a point or two. Full fine-tuning also risks more forgetting, where the model gets better at your task and worse at everything else.",
36
      heading="Same 8B model, same data", beforeLabel="full fine-tune", afterLabel="LoRA",
37
      rows=[["trainable parameters", "8 billion", "≈ 20 million"], ["training memory", "≈ 130 GB", "≈ 20 GB"], ["checkpoint", "16 GB", "≈ 40 MB"], ["narrow-task quality", "best", "within 1–2 pts"], ["forgetting risk", "higher", "lower"]],
38
      note="For narrow tasks, try LoRA first; go full when the gap is real and measured."),
39
    S("Bullets", "So when is full fine-tuning worth it? When you are moving the model a long way, like a new language or a highly specialised domain. When you have a lot of data, hundreds of thousands of examples or more. When the model will run on its own dedicated capacity anyway. And when you have evaluations that would catch the model forgetting its general skills.",
40
      heading="When full fine-tuning is worth it",
41
      items=["A big shift: new language, deep domain vocabulary, new modality", "Lots of data: hundreds of thousands of examples or more", "It will run on dedicated capacity anyway", "You have evals that would catch forgetting"]),
42
    S("CommandCard", "You can feel this on a laptop. Lesson seventeen trains the same small model three ways, full, LoRA and DoRA, on the ticket data, and prints peak memory, training time, checkpoint size and accuracy side by side. Watch the peak memory line for the full run.",
43
      heading="Run it · lesson 17 on your Mac",
44
      lines=["# same data, three ways: compare memory and quality", "cd lessons/17-full-vs-peft-mlx", "python peft_params.py --model 0.5b", "bash run_variants.sh        # full · lora · dora", "python compare_variants.py  # memory, time, size, accuracy"]),
45
    S("Recap", "Three things to take away. Full fine-tuning updates every weight: the most powerful option and the most expensive. Training needs about sixteen bytes per parameter against two for serving, so memory decides what is possible. And right-size the model and try LoRA first; go full when the shift is big and the gap is measured.",
46
      heading="Recap", items=["Full fine-tuning updates every weight: most power, most cost", "≈16 bytes per parameter to train vs ≈2 to serve", "Right-size and try LoRA first; go full when the gap is real"]),
47
    S("EndCard", "Next, supervised fine-tuning: the data format almost every customisation starts with. And when you are ready to run it: lesson seventeen, lab T2, full versus LoRA versus DoRA on your Mac.",
48
      line="Lesson 17 · Lab T2 — full vs LoRA vs DoRA on your Mac"),
49
]}
50

51
# ───────────────────────────── 15 · SFT ─────────────────────────────
52
V["sft"] = {"title": "Supervised Fine-Tuning", "subtitle": "Teaching by example", "accent": "copper", "scenes": [
53
    title(15, "Supervised fine-tuning", "Teaching by example",
54
          "what SFT data looks like, why only the answer is graded, and how to tell it is working",
55
          "Supervised fine-tuning is where almost every customisation starts. You show the model examples of exactly what you want, and it learns to imitate them. The idea is simple, and most of the skill is in the data."),
56
    S("TrainingPipeline", "First, where it sits. A base model is pretrained on trillions of tokens, then its creator instruction-tunes it. Supervised fine-tuning is the next step, done by you, on your task: a prompt in, the ideal answer out, thousands of times.", stage=1),
57
    S("LossMask", "Here is one training example for ticket triage: a system message, the customer's ticket, and the ideal JSON answer. The key detail is where the loss is counted. The system and user parts are context, so the model is not graded on predicting them. Only the assistant's answer is graded. This is called masking the prompt. Forget it, and you partly teach the model to write tickets instead of triaging them.",
58
      note="Check your trainer: mlx-lm needs --mask-prompt; TRL needs assistant_only_loss=True."),
59
    S("CommandCard", "On disk it is plain JSON lines, one conversation per line, in the same chat format the API uses. Fireworks, MLX, TRL and Unsloth all accept it, so one dataset works on every platform. Our labs rely on exactly that.",
60
      heading="The data format · JSONL, one example per line",
61
      lines=['{"messages": [', '  {"role": "system",    "content": "You triage tickets..."},', '  {"role": "user",      "content": "Card declined twice..."},', '  {"role": "assistant", "content": "{\\"category\\": \\"billing\\"}"}', "]}", "", "# Fireworks, MLX, TRL and Unsloth all read this format"]),
62
    S("Bullets", "Data quality decides the result. Consistent, so similar inputs get similar answers. Covering, so every category and edge case appears. Clean, because the model learns a wrong label just as confidently as a right one. And held out: split off your test set before training, and never train on it.",
63
      heading="Data beats everything",
64
      items=["Consistent: the same kind of input always gets the same kind of answer", "Covering: every category, edge case and refusal you care about", "Clean: a wrong label is learned as confidently as a right one", "Held out: split the test set off first; never train on it"]),
65
    S("Worked", "How much data? These are typical ranges for a task like triage, not guarantees. Fifty examples mostly fix the format. Five hundred usually buy eight to twelve points of accuracy. Two thousand get most of the way, and then the curve flattens. Twenty thousand noisy examples often do worse than two thousand clean ones. Start small and clean, then add examples exactly where the evaluation shows gaps.",
66
      heading="Worked example · how much data?", scenario="ticket triage · 7B base · LoRA SFT · typical ranges",
67
      rows=[["50 examples", "format fixed, accuracy barely moves"], ["500 examples", "+8 to 12 points"], ["2,000 examples", "+12 to 16, then flattens"], ["20,000 noisy examples", "worse than 2,000 clean"]],
68
      answer="A few hundred clean examples first; add more where evals show gaps", answerLabel="Rule of thumb"),
69
    S("Bullets", "Only a few knobs matter. Epochs, the number of passes over the data: one to three, because more usually memorises. Learning rate: around one times ten to the minus four for LoRA, about ten times smaller for full fine-tuning. A sequence length long enough for your real examples. And watch validation loss, but trust the task evaluation.",
70
      heading="The knobs that matter",
71
      items=["Epochs: 1–3 passes; more usually memorises", "Learning rate: LoRA ≈ 1e-4 · full fine-tune ≈ 1e-5", "Max length: fits your longest real example", "Watch validation loss; decide on the task eval"]),
72
    S("BeforeAfter", "Read the two loss curves together. If training and validation loss both fall, it is learning. If training loss keeps falling while validation loss rises, it is memorising the examples, so stop earlier or add data. If both stay flat and high, it is not learning at all: check the data format, or raise the learning rate.",
73
      heading="Reading the loss curves", beforeLabel="means", afterLabel="do",
74
      rows=[["train ↓ · validation ↓", "learning", "keep going"], ["train ↓ · validation ↑", "memorising", "stop earlier, add data"], ["both flat and high", "not learning", "check data, raise LR"]]),
75
    S("Worked", "Know its limits. Supervised fine-tuning is excellent at format, tone, domain style and house rules. It is poor at injecting facts that change, which is a job for retrieval. And when both candidate answers are acceptable but one is better, you need preference tuning, which is videos seventeen and eighteen.",
76
      heading="What SFT can and cannot do",
77
      rows=[["output format, JSON, tone", "✓ very good"], ["domain vocabulary and style", "✓ good"], ["house rules and refusals", "✓ good"], ["facts that change weekly", "✗ use retrieval"], ["taste between two OK answers", "✗ preference tuning"]],
78
      answer="SFT teaches behaviour, not knowledge"),
79
    S("CommandCard", "In the labs, the same training file goes through three trainers: Fireworks in lesson ten, MLX on your Mac in lesson eleven, and in lesson seventeen a data checker that catches the mistakes that ruin runs: missing assistant turns, duplicates, leaked test examples and over-long rows.",
80
      heading="Run it · lessons 10, 11 and 17",
81
      lines=["# one train.jsonl, three trainers", "python lessons/17-full-vs-peft-mlx/sft_data_check.py", "bash lessons/11-lora-local-mlx/1_train_mlx.sh   # Mac", "bash lessons/10-lora-fireworks/1_train.sh       # managed"]),
82
    S("Recap", "Three things to take away. Supervised fine-tuning means imitating examples, and the loss is counted on the answer only. A few hundred clean, consistent examples beat many noisy ones. And it teaches behaviour and format, not facts that change.",
83
      heading="Recap", items=["SFT imitates examples; loss is counted on the answer only", "A few hundred clean, consistent examples beat many noisy ones", "Teaches behaviour and format, not changing facts"]),
84
    S("EndCard", "Next, parameter-efficient fine-tuning: LoRA and its family, and why it became the default. And when you are ready to run it: lesson seventeen, lab T2. Check the data, then train.",
85
      line="Lesson 17 · Lab T2 — check the data, then train"),
86
]}
87

88
# ───────────────────────────── 16 · PEFT ─────────────────────────────
89
V["peft"] = {"title": "Parameter-Efficient Fine-Tuning", "subtitle": "LoRA and its family", "accent": "teal", "scenes": [
90
    title(16, "PEFT and LoRA", "Parameter-efficient fine-tuning",
91
          "how LoRA works, the arithmetic behind its size, and when to pick QLoRA, DoRA or the others",
92
          "Parameter-efficient fine-tuning, or PEFT, is a family of tricks that lets you customise a huge model by training a tiny fraction of it. LoRA is the famous one. It is the default on almost every platform, including Fireworks."),
93
    S("WeightUpdate", "Here is the idea. Take one weight matrix inside the model, say four thousand by four thousand. Freeze it. Beside it, add two thin matrices, B and A, that multiply together into a correction of the same shape. Only B and A train. The layer's output becomes the original output plus a small learned adjustment.",
94
      mode="lora", note="output = W·x + (α / r) · B·A·x  ·  W never changes"),
95
    S("Worked", "Do the arithmetic. The full matrix holds about sixteen point eight million numbers. With a rank of sixteen, A and B hold about sixty-five thousand each, under one percent of that matrix. Across a whole eight billion parameter model, adapting the attention projections, that is a fraction of one percent of all weights. The rank is the dial: higher rank means more capacity and more memory.",
96
      heading="Worked example · the size of one adapter", scenario="one 4096 × 4096 projection · rank r = 16",
97
      rows=[["full matrix W", "16.8 M"], ["LoRA A  (r × 4096)", "65,536"], ["LoRA B  (4096 × r)", "65,536"], ["trainable share", "0.8% of this matrix"], ["whole 8B model, attention only", "≈ 0.2–0.5% of weights"]],
98
      answer="Rank r is the dial: bigger r, more capacity, more memory"),
99
    S("Bullets", "Why does something so small work? Because the change a fine-tune needs tends to be low rank: a few directions in weight space matter, not all of them. The frozen base keeps its general knowledge, which also reduces forgetting. And after training you can merge the adapter into the weights for zero extra latency, or keep it separate as a tiny file.",
100
      heading="Why it works",
101
      items=["The change a fine-tune needs is low-rank: a few directions matter", "The frozen base keeps its general knowledge", "Merge for zero extra latency, or keep a tiny swappable file"]),
102
    S("CardGrid", "LoRA has relatives. QLoRA runs LoRA on top of a four-bit base model to save memory. DoRA separates each weight into a magnitude and a direction and applies LoRA to the direction, which often adds a point or two for a small speed cost. Adapter layers insert tiny bottleneck blocks. Prefix and prompt tuning learn virtual tokens instead of weights. And IA3 learns simple scaling vectors, the smallest of all.",
103
      heading="The PEFT family", cols=3,
104
      cards=[["LoRA", "two low-rank matrices beside frozen weights", "the default"], ["QLoRA", "LoRA on a 4-bit quantized base", "big models, small memory"], ["DoRA", "magnitude + direction; LoRA on direction", "often +1–2 pts, slower"],
105
             ["Adapters", "small bottleneck layers in each block", "older; adds latency"], ["Prefix / prompt tuning", "learn virtual tokens, not weights", "tiny; weaker on hard tasks"], ["IA³", "learn per-channel scaling vectors", "smallest of all"]]),
106
    S("BeforeAfter", "Choosing is mostly about constraints. If memory is the limit, move from LoRA to QLoRA. If there is still a quality gap against full fine-tuning, try DoRA or a higher rank. And if you have hundreds of customers, each with their own tune, keep the adapters separate and serve them with multi-LoRA.",
107
      heading="Choosing a variant", beforeLabel="default", afterLabel="switch to",
108
      rows=[["memory is the limit", "LoRA", "QLoRA"], ["quality gap vs full", "LoRA r = 16", "DoRA / higher r"], ["hundreds of tenants", "merged models", "multi-LoRA"]]),
109
    S("Worked", "This is where PEFT changes the business. Fifty customers with fifty full fine-tunes means fifty sixteen-gigabyte models and fifty deployments. Fifty LoRA adapters are about forty megabytes each and can all be served from one deployment, with the right adapter chosen per request. The serving bill is roughly one deployment instead of fifty.",
110
      heading="Worked example · 50 tenants", scenario="one 8B base · one tune per customer",
111
      rows=[["50 full fine-tunes", "50 × 16 GB · 50 deployments"], ["50 LoRA adapters", "50 × 40 MB · one deployment"], ["per request", "the right adapter is applied"], ["serving bill", "≈ 1 deployment, not 50"]],
112
      answer="Multi-LoRA makes per-customer tuning affordable"),
113
    S("Bullets", "The knobs are few. Rank: eight to sixteen for narrow tasks, thirty-two to sixty-four for harder ones. Alpha, a scaling factor usually set to about twice the rank. Which modules to adapt: attention only, or attention plus the feed-forward layers for more capacity. And how many layers, where adapting only the last few saves memory.",
114
      heading="The knobs",
115
      items=["rank r: 8–16 narrow tasks · 32–64 harder ones", "alpha: scaling, often ≈ 2 × r", "targets: attention only, or attention + MLP", "layers: last N only saves memory (mlx: --num-layers)"]),
116
    S("CommandCard", "Lesson seventeen has a calculator that prints the trainable parameters for any model and rank. Then it trains full, LoRA and DoRA on the same data, so you can see memory, time, adapter size and accuracy in one table.",
117
      heading="Run it · lesson 17",
118
      lines=["python peft_params.py --model 8b --rank 16", "python peft_params.py --model 70b --rank 64 --mlp", "bash run_variants.sh        # full · lora · dora", "python compare_variants.py  # one table"]),
119
    S("Recap", "Three things to take away. LoRA freezes the weights and trains a low-rank correction beside them. It touches under one percent of the parameters and gets close to full fine-tuning on narrow tasks. And pick QLoRA for memory, DoRA for quality, and multi-LoRA for many tenants.",
120
      heading="Recap", items=["LoRA freezes W and trains a low-rank B·A beside it", "<1% of parameters, near-full quality on narrow tasks", "QLoRA for memory · DoRA for quality · multi-LoRA for tenants"]),
121
    S("EndCard", "Next, preference alignment, starting with RLHF, the technique that made chat models helpful. And when you are ready to run it: lesson seventeen, lab T2.",
122
      line="Lesson 17 · Lab T2 — the PEFT calculator, then three runs"),
123
]}
124

125
# ───────────────────────────── 17 · RLHF ─────────────────────────────
126
V["rlhf"] = {"title": "RLHF", "subtitle": "Reinforcement learning from human feedback", "accent": "copper", "scenes": [
127
    title(17, "RLHF", "Reinforcement learning from human feedback",
128
          "why imitation is not enough, the three stages of RLHF, and what the KL leash is for",
129
          "Supervised fine-tuning teaches a model to imitate. But for many tasks there is no single right answer, only better and worse ones. Reinforcement learning from human feedback, RLHF, is how models learn that difference. It is the technique behind the first helpful chat assistants."),
130
    S("PrefPair", "Here is why. A customer writes in, angry about an outage. Answer A apologises, explains the cause, and offers a fix and a credit. Answer B is a technically accurate root cause with no apology and no next step. Both are correct. Writing the perfect reply is hard, but anyone can say which of these two is better. RLHF learns from exactly that kind of judgement.",
131
      heading="Why preferences, not answers", prompt="Customer: “Your outage cost us a day. Explain.”",
132
      chosen="Apologises, gives cause and timeline, states the fix and offers a credit.", rejected="Accurate root-cause paragraph. No apology, no next step.",
133
      note="Both are correct. People struggle to write the perfect answer, but easily pick the better one."),
134
    S("RlhfLoop", "RLHF has three stages. Stage one: start from a supervised fine-tuned model, one that already follows instructions and gives reasonable answers. You cannot reinforce behaviour the model never produces.", stage=0),
135
    S("RlhfLoop", "Stage two: collect comparisons. The model writes two answers to each prompt, and people pick the better one. Those choices train a reward model, a copy of the language model that outputs a single number. It learns to predict which answer a person would prefer, so it can score any new answer without a person in the loop.", stage=1),
136
    S("Worked", "Concretely, the reward model might score the empathetic answer two point one and the curt one minus zero point four. The difference, passed through a sigmoid, says there is about a ninety-two percent chance a person prefers A. That is the Bradley-Terry model, trained on tens of thousands of human comparisons.",
137
      heading="Worked example · the reward model's job", scenario="prompt: the outage complaint",
138
      rows=[["answer A · apology, cause, fix, credit", "score 2.1"], ["answer B · accurate, no apology", "score −0.4"], ["P(person prefers A) = σ(2.1 + 0.4)", "≈ 92%"], ["training data", "tens of thousands of comparisons"]],
139
      answer="The reward model turns human judgement into a number an optimizer can chase"),
140
    S("RlhfLoop", "Stage three is the reinforcement learning loop, usually with an algorithm called PPO. The model, now called the policy, writes answers. The reward model scores them. The policy is updated to make high-scoring answers more likely. Round and round, many thousands of times.", stage=2),
141
    S("Bullets", "There is a catch. The reward model only approximates human taste, and a policy that optimises it hard will find loopholes: flattery, padding, repeating phrases the reward model likes. This is called reward hacking. The fix is a leash. A KL penalty charges the policy for drifting too far from the original model, and beta sets the length of the leash.",
142
      heading="The KL leash",
143
      items=["Reward models are imperfect, and can be gamed", "Unleashed: flattery, padding, repeated phrases (reward hacking)", "KL penalty: pay for drifting from the reference model", "β = leash length: too tight learns nothing, too loose hacks"]),
144
    S("BeforeAfter", "It works, but it is heavy. Supervised fine-tuning needs one model in memory. PPO needs four: the policy, a frozen reference, the reward model and a value model. It needs lots of human comparisons, and it has many hyperparameters and can be unstable. That cost is exactly why the next technique, DPO, was invented.",
145
      heading="What RLHF costs", beforeLabel="SFT", afterLabel="RLHF with PPO",
146
      rows=[["models in memory", "1", "4"], ["human data", "prompt + answer", "ranked comparisons"], ["stability", "simple loss", "many knobs, can diverge"]],
147
      note="Four models: policy, frozen reference, reward model, value model."),
148
    S("CardGrid", "The family has grown. RLAIF replaces human labellers with an AI judge. GRPO, used for many reasoning models, compares a group of answers against each other and drops the value model. And reinforcement fine-tuning with graders uses a program to score each answer, such as tests passing or an exact match. Fireworks offers that as RFT.",
149
      heading="The reinforcement family today", cols=2,
150
      cards=[["RLHF + PPO", "human preferences → reward model → PPO", "the original recipe"], ["RLAIF", "an AI judge replaces human labels", "cheaper labels"],
151
             ["GRPO", "score a group of answers against each other; no value model", "popular for reasoning"], ["RFT with graders", "a program scores 0–1: tests pass, JSON valid, answer matches", "Fireworks RFT"]]),
152
    S("CommandCard", "Lesson sixteen runs the entire RLHF pipeline on a toy problem, in pure numpy, in a few seconds. It does supervised fine-tuning, fits a reward model to preference pairs, and optimises a policy with and without the KL leash, so you can watch reward hacking happen. Lesson eighteen points to the real trainers.",
153
      heading="Run it · lesson 16 (toy) and 18",
154
      lines=["# the whole RLHF pipeline in numpy, seconds", "python lessons/16-training-toy/toy_alignment.py", "python lessons/16-training-toy/toy_alignment.py --beta 0", "", "# real PPO / GRPO on a Mac (optional, advanced)", "pip install mlx-lm-lora"]),
155
    S("Recap", "Three things to take away. RLHF learns from comparisons when there is no single right answer. It runs in three stages: supervised fine-tuning, a reward model, then PPO with a KL leash. And it is powerful but heavy: four models and many knobs.",
156
      heading="Recap", items=["RLHF learns from comparisons when no single answer is right", "SFT → reward model → PPO, with a KL leash", "Powerful but heavy: four models, many knobs"]),
157
    S("EndCard", "Next, DPO, which gets most of the benefit of RLHF with none of the reinforcement learning machinery. And when you are ready to run it: lesson sixteen, lab T1.",
158
      line="Lesson 16 · Lab T1 — watch reward hacking in a toy RLHF"),
159
]}
160

161
# ───────────────────────────── 18 · DPO ─────────────────────────────
162
V["dpo"] = {"title": "Direct Preference Optimization", "subtitle": "RLHF's result, without the RL", "accent": "teal", "scenes": [
163
    title(18, "DPO", "Direct preference optimization",
164
          "how DPO learns from pairs, what β does, and how to build preference data from your own product",
165
          "Direct preference optimization, DPO, is the shortcut. It learns from the same preference pairs as RLHF, but skips the reward model and the reinforcement loop entirely. Today it is one of the most common ways teams align a model to their taste."),
166
    S("Bullets", "The insight, from a twenty twenty-three paper, is mathematical. The best possible policy under the RLHF objective can be written in terms of the reward, which means the reward can be written in terms of the policy. Substitute that in, and you can train straight on preference pairs with a simple loss: no reward model and no sampling loop.",
167
      heading="The insight",
168
      items=["RLHF's optimal policy has a closed form in terms of the reward", "So the reward can be expressed with the policy itself", "Train directly on pairs with a classification-style loss"]),
169
    S("DpoShift", "Here is what that loss does. For each pair, it compares how likely the model makes the chosen answer and the rejected one, each relative to a frozen reference copy. Training pushes the chosen answer up and the rejected one down. Beta controls how far it may move from the reference: the same leash as before, built into the loss.",
170
      note="β is the leash from RLHF, built into the loss."),
171
    S("BeforeAfter", "Side by side with PPO: two models in memory instead of four, no separate reward model, no sampling during training, and a training loop about as stable as supervised fine-tuning. That simplicity is why it spread so quickly.",
172
      heading="DPO vs RLHF with PPO", beforeLabel="RLHF + PPO", afterLabel="DPO",
173
      rows=[["models in memory", "4", "2 (policy + reference)"], ["reward model", "trained separately", "implicit"], ["sampling in training", "yes · slow", "no"], ["stability", "many knobs", "close to SFT"]]),
174
    S("PrefPair", "Where do pairs come from? Often, from your own product. When a support agent edits a model's draft, the edited version is chosen and the original is rejected. A thumbs-down, a regenerate or a correction all create pairs. Here, the concise, valid JSON is chosen, and the chatty answer that breaks the parser is rejected.",
175
      heading="Preference data from your own product", prompt="Ticket: “Card declined twice, launch is tomorrow.”",
176
      chosen='{"category": "billing", "severity": 4, "next_action": "route to billing"}', rejected="Sure! This looks like a billing issue, and I'd say it's fairly urgent…",
177
      note="Chosen: what an agent approved. Rejected: what they edited, or a thumbs-down."),
178
    S("Worked", "One step, with numbers. Relative to the reference, the chosen answer has become zero point eight more likely in log terms, and the rejected one zero point four less. The margin is one point two, and times a beta of zero point one gives zero point one two. The loss is minus log sigmoid of that: about zero point six three, down from zero point six nine when both were equal. More separation means lower loss.",
179
      heading="Worked example · one DPO step", scenario="β = 0.1 · log-probabilities relative to the reference",
180
      rows=[["chosen:   log π − log π_ref", "+0.8"], ["rejected: log π − log π_ref", "−0.4"], ["β × margin", "0.1 × 1.2 = 0.12"], ["loss = −log σ(0.12)", "0.63  (0.69 at start)"]],
181
      answer="The loss falls as chosen and rejected pull apart", answerLabel="Read it"),
182
    S("Bullets", "A few rules. Do supervised fine-tuning first: DPO refines behaviour, it does not teach the task. Make pairs differ in the thing you care about, not in random noise. Start beta around zero point one. And evaluate both the win rate against the old model and the task accuracy, because improving taste can quietly cost correctness.",
183
      heading="Getting DPO right",
184
      items=["SFT first: DPO refines a model that already does the task", "Pairs should differ in the thing you care about", "Start β ≈ 0.1; lower moves further from the reference", "Measure win-rate AND task accuracy"]),
185
    S("CardGrid", "There are relatives. IPO regularises against over-fitting the pairs. KTO learns from single thumbs up or down, no pairs needed. ORPO folds supervised fine-tuning and preference into one step with no reference model, and Fireworks supports it alongside DPO. SimPO is reference-free and length-normalised.",
186
      heading="DPO's relatives", cols=2,
187
      cards=[["IPO", "regularises against over-fitting the pairs", ""], ["KTO", "single thumbs-up / thumbs-down, no pairs", ""],
188
             ["ORPO", "SFT + preference in one step, no reference model", "Fireworks: --loss-method ORPO"], ["SimPO", "reference-free, length-normalised", ""]]),
189
    S("CommandCard", "In lesson eighteen you build pairs from the ticket data, train DPO on your Mac with MLX, optionally run the same job on Fireworks with one command, and compare valid-JSON rate, accuracy and answer length before and after.",
190
      heading="Run it · lesson 18",
191
      lines=["python make_pairs.py            # pairs from the ticket data", "bash dpo_mlx.sh                 # mlx_lm_lora --train-mode dpo", "bash dpo_fireworks.sh           # firectl dpo-job create", "python pref_eval.py --target mlx"]),
192
    S("Recap", "Three things to take away. DPO learns from chosen and rejected pairs with a simple loss. It needs two models, no reward model and no reinforcement loop, so it is nearly as stable as supervised fine-tuning. And do SFT first, then DPO, and measure both win rate and task accuracy.",
193
      heading="Recap", items=["DPO learns from chosen / rejected pairs with a simple loss", "Two models, no reward model, no RL loop: nearly as stable as SFT", "SFT first, then DPO; measure win-rate and accuracy"]),
194
    S("EndCard", "That completes the training series: full fine-tuning, supervised fine-tuning, LoRA and its family, RLHF, and DPO. When you are ready to run it: lesson eighteen, lab T3, DPO on your Mac and on Fireworks.",
195
      line="Lesson 18 · Lab T3 — DPO on your Mac and on Fireworks"),
196
]}
197

198
# ───────────── fix the existing 13: kickers "of 18" and end cards → lessons + current lab ids ─────────────
199
END = {
200
    "inference-101": ("Lesson 01 · Lab F1 — build your own TTFT and ITL harness", "lesson one, lab F1: build your own TTFT and ITL harness."),
201
    "kv-cache": ("Lesson 05 · Lab L5 — push context until it breaks, then fix it", "lesson five, lab L5: push context until it breaks, then fix it."),
202
    "gpu-bandwidth": ("Lessons 03–04 · Labs L3, L4 — sweep, then check the ceiling", "lessons three and four, labs L3 and L4: run the concurrency sweep, then check it against the ceiling."),
203
    "quantization": ("Lesson 04 · Lab L4 — three precisions, measured", "lesson four, lab L4: three precisions of one model, measured."),
204
    "batching": ("Lessons 03 & 13 · Labs L3, F7 — sweep, then add speculation", "lessons three and thirteen, labs L3 and F7: sweep concurrency, then add speculation."),
205
    "training": ("Lessons 10–11 · Labs F5, L8 — one fine-tune, managed and local", "lessons ten and eleven, labs F5 and L8: the same fine-tune, managed and local. The training deep dives follow in videos fourteen to eighteen."),
206
    "offloading": ("Lesson 12 · Lab L7 — MoE active bytes from decode speed", "lesson twelve, lab L7: measure MoE active bytes from decode speed."),
207
}
208
FULL_END = {  # videos whose end-card narration is written from scratch
209
    "platform": ("Lesson 15 · Capstone — one POC, one benchmark, one sizing memo",
210
                 "That is the whole map. Now go and build it, because the fastest way to sound credible is to have actually run it. When you are ready: lesson fifteen, the capstone. One POC, one benchmark, one sizing memo."),
211
    "serving-stack": ("Lessons 02 & 14 · Labs L2, L9 — serve it, then benchmark it",
212
                      "Run lesson two, lab L2, to get servers up on your Mac, and lesson fourteen, lab L9, for vLLM and SGLang on the Spark. Then benchmark them in lesson three."),
213
    "benchmarking": ("Lesson 03 · Lab L3 — the concurrency sweep",
214
                     "Run lesson three, lab L3, and keep the output. It becomes the first exhibit in your capstone memo."),
215
    "prefix-caching": ("Lesson 06 · Labs L6, F2 — measure it, then price it",
216
                       "Lesson six measures this on your own hardware with lab L6, and lab F2 shows the billing side on Fireworks."),
217
    "qlora": ("Lessons 11 & 10 · Labs L8, F5 — run both",
218
              "Lesson eleven, lab L8, runs this locally. Lesson ten, lab F5, runs the same dataset through Fireworks. For the full picture of training methods, see videos fourteen to eighteen."),
219
    "scale-out": ("Lessons 13–14 · Labs F7, L9 — speculation, then multi-node",
220
                  "Lessons thirteen and fourteen close out the serving track. Then come the training deep dives, videos fourteen to eighteen, and the capstone."),
221
}
222
KICK_SUFFIX = {"serving-stack": " · L2 and L9", "benchmarking": " · L3", "prefix-caching": " · L6 and F2", "qlora": " · L8 and F5", "scale-out": " · F7 and L9"}
223

224
changed = {}
225
for i, (vid, v) in enumerate(list(d.items()), start=1):
226
    t = v["scenes"][0]
227
    t["params"]["kicker"] = f"Video {i} of 18" + KICK_SUFFIX.get(vid, "")
228
    e = v["scenes"][-1]
229
    if vid in END:
230
        line, tail = END[vid]
231
        head = e["text"].split(" And when you are ready to run it:")[0]
232
        e["params"]["line"] = line
233
        e["text"] = f"{head} And when you are ready to run it: {tail}"
234
    elif vid in FULL_END:
235
        e["params"]["line"], e["text"] = FULL_END[vid]
236
    changed[vid] = [0, len(v["scenes"]) - 1]
237

238
d.update(V)
239
P.write_text(json.dumps(d, indent=2, ensure_ascii=False))
240
Path(P.parent.parent / "scripts" / ".changed_scenes.json").write_text(json.dumps(changed))
241
print(f"{len(d)} videos; new: {list(V)}")
242
for k, v in V.items():
243
    print(k, len(v["scenes"]), "scenes", sum(len(s["text"].split()) for s in v["scenes"]), "words")

scripts/render-all.sh Bash · 17 lines

1
#!/usr/bin/env bash
2
# Render every explainer to out/<id>.mp4 (1440x810, subtitles burned in)
3
set -uo pipefail
4
cd "$(dirname "$0")/.."
5
IDS=${IDS:-$(node -e "console.log(Object.keys(require('./src/narration.json')).join(' '))")}
6
mkdir -p out
7
BROWSER=${BROWSER_BIN:-}
8
[ -n "$BROWSER" ] && BFLAG="--browser-executable=$BROWSER" || BFLAG=""
9
[ -z "$BROWSER" ] && npx remotion browser ensure
10
for id in $IDS; do
11
  [ -s "out/$id.mp4" ] && { echo "skip $id (exists)"; continue; }
12
  echo "── rendering $id  $(date +%H:%M:%S)"
13
  npx remotion render "$id" "out/$id.mp4" $BFLAG \
14
    --concurrency=2 --scale=0.65 --jpeg-quality=78 --log=error || echo "FAILED $id"
15
done
16
echo "done $(date +%H:%M:%S)"
17
ls -lh out/*.mp4

scripts/tts_mlx.py Python · 96 lines

1
#!/usr/bin/env python3
2
"""
3
Generate narration on Apple silicon with Kokoro-82M through MLX (mlx-audio).
4

5
  pip install mlx-audio misaki soundfile numpy
6
  python scripts/tts_mlx.py                       # all videos
7
  python scripts/tts_mlx.py --video kv-cache      # one video
8
  python scripts/tts_mlx.py --voice am_michael --speed 1.0
9

10
Writes public/audio/<video>/NN.wav (24 kHz) and public/scenes/<video>.json,
11
which is what Remotion reads to size each scene. Identical output contract to
12
scripts/tts_onnx.py (the Linux/CPU path used to render the delivered MP4s).
13
"""
14
from __future__ import annotations
15
import argparse, json, pathlib, re
16
import numpy as np, soundfile as sf
17

18
ROOT = pathlib.Path(__file__).resolve().parents[1]
19
SR = 24_000
20

21
# spoken forms for things a TTS model reads badly
22
SUBS = [
23
    (r'\bRLHF\b', 'R L H F'), (r'\bRLAIF\b', 'R L A I F'), (r'\bPPO\b', 'P P O'), (r'\bGRPO\b', 'G R P O'),
24
    (r'\bPEFT\b', 'peft'), (r'\bDoRA\b', 'Dora'), (r'\bKL\b', 'K L'), (r'\bORPO\b', 'or po'), (r'\bIPO\b', 'I P O'),
25
    (r'\bKTO\b', 'K T O'), (r'\bSimPO\b', 'sim po'), (r'\bIA3\b', 'I A three'), (r'\bMLX\b', 'M L X'), (r'\bTRL\b', 'T R L'),
26
    (r'\bFSDP\b', 'F S D P'), (r'\bZeRO\b', 'zero'), (r'\bbf16\b', 'B F sixteen'), (r'\bPOC\b', 'P O C'), (r'\bJSON\b', 'jason'),
27
    (r'\bvs\b', 'versus'), (r'\b([FLT])([1-9])\b', r'\1 \2'), (r'β', 'beta'), (r'σ', 'sigma'), (r'π', 'pi'),
28
    (r'\bTTFT\b', 'T T F T'), (r'\bITL\b', 'I T L'), (r'\bTPOT\b', 'T POT'),
29
    (r'\bKV\b', 'K V'), (r'\bp95\b', 'p ninety five'), (r'\bp50\b', 'p fifty'),
30
    (r'\bGB/s\b', 'gigabytes per second'), (r'\bTB/s\b', 'terabytes per second'),
31
    (r'\btok/s\b', 'tokens per second'), (r'\bFP(\d+)\b', r'F P \1'), (r'\bINT4\b', 'int four'),
32
    (r'\bNVFP4\b', 'N V F P four'), (r'\bMXFP4\b', 'M X F P four'), (r'\bLoRA\b', 'LoRah'),
33
    (r'\bQLoRA\b', 'Q LoRah'), (r'\bSFT\b', 'S F T'), (r'\bDPO\b', 'D P O'), (r'\bRFT\b', 'R F T'),
34
    (r'\bMoE\b', 'mixture of experts'), (r'\bvLLM\b', 'v L L M'), (r'\bSGLang\b', 'S G Lang'),
35
    (r'\bGPU\b', 'G P U'), (r'\bGPUs\b', 'G P Uz'), (r'\bAPI\b', 'A P I'), (r'\bSLA\b', 'S L A'),
36
    (r'\bH100\b', 'H one hundred'), (r'\bH200\b', 'H two hundred'), (r'\bB200\b', 'B two hundred'),
37
    (r'×', ' times '), (r'÷', ' divided by '), (r'=', ' equals '), (r'→', ' then '),
38
    (r'α', 'alpha'), (r'\^', ' to the power '), (r'\s*·\s*', ', '), (r'—', ', '), (r'–', ' to '),
39
    (r'\$', ' dollars '), (r'%', ' percent'),
40
]
41

42
def speakable(t: str) -> str:
43
    for pat, rep in SUBS:
44
        t = re.sub(pat, rep, t)
45
    return re.sub(r'\s+', ' ', t).strip()
46

47
def main() -> int:
48
    ap = argparse.ArgumentParser()
49
    ap.add_argument('--model', default='mlx-community/Kokoro-82M-bf16')
50
    ap.add_argument('--voice', default='af_heart')
51
    ap.add_argument('--speed', type=float, default=0.96)
52
    ap.add_argument('--lang', default='a')
53
    ap.add_argument('--video')
54
    a = ap.parse_args()
55

56
    from mlx_audio.tts.utils import load_model   # imported late so --help works without mlx
57

58
    model = load_model(a.model)
59
    data = json.loads((ROOT / 'src' / 'narration.json').read_text())
60
    ids = [a.video] if a.video else list(data)
61
    (ROOT / 'public' / 'scenes').mkdir(parents=True, exist_ok=True)
62

63
    for vid in ids:
64
        v = data[vid]
65
        out = ROOT / 'public' / 'audio' / vid
66
        out.mkdir(parents=True, exist_ok=True)
67
        scenes = []
68
        print(f'\n▶ {vid} — {v["title"]}', flush=True)
69

70
        for i, sc in enumerate(v['scenes'], start=1):
71
            chunks = []
72
            for result in model.generate(text=speakable(sc['text']), voice=a.voice,
73
                                         speed=a.speed, lang_code=a.lang):
74
                chunks.append(np.asarray(result.audio, dtype=np.float32).reshape(-1))
75
            wav = np.concatenate(chunks)
76
            lead, tail = 0.35, (1.1 if sc['visual'] in ('Title', 'Recap', 'EndCard') else 0.7)
77
            wav = np.concatenate([np.zeros(int(lead * SR), np.float32), wav,
78
                                  np.zeros(int(tail * SR), np.float32)])
79
            name = f'{i:02d}.wav'
80
            sf.write(out / name, wav, SR)
81
            secs = len(wav) / SR
82
            scenes.append({'index': i, 'visual': sc['visual'], 'audio': f'audio/{vid}/{name}',
83
                           'durationSeconds': round(secs, 3), 'text': sc['text'], 'leadSeconds': lead})
84
            print(f'   {name} {secs:6.2f}s  {sc["visual"]}', flush=True)
85

86
        payload = {'video': vid, 'voice': a.voice, 'sampleRate': SR,
87
                   'totalSeconds': round(sum(s['durationSeconds'] for s in scenes), 2),
88
                   'scenes': scenes}
89
        (ROOT / 'public' / 'scenes' / f'{vid}.json').write_text(json.dumps(payload, indent=2))
90
        print(f'   → {payload["totalSeconds"]}s total', flush=True)
91

92
    print('\nDone. npm run dev to preview, npm run render to export.')
93
    return 0
94

95
if __name__ == '__main__':
96
    raise SystemExit(main())

scripts/tts_onnx.py Python · 74 lines

1
#!/usr/bin/env python3
2
"""Generate narration with Kokoro (ONNX runtime build). See scripts/tts_mlx.py for the Mac/MLX path."""
3
import argparse, json, pathlib, re, sys
4
import numpy as np, soundfile as sf
5
from kokoro_onnx import Kokoro
6

7
ROOT = pathlib.Path(__file__).resolve().parents[1]
8
SR = 24_000
9
SUBS = [
10
    (r'\bRLHF\b', 'R L H F'), (r'\bRLAIF\b', 'R L A I F'), (r'\bPPO\b', 'P P O'), (r'\bGRPO\b', 'G R P O'),
11
    (r'\bPEFT\b', 'peft'), (r'\bDoRA\b', 'Dora'), (r'\bKL\b', 'K L'), (r'\bORPO\b', 'or po'), (r'\bIPO\b', 'I P O'),
12
    (r'\bKTO\b', 'K T O'), (r'\bSimPO\b', 'sim po'), (r'\bIA3\b', 'I A three'), (r'\bMLX\b', 'M L X'), (r'\bTRL\b', 'T R L'),
13
    (r'\bFSDP\b', 'F S D P'), (r'\bZeRO\b', 'zero'), (r'\bbf16\b', 'B F sixteen'), (r'\bPOC\b', 'P O C'), (r'\bJSON\b', 'jason'),
14
    (r'\bvs\b', 'versus'), (r'\b([FLT])([1-9])\b', r'\1 \2'), (r'β', 'beta'), (r'σ', 'sigma'), (r'π', 'pi'),
15
    (r'\bTTFT\b', 'T T F T'), (r'\bITL\b', 'I T L'), (r'\bTPOT\b', 'T POT'),
16
    (r'\bKV\b', 'K V'), (r'\bp95\b', 'p ninety five'), (r'\bp50\b', 'p fifty'),
17
    (r'\bGB/s\b', 'gigabytes per second'), (r'\bTB/s\b', 'terabytes per second'),
18
    (r'\btok/s\b', 'tokens per second'), (r'\bFP(\d+)\b', r'F P \1'), (r'\bINT4\b', 'int four'),
19
    (r'\bNVFP4\b', 'N V F P four'), (r'\bMXFP4\b', 'M X F P four'), (r'\bLoRA\b', 'LoRah'),
20
    (r'\bQLoRA\b', 'Q LoRah'), (r'\bSFT\b', 'S F T'), (r'\bDPO\b', 'D P O'), (r'\bRFT\b', 'R F T'),
21
    (r'\bMoE\b', 'mixture of experts'), (r'\bvLLM\b', 'v L L M'), (r'\bSGLang\b', 'S G Lang'),
22
    (r'\bGPU\b', 'G P U'), (r'\bGPUs\b', 'G P Uz'), (r'\bAPI\b', 'A P I'), (r'\bSLA\b', 'S L A'),
23
    (r'\bH100\b', 'H one hundred'), (r'\bH200\b', 'H two hundred'), (r'\bB200\b', 'B two hundred'),
24
    (r'×', ' times '), (r'÷', ' divided by '), (r'=', ' equals '), (r'→', ' then '),
25
    (r'α', 'alpha'), (r'\^', ' to the power '), (r'\s*·\s*', ', '), (r'—', ', '), (r'–', ' to '),
26
    (r'\bα\b', 'alpha'), (r'\$', ' dollars '), (r'%', ' percent'),
27
]
28

29
def speakable(t: str) -> str:
30
    for pat, rep in SUBS:
31
        t = re.sub(pat, rep, t)
32
    return re.sub(r'\s+', ' ', t).strip()
33

34
def main() -> int:
35
    ap = argparse.ArgumentParser()
36
    ap.add_argument('--voice', default='af_heart')
37
    ap.add_argument('--speed', type=float, default=0.96)
38
    ap.add_argument('--video')
39
    ap.add_argument('--model', default=str(ROOT.parent / 'tts' / 'kokoro-v1.0.onnx'))
40
    ap.add_argument('--voices', default=str(ROOT.parent / 'tts' / 'voices-v1.0.bin'))
41
    a = ap.parse_args()
42

43
    data = json.loads((ROOT / 'src' / 'narration.json').read_text())
44
    ids = [a.video] if a.video else list(data)
45
    k = Kokoro(a.model, a.voices)
46
    (ROOT / 'public' / 'scenes').mkdir(parents=True, exist_ok=True)
47

48
    for vid in ids:
49
        v = data[vid]
50
        out = ROOT / 'public' / 'audio' / vid
51
        out.mkdir(parents=True, exist_ok=True)
52
        scenes = []
53
        print(f'\n▶ {vid} — {v["title"]}', flush=True)
54
        for i, sc in enumerate(v['scenes'], start=1):
55
            wav, _ = k.create(speakable(sc['text']), voice=a.voice, speed=a.speed, lang='en-us')
56
            wav = np.asarray(wav, dtype=np.float32)
57
            # breathing room: a beat before, a longer beat after title/recap/end scenes
58
            lead = 0.35
59
            tail = 1.1 if sc['visual'] in ('Title', 'Recap', 'EndCard') else 0.7
60
            wav = np.concatenate([np.zeros(int(lead * SR), np.float32), wav, np.zeros(int(tail * SR), np.float32)])
61
            name = f'{i:02d}.wav'
62
            sf.write(out / name, wav, SR)
63
            secs = len(wav) / SR
64
            scenes.append({'index': i, 'visual': sc['visual'], 'audio': f'audio/{vid}/{name}',
65
                           'durationSeconds': round(secs, 3), 'text': sc['text'], 'leadSeconds': lead})
66
            print(f'   {name} {secs:6.2f}s  {sc["visual"]}', flush=True)
67
        payload = {'video': vid, 'voice': a.voice, 'sampleRate': SR,
68
                   'totalSeconds': round(sum(s['durationSeconds'] for s in scenes), 2), 'scenes': scenes}
69
        (ROOT / 'public' / 'scenes' / f'{vid}.json').write_text(json.dumps(payload, indent=2))
70
        print(f'   → {payload["totalSeconds"]}s total', flush=True)
71
    return 0
72

73
if __name__ == '__main__':
74
    raise SystemExit(main())

scripts/tts_patch.py Python · 32 lines

1
#!/usr/bin/env python3
2
"""Regenerate only some scenes of an already-voiced video (e.g. after editing an end card).
3
    python scripts/tts_patch.py --video kv-cache --scenes last
4
    python scripts/tts_patch.py --video kv-cache --scenes 0,9
5
Keeps every other scene's audio and timing untouched."""
6
import argparse, json, pathlib, sys
7
import numpy as np, soundfile as sf
8
sys.path.insert(0, str(pathlib.Path(__file__).parent))
9
from tts_onnx import ROOT, SR, speakable  # same substitutions and pacing rules
10
from kokoro_onnx import Kokoro
11

12
ap = argparse.ArgumentParser()
13
ap.add_argument('--video', required=True)
14
ap.add_argument('--scenes', required=True, help="comma list of 0-based indices, or 'last'")
15
ap.add_argument('--voice', default='af_heart'); ap.add_argument('--speed', type=float, default=0.96)
16
a = ap.parse_args()
17
data = json.loads((ROOT / 'src' / 'narration.json').read_text())[a.video]
18
tj = ROOT / 'public' / 'scenes' / f'{a.video}.json'
19
timing = json.loads(tj.read_text())
20
idx = [len(data['scenes']) - 1] if a.scenes == 'last' else [int(x) for x in a.scenes.split(',')]
21
k = Kokoro(str(ROOT.parent / 'tts' / 'kokoro-v1.0.onnx'), str(ROOT.parent / 'tts' / 'voices-v1.0.bin'))
22
for i in idx:
23
    sc = data['scenes'][i]
24
    wav, _ = k.create(speakable(sc['text']), voice=a.voice, speed=a.speed, lang='en-us')
25
    lead, tail = 0.35, (1.1 if sc['visual'] in ('Title', 'Recap', 'EndCard') else 0.7)
26
    wav = np.concatenate([np.zeros(int(lead * SR), np.float32), np.asarray(wav, np.float32), np.zeros(int(tail * SR), np.float32)])
27
    name = f'{i + 1:02d}.wav'
28
    sf.write(ROOT / 'public' / 'audio' / a.video / name, wav, SR)
29
    timing['scenes'][i].update(durationSeconds=round(len(wav) / SR, 3), text=sc['text'], visual=sc['visual'])
30
    print(f'{a.video} scene {i + 1}: {len(wav) / SR:.2f}s')
31
timing['totalSeconds'] = round(sum(s['durationSeconds'] for s in timing['scenes']), 2)
32
tj.write_text(json.dumps(timing, indent=2))