Lesson
Define the machine before you build it
Turn “build an LLM” into a ladder of testable artifacts: scalar autograd, a character model, a Transformer, then a budgeted training run.
Primary workshop
YouTubeSkim the ten-video sequence; do not start the GPT-2 capstone yet.
Narrated mechanism
Use this shorter explanation to trace the mechanism, implementation order, and failure checks.
Distinguish a toy language model, a pretrained base model, and a chat assistant.
Choose a capstone tier that matches available compute.
Explain the course’s recurring training loop in one minute.
Build the mental model
See the mechanism before memorizing the code.
Before pressing play, predict how “text becomes tokens” eventually produces “evaluation decides what survives.” Keep the prediction visible and revise it only after your build produces evidence.
Explanatory diagram
Mechanism, end to end
Trace the causal path from “text becomes tokens” to “evaluation decides what survives.” Every arrow names a transformation that should preserve a testable invariant.
- 01text becomes tokens. Start with an input whose meaning and shape you can state explicitly.
- 02a network predicts the next token. Inspect this stage’s input, output, and invariant before moving to the next transformation.
- 03loss becomes gradients. Inspect this stage’s input, output, and invariant before moving to the next transformation.
- 04an optimizer changes weights. Inspect this stage’s input, output, and invariant before moving to the next transformation.
- 05evaluation decides what survives. This is the observable result you test before increasing scale.
Explanatory diagram
What becomes observable
A model is debuggable when each conceptual stage leaves a signal you can inspect instead of hiding behind the final loss.
- 01data. Make data visible in a printout, plot, assertion, or controlled comparison.
- 02tokens. Make tokens visible in a printout, plot, assertion, or controlled comparison.
- 03model. Make model visible in a printout, plot, assertion, or controlled comparison.
- 04loss. Make loss visible in a printout, plot, assertion, or controlled comparison.
- 05update. Make update visible in a printout, plot, assertion, or controlled comparison.
Explanatory diagram
The build–test–explain loop
Work at the smallest scale, compare against a trusted reference, and return to the first broken invariant.
- 01Build small. Create a repository with data.py, model.py, train.py, sample.py, and tests/.
- 02Predict. Distinguish a toy language model, a pretrained base model, and a chat assistant.
- 03Compare. Write down your lane: CPU, single GPU, or cluster.
- 04Inspect. Starting with GPT-2 optimization before a tiny model can overfit one batch
Why it works
The finish line is behavior, not parameter count
A useful first build is small enough to inspect. You should be able to overfit one batch, trace tensor shapes, sample text, and explain why the loss moves. GPT-2-scale reproduction is a later systems exercise, not proof that the fundamentals are understood.
Three different things are often called “an LLM”
A base model predicts continuations from pretraining data. Supervised fine-tuning changes the dataset and behavior. Preference or reinforcement stages further shape responses. This course builds the base-model mechanism first and labels assistant training as a separate extension.
Pick a compute lane now
The CPU lane trains tiny character or byte models. The single-GPU lane scales the same code to a compact Transformer. The cluster lane reproduces GPT-2-style throughput work. The concepts are shared; the batch size, model width, dataset, and runtime are not.
Implementation
Implementation compass
# One invariant survives every scale
logits = model(x) # [batch, time, vocab]
loss = cross_entropy(logits, y) # one scalar
loss.backward() # gradients for parameters
optimizer.step() # a slightly better modelBuild gate
- 01Create a repository with data.py, model.py, train.py, sample.py, and tests/.
- 02Write down your lane: CPU, single GPU, or cluster.
- 03Set one observable finish line: “generate 200 tokens after training and explain every tensor shape.”
- 04Record a hard budget for time and cloud spend before any long run.
Failure signatures
Primary evidence
Knowledge check
Make the model stick
1. Which artifact is produced directly by ordinary next-token pretraining?
2. What is the strongest first-course finish line?