01
01 · useful win
Define the machine before you build it
Turn “build an LLM” into a ladder of testable artifacts: scalar autograd, a character model, a Transformer, then a budgeted training run.
Distinguish a toy language model, a pretrained base model, and a chat assistant.
02
The invariant chain
text becomes tokens becomes a testable update
text becomes tokens
a network predicts the next token
loss becomes gradients
an optimizer changes weights
evaluation decides what survives
03
Why it works
The finish line is behavior, not parameter count
A useful first build is small enough to inspect. You should be able to overfit one batch, trace tensor shapes, sample text, and explain why the loss moves. GPT-2-scale reproduction is a later systems exercise, not proof that the fundamentals are understood.
1The finish line is behavior, not parameter count
2Three different things are often called “an LLM”
3loss becomes gradients becomes observable
04
Implementation compass
Pick a compute lane now
# One invariant survives every scale
logits = model(x) # [batch, time, vocab]
loss = cross_entropy(logits, y) # one scalar
loss.backward() # gradients for parameters
optimizer.step() # a slightly better modelShapeWrite every semantic axis beside the tensor.
ReferenceCompare one output against the smallest trusted version.
Scale gateOverfit one controlled batch before a real run.
05
Build gate
Distinguish a toy language model, a pretrained base model, and a chat assistant.
Create a repository with data.py, model.py, train.py, sample.py, and tests/.
Write down your lane: CPU, single GPU, or cluster.
Set one observable finish line: “generate 200 tokens after training and explain every tensor shape.”
Stop and inspect
Starting with GPT-2 optimization before a tiny model can overfit one batch
Calling a pretrained checkpoint “from scratch”
BUILD → TEST → EXPLAIN