HTML Docs course

Build an LLM from First Principles

Your progress0%

Lesson

Define the machine before you build it

Turn “build an LLM” into a ladder of testable artifacts: scalar autograd, a character model, a Transformer, then a budgeted training run.

Primary workshop

YouTube

Skim the ten-video sequence; do not start the GPT-2 capstone yet.

Narrated mechanism

Use this shorter explanation to trace the mechanism, implementation order, and failure checks.

1

Distinguish a toy language model, a pretrained base model, and a chat assistant.

2

Choose a capstone tier that matches available compute.

3

Explain the course’s recurring training loop in one minute.

Build the mental model

See the mechanism before memorizing the code.

Before pressing play, predict how “text becomes tokens” eventually produces “evaluation decides what survives.” Keep the prediction visible and revise it only after your build produces evidence.

Explanatory diagram

Mechanism, end to end

Trace the causal path from “text becomes tokens” to “evaluation decides what survives.” Every arrow names a transformation that should preserve a testable invariant.

FEEDSFEEDSFEEDSFEEDStext becomestokens01a networkpredicts the next token02loss becomesgradients03an optimizerchanges weights04evaluationdecides what survives05
  1. 01text becomes tokens. Start with an input whose meaning and shape you can state explicitly.
  2. 02a network predicts the next token. Inspect this stage’s input, output, and invariant before moving to the next transformation.
  3. 03loss becomes gradients. Inspect this stage’s input, output, and invariant before moving to the next transformation.
  4. 04an optimizer changes weights. Inspect this stage’s input, output, and invariant before moving to the next transformation.
  5. 05evaluation decides what survives. This is the observable result you test before increasing scale.

Explanatory diagram

What becomes observable

A model is debuggable when each conceptual stage leaves a signal you can inspect instead of hiding behind the final loss.

FEEDSFEEDSFEEDSFEEDSdata01tokens02model03loss04update05
  1. 01data. Make data visible in a printout, plot, assertion, or controlled comparison.
  2. 02tokens. Make tokens visible in a printout, plot, assertion, or controlled comparison.
  3. 03model. Make model visible in a printout, plot, assertion, or controlled comparison.
  4. 04loss. Make loss visible in a printout, plot, assertion, or controlled comparison.
  5. 05update. Make update visible in a printout, plot, assertion, or controlled comparison.

Explanatory diagram

The build–test–explain loop

Work at the smallest scale, compare against a trusted reference, and return to the first broken invariant.

FEEDSFEEDSFEEDSREVISEBuild small01Predict02Compare03Inspect04
  1. 01Build small. Create a repository with data.py, model.py, train.py, sample.py, and tests/.
  2. 02Predict. Distinguish a toy language model, a pretrained base model, and a chat assistant.
  3. 03Compare. Write down your lane: CPU, single GPU, or cluster.
  4. 04Inspect. Starting with GPT-2 optimization before a tiny model can overfit one batch

Why it works

01

The finish line is behavior, not parameter count

A useful first build is small enough to inspect. You should be able to overfit one batch, trace tensor shapes, sample text, and explain why the loss moves. GPT-2-scale reproduction is a later systems exercise, not proof that the fundamentals are understood.

02

Three different things are often called “an LLM”

A base model predicts continuations from pretraining data. Supervised fine-tuning changes the dataset and behavior. Preference or reinforcement stages further shape responses. This course builds the base-model mechanism first and labels assistant training as a separate extension.

03

Pick a compute lane now

The CPU lane trains tiny character or byte models. The single-GPU lane scales the same code to a compact Transformer. The cluster lane reproduces GPT-2-style throughput work. The concepts are shared; the batch size, model width, dataset, and runtime are not.

Implementation

Implementation compass

python
# One invariant survives every scale
logits = model(x)                 # [batch, time, vocab]
loss = cross_entropy(logits, y)  # one scalar
loss.backward()                  # gradients for parameters
optimizer.step()                 # a slightly better model

Build gate

  1. 01Create a repository with data.py, model.py, train.py, sample.py, and tests/.
  2. 02Write down your lane: CPU, single GPU, or cluster.
  3. 03Set one observable finish line: “generate 200 tokens after training and explain every tensor shape.”
  4. 04Record a hard budget for time and cloud spend before any long run.

Failure signatures

01Starting with GPT-2 optimization before a tiny model can overfit one batch
02Calling a pretrained checkpoint “from scratch”
03Treating chat behavior as an automatic property of next-token pretraining

Primary evidence

Knowledge check

Make the model stick

1. Which artifact is produced directly by ordinary next-token pretraining?

2. What is the strongest first-course finish line?

Open video player