01 / 11 · The build contract

Define the machine before you build it

Turn “build an LLM” into a ladder of testable artifacts: scalar autograd, a character model, a Transformer, then a budgeted training run.

workshop · 35–50 minevidence · 3 recordsgate · build + explain

Watch, then build

The Karpathy workshop is embedded beside a shorter narrated build-along. Use the workshop for the full derivation; use the build-along to understand the mechanism, implementation order, and failure checks before you type.

Karpathy · source workshop Course overview

Skim the ten-video sequence; do not start the GPT-2 capstone yet. Open on YouTube ↗

Narrated mechanism soft female voice · captions on

Pause after each scene and reproduce the visible invariant in your own repository.

Before pressing play

Predict first.

Write a one-sentence prediction for the mechanism below. Keep it visible while you work; revise it only after the build gives you evidence.

Demonstrated objectives

Distinguish a toy language model, a pretrained base model, and a chat assistant.
Choose a capstone tier that matches available compute.
Explain the course’s recurring training loop in one minute.

text becomes tokens

a network predicts the next token

loss becomes gradients

an optimizer changes weights

evaluation decides what survives

The mental model

These three ideas are the reference layer behind the videos. If a narrated step feels too fast, return here and trace the causal claim in prose.

The finish line is behavior, not parameter count

A useful first build is small enough to inspect. You should be able to overfit one batch, trace tensor shapes, sample text, and explain why the loss moves. GPT-2-scale reproduction is a later systems exercise, not proof that the fundamentals are understood.

Three different things are often called “an LLM”

A base model predicts continuations from pretraining data. Supervised fine-tuning changes the dataset and behavior. Preference or reinforcement stages further shape responses. This course builds the base-model mechanism first and labels assistant training as a separate extension.

Pick a compute lane now

The CPU lane trains tiny character or byte models. The single-GPU lane scales the same code to a compact Transformer. The cluster lane reproduces GPT-2-style throughput work. The concepts are shared; the batch size, model width, dataset, and runtime are not.

The workbench

Type this shape yourself. The snippet is a compass, not a complete solution.

# One invariant survives every scale
logits = model(x)                 # [batch, time, vocab]
loss = cross_entropy(logits, y)  # one scalar
loss.backward()                  # gradients for parameters
optimizer.step()                 # a slightly better model

Build gate

Create a repository with data.py, model.py, train.py, sample.py, and tests/.
Write down your lane: CPU, single GPU, or cluster.
Set one observable finish line: “generate 200 tokens after training and explain every tensor shape.”
Record a hard budget for time and cloud spend before any long run.

Failure signatures

Starting with GPT-2 optimization before a tiny model can overfit one batch

Calling a pretrained checkpoint “from scratch”

Treating chat behavior as an automatic property of next-token pretraining

Diagnostic checks

A correct click is practice, not proof. Explain the answer aloud and use the build gate for demonstrated mastery.

1. Which artifact is produced directly by ordinary next-token pretraining?

2. What is the strongest first-course finish line?

Primary evidence

Evidence record IDs

ev_0a9c8a2be1259dd831c9 · ev_595344e0b1cb01a5f986 · ev_1219445d15f2f3af28de

These IDs connect the page, narration, visual cues, checks, and frozen source snapshot.

02 / 11 · Learning mechanics

Build backpropagation from scalar values

Make a tiny reverse-mode autodiff engine so gradients stop feeling like hidden framework magic.

workshop · 4–6 hrevidence · 3 recordsgate · build + explain

Watch, then build

Karpathy · source workshop Zero to Hero 01

Code along through the Value object and the MLP; pause for the exercises. Open on YouTube ↗

Narrated mechanism soft female voice · captions on

Pause after each scene and reproduce the visible invariant in your own repository.

Before pressing play

Predict first.

Write a one-sentence prediction for the mechanism below. Keep it visible while you work; revise it only after the build gives you evidence.

Demonstrated objectives

Represent a computation as a directed acyclic graph.
Apply the chain rule backward while accumulating gradients.
Train a two-layer MLP using only the engine you wrote.

forward values

local derivatives

reverse topological order

gradient accumulation

parameter update

The mental model

These three ideas are the reference layer behind the videos. If a narrated step feels too fast, return here and trace the causal claim in prose.

A gradient answers a local counterfactual

For a scalar output, each parameter gradient measures the first-order change in that output for a tiny change in the parameter. Backpropagation reuses local derivatives so the same question can be answered for every upstream value efficiently.

The graph is the memory of the forward pass

Each Value stores its data, parents, operation, gradient, and a local backward rule. A topological ordering guarantees that a node receives downstream gradient contributions before it sends its own contribution to its parents.

Accumulation is not optional

When one value influences the output along multiple paths, those path contributions add. Assignment silently loses paths; += is the essential behavior. PyTorch follows the same accumulation idea, which is why training loops clear gradients between steps.

The workbench

Type this shape yourself. The snippet is a compass, not a complete solution.

class Value:
    def __init__(self, data, children=()):
        self.data, self.grad = data, 0.0
        self._prev = set(children)
        self._backward = lambda: None

# for out = a * b
out._backward = lambda: (
    setattr(a, "grad", a.grad + b.data * out.grad),
    setattr(b, "grad", b.grad + a.data * out.grad),
)

Build gate

Implement +, *, tanh, exp, pow, and a backward() topological traversal.
Check each derivative against a finite-difference estimate.
Add Neuron, Layer, and MLP modules with a parameters() method.
Train a tiny binary classifier and plot loss every step.

Failure signatures

Overwriting gradients instead of accumulating

Updating parameters before every gradient is complete

Forgetting to zero gradients between optimization steps

Diagnostic checks

A correct click is practice, not proof. Explain the answer aloud and use the build gate for demonstrated mastery.

1. A value feeds two branches that later rejoin. What must its backward rule do?

2. Why traverse nodes in reverse topological order?

Primary evidence

Evidence record IDs

ev_a7f5d7e6af731c1c980b · ev_f49dba4cb48bf7ec527f · ev_0a9c8a2be1259dd831c9

These IDs connect the page, narration, visual cues, checks, and frozen source snapshot.

03 / 11 · Learning mechanics

Turn text into a probabilistic machine

Build the smallest complete language model twice—first with counts, then with a one-layer neural network.

workshop · 3–5 hrevidence · 3 recordsgate · build + explain

Watch, then build

Karpathy · source workshop Zero to Hero 02

Focus on batching, broadcasting, NLL, and sampling. Open on YouTube ↗

Narrated mechanism soft female voice · captions on

Pause after each scene and reproduce the visible invariant in your own repository.

Before pressing play

Predict first.

Write a one-sentence prediction for the mechanism below. Keep it visible while you work; revise it only after the build gives you evidence.

Demonstrated objectives

Construct training pairs for next-token prediction.
Compute normalized probabilities and negative log likelihood.
Show the equivalence between row lookup and one-hot matrix multiplication.

token pairs

count or score matrix

softmax probabilities

negative log likelihood

categorical sampling

The mental model

These three ideas are the reference layer behind the videos. If a narrated step feels too fast, return here and trace the causal claim in prose.

Language modeling is conditional prediction

Given a context, the model returns a distribution over the next token. A bigram uses one previous character; the same train/evaluate/sample loop survives when the context and network become much larger.

Counts and a linear neural model meet at the same interface

The count model normalizes observed transition counts. The neural version indexes a row of weights, interprets it as logits, applies softmax, and optimizes negative log likelihood. Their parameterization differs; their probabilistic contract is the same.

Tensor shape is part of the program

Batch, time, and vocabulary axes carry meaning. Broadcasting is useful only when the aligned dimensions mean what you think they mean. Write shapes beside every intermediate until you can predict them.

The workbench

Type this shape yourself. The snippet is a compass, not a complete solution.

# xs: current token ids, ys: next token ids
logits = W[xs]                    # [N, vocab]
loss = F.cross_entropy(logits, ys)

for _ in range(100):
    loss.backward()
    W.data += -0.1 * W.grad
    W.grad = None

Build gate

Train count-based bigram and trigram models on the same split.
Implement row normalization once with loops and once with broadcasting.
Replace explicit one-hot vectors with direct weight indexing.
Report train, validation, and test NLL without tuning on the test split.

Failure signatures

Normalizing the entire matrix instead of each context row

Taking log of exact zero probabilities without smoothing

Using the test set to choose smoothing strength

Diagnostic checks

A correct click is practice, not proof. Explain the answer aloud and use the build gate for demonstrated mastery.

1. For a [vocab, vocab] count table, which normalization yields P(next | current)?

2. Why can W[x] replace one_hot(x) @ W?

Primary evidence

Evidence record IDs

ev_89f06e52f0461152c8d3 · ev_b8f2e6e185150559a880 · ev_0a9c8a2be1259dd831c9

These IDs connect the page, narration, visual cues, checks, and frozen source snapshot.

04 / 11 · Representations

Learn representations with an MLP

Replace literal token identities with embeddings, concatenate context, and learn nonlinear features without corrupting evaluation.

workshop · 3–5 hrevidence · 3 recordsgate · build + explain

Watch, then build

Karpathy · source workshop Zero to Hero 03

Rebuild the Bengio-style model and run the hyperparameter exercises. Open on YouTube ↗

Narrated mechanism soft female voice · captions on

Pause after each scene and reproduce the visible invariant in your own repository.

Before pressing play

Predict first.

Write a one-sentence prediction for the mechanism below. Keep it visible while you work; revise it only after the build gives you evidence.

Demonstrated objectives

Explain an embedding lookup as learned row selection.
Trace [batch, context] token ids through embedding, hidden, and output layers.
Use train, validation, and test splits for different decisions.

token ids

embedding vectors

flattened context

tanh hidden state

vocabulary logits

The mental model

These three ideas are the reference layer behind the videos. If a narrated step feels too fast, return here and trace the causal claim in prose.

Embeddings make similarity learnable

A token id has no geometry. An embedding table maps each id to a vector whose coordinates are changed by gradient descent. Tokens useful in similar contexts can acquire nearby representations.

Context is a tensor transformation

For context length three and embedding width C, lookup produces [B, 3, C], flattening produces [B, 3C], and the output layer returns [B, vocab]. Losing track of one axis usually produces a model that runs but means something else.

Evaluation is a decision protocol

Train data fits parameters. Validation data chooses hyperparameters and architecture. Test data estimates the final selected system once. Repeatedly consulting test loss turns it into another validation set.

The workbench

Type this shape yourself. The snippet is a compass, not a complete solution.

emb = C[X]                       # [B, context, emb]
h = torch.tanh(emb.view(B, -1) @ W1 + b1)
logits = h @ W2 + b2
loss = F.cross_entropy(logits, Y)

Build gate

Compute every tensor shape on paper before running the notebook.
Overfit one mini-batch, then restore the real data loader.
Sweep embedding width, hidden width, and learning rate using validation loss.
Visualize two embedding dimensions and describe one pattern cautiously.

Failure signatures

A starting loss far above -log(1 / vocab)

A training loss that improves while validation degrades

A reshape that silently mixes batch and context axes

Diagnostic checks

A correct click is practice, not proof. Explain the answer aloud and use the build gate for demonstrated mastery.

1. Which split should choose embedding width?

2. With B examples, 3 context tokens, and C-dimensional embeddings, what reaches W1?

Primary evidence

Evidence record IDs

ev_89f06e52f0461152c8d3 · ev_0a9c8a2be1259dd831c9 · ev_56a488f3acdc4fd47dc8

These IDs connect the page, narration, visual cues, checks, and frozen source snapshot.

05 / 11 · Representations

Train deep networks without superstition

Use initialization, activation statistics, gradient statistics, and update ratios to debug the learning process.

workshop · 4–6 hrevidence · 4 recordsgate · build + explain

Watch, then build

Karpathy · source workshop Zero to Hero 04

Stop at each diagnostic plot and predict the failure before the fix. Open on YouTube ↗

Narrated mechanism soft female voice · captions on

Pause after each scene and reproduce the visible invariant in your own repository.

Before pressing play

Predict first.

Write a one-sentence prediction for the mechanism below. Keep it visible while you work; revise it only after the build gives you evidence.

Demonstrated objectives

Predict the loss of uniform initial predictions.
Diagnose saturated activations and poorly scaled gradients.
Explain what BatchNorm changes during training and inference.

initial logits

activation distribution

gradient distribution

update:data ratio

validation curve

The mental model

These three ideas are the reference layer behind the videos. If a narrated step feels too fast, return here and trace the causal claim in prose.

Start with losses you can predict

Uniform predictions over V classes imply a cross-entropy near log(V). A much larger first loss suggests overconfident random logits. Fixing initialization makes the first optimization steps informative instead of spending them merely shrinking logits.

Inspect distributions, not only the scalar loss

A falling loss can hide dead units, saturated tanh outputs, exploding gradients, or layers that barely update. Plot activations, gradients, parameter-to-gradient scale, and update-to-parameter ratios by layer.

Normalization is a tool, not absolution

BatchNorm stabilizes layer statistics using batch-dependent normalization during training and running statistics at inference. It helps optimization but introduces distinct train/eval behavior; later Transformer blocks commonly rely on LayerNorm-style normalization instead.

The workbench

Type this shape yourself. The snippet is a compass, not a complete solution.

# A few health checks
print('expected init loss', -math.log(1 / vocab_size))
for name, p in model.named_parameters():
    if p.grad is not None:
        print(name, p.std().item(), p.grad.std().item(),
              (lr * p.grad.std() / p.std()).log10().item())

Build gate

Intentionally initialize all weights to zero; inspect which symmetry prevents full learning.
Create activation and gradient histograms for every layer.
Compare naive, Kaiming-scaled, and normalized training starts.
Fold a trained BatchNorm transform into the preceding linear layer and verify inference output.

Failure signatures

Loss is improving, so assuming every layer is healthy

Mixing training-mode and evaluation-mode BatchNorm behavior

Changing multiple suspected causes before measuring any one of them

Diagnostic checks

A correct click is practice, not proof. Explain the answer aloud and use the build gate for demonstrated mastery.

1. For V equally likely classes, what initial cross-entropy should you expect?

2. Why do equal zero weights damage a multilayer network?

Primary evidence

Evidence record IDs

ev_9181ce75364185d9042b · ev_d83e255fd3ecc25328f9 · ev_26ad5c841af2beb2941a · ev_56a488f3acdc4fd47dc8

These IDs connect the page, narration, visual cues, checks, and frozen source snapshot.

06 / 11 · Representations

Become a tensor backpropagation debugger

Derive gradients through cross-entropy, linear layers, tanh, BatchNorm, and embeddings—then compare with autograd.

workshop · 4–7 hrevidence · 3 recordsgate · build + explain

Watch, then build

Karpathy · source workshop Zero to Hero 05

This is an exercise session: derive first, unpause only when blocked. Open on YouTube ↗

Narrated mechanism soft female voice · captions on

Pause after each scene and reproduce the visible invariant in your own repository.

Before pressing play

Predict first.

Write a one-sentence prediction for the mechanism below. Keep it visible while you work; revise it only after the build gives you evidence.

Demonstrated objectives

Derive a vector-Jacobian product without materializing the full Jacobian.
Reduce broadcasted gradients back to the source shape.
Validate a manual backward pass against autograd numerically.

loss seed

softmax + NLL

linear transpose

elementwise derivative

scatter-add to embeddings

The mental model

These three ideas are the reference layer behind the videos. If a narrated step feels too fast, return here and trace the causal claim in prose.

Backprop computes products, not giant Jacobians

For each operation, the upstream gradient is multiplied by the operation’s local Jacobian. Efficient code uses algebra and tensor operations to form that product directly instead of constructing the Jacobian matrix.

Broadcasting has a backward rule

If a value was broadcast across a dimension during the forward pass, its gradient must sum across that dimension on the way back. Shape reconstruction is part of the derivative.

Gradient checking turns derivation into engineering

Compare each manual gradient to the autograd result using a strict tolerance. A mismatch should name the operation and shape. This isolates bugs far better than staring at final loss.

The workbench

Type this shape yourself. The snippet is a compass, not a complete solution.

# Linear layer: out = x @ W + b
dout = ...
dx = dout @ W.T
dW = x.T @ dout
db = dout.sum(0)          # undo bias broadcasting

# Verify
assert torch.allclose(dW, W.grad, atol=1e-7)

Build gate

Derive and implement each local backward rule in a separate notebook cell.
Write a cmp(name, manual, autograd) helper reporting max absolute difference.
Finish with a fused cross-entropy gradient and compare it to the expanded path.
Explain why embedding lookup uses scatter-add in the backward direction.

Failure signatures

Returning a gradient with the broadcasted output shape

Forgetting transposes in matrix products

Checking only that a gradient exists rather than that it matches

Diagnostic checks

A correct click is practice, not proof. Explain the answer aloud and use the build gate for demonstrated mastery.

1. A bias [C] is added to activations [B,C]. What shape operation appears in db?

2. Why avoid constructing a full Jacobian?

Primary evidence

Evidence record IDs

ev_3ff28e0410ad38fc83b2 · ev_0a9c8a2be1259dd831c9 · ev_f49dba4cb48bf7ec527f

These IDs connect the page, narration, visual cues, checks, and frozen source snapshot.

07 / 11 · Context

Grow the model’s receptive field

Use hierarchical grouping to deepen the network, manage tensor shapes, and see why architecture changes how context can interact.

workshop · 3–5 hrevidence · 3 recordsgate · build + explain

Watch, then build

Karpathy · source workshop Zero to Hero 06

Track the context and channel dimensions through every layer. Open on YouTube ↗

Narrated mechanism soft female voice · captions on

Pause after each scene and reproduce the visible invariant in your own repository.

Before pressing play

Predict first.

Write a one-sentence prediction for the mechanism below. Keep it visible while you work; revise it only after the build gives you evidence.

Demonstrated objectives

Compute receptive field growth across hierarchical layers.
Implement a flatten-consecutive operation with explicit shapes.
Distinguish a conceptual hierarchy from an efficient dilated convolution implementation.

token window

pairwise grouping

local features

deeper grouping

wide-context prediction

The mental model

These three ideas are the reference layer behind the videos. If a narrated step feels too fast, return here and trace the causal claim in prose.

Depth can aggregate context progressively

Instead of flattening the entire context at once, group adjacent positions, transform them, then group again. Each level summarizes a larger receptive field while preserving a structured path for local interactions.

Shape discipline becomes architecture discipline

A [B,T,C] tensor can become [B,T/n,Cn] when n consecutive positions are flattened. The operation is simple; consistently preserving batch, time, and channel meanings across layers is the real work.

The WaveNet connection is conceptual

The lecture’s tree-like hierarchy resembles the receptive-field growth of causal dilated convolution. The paper’s convolutional implementation is more efficient and includes additional architectural details; do not claim the educational model is a full reproduction.

The workbench

Type this shape yourself. The snippet is a compass, not a complete solution.

class FlattenConsecutive:
    def __init__(self, n): self.n = n
    def __call__(self, x):
        B, T, C = x.shape
        x = x.view(B, T // self.n, C * self.n)
        return x.squeeze(1) if x.shape[1] == 1 else x

Build gate

Draw the receptive field of one output after each grouping layer.
Print shapes after every module in the network.
Compare parameter count and validation loss against the flat-context MLP.
Read the WaveNet abstract and name two differences from your educational model.

Failure signatures

Treating a reshape as harmless when it changes which positions are adjacent

Claiming architectural equivalence from a shared receptive-field idea

Removing shape prints before the model is stable

Diagnostic checks

A correct click is practice, not proof. Explain the answer aloud and use the build gate for demonstrated mastery.

1. Two levels each combine pairs of adjacent positions. How many original positions can one top feature cover?

2. What must remain invariant when reshaping [B,T,C] to [B,T/n,Cn]?

Primary evidence

Evidence record IDs

ev_3ff28e0410ad38fc83b2 · ev_c3df373399abb1146242 · ev_0a9c8a2be1259dd831c9

These IDs connect the page, narration, visual cues, checks, and frozen source snapshot.

08 / 11 · The Transformer

Build causal self-attention, then a Transformer

Derive masked weighted aggregation, add multiple heads, residual pathways, normalization, and feed-forward computation.

workshop · 6–9 hrevidence · 3 recordsgate · build + explain

Watch, then build

Karpathy · source workshop Zero to Hero 07

Work through the matrix-multiply progression before copying the final Head class. Open on YouTube ↗

Narrated mechanism soft female voice · captions on

Pause after each scene and reproduce the visible invariant in your own repository.

Before pressing play

Predict first.

Write a one-sentence prediction for the mechanism below. Keep it visible while you work; revise it only after the build gives you evidence.

Demonstrated objectives

Derive causal attention as content-dependent weighted aggregation.
Explain Q, K, and V using both shapes and communication roles.
Assemble attention, feed-forward, residual, and normalization sublayers into a decoder block.

token states

queries + keys

masked affinities

softmax weights

weighted values

The mental model

These three ideas are the reference layer behind the videos. If a narrated step feels too fast, return here and trace the causal claim in prose.

Attention is a communication mechanism

Each token emits a query describing what it seeks, a key describing what it offers, and a value carrying the information to aggregate. Query-key similarity creates data-dependent weights over allowed source positions.

Causality is enforced before softmax

Future positions receive negative infinity in the affinity matrix, which turns into zero probability after softmax. The mask constrains information flow; it is not a post-processing filter on the output.

A Transformer block has two kinds of work

Multi-head attention communicates across positions. The feed-forward network computes independently at each position. Residual connections preserve an update path, and normalization manages scale. Stacking blocks alternates communication and computation.

The workbench

Type this shape yourself. The snippet is a compass, not a complete solution.

q, k, v = x @ Wq, x @ Wk, x @ Wv       # [B,T,H]
wei = q @ k.transpose(-2, -1) * H**-0.5   # [B,T,T]
wei = wei.masked_fill(causal_mask == 0, float('-inf'))
wei = F.softmax(wei, dim=-1)
out = wei @ v                              # [B,T,H]

Build gate

Implement the progression: uniform average → learned weights → masked self-attention.
Assert that changing a future token cannot change an earlier output.
Combine heads in parallel and verify the final channel dimension.
Train on a tiny corpus and compare bigram, single-head, and multi-block validation loss.

Failure signatures

Masking after softmax

Scaling by the model width instead of head width

Allowing tokens to communicate across the batch dimension

Diagnostic checks

A correct click is practice, not proof. Explain the answer aloud and use the build gate for demonstrated mastery.

1. Where should the causal mask be applied?

2. What changes from one destination token to another in self-attention?

Primary evidence

Evidence record IDs

ev_4ca062937d79808b191f · ev_5deb0a7894ae160c9384 · ev_0a9c8a2be1259dd831c9

These IDs connect the page, narration, visual cues, checks, and frozen source snapshot.

09 / 11 · The Transformer

Build the tokenizer as a separate learned system

Start from UTF-8 bytes, train byte-pair merges, and make encode/decode behavior explicit and testable.

workshop · 4–7 hrevidence · 3 recordsgate · build + explain

Watch, then build

Karpathy · source workshop Zero to Hero 09

Implement each exercise before consulting minBPE. Open on YouTube ↗

Narrated mechanism soft female voice · captions on

Pause after each scene and reproduce the visible invariant in your own repository.

Before pressing play

Predict first.

Write a one-sentence prediction for the mechanism below. Keep it visible while you work; revise it only after the build gives you evidence.

Demonstrated objectives

Explain the boundary between tokenizer training and model training.
Implement deterministic BPE training, encoding, and decoding.
Test Unicode, special-token, and round-trip behavior.

Unicode text

UTF-8 bytes

pair counts

ranked merges

token ids

The mental model

These three ideas are the reference layer behind the videos. If a narrated step feels too fast, return here and trace the causal claim in prose.

Tokenization is not a neural layer

The tokenizer has its own corpus, training algorithm, vocabulary, and serialization. The Transformer consumes integer ids; it does not learn the byte-pair merge table through backpropagation.

BPE repeatedly compresses frequent adjacent pairs

Begin with byte ids. Count adjacent pairs, merge the most frequent pair into a new id, and repeat until the target vocabulary size. Encoding must replay learned merges by rank, not retrain them on each input.

Round-trip correctness is the first invariant

For ordinary text, decode(encode(text)) should reproduce the input bytes. Special tokens require an explicit policy because interpreting a literal string as a control token can change behavior.

The workbench

Type this shape yourself. The snippet is a compass, not a complete solution.

ids = list(text.encode("utf-8"))
for new_id in range(256, vocab_size):
    pair = max(get_stats(ids), key=get_stats(ids).get)
    ids = merge(ids, pair, new_id)
    merges[pair] = new_id

assert decode(encode(text)) == text

Build gate

Implement get_stats and merge without reading the solution.
Train a 512-token vocabulary on a small multilingual corpus.
Measure compression ratio and inspect the first twenty learned merges.
Add round-trip tests for emoji, combining marks, whitespace, and control-like text.

Failure signatures

Counting overlapping pairs inconsistently between training and encoding

Applying merges in arbitrary dictionary order instead of learned rank

Letting special-token strings pass without an explicit allowed/disallowed policy

Diagnostic checks

A correct click is practice, not proof. Explain the answer aloud and use the build gate for demonstrated mastery.

1. Which parameters are learned by BPE training?

2. After training, how should encode choose the next merge?

Primary evidence

Evidence record IDs

ev_69a066b80a6c5222b0aa · ev_0ff37a2e8aefe128f33b · ev_cbff00a40ec78f6b9469

These IDs connect the page, narration, visual cues, checks, and frozen source snapshot.

10 / 11 · Training systems

Scale the same loop into a GPT-2 reproduction

Implement the GPT-2 module, verify compatibility, make training fast, and run a measured, restartable experiment.

workshop · 8–16 hr + trainingevidence · 4 recordsgate · build + explain

Watch, then build

Karpathy · source workshop Zero to Hero 10

Use the commit history as a lab manual; stop after each verified stage. Open on YouTube ↗

Narrated mechanism soft female voice · captions on

Pause after each scene and reproduce the visible invariant in your own repository.

Before pressing play

Predict first.

Write a one-sentence prediction for the mechanism below. Keep it visible while you work; revise it only after the build gives you evidence.

Demonstrated objectives

Implement a GPT-2-compatible module and load reference weights.
Separate correctness optimizations from throughput optimizations.
Design a restartable training run with validation, checkpoints, and a fixed budget.

data shards

B×T batches

forward + loss

backward + accumulate

validate + checkpoint

The mental model

These three ideas are the reference layer behind the videos. If a narrated step feels too fast, return here and trace the causal claim in prose.

Compatibility is a powerful correctness test

Match the reference configuration and parameter layout, load a released checkpoint, and compare logits. This isolates architecture mistakes before expensive training begins.

Make it correct, then make it fast

Overfit one batch first. Add mixed precision, compilation, fused attention, vocabulary-friendly dimensions, gradient accumulation, and distributed execution one at a time while checking that loss behavior remains plausible.

A training run is an experiment with recovery

Pin the code, data fingerprint, tokenizer, configuration, random seed, hardware, and library versions. Save optimizer and scheduler state with the model so interruption does not silently change the run.

The workbench

Type this shape yourself. The snippet is a compass, not a complete solution.

for step in range(max_steps):
    optimizer.zero_grad(set_to_none=True)
    for micro in range(grad_accum_steps):
        with torch.autocast(device_type=device_type, dtype=torch.bfloat16):
            _, loss = model(x, y)
            loss = loss / grad_accum_steps
        loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step(); scheduler.step()

Build gate

Walk the build-nanogpt commits in order and record one invariant per commit.
Load GPT-2 weights and compare a small slice of logits.
Overfit one batch before enabling any performance feature.
Benchmark each optimization and reject changes that alter correctness checks.
Run a budgeted tiny reproduction before considering 124M parameters.

Failure signatures

Changing model code and performance code in the same step

Comparing losses across different tokenizers or data distributions as if equivalent

Saving only model weights and losing optimizer/scheduler state

Diagnostic checks

A correct click is practice, not proof. Explain the answer aloud and use the build gate for demonstrated mastery.

1. What should happen before a long pretraining run?

2. Why divide each micro-batch loss by grad_accum_steps?

Primary evidence

Evidence record IDs

ev_cbff00a40ec78f6b9469 · ev_595344e0b1cb01a5f986 · ev_a14e7412d07bf1f6c6c3 · ev_9d01799d5671855fd480

These IDs connect the page, narration, visual cues, checks, and frozen source snapshot.

11 / 11 · Training systems

Evaluate honestly; add chat behavior deliberately

Separate validation loss, benchmark behavior, sampling choices, and post-training so the final system’s claims stay precise.

workshop · 4–8 hrevidence · 5 recordsgate · build + explain

Watch, then build

Karpathy · source workshop Zero to Hero 08

Use this after the base-model capstone to map pretraining to assistant stages. Open on YouTube ↗

Narrated mechanism soft female voice · captions on

Pause after each scene and reproduce the visible invariant in your own repository.

Before pressing play

Predict first.

Write a one-sentence prediction for the mechanism below. Keep it visible while you work; revise it only after the build gives you evidence.

Demonstrated objectives

Choose metrics that match base-model and assistant-stage claims.
Explain how temperature and top-k change sampling without changing weights.
Design a small supervised fine-tuning extension without calling it pretraining.

validation loss

task evaluation

sampling policy

SFT data

chat interface

The mental model

These three ideas are the reference layer behind the videos. If a narrated step feels too fast, return here and trace the causal claim in prose.

One metric cannot certify the whole system

Validation loss measures the model on held-out data from a defined distribution. Task evaluations probe selected capabilities. Human review catches qualitative failure modes. Report the dataset, tokenizer, prompt format, decoding settings, and uncertainty around every result.

Sampling is a policy over fixed logits

Temperature rescales logits before softmax; top-k truncates candidates. They change diversity and error rate at inference, but they do not improve the trained weights. Keep decoding settings fixed when comparing checkpoints.

Chat is a later data curriculum

A pretrained model continues documents. Supervised fine-tuning trains on instruction/response or conversation-formatted sequences. Preference-based stages can further shape behavior. The nanochat path is an optional full-stack continuation after the base model is understood.

The workbench

Type this shape yourself. The snippet is a compass, not a complete solution.

@torch.no_grad()
def generate(model, idx, max_new, temperature=0.8, top_k=40):
    for _ in range(max_new):
        logits, _ = model(idx[:, -model.block_size:])
        logits = logits[:, -1, :] / temperature
        v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
        logits[logits < v[:, [-1]]] = -float("inf")
        idx = torch.cat((idx, torch.multinomial(logits.softmax(-1), 1)), 1)
    return idx

Build gate

Freeze a validation split and decoding configuration before comparing checkpoints.
Create a one-page model card with data, budget, metrics, known limits, and sample settings.
Run a temperature × top-k grid and annotate diversity versus coherence.
Optional: format a tiny instruction dataset and continue training with a lower learning rate.
Stretch: use nanochat or CS336 only after every prior build gate passes.

Failure signatures

Cherry-picking the best sample from many without disclosure

Reporting benchmark scores without the exact evaluation harness

Calling supervised fine-tuning “training from scratch”

Diagnostic checks

A correct click is practice, not proof. Explain the answer aloud and use the build gate for demonstrated mastery.

1. What does lowering temperature do at inference?

2. What most directly distinguishes SFT from pretraining?

Primary evidence

Evidence record IDs

ev_c98e949f4e94d9c2ab7a · ev_5072864ec026b5b95c7a · ev_1219445d15f2f3af28de · ev_df8932cbfdd1abcd5390 · ev_b757f4f5d01b6a65a508

These IDs connect the page, narration, visual cues, checks, and frozen source snapshot.

Build the machine.Know every joint.

Define the machine before you build it

Watch, then build

Predict first.

Demonstrated objectives

The mental model

The finish line is behavior, not parameter count

Three different things are often called “an LLM”

Pick a compute lane now

The workbench

Build gate

Failure signatures

Diagnostic checks

1. Which artifact is produced directly by ordinary next-token pretraining?

2. What is the strongest first-course finish line?

Primary evidence

Evidence record IDs

Build backpropagation from scalar values

Watch, then build

Predict first.

Demonstrated objectives

The mental model

A gradient answers a local counterfactual

The graph is the memory of the forward pass

Accumulation is not optional

The workbench

Build gate

Failure signatures

Diagnostic checks

1. A value feeds two branches that later rejoin. What must its backward rule do?

2. Why traverse nodes in reverse topological order?

Primary evidence

Evidence record IDs

Turn text into a probabilistic machine

Watch, then build

Predict first.

Demonstrated objectives

The mental model

Language modeling is conditional prediction

Counts and a linear neural model meet at the same interface

Tensor shape is part of the program

The workbench

Build gate

Failure signatures

Diagnostic checks

1. For a [vocab, vocab] count table, which normalization yields P(next | current)?

2. Why can W[x] replace one_hot(x) @ W?

Primary evidence

Evidence record IDs

Learn representations with an MLP

Watch, then build

Predict first.

Demonstrated objectives

The mental model

Embeddings make similarity learnable

Context is a tensor transformation

Evaluation is a decision protocol

The workbench

Build gate

Failure signatures

Diagnostic checks

1. Which split should choose embedding width?

2. With B examples, 3 context tokens, and C-dimensional embeddings, what reaches W1?

Primary evidence

Evidence record IDs

Train deep networks without superstition

Watch, then build

Predict first.

Demonstrated objectives

The mental model

Start with losses you can predict

Inspect distributions, not only the scalar loss

Normalization is a tool, not absolution

The workbench

Build gate

Failure signatures

Diagnostic checks

1. For V equally likely classes, what initial cross-entropy should you expect?

2. Why do equal zero weights damage a multilayer network?

Primary evidence

Build the machine.
Know every joint.