01

10 · useful win

Scale the same loop into a GPT-2 reproduction

Implement the GPT-2 module, verify compatibility, make training fast, and run a measured, restartable experiment.

Implement a GPT-2-compatible module and load reference weights.
02

The invariant chain

data shards becomes a testable update

data shards
B×T batches
forward + loss
backward + accumulate
validate + checkpoint
03

Why it works

Compatibility is a powerful correctness test

Match the reference configuration and parameter layout, load a released checkpoint, and compare logits. This isolates architecture mistakes before expensive training begins.

1Compatibility is a powerful correctness test
2Make it correct, then make it fast
3forward + loss becomes observable
04

Implementation compass

A training run is an experiment with recovery

for step in range(max_steps):
    optimizer.zero_grad(set_to_none=True)
    for micro in range(grad_accum_steps):
        with torch.autocast(device_type=device_type, dtype=torch.bfloat16):
            _, loss = model(x, y)
            loss = loss / grad_accum_steps
        loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step(); scheduler.step()
ShapeWrite every semantic axis beside the tensor.
ReferenceCompare one output against the smallest trusted version.
Scale gateOverfit one controlled batch before a real run.
05

Build gate

Implement a GPT-2-compatible module and load reference weights.

Walk the build-nanogpt commits in order and record one invariant per commit.
Load GPT-2 weights and compare a small slice of logits.
Overfit one batch before enabling any performance feature.
Stop and inspect

Changing model code and performance code in the same step

Comparing losses across different tokenizers or data distributions as if equivalent

BUILD → TEST → EXPLAIN