03 · useful win

Turn text into a probabilistic machine

Build the smallest complete language model twice—first with counts, then with a one-layer neural network.

Construct training pairs for next-token prediction.

02

The invariant chain

token pairs becomes a testable update

token pairs

count or score matrix

softmax probabilities

negative log likelihood

categorical sampling

03

Why it works

Language modeling is conditional prediction

Given a context, the model returns a distribution over the next token. A bigram uses one previous character; the same train/evaluate/sample loop survives when the context and network become much larger.

1Language modeling is conditional prediction

2Counts and a linear neural model meet at the same interface

3softmax probabilities becomes observable

04

Implementation compass

Tensor shape is part of the program

# xs: current token ids, ys: next token ids
logits = W[xs]                    # [N, vocab]
loss = F.cross_entropy(logits, ys)

for _ in range(100):
    loss.backward()
    W.data += -0.1 * W.grad
    W.grad = None

ShapeWrite every semantic axis beside the tensor.

ReferenceCompare one output against the smallest trusted version.

Scale gateOverfit one controlled batch before a real run.

05

Build gate

Construct training pairs for next-token prediction.

Train count-based bigram and trigram models on the same split.

Implement row normalization once with loops and once with broadcasting.

Replace explicit one-hot vectors with direct weight indexing.

Stop and inspect

Normalizing the entire matrix instead of each context row

Taking log of exact zero probabilities without smoothing

BUILD → TEST → EXPLAIN