01
04 · useful win
Learn representations with an MLP
Replace literal token identities with embeddings, concatenate context, and learn nonlinear features without corrupting evaluation.
Explain an embedding lookup as learned row selection.
02
The invariant chain
token ids becomes a testable update
token ids
embedding vectors
flattened context
tanh hidden state
vocabulary logits
03
Why it works
Embeddings make similarity learnable
A token id has no geometry. An embedding table maps each id to a vector whose coordinates are changed by gradient descent. Tokens useful in similar contexts can acquire nearby representations.
1Embeddings make similarity learnable
2Context is a tensor transformation
3flattened context becomes observable
04
Implementation compass
Evaluation is a decision protocol
emb = C[X] # [B, context, emb]
h = torch.tanh(emb.view(B, -1) @ W1 + b1)
logits = h @ W2 + b2
loss = F.cross_entropy(logits, Y)ShapeWrite every semantic axis beside the tensor.
ReferenceCompare one output against the smallest trusted version.
Scale gateOverfit one controlled batch before a real run.
05
Build gate
Explain an embedding lookup as learned row selection.
Compute every tensor shape on paper before running the notebook.
Overfit one mini-batch, then restore the real data loader.
Sweep embedding width, hidden width, and learning rate using validation loss.
Stop and inspect
A starting loss far above -log(1 / vocab)
A training loss that improves while validation degrades
BUILD → TEST → EXPLAIN