01
08 · useful win
Build causal self-attention, then a Transformer
Derive masked weighted aggregation, add multiple heads, residual pathways, normalization, and feed-forward computation.
Derive causal attention as content-dependent weighted aggregation.
02
The invariant chain
token states becomes a testable update
token states
queries + keys
masked affinities
softmax weights
weighted values
03
Why it works
Attention is a communication mechanism
Each token emits a query describing what it seeks, a key describing what it offers, and a value carrying the information to aggregate. Query-key similarity creates data-dependent weights over allowed source positions.
1Attention is a communication mechanism
2Causality is enforced before softmax
3masked affinities becomes observable
04
Implementation compass
A Transformer block has two kinds of work
q, k, v = x @ Wq, x @ Wk, x @ Wv # [B,T,H]
wei = q @ k.transpose(-2, -1) * H**-0.5 # [B,T,T]
wei = wei.masked_fill(causal_mask == 0, float('-inf'))
wei = F.softmax(wei, dim=-1)
out = wei @ v # [B,T,H]ShapeWrite every semantic axis beside the tensor.
ReferenceCompare one output against the smallest trusted version.
Scale gateOverfit one controlled batch before a real run.
05
Build gate
Derive causal attention as content-dependent weighted aggregation.
Implement the progression: uniform average → learned weights → masked self-attention.
Assert that changing a future token cannot change an earlier output.
Combine heads in parallel and verify the final channel dimension.
Stop and inspect
Masking after softmax
Scaling by the model width instead of head width
BUILD → TEST → EXPLAIN