06 · useful win

Become a tensor backpropagation debugger

Derive gradients through cross-entropy, linear layers, tanh, BatchNorm, and embeddings—then compare with autograd.

Derive a vector-Jacobian product without materializing the full Jacobian.

02

The invariant chain

loss seed becomes a testable update

loss seed

softmax + NLL

linear transpose

elementwise derivative

scatter-add to embeddings

03

Why it works

Backprop computes products, not giant Jacobians

For each operation, the upstream gradient is multiplied by the operation’s local Jacobian. Efficient code uses algebra and tensor operations to form that product directly instead of constructing the Jacobian matrix.

1Backprop computes products, not giant Jacobians

2Broadcasting has a backward rule

3linear transpose becomes observable

04

Implementation compass

Gradient checking turns derivation into engineering

# Linear layer: out = x @ W + b
dout = ...
dx = dout @ W.T
dW = x.T @ dout
db = dout.sum(0)          # undo bias broadcasting

# Verify
assert torch.allclose(dW, W.grad, atol=1e-7)

ShapeWrite every semantic axis beside the tensor.

ReferenceCompare one output against the smallest trusted version.

Scale gateOverfit one controlled batch before a real run.

05

Build gate

Derive a vector-Jacobian product without materializing the full Jacobian.

Derive and implement each local backward rule in a separate notebook cell.

Write a cmp(name, manual, autograd) helper reporting max absolute difference.

Finish with a fused cross-entropy gradient and compare it to the expanded path.

Stop and inspect

Returning a gradient with the broadcasted output shape

Forgetting transposes in matrix products

BUILD → TEST → EXPLAIN