02 · useful win

Build backpropagation from scalar values

Make a tiny reverse-mode autodiff engine so gradients stop feeling like hidden framework magic.

Represent a computation as a directed acyclic graph.

02

The invariant chain

forward values becomes a testable update

forward values

local derivatives

reverse topological order

gradient accumulation

parameter update

03

Why it works

A gradient answers a local counterfactual

For a scalar output, each parameter gradient measures the first-order change in that output for a tiny change in the parameter. Backpropagation reuses local derivatives so the same question can be answered for every upstream value efficiently.

1A gradient answers a local counterfactual

2The graph is the memory of the forward pass

3reverse topological order becomes observable

04

Implementation compass

Accumulation is not optional

class Value:
    def __init__(self, data, children=()):
        self.data, self.grad = data, 0.0
        self._prev = set(children)
        self._backward = lambda: None

# for out = a * b
out._backward = lambda: (
    setattr(a, "grad", a.grad + b.data * out.grad),
    setattr(b, "grad", b.grad + a.data * out.grad),
)

ShapeWrite every semantic axis beside the tensor.

ReferenceCompare one output against the smallest trusted version.

Scale gateOverfit one controlled batch before a real run.

05

Build gate

Represent a computation as a directed acyclic graph.

Implement +, *, tanh, exp, pow, and a backward() topological traversal.

Check each derivative against a finite-difference estimate.

Add Neuron, Layer, and MLP modules with a parameters() method.

Stop and inspect

Overwriting gradients instead of accumulating

Updating parameters before every gradient is complete

BUILD → TEST → EXPLAIN