05 · useful win

Train deep networks without superstition

Use initialization, activation statistics, gradient statistics, and update ratios to debug the learning process.

Predict the loss of uniform initial predictions.

02

The invariant chain

initial logits becomes a testable update

initial logits

activation distribution

gradient distribution

update:data ratio

validation curve

03

Why it works

Start with losses you can predict

Uniform predictions over V classes imply a cross-entropy near log(V). A much larger first loss suggests overconfident random logits. Fixing initialization makes the first optimization steps informative instead of spending them merely shrinking logits.

1Start with losses you can predict

2Inspect distributions, not only the scalar loss

3gradient distribution becomes observable

04

Implementation compass

Normalization is a tool, not absolution

# A few health checks
print('expected init loss', -math.log(1 / vocab_size))
for name, p in model.named_parameters():
    if p.grad is not None:
        print(name, p.std().item(), p.grad.std().item(),
              (lr * p.grad.std() / p.std()).log10().item())

ShapeWrite every semantic axis beside the tensor.

ReferenceCompare one output against the smallest trusted version.

Scale gateOverfit one controlled batch before a real run.

05

Build gate

Predict the loss of uniform initial predictions.

Intentionally initialize all weights to zero; inspect which symmetry prevents full learning.

Create activation and gradient histograms for every layer.

Compare naive, Kaiming-scaled, and normalized training starts.

Stop and inspect

Loss is improving, so assuming every layer is healthy

Mixing training-mode and evaluation-mode BatchNorm behavior

BUILD → TEST → EXPLAIN