09 · useful win

Build the tokenizer as a separate learned system

Start from UTF-8 bytes, train byte-pair merges, and make encode/decode behavior explicit and testable.

Explain the boundary between tokenizer training and model training.

02

The invariant chain

Unicode text becomes a testable update

Unicode text

UTF-8 bytes

pair counts

ranked merges

token ids

03

Why it works

Tokenization is not a neural layer

The tokenizer has its own corpus, training algorithm, vocabulary, and serialization. The Transformer consumes integer ids; it does not learn the byte-pair merge table through backpropagation.

1Tokenization is not a neural layer

2BPE repeatedly compresses frequent adjacent pairs

3pair counts becomes observable

04

Implementation compass

Round-trip correctness is the first invariant

ids = list(text.encode("utf-8"))
for new_id in range(256, vocab_size):
    pair = max(get_stats(ids), key=get_stats(ids).get)
    ids = merge(ids, pair, new_id)
    merges[pair] = new_id

assert decode(encode(text)) == text

ShapeWrite every semantic axis beside the tensor.

ReferenceCompare one output against the smallest trusted version.

Scale gateOverfit one controlled batch before a real run.

05

Build gate

Explain the boundary between tokenizer training and model training.

Implement get_stats and merge without reading the solution.

Train a 512-token vocabulary on a small multilingual corpus.

Measure compression ratio and inspect the first twenty learned merges.

Stop and inspect

Counting overlapping pairs inconsistently between training and encoding

Applying merges in arbitrary dictionary order instead of learned rank

BUILD → TEST → EXPLAIN