01

11 · useful win

Evaluate honestly; add chat behavior deliberately

Separate validation loss, benchmark behavior, sampling choices, and post-training so the final system’s claims stay precise.

Choose metrics that match base-model and assistant-stage claims.
02

The invariant chain

validation loss becomes a testable update

validation loss
task evaluation
sampling policy
SFT data
chat interface
03

Why it works

One metric cannot certify the whole system

Validation loss measures the model on held-out data from a defined distribution. Task evaluations probe selected capabilities. Human review catches qualitative failure modes. Report the dataset, tokenizer, prompt format, decoding settings, and uncertainty around every result.

1One metric cannot certify the whole system
2Sampling is a policy over fixed logits
3sampling policy becomes observable
04

Implementation compass

Chat is a later data curriculum

@torch.no_grad()
def generate(model, idx, max_new, temperature=0.8, top_k=40):
    for _ in range(max_new):
        logits, _ = model(idx[:, -model.block_size:])
        logits = logits[:, -1, :] / temperature
        v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
        logits[logits < v[:, [-1]]] = -float("inf")
        idx = torch.cat((idx, torch.multinomial(logits.softmax(-1), 1)), 1)
    return idx
ShapeWrite every semantic axis beside the tensor.
ReferenceCompare one output against the smallest trusted version.
Scale gateOverfit one controlled batch before a real run.
05

Build gate

Choose metrics that match base-model and assistant-stage claims.

Freeze a validation split and decoding configuration before comparing checkpoints.
Create a one-page model card with data, budget, metrics, known limits, and sample settings.
Run a temperature × top-k grid and annotate diversity versus coherence.
Stop and inspect

Cherry-picking the best sample from many without disclosure

Reporting benchmark scores without the exact evaluation harness

BUILD → TEST → EXPLAIN