Benchmark the promise and the business

A credible inference benchmark begins with a contract. Freeze the model revision, tokenizer, precision, quantization, prompt corpus, output cap, decoding settings, hardware, runtime version, batching policy, cache state, concurrency pattern, quality metric, and latency objective. MLPerf's enduring discipline is to measure defined workloads against accuracy targets. Without that discipline, two providers can report impressive numbers while running different tasks. The benchmark is not a chart. It is a reproducible experimental boundary.

Make goodput your north-star system metric. Goodput counts requests that meet both the quality floor and the latency service objective. Peak tokens per second can rise while time to first token deteriorates, tail latency explodes, or users time out. Report time to first token at multiple percentiles, inter-token latency, total request latency, tokens per GPU second, errors, and goodput. Run a matrix across prompt lengths, output lengths, concurrency levels, and cache conditions. One number cannot describe an inference system.

Calculate cost per successful million tokens, not cost at theoretical peak. Start with GPU hourly cost divided by effective tokens per GPU second and by thirty-six hundred, then multiply by one million. Effective throughput uses achieved utilization, not benchmark utilization. Add CPU, storage, network, failed requests, cold starts, support burden, and reserved headroom. Baseten's live retail prices are a comparison point, not your cost structure. Record the date because prices and hardware change quickly.

Work a deliberately illustrative example. Suppose a GPU costs six dollars and fifty cents per hour and reaches fourteen hundred output tokens per second in a warm peak test. If representative traffic achieves only fifty-five percent utilization, effective throughput is seven hundred seventy tokens per second before failures. The compute-only cost per million output tokens is then the hourly price divided by effective tokens per second and thirty-six hundred, multiplied by one million. But if fifteen percent of requests miss the service objective and are retried or abandoned, cost per successful token rises again. Add CPU, storage, network, observability, and warm headroom. Now compare two configurations: one may show lower peak throughput yet higher goodput because its queue remains stable. The numbers here are not a market price claim; replace every input with a dated measurement. The lesson is the shape of the calculation: utilization, success, and overhead decide the business, not the most flattering microbenchmark.

Read breakthrough claims with disciplined curiosity. Taalas reports extraordinary output speed and cost and power advantages for a hard-wired Llama, while also acknowledging quality degradation from aggressive early quantization. The correct response is to reproduce the workload boundary, compare task quality, and identify applications where the speed changes user value. Publish your own corpus, commands, versions, raw results, percentiles, failures, quality result, and assumptions. A rerunnable benchmark is technical proof and sales collateral at the same time.