Operate production inference

Production inference begins with a service contract, not an autoscaler. For one workload, define tail time to first token, inter-token latency, availability, maximum queue time, quality floor, data residency, retention, and recovery objectives. Then write overload behavior: which requests may wait, which can degrade, which route elsewhere, and which fail fast. Every control-plane policy should trace back to this contract. If the promise is vague, the system can be green while the customer experience is broken.

Operate four explicit loops. The release loop builds, evaluates, shadows, promotes, observes, and rolls back. The capacity loop forecasts, reserves, warms, bursts, sheds, and reconciles. The request loop admits, routes, executes, streams, and measures. The incident loop detects, protects, diagnoses, recovers, and learns. Baseten's documented environments and rolling deployments illustrate how much product sits around model execution. Your design partner buys these loops even when they think they are buying faster tokens.

Plan capacity in three classes. Warm interactive capacity protects the latency objective. Elastic burst capacity absorbs growth but has a measured startup time. Batch capacity can be delayed or interrupted for better economics. Track headroom by request shape and token demand, not only GPU count. Provisioned-throughput products from hyperscalers show that enterprises will pay for predictability. Your opportunity may be to offer a sharper capacity contract for one workload rather than a larger undifferentiated GPU pool.

Walk through a peak-traffic GPU loss. Health checks remove the replica from new routing before client timeouts reveal the problem. The gateway stops admitting low-priority requests that would push surviving queues beyond their deadlines. Sessions with reusable prefixes route to healthy cache candidates when possible; others receive an explicit overload response or a documented degraded model, depending on the contract. The capacity loop requests replacement hardware, but the incident response does not wait for it. Operators see the affected deployment version, request identifiers, queue depth, and customer scope. Status communication begins from the service contract, not from guesswork. After capacity returns, the system warms weights, passes a smoke test, re-enters traffic gradually, and reconciles parked or failed work. The postmortem asks why the failure consumed the error budget and whether more warm headroom, a different admission threshold, or regional redundancy is cheaper. This is reliability as a product: the customer can predict both failure behavior and communication.

Before live traffic, run a game day. Kill a GPU. Break weight delivery. Send an overlong prompt. Disconnect a streaming client. Exhaust regional capacity. For each event, record the customer symptom, automatic protection, operator action, recovery time, and prevention. Disaggregate prefill and decode only if representative traces show they need different resource shapes. Reliability becomes a wedge when the buyer sees that your system fails predictably, communicates clearly, and restores service without improvisation.