Build the serving MVP

The smallest credible serving system has two planes. The data plane takes a request through authentication, admission control, routing, a model runtime, and a GPU, then streams tokens back. The control plane packages models, records versions, applies scaling policy, finds capacity, promotes releases, and rolls them back. Mixing the two in your mental model produces fragile systems. The data plane is optimized for predictable latency. The control plane is optimized for safe change and recoverability.

Build in six vertical weeks. First, run one model and freeze a replayable request corpus. Second, add an OpenAI-compatible gateway with keys, request identifiers, timeouts, streaming, and structured errors. Third, measure queue time, time to first token, inter-token latency, token counts, cache behavior, and model version. Fourth, add bounded concurrency and your differentiating router or batcher. Fifth, add a versioned deployment and one-command rollback. Sixth, replay a design partner's traffic and operate the trial live.

Protect the service before you scale it. Baseten's autoscaling documentation makes the cost-latency tradeoff explicit: more concurrency packs hardware efficiently but increases queueing, while less concurrency costs more and reduces queueing. New replicas also need time to acquire weights and become healthy. So admission control is not an advanced feature. It is the first reliability boundary. When warm capacity is exhausted, decide whether to queue, route, degrade, or reject before an overload cascade destroys every request.

Use one coding-agent request as the architecture test. The gateway authenticates the tenant, validates the context limit, attaches a request identifier, and records a deadline class. Admission control checks warm capacity. The router sees that one replica already holds the repository prefix and that its queue still fits the deadline, so it preserves cache locality. The runtime performs prefill only for the new suffix, begins streaming, and exports time to first token, inter-token latency, cache-hit estimate, token counts, replica, and model version. Meanwhile, the control plane knows exactly which immutable deployment served the request and can roll it back. If you cannot draw and measure this trace, adding another region or scheduler will not create a product. If the trace proves that cache-aware routing cuts goodput failures on partner traffic, you have the first workload-specific loop. The next feature should improve or protect that loop, not broaden the platform for an imaginary user.

Rent or reuse everything except the loop that creates your advantage. vLLM and Dynamo already provide serious serving primitives. Your durable seed is a workload signal others ignore and a decision that improves the outcome: prefix reuse controls routing, deadlines control admission, adapter popularity controls preloading, or residency controls placement. The endpoint can be copied. A loop trained on real workload behavior, integrated into a customer's release process, and proven against their service objective becomes difficult to replace.