AI System Design Interview â Detailed Example

00 · Interview contract

Start with the invariant.

The hard part is not drawing boxes. It is maintaining per-conversation correctness, fair admission, and a recoverable stream while an expensive dependency is saturated.

ONE-SENTENCE THESIS

“I’ll separate durable conversation state from ephemeral inference work, put token-aware admission before the scarce model pool, and make the event stream resumable and idempotent.”

Functional core

Create conversations, append user messages, stream assistant events, reconnect, retrieve history, cancel generation, and isolate tenants.

Quality attributes

Low perceived latency, per-conversation ordering, bounded overload, durable accepted messages, fair capacity, observability, and safe failure.

01 · Scope and estimates

Turn ambiguity into budgets.

Clarifying questions worth asking

Area	Question	Why it changes the design
Interaction	Text only, or tools, files, images, and voice?	Payload size, stream semantics, storage, safety, and execution isolation change materially.
Latency	Which matters more: time to first token or total completion?	Routing, caching, speculative work, and model choice optimize different budgets.
Consistency	Can one conversation have concurrent user turns?	Determines optimistic concurrency, queueing, and branch semantics.
Capacity	Owned GPU workers, external APIs, or both?	Scheduler control, retry boundaries, quotas, and provider abstraction differ.
Tenancy	Do tenants need regional isolation or dedicated capacity?	Affects partitioning, encryption, routing, and noisy-neighbor protection.

Back-of-the-envelope capacity

ASSUMPTIONS arrival_rate = 1,000 requests / second average_stream = 20 seconds average_input = 2,000 tokens / request average_output = 300 tokens / request concurrent_streams = 1,000 × 20 = 20,000 input_throughput = 1,000 × 2,000 = 2,000,000 tokens / second output_throughput = 1,000 × 300 = 300,000 tokens / second DESIGN CONSEQUENCE Tokens—not HTTP requests—are the useful admission and capacity unit.

Anthropic’s public API documentation makes the same operational distinction by describing limits in requests, input tokens, and output tokens; it also documents token-bucket behavior and retry-after responses.⁵ Treat that as evidence for the design primitive, not as a claim about internal interview expectations.

02 · Reference architecture

Keep durable truth off the hot path.

Fig. 1 — The request path is horizontal; state and policy dependencies are vertical. The admission/scheduler boundary protects the most expensive resource.

03 · Spoken walkthrough

Explain the lifecycle in six moves.

01 / ACCEPT

“The edge authenticates the tenant, validates an idempotency key, assigns a request ID, and checks coarse request limits. It does not hold conversation truth.”

02 / ORDER

“The conversation service performs an optimistic-version append. If two turns target the same conversation version, I reject or branch explicitly rather than silently reorder context.”

03 / ADMIT

“Admission estimates input and output token cost, checks tenant and global token buckets, then either admits, queues within a bounded deadline, or returns a retryable overload response. I avoid an unbounded queue because it converts overload into latency and memory failure.”

04 / BUILD

“The context builder reads the durable message log, applies retention and truncation policy, retrieves permitted context, and forms a cache-friendly prompt prefix. Static instructions and tools precede volatile user content.”

05 / RUN

“The scheduler selects a compatible worker pool using capability, region, budget, and current queue time. It issues a lease, not an ownership transfer, so failed workers can be reclaimed.”

06 / STREAM

“Output events receive monotonically increasing sequence numbers. The gateway streams them to the client; reconnect uses the last acknowledged sequence. On completion, the final assistant message and usage record commit idempotently. Partial output remains marked partial unless policy allows recovery.”

04 · Interface and data

Make retries safe by construction.

POST /v1/conversations/{conversation_id}/responses Idempotency-Key: req_7f21 If-Conversation-Version: 42 { "input": [{"type":"text","text":"Compare the two plans"}], "stream": true, "model_policy": {"capability":"reasoning","region":"us"}, "max_output_tokens": 800 } SSE EVENTS id: 1042 event: response.output_text.delta data: {"request_id":"req_7f21","seq":17,"delta":"The first"} event: response.completed data: {"request_id":"req_7f21","final_version":43,"usage":{...}}

Entity	Key fields	Invariant
Conversation	tenant_id, conversation_id, version, region, policy_id	Version increases on accepted turn boundaries.
Message	message_id, conversation_id, turn_id, role, status, content_ref, created_at	Final messages are immutable; edits create a branch or new revision.
Inference job	request_id, idempotency_key, estimated_tokens, priority, deadline, worker_lease	One terminal outcome per idempotency key.
Stream event	request_id, seq, event_type, payload_ref	Sequence is monotonic within one request.
Usage ledger	tenant_id, request_id, input_tokens, output_tokens, cache_tokens, model	Billing/quotas reconcile exactly once from terminal usage.

Both OpenAI and Anthropic publicly document server-sent event streaming for incremental model output.³⁷ The resumable sequence layer above is an interview design choice layered around provider streams.

05 · Scarce-capacity control

Schedule tokens, not requests.

Admission

Hierarchical token buckets: global → region → tenant → priority class. Reserve a small recovery pool so retries and health checks are not starved by normal traffic.

Fair queueing

Deficit round robin or weighted fair queueing prevents one tenant’s long prompts from monopolizing capacity. Charge estimated tokens, then reconcile actual usage.

Routing

Filter by capability and policy first; optimize cost/latency second. A cheap model is not eligible if it fails the task contract.

Batching

Use narrow dynamic batching at the worker boundary only when queue delay stays inside the latency budget. Send offline evals and embeddings to a separate asynchronous lane.

Cache boundary

Cache repeated, tenant-safe prefixes such as system instructions and tool definitions; keep dynamic user content late in the prompt. Anthropic documents prompt-prefix caching and cache-aware token throughput,⁶ while OpenAI separates non-interactive batch workloads from synchronous traffic.⁴ Cache keys must include tenant, policy, model, and prompt-version identity to prevent cross-boundary reuse.

06 · Failure model

Say what breaks—and what the user sees.

Failure	Detection	System behavior	User contract
Duplicate submit	Same idempotency key	Return existing job/terminal result; never double-charge.	One visible response.
Concurrent turn	Conversation version mismatch	Reject with current version or create an explicit branch.	No silent reordering.
Client disconnect	Stream heartbeat/close	Buffer a bounded event window; resume from last sequence or replay final state.	Reconnect without duplicate text.
Worker lost	Lease timeout/heartbeat	Retry only before externally visible side effects; otherwise surface partial.	Honest partial state, not fabricated completion.
Overload	Token buckets/queue deadline	Shed low priority, preserve recovery capacity, return retry guidance.	Fast retryable failure instead of a hanging request.
Safety stop	Policy event/refusal	Terminate stream with typed final event; record policy decision separately.	Clear refusal/error state.
Data-store lag	Read-after-write/version check	Route conversation to leader/consistent partition or wait within budget.	Never omit the user’s accepted prior turn.

Avoid: “Retry three times” without naming idempotency, side effects, deadlines, or whether partial output already escaped.

07 · Decisions

Make trade-offs explicit.

Server-sent events

Simple one-way incremental output over HTTP; natural fit for text deltas.

CHOOSE FOR V1

WebSocket

Better for bidirectional realtime controls, audio, and long-lived interactive tools; higher connection complexity.

Per-conversation ordering

Strong sequence for turns inside one conversation.

KEEP STRONG

Global ordering

Expensive and unnecessary; analytics and cross-conversation views can be eventual.

Native provider capabilities

Faster access to streaming events, caching, tools, and new model behavior.

THIN ADAPTER

Deep abstraction

Portability, but often collapses important semantics into a lowest common denominator.

Bounded queue

Predictable deadlines and honest overload.

PROTECT SLO

Unbounded queue

Looks available until waiting time and memory grow without limit.

08 · Interview pacing

A 45-minute delivery plan.

FRAME0–5 MIN

Clarify modality, scale, latency, consistency, tenancy, and ownership of model capacity.

BUDGET5–10 MIN

Estimate concurrent streams and token throughput. State the one design consequence.

BASELINE10–22 MIN

Draw the request path, state stores, stream, and model boundary. Walk one request.

PRESSURE22–35 MIN

Deep dive on scheduling, ordering, reconnect, idempotency, and overload.

TRADE35–45 MIN

Discuss alternatives, observability, safety, cost, and what changes at 10× scale.

Good sign: the interviewer interrupts and chooses a deep dive. Follow them. The architecture is a conversation map, not a rehearsed lecture.

09 · Preparation

Practice pressure, not pictures.

Drill 1 · Capacity

For five prompts, estimate RPS, concurrent work, payload, tokens, storage, and bandwidth. End every estimate with a design consequence.

Drill 2 · Lifecycle

Trace one request from client to durable commit. Name IDs, timeouts, retry boundaries, and which component owns truth.

Drill 3 · Failure injection

At each arrow, ask: duplicate, delay, reorder, drop, overload, partial side effect. Define recovery and user-visible state.

Drill 4 · AI-native constraints

Practice token-aware quotas, streaming, context construction, prefix caching, evaluation, model routing, tool isolation, and safety stops.

Canvas checklist

Number the request path so the interviewer can follow your narration.
Use a different visual treatment for durable stores, online services, and scarce resources.
Label arrows with the payload or event—not “calls.”
Box the consistency boundary and the overload boundary.
Write assumptions in a corner and revise them when the interviewer changes scale.
Keep an explicit “trade-offs / failures” area; do not squeeze it in at the end.

Evidence & limits

Primary sources.

Interview Loop local catalog, snapshot IH-CATALOG-01. Used to select a composite prompt from OpenAI and Anthropic reported-source system-design families.
OpenAI API — Latency optimization. Supports decomposing latency into model speed, tokens, requests, parallelism, perceived wait, and non-LLM paths.
OpenAI API — Streaming API responses. Supports SSE-based incremental response delivery.
OpenAI API — Batch API. Supports separating asynchronous non-interactive work from synchronous paths.
Anthropic — Rate limits. Supports token-aware quotas, token-bucket behavior, and retry guidance.
Anthropic — Prompt caching. Supports caching repeated prompt prefixes and reasoning about effective throughput.
Anthropic — Streaming messages. Supports incremental SSE events for text and tool-use deltas.
Anthropic Engineering — Building effective agents. Useful for explaining when predetermined workflows are simpler than open-ended agents.

Limits: All scale numbers, SLIs, data models, and architecture choices are illustrative. They are not descriptions of OpenAI or Anthropic internals. Interview prompts are reported-source practice material, not guaranteed future questions.

Design the pressure paths.

Start with the invariant.

Turn ambiguity into budgets.

Clarifying questions worth asking

Back-of-the-envelope capacity

Keep durable truth off the hot path.

Explain the lifecycle in six moves.

Make retries safe by construction.

Schedule tokens, not requests.

Cache boundary

Say what breaks—and what the user sees.

Make trade-offs explicit.

A 45-minute delivery plan.

Practice pressure, not pictures.

Canvas checklist

Primary sources.