Anthropic IC7 System Design

IC7 opening (first 30 seconds): "Before I draw anything — let me scope this. We're building the conversational interface for Claude: a user sends a text message, the system orchestrates inference, and streams tokens back. I'll focus on 1:1 chat, not group. I'll design for 50M monthly active users, 10 messages/conversation average, with P99 time-to-first-token under 500ms. Safety filtering is a first-class architectural constraint, not a bolt-on. I'll explicitly call out where I'm making trade-offs."

Requirements I'm driving:

• Functional: Create conversation, send message, receive streamed response, view history, search conversations, multi-device sync, conversation forking/branching
• Non-functional: P99 TTFT < 500ms, message durability 99.999%, read-after-write consistency for same-user, eventual consistency cross-device (< 2s), 50M MAU / 500M messages/day
• Safety: Every user message passes through input classifier before reaching inference. Every Claude response passes through output classifier before reaching user. Audit log for every message, immutable.

Back-of-envelope:

Key architectural decisions:

• SSE over WebSocket for token streaming: Claude responses are server-initiated, unidirectional streams. SSE is simpler, works through HTTP/2 proxies and CDN edges, auto-reconnects natively. WebSocket adds bidirectional complexity we don't need for token delivery. User messages go via standard POST.
• Cassandra for messages, PostgreSQL for conversation metadata: Messages are append-heavy, partitioned by conversation_id, queried by recency. Cassandra's write-optimized wide-column model is ideal. Conversation metadata (title, user_id, model, created_at) is relational and low-volume — PostgreSQL is simpler and supports the join patterns we need for search/filtering.
• Kafka as the event spine: Every message write publishes to Kafka. Consumers: search indexer, billing/metering, audit log writer, analytics, notification service. Decouples the hot path from downstream processing.

IC7 signal: "I'll walk through both the Cassandra message table and the PostgreSQL metadata table, and explain why I split them — then I'll show you the partition key reasoning, because that's where this design lives or dies at scale."

Why ULIDs, not UUIDs or timestamps:

• ULIDs encode millisecond-precision time in the first 48 bits, followed by 80 bits of randomness. They sort lexicographically in time order — so Cassandra's clustering key ordering gives us chronological message order for free.
• Unlike UUIDv1 (which also encodes time), ULIDs are monotonic within the same millisecond, preventing ordering ambiguity for rapid successive messages.
• Unlike plain timestamps, they're globally unique without coordination — no sequence generator needed across distributed nodes.

IC7 signal: "This is one of the trickiest UX problems in streaming chat — and the answer is different depending on whether we're using SSE or WebSocket. I chose SSE specifically because it handles this more gracefully."

Key design elements:

• SSE's Last-Event-ID is the protocol-native reconnection mechanism. The browser automatically sends it on reconnect — zero client-side code needed.
• Server-side ring buffer (Redis): Each active stream writes tokens to a Redis list keyed by stream:{message_id} with 5-minute TTL. This is cheap — a typical response is ~500 tokens × 5 bytes = 2.5 KB per stream.
• Inference continues during disconnect: We do NOT cancel GPU inference when the client disconnects for < 30 seconds. GPU time is expensive; re-running inference is worse than buffering 2.5 KB. For disconnects > 30s, we cancel inference and write the partial response to Cassandra.

IC7 signal: "In a 1:1 chat with an LLM, ordering is simpler than in a multi-party system — but there are still real edge cases I want to address."

Why ordering is (mostly) simpler here: In a Claude 1:1 chat, the conversation is strictly turn-based: user sends → Claude responds → user sends. There's natural serialization because Claude can't respond until the user message arrives, and the user typically waits for Claude's response. This eliminates 90% of multi-party ordering problems.

Where ordering still matters:

• Rapid successive messages: User sends "What is X?" then immediately sends "Actually, what about Y?" before Claude starts responding. Both hit the Chat Service within milliseconds.
• Multi-device: User sends from phone, then immediately sends from laptop — different TCP connections, different server instances.
• Conversation branching: User edits a previous message, creating a fork — which message is the "real" next one?

Solution: ULIDs + optimistic concurrency on conversation version:

• Each message gets a ULID (time-ordered, monotonic within a node). Cassandra's clustering key sorts these correctly.
• Each conversation has a version counter in PostgreSQL. When the Chat Service processes a user message, it does: UPDATE conversations SET version = version + 1 WHERE conversation_id = ? AND version = ?. If the CAS fails (concurrent write from another device), retry with conflict resolution.
• For the "rapid messages" case: queue them in the Chat Service and process sequentially per conversation. A per-conversation in-memory lock (or Redis distributed lock with ~100ms TTL) ensures only one message is being processed at a time for a given conversation.

IC7 signal: "This is the wide partition problem, and it's one of the first things I think about when I choose conversation_id as my partition key. Let me walk through why I'm not worried for our use case, but also what I'd do if I were."

The math:

Why this is unlikely but addressable:

• Real-world distribution: 99.9% of Claude conversations have < 200 messages (Claude's context window is ~200K tokens; at ~50 tokens/message, that's ~4000 messages max before the context is full). A 100K-message conversation would require the user to explicitly clear context and continue hundreds of times.
• But if it happens (defensive design):

Bucketed partitioning strategy:

Query pattern changes:

• "Last 20 messages" → query the latest bucket first. If it has < 20, query the previous bucket. At most 2 partition reads for the common case.
• "All messages" → parallel query across all buckets for the conversation, merge by ULID order client-side.
• A conversation_buckets counter in PostgreSQL tracks the current bucket number, so we always know the latest bucket without scanning.

But here's my IC7 judgment call: I would NOT implement bucketing on day one. I'd ship with simple conversation_id partitioning, set a monitoring alert at partition size > 50 MB, and add bucketing when we see real data hit that threshold. Premature optimization of a partition scheme that affects every query is worse than a migration later. The 99.9th percentile conversation will never hit this limit.

IC7 signal: "Search is fundamentally a different access pattern from chat retrieval, and I would never bolt it onto Cassandra. This is a separate indexing pipeline."

Elasticsearch index schema:

• Index per month (time-based rollover for retention and performance): messages-2026-05
• Routing key: user_id — all of one user's messages land on the same shard, so search is a single-shard query
• Fields: user_id (keyword), conversation_id (keyword), role (keyword), content (text, analyzed), created_at (date), model (keyword)
• Query: {"bool": {"filter": [{"term": {"user_id": "..."}}], "must": [{"match": {"content": "recipe"}}]}}

Search latency target: P99 < 200ms. This is achievable with user_id routing (single-shard) and warm caches for active users.

Why not Cassandra SASI or SAI? Jake may probe this directly given his DataStax background. SAI (Storage-Attached Indexes) in Cassandra 5.0+ support basic text search, but they're partition-local — you can't search across all conversations for a user without a full cluster scan. Elasticsearch gives us cross-partition full-text search with relevance scoring, which is what users actually need.

IC7 signal: "Safety is the reason Anthropic exists. I'm going to design this as a synchronous, mandatory pipeline — not an optional middleware. Every message, every token, every time."

Gate 1 — Input Classification (pre-inference):

• Constitutional Classifier: Lightweight model trained on Anthropic's constitution. Classifies input across harm categories (violence, CSAM, weapons, PII extraction, jailbreaks). Binary pass/block with confidence score.
• Prompt injection detector: Pattern matching + learned classifier for attempts to override system prompts. Particularly important for API users who set custom system prompts.
• PII scanner: Regex + NER for credit cards, SSNs, API keys. Can redact before passing to inference rather than blocking entirely.
• Latency budget: All three run in parallel. Combined P99: ~20ms. This is pre-inference, so it's additive to TTFT.

Gate 2 — Output Classification (during + post-inference):

• Token-level sliding window: Every N tokens (e.g., 10), run a lightweight classifier on the last 50 tokens. If harmful content is emerging, truncate the stream immediately and send a safety stop event. ~2ms per check.
• Completion-level classifier: After the full response is generated, run a comprehensive classifier. If it fails, replace the response with a safety message (even if tokens were already streamed — send a correction event).
• This two-layer approach catches both gradual drift and full-response failures.

IC7 signal: "Billing accuracy is a trust issue, not just an accounting issue. If we overcharge, we lose customers. If we undercharge, we lose money on GPU inference — the most expensive line item in the company. I'll design for exact counting, not estimation."

Token counting architecture:

• Input tokens: Counted by the Chat Service before sending to inference. The tokenizer runs on CPU — it's deterministic and fast (~1ms for a typical message). The count is written to the message record and published to Kafka.
• Output tokens: Counted by the inference worker as it generates. Each token emission increments a counter. The final count is written to the message record when the stream completes. For interrupted streams (client disconnect, safety truncation), we count tokens actually generated (which cost GPU time), not tokens delivered.
• Metering pipeline: Kafka consumer aggregates token counts per user/org per hour into a time-series store (ClickHouse or TimescaleDB). This feeds the billing API and usage dashboards.

Edge cases that matter for billing:

• Safety-filtered responses: If the input classifier blocks a message, the user is NOT charged (no inference ran). If the output classifier truncates mid-stream, the user IS charged for tokens generated (GPU time was consumed).
• Reconnection: Tokens replayed from the Redis buffer are not double-counted — the count was set when they were first generated.
• Model version pricing: Different models have different per-token prices. The model_version on each message record ties to a pricing table. This is immutable — price changes only apply to new messages.

IC7 signal: "I'll be precise about what consistency level I'm choosing at each layer and why — because 'strong consistency' is not a useful answer without specifying the scope."

Consistency guarantees by layer:

The critical insight: In a 1:1 chat with an LLM, the strongest consistency requirement is read-your-own-writes for the active user on the active device. Everything else can be eventual. This is because the user is the only writer to their conversation (Claude's responses are system-generated through the same pipeline), and the most important UX guarantee is "I see what I just typed."

Cassandra consistency levels:

• Writes: LOCAL_QUORUM (2 of 3 replicas in the local DC). This ensures durability without paying cross-DC latency. If we have a US-East and EU-West DC, a US user's write hits 2/3 US replicas synchronously and replicates to EU asynchronously.
• Reads: LOCAL_QUORUM for the active session (guarantees read-after-write). LOCAL_ONE for history loads and non-active sessions (lower latency, acceptable staleness).

IC7 signal: "Multi-device sync in a chat system is often over-engineered. For a 1:1 LLM chat, the sync model is simpler than WhatsApp or iMessage because there's only one conversation partner (Claude), and responses are generated server-side."

How sync works:

• User sends from phone: POST goes to Chat Service → message written to Cassandra → event published to Kafka. The Connection Manager's Kafka consumer sees the event, looks up all active SSE connections for this user, and pushes the message to the laptop's SSE stream.
• Claude responds: Token stream is fanned out to ALL active device connections simultaneously. Both phone and laptop see tokens appear at the same time.
• Device comes online after being offline: On SSE reconnect, the client sends its Last-Event-ID. The Connection Manager replays missed events from Kafka (or a Redis catchup buffer if within 5 min, or queries Cassandra directly for older messages).

Conflict resolution (user types on both devices):

• Last-writer-wins with the per-conversation version counter from Q4. If both devices send a message simultaneously, one gets the version bump, the other gets a conflict error and retries.
• The UX: show "sending..." on both, but only one actually gets queued for inference. The other device's message is requeued as the next message in the conversation.

IC7 opening (first 30 seconds): "I'll design the public-facing API that serves Claude to external developers. This is Anthropic's revenue engine — every dollar flows through this API. I'll scope to: multi-model serving (Haiku, Sonnet, Opus), synchronous + streaming responses, multi-tenant isolation with per-org rate limiting, and a target of P99 time-to-first-token under 800ms for Sonnet. I'll explicitly call out where safety constraints shape the architecture differently from a generic serving system."

Requirements I'm driving:

• Functional: /v1/messages endpoint (sync + streaming), /v1/batch for async bulk jobs, model selection by name+version, system prompts, multi-turn conversations, tool use / function calling, token counting in response
• Non-functional: P99 TTFT < 800ms (Sonnet), P99 per-token latency < 30ms, availability 99.95% (26 min downtime/month budget), throughput 500K requests/min peak, multi-region (US, EU, APAC)
• Safety: Constitutional Classifier on every request. Content filtering before AND after inference. Per-org audit logs. Compliance with ASL-3 for frontier models.
• Business: Per-org API key auth, usage-based billing (input/output tokens), rate limiting at org and model tier, cost transparency

Back-of-envelope:

Key architectural decisions:

• Separate tenant scheduler before GPU allocation: This is not just rate limiting at the gateway — it's a weighted fair queue that ensures org A's 10K req/min burst doesn't starve org B's 100 req/min. This is the most important component Bharat will probe.
• Model Registry as source of truth: Immutable model versions. Each version has a unique ID (e.g., claude-4-sonnet-20260501), a weight file URI, a configuration hash, and a safety classifier version. Rollbacks are instant: just update the routing table to point to the previous version.
• Streaming Proxy as a separate component: Decouples long-lived SSE connections from inference workers. Workers generate tokens and push to an internal queue; the Streaming Proxy holds the client connection and drains the queue. This means we can scale workers and connection-holders independently.

IC7 signal: "I'll design the external API with the same rigor as a public specification — versioned, idempotent, with explicit error contracts. Bharat built GWS; he knows what a production API contract looks like."

API versioning strategy:

• Header-based versioning (Anthropic-Version: 2026-05-01), not URL-based. This allows us to make backward-compatible changes without forcing URL migrations. Breaking changes require a new version date.
• Model version is explicit: Clients pin to claude-4-sonnet-20260501. The alias claude-4-sonnet (no date) resolves to the latest — but we log which exact version was used in the response, so clients can reproduce results.

Idempotency:

• X-Request-ID is the idempotency key. If we receive a duplicate within 24 hours, we return the cached response without re-running inference. The dedup store is Redis with 24h TTL, keyed by {org_id}:{request_id}.
• This is critical for retries after network timeouts — the client might not know if the request was processed. Without idempotency, a retry would generate a different response (temperature > 0) and double-charge the org.

IC7 signal: "This is the hardest problem in a multi-tenant serving system, and it's one Bharat lived at Google. The naive approach — per-org rate limits at the gateway — doesn't work because the bottleneck isn't HTTP requests, it's GPU-seconds. I need to solve fairness at the GPU scheduling layer."

Three layers of isolation:

Layer 1 — Admission Control (API Gateway):

• Token bucket rate limiter per org, per model, stored in Redis. Three dimensions: requests/min, input tokens/min, output tokens/min.
• Returns 429 Too Many Requests with retry_after header. This is the coarse filter — keeps volume within contracted limits.
• But this alone doesn't prevent starvation. An enterprise org with 50K RPM limit can still monopolize GPUs if their requests are all Opus (10x more GPU time than Haiku).

Layer 2 — Weighted Fair Queuing (Tenant Scheduler):

• This is the critical layer. Each org gets a virtual queue. The scheduler dequeues using Weighted Fair Queuing (WFQ): weight = org's contracted spend tier.
• When GPUs become available, the scheduler picks the next request from the org with the highest deficit (most underserved relative to its weight). This ensures even during overload, every org gets GPU time proportional to their tier.
• Concurrency cap per org: In addition to WFQ, each org has a max concurrent inference requests (e.g., Enterprise = 200, Startup = 10). This prevents one org from consuming all GPU memory with long-running Opus requests.
• Queue depth limit: If an org's queue exceeds 1000 pending requests, new requests are rejected with 529 (overloaded). This prevents unbounded memory growth.

Layer 3 — Worker-Level Isolation:

• Each inference request runs in an isolated memory space on the GPU. KV cache allocation is bounded per request (max_tokens controls this). No request can OOM the worker by consuming another request's KV cache.
• Request-level timeout: if inference exceeds 120s, the request is killed and the GPU memory is reclaimed.

IC7 signal: "I'll use Little's Law to derive GPU count from SLO requirements, not just throughput. The constraint isn't 'can we serve all requests' — it's 'can we serve them within our latency SLO while maintaining headroom for bursts.'"

Little's Law: L = λ × W

Capacity planning methodology (what Bharat will want to hear):

• SLO-driven, not throughput-driven: I don't size for "can we serve 500K RPM." I size for "can we serve 500K RPM with P99 TTFT < 800ms." These are different numbers because P99 requires headroom for queuing delay.
• Queuing theory: At 70% utilization, the M/G/1 queue model gives P99 queue time of ~10× average service time × (1/(1-ρ)). At 90% utilization, queue times explode. This is why 70% target, not 90%.
• Regional distribution: US-East (50%), EU-West (30%), APAC (20%). Each region is self-sufficient — no cross-region inference. Regional isolation means a US outage doesn't cascade to EU.
• Autoscaling signal: Queue depth × avg service time (estimated queue wait). Scale up at > 2s estimated wait, scale down at < 0.5s. Use predictive scaling for known patterns (weekday 9am traffic ramp).

IC7 signal: "GPU OOM is the most common inference failure mode, and it's fundamentally different from CPU OOM because GPU memory is shared across batched requests. A single OOM can kill an entire batch."

Prevention mechanisms (most OOMs should never happen):

• Admission control on KV cache: Before adding a request to a batch, the worker estimates KV cache memory: KV_size = 2 × num_layers × hidden_dim × (input_tokens + max_tokens) × sizeof(float16). If estimated total exceeds 85% of GPU memory, reject admission to this batch.
• Max concurrent batch size: Hard cap based on model size. For Sonnet on H100 (80GB): model weights ~30GB, leaving ~50GB for KV cache. At ~1MB per 1K tokens, that's ~50 concurrent requests with 1K context each. Dynamic cap adjusts based on actual context lengths.
• Memory monitoring: Worker reports GPU memory utilization every 100ms to the scheduler. Scheduler reduces routing weight to workers above 80% utilization.

IC7 signal: "Model rollouts are fundamentally harder than code rollouts because regressions are subtle — a model might generate slightly worse responses without any errors or latency changes. I need both automated and human-in-the-loop validation."

The hard part — detecting quality regression:

• Latency and error rate are easy. Quality regression (model gives slightly worse answers) is hard because there's no objective metric.
• Eval harness: Run a fixed eval set (1000 prompts with known-good responses) against the new model. Score using automated judges (another Claude instance scores quality 1-5). Flag if mean score drops > 0.2.
• Thumbs-down rate: During canary, compare user thumbs-down rate between old and new. Statistically significant increase (p < 0.05) triggers rollback.
• Safety regression: If the safety classifier fires more often on the new model's output, that's a signal the model is generating more borderline content — even if none is actually served to users.

IC7 signal: "Rate limiting at an inference API is multi-dimensional. I can't just count requests — I need to count tokens, because a 100K-token Opus request consumes 1000x more resources than a 100-token Haiku request."

Implementation — Sliding Window + Token Bucket hybrid:

• Request rate: Sliding window counter (Redis ZSET with timestamps). Counts requests in the last 60 seconds. Simple, accurate, no burst problem.
• Token rate: Token bucket (Redis key with counter + refill timestamp). Refills at contracted TPM / 60 per second. This allows short bursts (up to bucket capacity) while enforcing sustained rate.
• Pre-check vs. post-check: Input tokens are known before inference (we run the tokenizer). Output tokens are estimated (max_tokens as upper bound). We deduct the estimate from the token bucket, then credit back the difference after inference completes. This prevents over-admission while avoiding over-charging.

Rate limit headers in every response:

IC7 signal: "Continuous batching is the single biggest throughput optimization in LLM serving, and understanding the prefill/decode asymmetry is essential to capacity planning. Let me walk through the GPU timeline."

Prefill vs. Decode — fundamentally different GPU workloads:

Key insight for continuous batching: Since decode is memory-bandwidth-bound, adding more requests to a decode batch costs almost zero additional time (the GPU reads model weights once regardless of batch size). This is why continuous batching achieves 10-30x better throughput than static batching — the GPU is never waiting for a full batch to form, and new requests slot into the decode batch immediately after their prefill.

Scheduling optimization — chunked prefill:

• A long prefill (e.g., 100K tokens) can block decode steps for hundreds of milliseconds, spiking TTFT for other requests in the batch.
• Solution: break prefill into chunks (e.g., 512 tokens per chunk). Between chunks, run a decode step for the rest of the batch. This interleaves prefill and decode, keeping per-token latency stable for existing streams.

IC7 signal: "I'll walk through a systematic debugging framework, not just guess. The right approach is: scope the problem, check the dashboards, narrow to a component, and identify root cause. This is how I'd lead an incident."

Step 1 — Scope:

• Is this one customer or multiple? → Check TTFT dashboards filtered by org_id. If one org, likely their request pattern. If global, likely infrastructure.
• Is this one model or all models? → Filter by model. If one model, likely that model's cluster. If all, likely shared infrastructure (gateway, scheduler).
• Is this one region or all regions? → Filter by region. If one region, likely regional capacity or network issue.

Step 2 — Component-by-component latency breakdown:

Step 3 — Most likely causes (in order of probability):

1. Queue wait spike: Their requests are waiting for GPU allocation. Check: tenant scheduler queue depth for this org. Cause: another high-priority org is consuming GPU capacity, or their traffic spiked and exceeded concurrency cap. Fix: if legitimate, scale GPU pool or adjust weights.
2. Long input context: Customer started sending 50K-token inputs instead of 2K. Prefill scales linearly with input length. Check: avg input_tokens for this org's recent requests. Fix: this is expected behavior — communicate to customer that longer contexts have higher TTFT.
3. Model cold start: Their requests are hitting a newly scaled-up worker that's still loading model weights (~30-60 seconds for a large model). Check: worker age distribution. Fix: improve warm pool sizing, preload models on standby workers.
4. KV cache fragmentation: GPU memory is fragmented after many allocations/deallocations, forcing garbage collection pauses. Check: GPU memory fragmentation metrics. Fix: implement PagedAttention (vLLM-style) to eliminate external fragmentation.
5. Network/infrastructure: Rarely the cause for TTFT, but check: gateway latency spikes, Redis (rate limiter) latency, cross-AZ network latency if scheduler and workers are in different AZs.

IC7 signal: "GPU cost is the dominant line item — likely 60-70% of COGS for an inference company. I'll walk through the optimization hierarchy, from biggest impact to smallest, with the SLO trade-off for each."

Cost optimization hierarchy (ordered by impact):

The biggest lever — KV cache reuse:

• Many API requests share common prefixes: system prompts, few-shot examples, RAG context. If we cache the KV states for these prefixes, subsequent requests skip prefill for the shared portion.
• Implementation: hash the prefix tokens → look up in a distributed KV cache (Redis or custom GPU-memory cache). If hit, load the cached KV states directly to GPU memory. If miss, run normal prefill and cache the result.
• For a customer whose system prompt is 2,000 tokens (common for API users), caching saves ~100-200ms of prefill per request. At 8,333 req/sec, that's ~1,000 GPU-seconds saved per second.

Speculative decoding:

• Use a small draft model (Haiku) to generate N candidate tokens, then verify them in one forward pass of the target model (Sonnet/Opus). If K of N candidates are accepted, we generated K tokens in the time of ~2 forward passes instead of K forward passes.
• Acceptance rate depends on task: ~70-80% for straightforward text, ~40-50% for creative/reasoning. Average 2-3x speedup.