Separate API aggregation, managed inference, runtime infrastructure, and custom silicon
Lesson 1 of 7 · I · Decide
Choose the company, not the category
Map the inference market and select a lane you can actually win.
Explain why a generic Baseten clone has no credible initial wedge
Select a lane using evidence about founder advantage, capital, and time
Detailed lesson
Use the model in the field
Before the answer: Before reading, write one sentence: what exactly would your customer buy from an 'inference company'?
WORKING MODEL
Four ideas to carry
Layer 1 — API access
A catalog of existing models behind one endpoint. Fastest to ship, easiest to copy, weakest moat unless routing or distribution is exceptional.
Layer 2 — managed custom inference
A developer gives you a model or workload; you package, optimize, deploy, scale, observe, and support it. This is Baseten's center of gravity.
Layer 3 — runtime and control plane
Scheduling, KV-aware routing, weight distribution, autoscaling, compilation, reliability, and capacity arbitrage. Open source already covers much of the raw machinery.
Layer 4 — model-specific silicon
The model becomes the hardware design target. Potentially step-function economics, but the team, capital, verification, manufacturing, and model-obsolescence risks are severe.
REFERENCE
The answer in one line
Yes, it is a good time to start an inference company—if you begin with a workload-shaped problem rather than “inference” as a category. Demand is real, open runtimes reduce prototyping cost, and the gap between a model demo and a reliable production service remains painful. The catch is that the horizontal market is crowded by specialist platforms, open-source stacks, and hyperscalers.
A basic OpenAI-compatible endpoint is a weekend project. A system customers trust with production traffic is a multi-year operating capability. Baseten's own documentation spans packaging, optimized runtimes, multi-cloud capacity, weight distribution, request routing, environments, promotion, rollback, observability, and autoscaling—not merely model execution.E4–E6
Two superficially similar companies
Baseten-like platform
Customers bring changing models. You win by making deployment faster, performance more predictable, and operations less painful across diverse workloads.
- Prototype
- 4–12 weeks
- Credible product
- 12–24 months
- Core risk
- Distribution + operations
Taalas-like company
You specialize hardware around a chosen model or model family. You win only if the efficiency gain survives quality requirements and model churn.
- Prototype
- Simulation / FPGA first
- Credible product
- Multiple years
- Core risk
- Technical + capital + timing
Taalas reports that its first product required roughly two and a half years, 24 people, and $30 million spent, backed by more than $200 million raised. It also emphasizes that many team members had collaborated for decades.E3 Treat that as the lower bound achieved by an unusually experienced team, not as a normal seed-stage plan.
Your lane-selection gate
Choose a software wedge unless you can answer yes to all four hardware questions: (1) do you have a world-class compiler/architecture/verification nucleus; (2) can you name a stable workload worth hardening; (3) can you fund at least two major technical proof cycles before revenue; and (4) do you have manufacturing and systems partners? If one answer is no, prove value in software first.
SHIP EVIDENCE
Build your lane memo
Complete a one-page memo with: buyer, workload, current workaround, latency or cost failure, why existing platforms fail, your 10× claim, and the cheapest experiment that could disprove it.
Course artifact: founder/01-lane-memo.md
EVIDENCE LEDGER
Sources and limits
Vendor performance and product claims are labeled as first-party. Prices and product surfaces can change; follow the live links before making a purchasing or fundraising decision.
- ev-taalas-thesis The path to ubiquitous AI · Taalas
Taalas says its platform specializes silicon for each model, merges storage and compute, and removes technologies such as HBM and advanced packaging from its system. These are first-party technical and performance claims, not independent verification. - ev-taalas-team The path to ubiquitous AI · Taalas
Taalas says its first product took about two and a half years, a 24-person team, and 30 million dollars spent from more than 200 million dollars raised. It describes many team members as long-time collaborators and depends on experienced external partners. - ev-baseten-product Baseten overview · Baseten
Baseten presents a managed path from an open, fine-tuned, or custom model to a production API with containerization, GPU scheduling, autoscaling, observability, model-specific runtime optimization, and multi-cloud placement. - ev-vllm-stack vLLM Production Stack · vLLM project
The open vLLM Production Stack includes Kubernetes deployment, request routing, KV-cache offload and awareness, monitoring, autoscaling with KEDA, disaggregated prefill, tracing, and cloud deployment examples. - ev-dynamo-disagg Disaggregated serving · NVIDIA Dynamo
NVIDIA Dynamo separates prefill and decode into independently scalable pools. Prefill pressure follows input length and context; decode pressure follows concurrency, output length, and KV-memory demand. The architecture helps when those phases need different resource shapes.
Knowledge check
Make the model stick
Check 1 · discrimination
Which is the strongest first description of an inference startup?
Check 2 · recall
What most distinguishes a Taalas-like company from a Baseten-like company?
Check 3 · transfer
When should a first-time team start with custom silicon?
Lesson 2 of 7 · I · Decide
Find a workload-shaped wedge
Turn founder advantage and customer pain into a falsifiable market thesis.
Run evidence-seeking customer interviews without pitching
Score candidate workloads on urgency, frequency, spend, access, and defensibility
Write a design-partner offer with measurable success criteria
Detailed lesson
Use the model in the field
Before the answer: List three teams that spend real money on inference today and one person you can contact at each.
WORKING MODEL
Four ideas to carry
Search, then execute
A startup is initially a search for a repeatable model. Your interview plan is an experiment plan, not a sales script.
Past behavior beats preference
Ask about the last outage, migration, bill, or launch. Hypothetical willingness to pay is weak evidence.
Workload beats vertical
“Healthcare” is broad. “Streaming pathology segmentation in a hospital VPC with a 200 ms p95” is a tractable product boundary.
A wedge has an expansion path
Start narrow enough to win, but pick a control point that naturally expands into routing, optimization, compliance, or capacity.
REFERENCE
Customer discovery for infrastructure
Infrastructure buyers rarely say “I need a new platform.” They report a missed launch, a cost shock, an SLO violation, a security review, or an engineer who spends every Monday nursing deployments. Interview around the event.
- Context. “Walk me through the last model you put in production.”
- Failure. “Where did the schedule slip or the system become unpredictable?”
- Cost. “What did the failure consume—engineer weeks, GPU idle time, lost requests, or delayed revenue?”
- Workaround. “What did you try, and why is it still unsatisfactory?”
- Commitment. “If we remove this failure in four weeks, will you provide traffic, an owner, and a paid pilot?”
YC's core loop is launch, talk to users, and iterate; Steve Blank frames this stage as searching through experiments rather than executing a fixed plan.E11–E12 For deep infrastructure, “launch” can be a benchmark report or a concierge deployment before it is a self-serve product.
Candidate wedges worth testing
Long-context coding agents
Repeated prefixes, bursty concurrency, high inter-token sensitivity. Potential edge: KV-aware routing and cache economics.
Private multimodal inference
Large payloads, custom preprocessing, VPC constraints. Potential edge: data plane installed in customer cloud.
Real-time voice
Strict latency budgets and streaming. Potential edge: admission control and regional warm pools.
Fine-tuned model fleets
Many LoRAs, sparse traffic, rollout risk. Potential edge: rapid adapter loading and safe promotions.
Regulated batch inference
Auditability and predictable completion windows. Potential edge: cost-aware scheduling and evidence logs.
Do not pick from this list because it sounds fashionable. Score only workloads where you can reach users, access representative traffic, and measure a failure better than incumbents.
The five-customer gate
Proceed to the technical MVP when five independent buyers describe the same painful event, at least two share a representative workload, and one signs a paid or tightly time-bounded design partnership. Until then, your job is learning.
SHIP EVIDENCE
Run the 20-interview sprint
Use the interview guide and wedge scorecard. Interview 20 operators in two weeks, summarize only observed events, rank three workloads, and ask the top five for traffic traces or replayable requests.
Course artifact: labs/customer-interview-script.md
EVIDENCE LEDGER
Sources and limits
Vendor performance and product claims are labeled as first-party. Prices and product surfaces can change; follow the live links before making a purchasing or fundraising decision.
- ev-baseten-product Baseten overview · Baseten
Baseten presents a managed path from an open, fine-tuned, or custom model to a production API with containerization, GPU scheduling, autoscaling, observability, model-specific runtime optimization, and multi-cloud placement. - ev-yc-launch YC's essential startup advice · Y Combinator
Y Combinator advises founders to launch a useful early product, talk directly to users, iterate from observed needs, keep the team small before product-market fit, and care about unit economics rather than scaling an unprofitable product. - ev-blank-search Search versus execution · Steve Blank
Steve Blank distinguishes a startup's search for a repeatable business model from the later execution of a known model. Customer development turns business-model assumptions into experiments that can be falsified and revised. - ev-fireworks-serverless Serverless overview · Fireworks AI
Fireworks documents pay-per-token serverless inference with no GPU sizing and features such as prompt caching and session affinity. It is evidence that the commodity API layer already has mature specialist competitors. - ev-modal-platform Modal introduction · Modal
Modal offers general serverless compute with per-second billing, GPU functions, sandboxes, and elastic scale. It broadens the competitive set beyond companies that market themselves only as inference platforms. - ev-aws-provisioned Amazon Bedrock Provisioned Throughput · AWS
Amazon Bedrock lets customers buy fixed-cost model throughput in model units and commitment periods. Hyperscalers therefore compete for predictable enterprise inference capacity in addition to on-demand API traffic.
Knowledge check
Make the model stick
Check 1 · application
Which interview question produces the strongest evidence?
Check 2 · discrimination
Which workload is best defined?
Check 3 · transfer
What is sufficient evidence to start a general platform build?
Lesson 3 of 7 · II · Build
Build the serving MVP
Assemble the smallest production-shaped system that can prove your wedge.
Draw the request and deployment paths separately
Implement an OpenAI-compatible gateway with admission control and metrics
Choose one runtime and defer multi-cloud complexity
Detailed lesson
Use the model in the field
Before the answer: Sketch every component between a client request and the first token. Circle the one component that creates your advantage.
WORKING MODEL
Four ideas to carry
Data plane
Gateway, queue, router, runtime, cache, GPU, streaming response. Optimize the request path.
Control plane
Model registry, builds, deployment versions, autoscaling policy, rollout, rollback, capacity, secrets. Optimize safe change.
Admission before autoscaling
Scaling cannot rescue a queue that has already violated the SLO. Reject, degrade, or route before overload cascades.
One workload, one engine
Start with one model family and one GPU shape. Breadth hides whether your core claim is true.
REFERENCE
The minimum production-shaped architecture
Client→API gateway→Admission→KV-aware router→Runtime→GPU
control plane: registry · builds · versions · rollout · scaling policy · capacity
Separate the data plane from the control plane. The data plane owns hot-path latency and streaming. The control plane owns change, placement, safety, and recovery. A Baseten-scale product implements both deeply.E5–E6
Week-by-week build
- Week 1 — replayable baseline. Run one open model with vLLM on one GPU. Capture a fixed request corpus and reference outputs.
- Week 2 — compatibility edge. Put an OpenAI-compatible gateway in front. Add API keys, request IDs, timeouts, streaming, and structured errors.
- Week 3 — measurement edge. Record queue time, time to first token, inter-token latency, total latency, tokens, cache result, model version, and GPU assignment.
- Week 4 — control loop. Add bounded concurrency, load shedding, and the one routing or batching policy your wedge requires.
- Week 5 — safe change. Package the model, pin the environment, add a health probe, versioned deployment, smoke test, and one-command rollback.
- Week 6 — customer replay. Re-run a design partner's trace, publish the result, and operate the system during a live trial.
What not to build yet
Do not begin with a GPU marketplace, arbitrary containers, ten regions, a custom scheduler, or a beautiful dashboard. vLLM Production Stack and NVIDIA Dynamo already expose advanced routing and distributed serving primitives.E8–E9 Use them until your evidence shows exactly where they fail your workload.
The wedge must live in a control loop
A durable product observes something competitors ignore, makes a better decision, and improves from the result. Examples: prefix reuse predicts routing; deadline class controls admission; adapter popularity controls preloading; data residency controls placement. The loop—not the endpoint—is the seed of your moat.
SHIP EVIDENCE
Run the local gateway and benchmark
Start the provided OpenAI-compatible mock gateway, send a small load test, inspect time-to-first-token and queue metrics, then replace the mock backend with your vLLM endpoint.
Course artifact: labs/README.md
EVIDENCE LEDGER
Sources and limits
Vendor performance and product claims are labeled as first-party. Prices and product surfaces can change; follow the live links before making a purchasing or fundraising decision.
- ev-baseten-control-plane How Baseten works · Baseten
Baseten documents a control plane spanning model packaging, build pipelines, multi-cloud capacity management, request routing, weight delivery and caching, deployment environments, promotion, rollback, and an OpenAI-compatible API for engine-based deployments. - ev-baseten-scaling Autoscaling · Baseten
Baseten documents scale-to-zero, min and max replicas, concurrency targets, utilization thresholds, autoscaling windows, and scale-down delays. It explicitly frames higher concurrency as lower cost with more queueing, and lower concurrency as higher cost with less queueing. - ev-vllm-stack vLLM Production Stack · vLLM project
The open vLLM Production Stack includes Kubernetes deployment, request routing, KV-cache offload and awareness, monitoring, autoscaling with KEDA, disaggregated prefill, tracing, and cloud deployment examples. - ev-dynamo-disagg Disaggregated serving · NVIDIA Dynamo
NVIDIA Dynamo separates prefill and decode into independently scalable pools. Prefill pressure follows input length and context; decode pressure follows concurrency, output length, and KV-memory demand. The architecture helps when those phases need different resource shapes.
Knowledge check
Make the model stick
Check 1 · recall
Which component belongs primarily to the control plane?
Check 2 · application
Traffic suddenly exceeds warm capacity. What is the first protective action?
Check 3 · transfer
What should you custom-build first?
Lesson 4 of 7 · II · Build
Benchmark the promise and the business
Turn performance into goodput, cost, and a result a skeptical buyer can reproduce.
Define a benchmark matrix that controls model, quality, prompts, concurrency, and hardware
Calculate goodput and cost per successful million tokens
Reject performance claims that trade away quality or hide queueing
Detailed lesson
Use the model in the field
Before the answer: Name three ways a provider can make a tokens-per-second chart look impressive without improving your user's experience.
WORKING MODEL
Four ideas to carry
Goodput, not peak throughput
Count requests that meet quality and latency targets. Saturating a GPU while users time out is not success.
A matrix, not one number
Vary input/output length, concurrency, cache reuse, model/quantization, and GPU. Report distributions.
Quality is a constraint
Quantization and speculative methods can change outputs. Compare accuracy or task success before speed.
Economics include idle and failure
Use achieved utilization, cold starts, retries, support, and reserved capacity—not peak benchmark utilization.
REFERENCE
The benchmark contract
Freeze these before running: model revision, tokenizer, precision, quantization, prompt corpus, output cap, decoding settings, hardware, runtime version, batching policy, cache state, concurrency pattern, quality metric, and SLO. MLPerf's central discipline is that performance is measured against defined workloads and accuracy targets.E10
| Metric | What it reveals | How it lies |
|---|---|---|
| TTFT p50/p95/p99 | Queue + prefill responsiveness | Warm cache, short prompts, low concurrency |
| Inter-token latency | Streaming smoothness | Averages hide stalls |
| Output tokens/sec/user | Single-stream generation pace | Can ignore aggregate efficiency |
| Tokens/sec/GPU | Capacity efficiency | Can ignore user latency |
| Goodput | Requests meeting quality + SLO | Only honest if the SLO is external |
| Cost/successful 1M tokens | Business reality | Must include idle, failures, and overhead |
Core equations
goodput = successful_requests_meeting_SLO / second
effective_tokens_per_gpu_second = measured_tokens_per_second × achieved_utilization
compute_COGS_per_1M = gpu_hourly_cost / effective_tokens_per_gpu_second / 3600 × 1,000,000
gross_margin = 1 − total_variable_COGS / revenue
Baseten's live pricing page is a useful retail reference, not your eventual cost base.E7 Record the retrieval date and make the calculator editable. Your true COGS adds CPU, storage, networking, failed requests, support burden, and idle headroom.
Read Taalas claims correctly
Taalas reports an extreme output-rate result and large cost/power advantages on its hard-wired Llama. The same page discloses quality degradation from aggressive first-generation quantization.E2 The intellectually honest response is neither belief nor dismissal: reproduce the workload boundary, compare quality, and ask which applications convert the speed into user or agent value.
Your benchmark report
Publish the corpus description, command, versions, raw results, percentile charts, quality result, failures, and cost assumptions. State what you did not test. A benchmark that a design partner can rerun is both product work and sales collateral.
SHIP EVIDENCE
Produce Benchmark Report 0.1
Use the load generator and unit economics calculator. Run at least four concurrency levels, two prompt shapes, warm and cold cache, then write a one-page conclusion with the best and worst regime.
Course artifact: labs/benchmark.py
EVIDENCE LEDGER
Sources and limits
Vendor performance and product claims are labeled as first-party. Prices and product surfaces can change; follow the live links before making a purchasing or fundraising decision.
- ev-taalas-results The path to ubiquitous AI · Taalas
Taalas reports 17,000 tokens per second per user on a hard-wired Llama 3.1 8B, nearly 10 times its comparison baseline, with lower build cost and power. It also acknowledges quality degradation from aggressive three-bit and six-bit quantization in its first generation. - ev-baseten-scaling Autoscaling · Baseten
Baseten documents scale-to-zero, min and max replicas, concurrency targets, utilization thresholds, autoscaling windows, and scale-down delays. It explicitly frames higher concurrency as lower cost with more queueing, and lower concurrency as higher cost with less queueing. - ev-baseten-pricing Cloud pricing · Baseten
Baseten lists pay-as-you-go dedicated GPU deployment prices by minute and offers scale-to-zero. Its July 2026 page lists H100 and B200 instances alongside smaller GPUs, while enterprise plans add self-hosting, custom SLAs, security controls, and use of existing cloud commitments. Prices are volatile; consult the live page. - ev-mlperf-method MLPerf Inference Benchmark Suite · MLCommons
MLPerf Inference measures system performance across multiple deployment scenarios while pairing performance with defined models, datasets, and accuracy targets. The 2026 suite includes datacenter and edge workloads for several language, vision, recommendation, and speech models.
Knowledge check
Make the model stick
Check 1 · discrimination
Which is the most buyer-relevant primary metric?
Check 2 · application
What must accompany an aggressive quantization speedup?
Check 3 · transfer
Which cost input is most likely to make a prototype model unrealistically optimistic?
Lesson 5 of 7 · II · Build
Operate production inference
Design overload, rollout, capacity, and incident behavior before enterprise traffic arrives.
Write SLOs and overload policy for one workload
Design a versioned rollout and rollback path
Create a capacity plan that distinguishes warm, burst, and reserved traffic
Detailed lesson
Use the model in the field
Before the answer: If one GPU dies during a burst, list the exact order in which the system should protect customers.
WORKING MODEL
Four ideas to carry
SLO first
Define latency, availability, quality, and data boundaries. Autoscaling and routing policies derive from them.
Cold starts are product behavior
A scale-to-zero promise needs weight delivery, parked-request timeouts, and an honest first-request experience.
Versions are immutable
Build, test, promote, observe, and roll back. Never mutate the only known-good deployment.
Capacity has classes
Warm interactive, elastic burst, and asynchronous batch require different economics and promises.
REFERENCE
Write the service contract
For one workload, define p95 and p99 time to first token, inter-token latency, availability, error budget, maximum queue time, quality floor, residency, retention, and recovery objectives. Then specify overload behavior: which requests wait, degrade, route elsewhere, or fail fast.
Four production loops
Baseten's environments, stable endpoints, deployment versions, rolling updates, metrics, and rollback model show how much product lives around the runtime.E5
Capacity without fantasy
Maintain three pools: warm interactive capacity for the SLO; elastic burst capacity with a measured startup time; and batch capacity that can be interrupted or delayed. Track headroom in requests and tokens by prompt shape, not only GPU count. AWS's provisioned throughput product shows that enterprise buyers will pay for predictable capacity as a contract, while on-demand traffic remains variable.E16
Disaggregate only when the trace says so
NVIDIA Dynamo documents that prefill and decode pressure come from different variables and can be scaled separately.E9 This is powerful for long-context or high-concurrency workloads, but it adds cache-transfer and scheduling complexity. Start aggregated; split when measured bottlenecks persist across representative traffic.
The pilot reliability packet
Before live data, deliver an architecture diagram, data flow, subprocessor list, logging/retention policy, threat model, incident contacts, SLO, rollback plan, load-test result, and explicit exclusions. Trust can be your wedge even when raw runtime performance is similar.
SHIP EVIDENCE
Run a game day
Use the incident cards to simulate: GPU loss, weight fetch failure, runaway prompt length, downstream disconnect, and capacity shortage. For each, record customer symptom, automatic protection, operator action, and prevention.
Course artifact: labs/pilot-scorecard.md
EVIDENCE LEDGER
Sources and limits
Vendor performance and product claims are labeled as first-party. Prices and product surfaces can change; follow the live links before making a purchasing or fundraising decision.
- ev-baseten-control-plane How Baseten works · Baseten
Baseten documents a control plane spanning model packaging, build pipelines, multi-cloud capacity management, request routing, weight delivery and caching, deployment environments, promotion, rollback, and an OpenAI-compatible API for engine-based deployments. - ev-baseten-scaling Autoscaling · Baseten
Baseten documents scale-to-zero, min and max replicas, concurrency targets, utilization thresholds, autoscaling windows, and scale-down delays. It explicitly frames higher concurrency as lower cost with more queueing, and lower concurrency as higher cost with less queueing. - ev-dynamo-disagg Disaggregated serving · NVIDIA Dynamo
NVIDIA Dynamo separates prefill and decode into independently scalable pools. Prefill pressure follows input length and context; decode pressure follows concurrency, output length, and KV-memory demand. The architecture helps when those phases need different resource shapes. - ev-aws-provisioned Amazon Bedrock Provisioned Throughput · AWS
Amazon Bedrock lets customers buy fixed-cost model throughput in model units and commitment periods. Hyperscalers therefore compete for predictable enterprise inference capacity in addition to on-demand API traffic.
Knowledge check
Make the model stick
Check 1 · application
What should define autoscaling policy?
Check 2 · discrimination
When is disaggregated prefill/decode justified?
Check 3 · recall
Which release practice gives the safest recovery?
Lesson 6 of 7 · III · Company
Turn pilots into a company
Price the wedge, run design partnerships, hire the nucleus, and raise against evidence.
Structure a paid design partnership around a baseline and success gate
Choose pricing that aligns with the value and cost driver
Build a milestone-based hiring and fundraising plan
Detailed lesson
Use the model in the field
Before the answer: Write the one metric that would make a design partner expand from pilot to production.
WORKING MODEL
Four ideas to carry
Paid learning
A design partnership buys access, priority, and a defined result. Free pilots hide urgency and create orphan integrations.
Price the control point
Per token fits APIs; per GPU-hour fits dedicated capacity; platform or minimum commitments fund reliability and support.
The first team is a nucleus
Founder-led product, inference/runtime, distributed systems/SRE, and a customer-owning engineer. Add functions after repeatability.
Milestones finance risk retirement
Seed proves repeated pain and a working wedge. Series A should prove repeatable deployment and expansion, not promise them.
REFERENCE
The design-partner contract
Define one workload, a frozen baseline, representative traffic, a four-to-eight-week window, weekly operator access, security boundaries, success metrics, a fee, and a production conversion decision. The customer receives priority and influence; you receive data, truth, and a reference if successful.
Baseline: current p95 TTFT, quality, cost, and deployment time.
Target: one primary outcome plus non-regression constraints.
Inputs: traces, model artifacts, test corpus, owner, environment access.
Decision: expand, extend for one named gap, or stop.
Pricing ladders
| Model | When it fits | Risk |
|---|---|---|
| Per token/request | Shared API, elastic traffic | You absorb utilization risk |
| Per GPU-hour + platform fee | Dedicated custom deployment | Can look like low-margin resale |
| Reserved capacity commitment | Predictable enterprise load | Capacity forecasting error |
| Value or outcome component | Wedge tied tightly to business output | Measurement and procurement complexity |
Baseten itself mixes usage pricing with enterprise deployment and support capabilities.E4, E7 Your pilot price should cover attention and test willingness to pay; it does not need to maximize early gross margin.
The first four roles
- CEO/founder-product: discovery, sales, priority, capital, partnerships.
- Inference/compiler engineer: model runtime, kernels, profiling, quality-performance tradeoffs.
- Distributed systems/SRE: gateway, orchestration, observability, incident behavior.
- Customer engineer: integration, evaluation, traffic replay, production handoff.
For a hardware path, add architecture, verification, physical design, compiler, board/system, and manufacturing leadership before tapeout planning. Taalas's long-collaborating team is a warning against assembling that capability casually.E3
Fundraising narrative
Show: repeated incidents; design partners; a reproducible benchmark; unit economics by regime; reliability evidence; and the next risk-retiring milestone. Ask for enough runway to cross that milestone with buffer. Do not pitch a GPU reseller as a software-margin company or a research project as a near-term infrastructure business.
SHIP EVIDENCE
Write the pilot and financing packet
Complete the design-partner one-pager, pricing model, 18-month hiring plan, use-of-funds table, and five milestone charts. Ask one target customer to redline the pilot offer.
Course artifact: labs/design-partner-one-pager.md
EVIDENCE LEDGER
Sources and limits
Vendor performance and product claims are labeled as first-party. Prices and product surfaces can change; follow the live links before making a purchasing or fundraising decision.
- ev-taalas-team The path to ubiquitous AI · Taalas
Taalas says its first product took about two and a half years, a 24-person team, and 30 million dollars spent from more than 200 million dollars raised. It describes many team members as long-time collaborators and depends on experienced external partners. - ev-baseten-product Baseten overview · Baseten
Baseten presents a managed path from an open, fine-tuned, or custom model to a production API with containerization, GPU scheduling, autoscaling, observability, model-specific runtime optimization, and multi-cloud placement. - ev-baseten-pricing Cloud pricing · Baseten
Baseten lists pay-as-you-go dedicated GPU deployment prices by minute and offers scale-to-zero. Its July 2026 page lists H100 and B200 instances alongside smaller GPUs, while enterprise plans add self-hosting, custom SLAs, security controls, and use of existing cloud commitments. Prices are volatile; consult the live page. - ev-yc-launch YC's essential startup advice · Y Combinator
Y Combinator advises founders to launch a useful early product, talk directly to users, iterate from observed needs, keep the team small before product-market fit, and care about unit economics rather than scaling an unprofitable product. - ev-blank-search Search versus execution · Steve Blank
Steve Blank distinguishes a startup's search for a repeatable business model from the later execution of a known model. Customer development turns business-model assumptions into experiments that can be falsified and revised.
Knowledge check
Make the model stick
Check 1 · application
What makes a design partnership diagnostic?
Check 2 · recall
Which pricing model best matches dedicated predictable capacity?
Check 3 · transfer
What should a seed round primarily finance?
Lesson 7 of 7 · IV · Moonshot
Earn the Taalas path
Convert a software bottleneck into a model-hardware co-design program—with stop gates.
Identify a stable model/workload boundary suitable for specialization
Sequence software profiling, architecture simulation, FPGA/emulation, and silicon gates
Write stop criteria for quality, economics, model churn, and capital
Detailed lesson
Use the model in the field
Before the answer: Name the exact data movement or operator that dominates your chosen workload and how you know.
WORKING MODEL
Four ideas to carry
Specialize around evidence
Profile real traffic to find stable operators, memory movement, precision, and latency constraints.
Model churn is a first-class test
Replay at least one plausible architecture change. If the advantage disappears, the design is too brittle.
Compiler and system are part of the chip
A fast die without model mapping, host integration, observability, and deployment workflow is not a product.
Stop gates protect ambition
Quality, performance, manufacturability, customer value, capital, and schedule must all pass.
REFERENCE
What Taalas is actually betting
Taalas's stated bet is total specialization: map a model into custom silicon, merge storage and compute, and simplify the surrounding system.E1 This trades generality for large gains. The startup opportunity is not “build a faster chip.” It is “find a stable computation valuable enough to harden, and build the compiler, system, and supply chain that deliver it.”
The evidence ladder
- Gate 0 — workload proof. Paid demand, representative traces, quality floor, and a bottleneck stable across versions.
- Gate 1 — software ceiling. Best available GPU/runtime optimization still misses the economic or latency target.
- Gate 2 — architecture model. Cycle, memory, power, area, and yield assumptions produce a step-function system advantage.
- Gate 3 — compiler mapping. Real model graphs map automatically enough to support the product promise.
- Gate 4 — FPGA/emulation. Functional correctness, host I/O, scheduling, and representative kernels work end to end.
- Gate 5 — customer pre-commit. Buyers accept quality, model boundary, deployment shape, and purchase economics.
- Gate 6 — tapeout readiness. Verification closure, DFT, physical plan, package/board, firmware, manufacturing slots, and capital buffer.
Three red-team scenarios
Model architecture changes
Attention, sparsity, context, experts, or precision moves. Quantify how much of the chip remains useful.
GPU software improves
Open runtimes, kernels, and new hardware close your gap before launch. Compare against a forward baseline.
Quality floor tightens
Your quantization or approximation no longer passes. Measure the cost of restoring quality.
A realistic company sequence
Year 0: software wedge, paid traces, profiling, compiler prototype. Year 1: architecture simulator, model mapper, FPGA/emulation, customer pre-commit. Year 2+: silicon only after the gates. This is not a promise of schedule; it is a risk-ordering strategy.
Your next 90 days
Days 1–14 20 interviews + lane memo
Days 15–30 baseline + trace corpus
Days 31–60 gateway + control loop + benchmark
Days 61–90 paid pilot + reliability game day + go/no-go
At day 90, continue only if you have repeated pain, representative traffic, a measurable advantage, and commitment. Otherwise change the wedge—not the chart.
SHIP EVIDENCE
Write the hardware pre-mortem and 90-day operating plan
Complete every hardware gate with current evidence, mark unknowns, run the three red-team scenarios, then schedule weekly deliverables for the first 90 days. No gate may be marked passed by enthusiasm.
Course artifact: labs/hardware-gates.md
EVIDENCE LEDGER
Sources and limits
Vendor performance and product claims are labeled as first-party. Prices and product surfaces can change; follow the live links before making a purchasing or fundraising decision.
- ev-taalas-thesis The path to ubiquitous AI · Taalas
Taalas says its platform specializes silicon for each model, merges storage and compute, and removes technologies such as HBM and advanced packaging from its system. These are first-party technical and performance claims, not independent verification. - ev-taalas-results The path to ubiquitous AI · Taalas
Taalas reports 17,000 tokens per second per user on a hard-wired Llama 3.1 8B, nearly 10 times its comparison baseline, with lower build cost and power. It also acknowledges quality degradation from aggressive three-bit and six-bit quantization in its first generation. - ev-taalas-team The path to ubiquitous AI · Taalas
Taalas says its first product took about two and a half years, a 24-person team, and 30 million dollars spent from more than 200 million dollars raised. It describes many team members as long-time collaborators and depends on experienced external partners. - ev-dynamo-disagg Disaggregated serving · NVIDIA Dynamo
NVIDIA Dynamo separates prefill and decode into independently scalable pools. Prefill pressure follows input length and context; decode pressure follows concurrency, output length, and KV-memory demand. The architecture helps when those phases need different resource shapes. - ev-mlperf-method MLPerf Inference Benchmark Suite · MLCommons
MLPerf Inference measures system performance across multiple deployment scenarios while pairing performance with defined models, datasets, and accuracy targets. The 2026 suite includes datacenter and edge workloads for several language, vision, recommendation, and speech models.
Knowledge check
Make the model stick
Check 1 · recall
What is the first hardware gate?
Check 2 · application
Why benchmark against a forward GPU baseline?
Check 3 · transfer