Back to the course

Course artifact

Pilot reliability scorecard

A reliability and game-day checklist for deciding whether an inference pilot is ready for production.

labs/pilot-scorecard.mdPrivate course toolkit

Pilot reliability scorecard

Area Evidence required Result
Service contract p95/p99 latency, availability, quality, queue and error policy
Release immutable version, smoke test, shadow/canary, rollback
Capacity warm pool, burst startup, batch policy, shortage behavior
Data flow diagram, residency, retention, deletion, access logs
Security secrets, tenant isolation, image provenance, vulnerability path
Incident owner, pager, status updates, recovery and postmortem
Economics achieved utilization, idle, failures, support, gross margin

Game-day cards

  1. A GPU disappears during a burst.
  2. Model weights fail integrity validation.
  3. A prompt exceeds the normal context by 20×.
  4. A client disconnects mid-stream.
  5. The preferred region has no new capacity.

For each: symptom, automatic protection, operator action, recovery target, prevention.