1. Code Yellow Context & Strategy
What is the UBB Code Yellow?
A cross-organizational effort (April 28 – June 20, 2025) to win AI application-layer companies (Cursor, Perplexity, Replit, Midjourney) for Usage-Based Billing, competing against Metronome and Orb.
Success Criteria
- 2 hero AI logos acquired (sending usage + 1 invoice paid)
- 20 AI prospects send usage events
- 5 AI prospects adopted
- $2M Billing deal signed/renewed
- Revamped UBB marketing page
Operating Model
Stripe evolved UBB "from skateboard phase to bicycle phase" — optimized for speed of learning:
- 1-week sprints (Tuesday–Monday) for rapid build-measure-learn loops
- 1-day timeboxed API design phases
- Co-development with 3-5 AI startups (engineers in customer Slack channels)
- 5 autonomous squads formed — "Usage monitoring and analytics" was one
- Coordination channel:
#win-ai-ubb-analytics
Teams Involved
UBB Team
Owns upstream pipeline — UAP (Usage Aggregation Pipeline), RTR (Real-Time Ratings) Flink apps
Revenue Intelligence
Owns analytics Flink apps, dashboard charts, Meter Usage API, Spend API
Product Catalog
V1 products/prices and V2 RateCards/PricingPlans — heavy overlap with code yellow primitives
Analytics Core
Platform APIs (v1/analytics/metrics, metric_run) powering analytics data surfaces
BillingScript
Borrowed engineers during code yellow for UBB-related projects
Billing UX Platform
Frontend dashboard surfaces for billing including UBB products
Resource Impact
4.5 HC reallocated from Billing Analytics (subscription adoption) → UBB/genAI acquisition through H1. Subscription PLG, Essential Journey, and UI lib migration all deferred to Q4 2025+.
Timeline
Late April 2025
Code Yellow declared; API review filed (target: May 2)
May 2025
Meter Usage API approved for Private Preview; rate limiting implemented; code complete May 9
June-July 2025
Kafka over-provisioning, incident remediation, SDK private preview releases
Jan 2025
Rated Usage PRD authored — rated usage metrics for dashboard
Oct 2025
Spend Analytics API PRD — usage & spend for customer portals
April 2026
Spend API launched in Private Preview for Sessions
March 2026
ir-canny-tradition incident with Perplexity (3-track data quality failure)
2. Product Overview
What is UBB Analytics?
UBB Analytics provides real-time usage and spending data for merchants using Usage-Based Billing. It powers three fundamental capabilities:
Aggregate Usage
Query raw usage by meter, customer, dimension, tenant — 1-3 min freshness
Rated Usage (Spend)
Usage × price to forecast upcoming invoice amounts — 5-10 min freshness
Spending Trends
Time-series data for individual customers or cohorts — hourly/daily/weekly/monthly
Key Use Cases
| Use Case | Who | Example |
|---|
| Pre-invoice usage check | End customers | "How much have I used this billing period?" |
| High-usage monitoring | Merchants | Prevent surprise bills, identify expansion opps |
| Spend forecasting | Sales teams | "Top 10 customers by forecasted spend this month" |
| Billing dispute resolution | Support | "Why was I charged $X?" |
| Customer portal | Merchants → their customers | "Your current usage = ~$X charge" |
User Problem (from PRD)
The Gap
AI companies (Perplexity, Cursor, Lovable, Together AI) need to give their customers clear visibility into usage and spend. Competitors (Metronome, Orb) already provide this. Without it, Stripe loses deals.
Target Users
Perplexity, Together AI, Vercel, Figma, UiPath, Cursor, Lovable — primarily AI application layer companies with usage-based pricing models.
Key User Stories (Spend API — Prioritized)
| Priority | Story | Requested By |
|---|
| P0 | Period-to-date spend with ≤2 min freshness | Lovable, Perplexity |
| P0 | Filter/group by pricing dimensions, meter dimensions, tenant ID | Multiple |
| P1 | Historical period spend | Multiple |
| P1 | Time series breakdown (hourly/daily granularity) | Multiple |
| P2 | Pure UBB spend for any chosen time window | Future |
3. APIs — Shape & Design
3.1 Meter Usage API Private Preview
| Endpoint | GET /v1/billing/analytics/meter_usage |
|---|
| Purpose | Query aggregate usage grouped/filtered by meter dimensions and tenants |
|---|
| Gate | BillingAnalyticsUsagePrivatePreview |
|---|
| Permission | billing_analytics_meter_usage_read |
|---|
| Rate Limit | 100 RPS / merchant, 100 concurrent requests |
|---|
| Error | HTTP 429 on violation |
|---|
| Approved | May 7, 2025 (APIREVIEW-3649) |
|---|
Parameters
| Param | Required | Type | Notes |
|---|
start_time | Yes | timestamp | Query range start |
end_time | Yes | timestamp | Query range end |
customer_id | No | string | Filter by customer |
meters | Yes | array | meter_id + dimension_group_by_keys + dimension_filters + tenant_filters |
value_grouping_window | No | enum | hour / day / week / month |
timezone | No | string | IANA timezone |
Response Shape
{
"data_refreshed_at": 1715100000,
"data": [
{
"bucket_start_time": 1715000000,
"bucket_end_time": 1715003600,
"meter_id": "mbm_xxx",
"dimensions": {"model": "gpt-4"},
"bucket_value": 1542.0
}
]
}
Key Design Decisions
- No pagination — Pinot limitation; mitigated by cardinality restrictions and max 90-day range
- Tenant filters added May 20 addendum for high-cardinality dimensions
- Breaking change:
tenant_filters/dimension_filters updated to accept Array values instead of single String
- Alpha users: ~5 (Perplexity, Lovable, Cursor) via gate
3.2 Spend API Private Preview
| Endpoint | GET /v2/billing/analytics/usage_spend |
|---|
| Purpose | Rated usage (usage × price) for forecasting upcoming invoice amounts |
|---|
| Launched | April 2026 (Sessions) |
|---|
| Rate Limit | 25 RPS initially (scalable to 10,000 QPS planned) |
|---|
| Namespace | v2/ — aligned with Billing Product Catalog domain |
|---|
Parameters
| Param | Required | Type |
|---|
customer_id | Yes | string |
period | Yes | time_range / subscription / billing_cadence |
value_grouping_window | No | hour / day / week / month |
group_by_keys | No | metered_item / rate_card |
filters.metered_items | No | array of IDs |
Response Shape
{
"refreshed_at": 1715100000,
"rows": [
{
"starts_at": 1715000000,
"ends_at": 1715086400,
"rated_usage_amount": 15420,
"discount_amount": 1000,
"promotional_credit_amount": 500,
"paid_credit_amount": 200,
"total_excluding_tax_amount": 13720,
"currency": "usd"
}
]
}
Spend Terminology
| Term | Formula |
|---|
| Rated usage | price × quantity (gross) |
| Paid usage | rated_usage − discounts − free credits |
| Amount pretax | rated_usage − discounts − all credits |
3.3 Customer Portal Usage Analytics API
| Endpoint | GET /v1/billing_portal/sessions/{session}/exp/usage_analytics |
|---|
| Purpose | Powers merchant customer portals — end-customers see their own usage |
|---|
| Auth | Ephemeral key (validates customer ownership) |
|---|
| Gate | customer_portal_usage_analytics |
|---|
Key feature: Portal configuration stores allowed_dimensions — merchants control which dimensions their end-customers can group/filter by.
3.4 Dashboard Analytics Metric
The Analytics Core metric usage_based_billing.gross_usage_revenue powers the dashboard charts:
- Measure:
usage_revenue (currency, minor units)
- Dimensions: price, product, customer, meter
- Rate limit: 1 req/s, concurrency: 5
- Limitation: Meters with formula 'LAST' or 'MAX' not supported
- Available after: 2025-04-02 (historical data purged due to pipeline bugs)
6. Key Decisions & Tradeoffs
Decision 1: Real-time vs. Batch
Choice: Real-time streaming (Flink → Kafka → Pinot)
Why: Merchants need current-period usage data to forecast bills. The "surprise bill" problem is solved only with real-time data. Competitors (Metronome, Orb) already offer this.
Tradeoff: Higher operational complexity, more failure modes, requires Flink expertise. Bootstrap takes 12-24 hours.
Decision 2: RTR Eviction Timer — Accuracy vs. Freshness
Choice: Currently 1-2 min, planned increase to 10-15 min
Why: Longer windows let RTR collect more usage events before rating, improving accuracy for bursty patterns. Dashboard SLA (<1 hour) gives headroom.
Tradeoff: Freshness degrades from 2-3 min to 11-16 min. Acceptable for dashboard; may not be for customer portal (P0 = ≤2 min).
Decision 3: Hybrid Pinot Tables
Choice: Real-time ingestion from Flink + daily batch segments
Why: Real-time provides freshness; batch provides correctness (backfills, corrections). AdvanceTimeBoundary mechanism controls query routing.
Tradeoff: Complexity managing time boundaries; potential overlap during transitions. AdvanceTimeBoundary failures block settings changes.
Decision 4: Metered Billing Excluded from MRR
Choice: Usage-based charges excluded from MRR metric
Why: MRR = "predictable recurring revenue." Usage is inherently unpredictable month-to-month. This is industry-standard.
Tradeoff: 100% metered merchants see $0 MRR / $0 active subscribers. "Total Monthly Revenue" metric (MRR + usage) is in development.
Decision 5: Separate Aggregate vs. Rated APIs
Choice: /meter_usage (raw) vs. /spend (rated)
Why: Different freshness guarantees (1-3 min vs 5-10 min), different consumers, different query patterns. Aggregate = operational monitoring; Spend = financial forecasting.
Tradeoff: Two APIs to learn, two data paths to monitor, potential confusion on which to use.
Decision 6: v2 Namespace for Spend API
Choice: GET /v2/billing/analytics/usage_spend
Why: Aligned with Billing Product Catalog domain (rate_cards, metered_items are v2 concepts). Also allows cleaner resource-oriented design.
Tradeoff: v2 APIs initially required Sandbox (not Legacy Testmode). DEeP team built dual-projection to support LTM for code yellow by July 2025.
Decision 7: 15-Minute Bucket Granularity
Choice: Data stored in 15-min time buckets in Pinot
Why: Enables timezone flexibility (can re-bucket to hour/day/week/month at query time). Sub-minute precision not needed for billing analytics.
Tradeoff: Cannot drill into sub-15-min patterns. Acceptable for the "how much have I spent?" use case.
Decision 8: No Pagination (Pinot Limitation)
Choice: Limit response via cardinality restrictions and validation errors
Why: Pinot doesn't efficiently support cursor-based pagination. Max ~200 rows estimated per query.
Tradeoff: Can't return unbounded result sets. Mitigated by max 90-day range for day/hour granularity and validation errors for excessive cardinality.
9. Customer Portal & Configuration
Portal Usage Analytics
Merchants can enable a usage analytics experience in their customer portal, allowing end-customers to see their own usage and spend data.
Implementation Pattern
// Portal configuration model
{
enabled: true,
usage_analytics: {
group_by_dimensions: ["model", "region"] // merchant controls
},
credit_balance: { ... }
}
// At API layer:
// 1. Auth via ephemeral key
// 2. Validate customer ownership
// 3. Filter group_by/filters against portal config's allowed_dimensions
// 4. Delegate to GetUsageAnalytics command
Branching Logic
| report_type | DataStory Used | Data Source |
|---|
"usage" | billing_meter.combined_usage_event_aggregations.2 | Aggregate usage Pinot table |
"spend" | billing_meter.rated_usage_customer_portal.1 | Rated usage Pinot table |
Response Shape
{
"object": "billing.analytics.usage_analytics",
"livemode": true,
"refreshed_at": 1715100000,
"rows": [
{
"starts_at": 1715000000,
"ends_at": 1715086400,
"value": 1542.0,
"meter": "mbm_xxx",
"dimensions": {"model": "gpt-4"}
}
]
}
Key Architectural Patterns
- Dimension filtering at API layer: Portal config stores
group_by_dimensions that restrict which dimensions customers can query
- Gap-filling: Command generates expected time buckets and zero-fills missing intervals with timezone-aware bucketing
- Currency conversion in-query: FX rates joined at Pinot query time
10. Interview Prep Questions
Why real-time over batch for UBB analytics?
Merchant trust requires current-period visibility. The core use case — "how much will my bill be?" — is only valuable with fresh data. Batch (hourly/daily) was too stale. Competitors (Metronome, Orb) already offered this. The "surprise bill" problem is solved only with real-time data, and AI companies (our target market) expect sub-minute freshness for their customer portals.
How do you balance accuracy vs. freshness?
The RTR eviction timer is the key lever. Currently 1-2 min (fast but may miss late events), planned 10-15 min (more accurate for bursty patterns). The insight: dashboard SLA is 1 hour, so we have headroom to prioritize accuracy without breaking user expectations. For the customer portal (P0 = ≤2 min), we'd keep the shorter window. This is a per-surface tradeoff, not a system-wide one.
What went wrong with Perplexity (ir-canny-tradition)?
Three-track failure in 7 days: (1) correctness — credits zeroed by Blueprint v2 bug, (2) freshness — 5-day blackout during planned rollout, (3) completeness — tiered pricing silently dropped. Root cause: treating "Private Preview" as permission to have lower quality bars. Key learning: once data is shared, merchants build on it immediately regardless of preview status. Fix: same observability as GA (freshness SLAs, correctness checkers, known-limitation guardrails), plus never create data blackout windows (parallel pipelines instead).
Why separate aggregate and rated/spend APIs?
Different latency profiles (1-3 min vs 5-10 min), different consumers (operational vs financial), different query patterns (raw counts vs dollar forecasts). Aggregate usage answers "how much did I use?" (operations teams). Spend answers "how much will I owe?" (finance, customer success). Combining them would force the slower freshness on the faster use case, or expose confusing schema mixing raw and monetary values.
How does the system scale for enterprise merchants?
Three key design choices: (1) Pinot for low-latency analytical queries at scale (sub-second even at millions of rows), (2) Flink for stateful stream processing with exactly-once guarantees, (3) Kafka for decoupling producers from consumers. Hybrid tables bridge real-time freshness with batch correctness. Per-merchant rate limits (100 RPS) protect the shared cluster. For Perplexity-scale (their largest customer), we introduced high-cardinality Pinot tables with tenant filters.
Why exclude metered billing from MRR?
MRR means "predictable recurring revenue" — the industry standard definition. Usage-based charges are inherently unpredictable (a customer might use 10x one month and 0.1x the next). Including them would make MRR volatile and useless for forecasting. The tradeoff: 100% metered merchants see $0 MRR. Solution: "Total Monthly Revenue" metric (MRR + rated usage) in development, with forecasting toggle (only viable for 22.5% of merchants due to volatility).
How do you ensure data quality at scale?
Four dimensions, each with continuous automated monitoring: (1) Correctness — checkers comparing dataset against upstream source of truth (invoices), (2) Freshness — SLA monitoring with PagerDuty integration (9-hour SLA), (3) Completeness — row count validation within 5% of source, detection of unexpected nulls, known-limitation guardrails, (4) Consistency — cross-surface validation between Dashboard/Sigma/API/SDP. The SigmaDataCheckSuite runs every 6 hours. Critical learning: manual Hubble queries don't scale.
What was the code yellow's impact on the product org?
Massive reallocation: 4.5 HC moved from subscription analytics to UBB/genAI. Subscription PLG, Essential Journey, and multiprocessor support all deferred 2+ quarters. The bet: AI companies represent a larger growth opportunity than incremental subscription improvements. Operating model shifted to 1-week sprints with engineers co-developing in customer Slack channels. This enabled same-day API design and rapid private preview iteration.
How do SDP/Sigma datasets differ from the API?
APIs serve real-time queries (seconds-old data, ≤200 rows, rate-limited). SDP/Sigma serve bulk data extraction (daily freshness, full history, SQL-queryable). Different consumers: APIs power live dashboards and customer portals; SDP powers data warehouses and board reports. The paid_usage dataset schema stores 15-min buckets with the full spend breakdown (gross, discounts, credits), enabling merchants to build custom analytics beyond what the API exposes.
What's the customer portal design and why does it matter?
The portal lets merchants expose usage/spend data directly to their end-customers — a key requirement from AI companies (Cursor users seeing their token usage, Perplexity users seeing query counts). Key design: merchants configure which dimensions are visible via allowed_dimensions in the portal config. This prevents data leakage (e.g., a customer shouldn't see other customers' usage) while enabling flexible self-serve analytics. Auth uses ephemeral keys with customer ownership validation.