Smart Benchmarking

1 Executive Summary

Smart Benchmarking is a feature in Stripe's Billing dashboard that enables subscription merchants to compare their performance against a peer group of similar businesses. Unlike the legacy experience where merchants self-selected cohorts based on ARR/ARPU/business model, Smart Benchmarking uses an automated nearest-neighbor peer identification system combining website/product embeddings, ARR, ARPU, and MCC codes.

This is the first time Stripe leveraged AI embeddings (from merchant websites and product descriptions) for a customer-facing analytics experience. The underlying technology is protected by a patent filing: "Adaptive Intelligent Benchmarking Framework Using Encoded Operational Profiles."

2 Problem & User Needs

Voice of the Customer

Merchants leverage benchmarks to:

1. Assess business health — peer comparison as a standard barometer
2. Analyze market trends — identifying seasonality and performance shifts
3. Report to investors — unbiased, third-party performance measurement
4. Motivate teams — celebrating performance and setting peer-based goals

Problems with the Legacy Experience (v1, launched 2024)

The v1 experience let users self-select ARR, ARPU, and Business Model (B2B/B2C). Three key gaps emerged:

Competitive Landscape

3 Product Shape & Experience

Where It Lives

Benchmarks are embedded directly into existing Billing charts on the Billing Overview page — no separate navigation required. This was a deliberate choice over the legacy approach of a separate /benchmarking tab.

What Merchants See

• Benchmark lines on charts: Median, 25th, and 75th percentile lines overlaid on existing metric charts
• Percentile ranking: Clear indication of standing (e.g., "72nd percentile")
• Peer group composition: Count of peers, ARR/ARPU distributions, inferred categories, geographic regions
• "How benchmarks work" explainer: Plain-language description building trust

Metrics Benchmarked (9 total)

Eligibility

Geographic Region Communication

• If ≥25% of merchants in the cohort are from a geographic region → show as significant
• If no regions ≥25% → tell user their cohort is "globally distributed"
• Regions: US, Canada, UK+Ireland, Southeast Asia, Central EU, Eastern Europe, Latin America, Middle East+Africa, Australia+NZ

4 Technical Architecture

Architecture Decision: Batch Compute (Selected)

Three options were evaluated:

Data Pipeline

Data Interfaces (Key Schemas)

Key Technical Properties

• Table size: O(k × M) rows where k=200 (cluster size), M=total merchants
• Refresh cadence: Daily Spark job
• Serving: Pinot datastories for low-latency dashboard queries
• Monitoring: Airflow alerting, validation pipelines for regression detection
• Data locality: India merchants excluded (data residency compliance)

5 Algorithm & Methodology

The Nearest Neighbor Approach

The algorithm maps merchants into a multi-dimensional vector space using weighted features, then finds the 200 closest neighbors for each merchant using weighted Euclidean distance.

Features Used

Three KNN Variants (Tiered Fallback)

Embedding Sources

• Website embeddings: Crawled from merchant websites, summarized by LLM, vectorized by OpenAI Ada. Updated weekly. ~50K merchants have blocked/invalid websites.
• Product embeddings: From merchant-provided product descriptions during onboarding (cdm.merchants_pii). Used as fallback for 102K merchants without valid web scrapes.
• Why not average both? Tested but didn't yield consistently better results. Common buzzwords ("AI", "platform") created false similarity.

Cohort Evaluation Methods

1. Manual Review — Primary method. Review merchant lists for industry/product similarity across 6 CAB merchants.
2. Peer Group Distance — Median weighted Euclidean distance between primary merchant and peers.
3. Metric Bands — Distance between 75th and 25th percentile (tighter = more useful).
4. LLM Summary — LLM summarizes peer group themes, evaluates match quality.
5. Restricted Inferred Categories — Limited to 100 distinct values for evaluating industry homogeneity.

Key Statistical Insights

6 Key Product Decisions

7 Launch & Adoption

Timeline

Adoption Metrics

Team

PM: Asavari Tayal • EM: Sauhard Bindal • DS: Alyssa Wisk • Eng: Rohan Nair, Sai Sankalp Arrabolu, Steven Gresh, Reed Trevelyan • ML: Zahra Iman Smullin, Apurv Gandhi, David Kwak, Akhil Lohchab, Alex Chao, Manish Kurse

8 Patent Summary

"Adaptive Intelligent Benchmarking Framework Using Encoded Operational Profiles"

Filing: ST000524-US (134563-0327)

Core Method (Claim 1)

1. Receive identification of a subject computing infrastructure
2. Retrieve network-accessible electronic pages with descriptive configuration data (web crawling)
3. Retrieve operational characteristics (usage metrics + classification identifiers)
4. Generate content embeddings using ML model from descriptive configuration data
5. Encode operational characteristics into encoded values
6. Combine encoded values with content embeddings → composite embedding
7. Apply adaptive embedding weights (dynamically adjusted by context like age/size)
8. Identify similar subset via clustering protocol (k-nearest neighbor)
9. Present UI with benchmark metrics calculated from the peer subset

Key Dependent Claims

• Claim 2: Web crawling at predefined intervals to detect content changes
• Claim 3: LLM used to summarize text before generating embeddings
• Claim 4: Dynamic weight adjustment based on infrastructure age or size
• Claim 5: K-nearest neighbor search in multi-dimensional embedding space
• Claim 6: Visualization of distribution of peer group metrics
• Claim 7: Percentile ranking display
• Claim 8: Automatic configuration adjustment recommendations

Novel Technical Contributions

Patent ↔ Product Mapping

9 Future Directions

Short-term

• Benchmarking PLG: Embed benchmarks into more surfaces (metrics digest email, downloadable reports)
• Additional metrics: Voluntary/involuntary churn, more granular filters (product, price, geography)
• UBB Metrics: Usage-Based Billing merchants don't have benchmarks yet

Medium-term

• Realtime Benchmarks: Multi-stage Pinot Query engine could enable on-the-fly computation
• Customizable Metrics: Merchant-defined SQL queries in Sigma with benchmarked p25/p50/p75 reports
• Exportable reports: PDF benchmark "report card" for investors
• Annotations: Stripe-defined and user-defined annotations on charts

Long-term Vision

• Filter peer group by attribute (e.g., "peers in US only")
• Highlight under/overperformance on homepage or via insights module
• Trigger Stripe Automations workflows when underperformance detected
• Allow merchant-defined annotations for context

10 Why KNN Over Other Algorithms

A common question is why we chose K-Nearest Neighbors (KNN) over other clustering algorithms like K-Means, DBSCAN, or hierarchical clustering. The answer comes down to a fundamental difference in what we needed.

The Core Insight: We Need Peers, Not Clusters

Why KNN Is the Right Fit

1. Personalized peer groups: Every merchant gets their OWN unique set of 200 nearest neighbors. Merchant A and Merchant B may share some peers but have different groups overall.
2. Guaranteed coverage: Every merchant gets exactly K peers (unlike DBSCAN which may leave merchants unmatched).
3. Weighted features: KNN naturally supports weighted distance functions, letting us give 10x importance to embeddings over ARR/ARPU.
4. Interpretable: "These are your 200 most similar businesses" is easy to explain to merchants. "You're in Cluster #47 of 150 clusters" is not.
5. No global assumptions: K-Means assumes spherical clusters of similar size. Our merchant distribution is highly skewed (most have low ARR, a few have very high ARR). KNN doesn't care about global distribution shape.

Two KNN Approaches Compared

Within KNN, the team evaluated two specific variants:

After manual evaluation across 6 CAB merchants, Algorithm 2 (Weighted KNN) was actually selected for the final implementation — with a 10x weight on embeddings. The team found that the weighted approach gave the best balance of industry relevance and financial similarity, while being computationally more efficient at scale.

Interview Tip

11 Embeddings: A Non-Technical Explainer

What Are Embeddings?

Imagine you could describe every business on Stripe with a single point on a map. Businesses that are similar — say, two AI SaaS companies — would be plotted close together. Businesses that are very different — say, a yoga studio and a cloud infrastructure company — would be far apart.

An embedding is that map coordinate. It's a list of numbers (in our case, 1,535 numbers) that represents "what this business is about." The magic is that these numbers are generated by an AI model that has read millions of websites and understands language, so it captures nuanced similarities that simple labels like "SaaS" or "retail" cannot.

Analogy: A GPS Coordinate for Your Business Identity

How We Create Embeddings

Here's the step-by-step pipeline:

How Embeddings Power the Clustering

Once every merchant has their embedding (their "business identity coordinate"), the KNN algorithm can calculate the distance between any two merchants. Shorter distance = more similar businesses.

The algorithm combines the embedding distance with ARR, ARPU, and MCC distances to find the 200 merchants closest to you in this combined space. The embedding gets 10x the weight of the other features, meaning industry/product similarity matters most, but financial scale still plays a role.

Why Embeddings Beat Simple Categories

The Fallback: Product Embeddings

~50K merchants have websites we can't successfully crawl (paywalls, invalid URLs, blocked scrapers). For these merchants, we use product embeddings instead — these are generated from the product descriptions merchants provide during Stripe onboarding. Same embedding process, different source text.

Real Example

12 CAB Merchants: Validation Testing

The algorithm was manually validated against 6 Customer Advisory Board (CAB) merchants spanning different industries, sizes, and business models. For each merchant, the team compared peer groups generated by different algorithm variants using the Streamlit app, manual inspection, and metric band analysis.

The 6 CAB Merchants

What We Evaluated

For each merchant, the team compared two algorithm variants side-by-side:

• Clustering 1: Pre-filter by ARR/ARPU range, then apply KNN on web embeddings + MCC
• Clustering 2: Weighted KNN using all features simultaneously (embeddings weighted 10x)

Key Findings by Merchant

Validation Metrics Summary

Key takeaway: Cluster 2 (Weighted KNN with 10x embedding weight) consistently produced peers with closer ARR/ARPU matches AND better industry homogeneity. The weighted approach was selected as the final algorithm.

How We Used the Streamlit App

The team built a Streamlit app that allowed internal users to:

• Enter any merchant ID and see their peer group
• View the peer group's metric charts (what the merchant would see in the dashboard)
• Browse the list of peer merchants with descriptions
• Compare algorithm variants side-by-side
• Evaluate whether the peers "feel right" based on manual inspection

This tool was critical for iteration — it let the DS team quickly try different weights, feature combinations, and eligibility thresholds and immediately see the impact on real merchant peer groups.

13 Legal & Privacy Guidelines

Smart Benchmarking went through extensive legal review given it uses merchant data to generate comparative insights. The Product Legal Review (PLR) was led by Molly Peck (Legal) and Devyn Cegelski (Compliance), with the overall risk assessed as medium-low.

Core Legal Principles

The feature operates under the [P&C] Best Practices for Stripe Benchmarking Features framework, which establishes:

• Aggregation requirement: All data shown must be in aggregate form. Reverse engineering of individual merchant data points must not be possible.
• Minimum cohort size: Peer distributions must consist of at least 100 data points (exception: 50 for Trial Conversion Rate specifically, due to lower feature adoption).
• Only permitted merchant data: Identify and exclude merchants with negotiated contracts prohibiting Stripe from using their data for product development.
• Data freshness buffer: Approved to use a two-week buffer (e.g., February data shown starting March 15th). This was an exception from the original one-month guidance.

Sales-Sold User Data Exception

Web Scraping Guidelines

Third-Party Licensed Data

LLM Usage & AI-Generated Content

Hallucination safeguards: The LLM is constrained to output from 100 predefined categories. As of GA, zero hallucinations detected across 343K merchants. Alerting and a killswitch are in place to detect deviations.

Cluster Composition Modal — GA Privacy Decisions

The GA redesign of the Cluster Composition Modal required legal sign-off on three elements:

Data Locality & Geographic Compliance

• India: Merchants operating out of India are excluded from Smart Benchmarking entirely (data residency requirements).
• Data locality column is pulled from precomputed metrics table and validated before inclusion.
• Gate: benchmarks_disable_smart_benchmarking_for_merchant can be used to exclude specific merchants.

Merchant Opt-Out Mechanisms

• Opt-out from being a peer: Built for large/sensitive merchants. Removes merchant from other merchants' peer groups within 24 hours.
• Data deletion upon request: 24-hour SLA for removing a merchant's data from benchmarking datasets.
• Controls aligned with upstream deletion: If a merchant deletes their Stripe account, their data is removed from benchmarking.

Key Risks Summary

! Interview Prep: Key Talking Points