Smart Benchmarking — Deep Technical Architecture

KNN-based merchant cohort clustering for Stripe Billing analytics. Surfaces peer benchmarks (25th/50th/75th percentiles) for churn metrics, enabling merchants to understand performance relative to similar businesses.

Prepared for OpenAI PM interview — demonstrates end-to-end ML product architecture, data pipeline design, and frontend integration

System Architecture — End-to-End Data PipelineFrom merchant data ingestion through ML clustering to Dashboard presentation

MERCHANT DATA SOURCES Operating Characteristics ARR (Annual Recurring Revenue) ARPU (Avg Revenue Per User) Business model, segment, age Industry Features industry_analytical_mcc Consulting, Software, Transportation... Web Embeddings 4K-dimensional vectors Crawled merchant websites Cosine similarity matching Additional Signals B2B/B2C, country, currency primary_vertical, merchant_segment Feature Flags enable_smart_benchmarking enable_billing_analytics_benchmarking_churn_page ML PIPELINE — KNN CLUSTERING K-Nearest Neighbors Algorithm Cosine similarity on 4K web embeddings + MCC industry segmentation Configurable features — adjust within hours No downtime required Output: Merchant Cohorts P25 / P50 (median) / P75 percentile values DATA PIPELINE / FLYTE Flexible pipeline architecture Easily onboard new metrics Supported Metrics Subscriber churn rate Gross MRR churn rate / Net MRR churn rate Subscriber retention cohort Revenue retention cohort FRONTEND / DASHBOARD Billing > Churn Tab Metric Widgets Benchmark overlay lines P25 / P50 / P75 percentiles Metric Detail View Time filtering + grouping Peer Group Composition Modal: explains cohort makeup "Open Report" > Sigma Deeper analysis in Sigma reports Sigma Reports Custom queries Drill-down analysis Data stories features embeddings cohorts percentiles FLYTE Key Properties Zero downtime configuration changes New metrics onboarded in hours Transparent peer group composition
ML Pipeline / KNN
Web Embeddings / Frontend
Operating Characteristics
Industry / Flyte Pipeline
Feature Flags
Animated data flow

ML / Clustering Algorithm ArchitectureFeature engineering, algorithm comparison, evaluation framework, and output

FEATURE ENGINEERING HIGH PRIORITY FEATURES industry_analytical_mcc ARR (Annual Recurring Revenue) ARPU (Avg Revenue Per User) Merchant Web Embeddings 4,096 dimensions ADDITIONAL FEATURES Business model (B2B / B2C) Merchant segment Merchant age Country / Currency primary_vertical Configurable feature set WEB CRAWLER Merchant website crawling 4K embedding extraction ALGORITHM COMPARISON K-Means Clustering Embeddings fed into K-Means Elbow method for optimal K Fixed clusters, less adaptable evaluated CHOSEN K-Nearest Neighbors (KNN) Evaluate each merchant individually Find K nearest neighbors in feature space Cosine similarity on 4K embeddings Most flexible, per-merchant cohorts No fixed cluster count needed Hierarchical Clustering Feature hierarchy approach Can merge small cohorts upward More rigid, harder to configure evaluated Why KNN Won Per-merchant flexibility No predefined cluster count Easily add/remove features Hours to reconfigure, zero downtime EVALUATION FRAMEWORK 3-Sigma Outlier Detection Distance from cluster centroid Removes anomalous merchants Homogeneity Score Are merchants in cluster same industry? Validates clustering quality Distribution Metrics Std dev of benchmarks per cohort Lower variance = better cohort LLM-Based Classification Web crawl + LLM to classify Describe cohort composition Natural language cohort labels StreamLit Validation App Internal manual validation Human-in-the-loop quality check Quality Gate Homogeneity threshold met OUTPUT — COHORT COMPOSITION & PEER TRANSPARENCY Cohort Composition Operating characteristics Geographic breakdown Industry distribution Peer Group Transparency Show composition to build trust "Why are these my peers?" Modal in Dashboard Data Stories Actual ARR / ARPU values Percentile context Actionable insights Benchmarking Metrics P25 / P50 / P75 values Per metric, per cohort Time-series overlay KNN eval
KNN (Chosen Algorithm)
Feature Engineering
Evaluation / Distribution
Alternative Algorithms
Web Crawler Pipeline
Animated data flow
CHOSEN = selected approach

Smart Benchmarking Architecture Diagrams

Sources: Internal design docs, ML pipeline specs, Flyte architecture, Dashboard implementation PRDs

Built with SVG + CSS animations. No external dependencies.