Build the retrofit. Constrain its job.
The winning company is not “another dataset vendor.” It is the neutral layer that makes heterogeneous physical runs trustworthy, comparable, trainable, simulatable, and evaluable.
Standardize the data plane first. Keep the robot’s certified controller and safety system in charge. Monetize the verified capability above the appliance.
The retrofit should connect existing robots, teleoperation devices, and external sensors; establish a common clock and coordinate system; preserve raw evidence; emit model-ready views; create calibrated simulator counterparts; and close the loop from failure to targeted recollection. Commanded teleoperation belongs in a separate, hardware-specific safety envelope—not in the universal core. [E‑007]
“Robot data” is seven different goods.
A buyer does not purchase bytes in the abstract. Different data planes teach different things, carry different scarcity, and support different business models.
Video, images, audio, maps, CAD and spatial context. Abundant generically; scarce in rights-cleared specialist environments.
Calibrated RGB-D, LiDAR, radar, thermal, IMU and geometry. Valuable when sensor-matched and synchronized.
Robot state, commands, force, tactile, slip, controller mode and latency. Premium learning signal.
Interventions, near misses, partial progress and correction. Scarce and disproportionately useful.
Assets, physics, tasks, synthetic rollouts, counterfactuals and hidden tests. Reusable infrastructure.
The full taxonomy also separates simulation ingredients from generated rollouts and evaluation evidence. This matters because simulation throughput is increasingly abundant while task authoring, contact fidelity, system identification, scenario validity, and real-world correlation remain difficult. [E‑001]
Real and synthetic data are complements
NVIDIA reported generating 780,000 synthetic trajectories—described as 6,500 human-demonstration hours—in 11 hours, and reported a 40% performance improvement from mixing synthetic and real data relative to real-only training. The correct conclusion is not that real data disappears: scarce demonstrations, target distributions, system identification, and real holdouts determine whether synthetic scale is useful. [S‑01]
The minimum valuable episode is causal
A useful episode contains task intent and success predicates; embodiment, firmware and controller state; calibrated observations; native robot state and requested/executed actions; outcome and intervention; clock and transform uncertainty; provenance, rights and lineage. LeRobot and RLDS provide important interchange structures, but the commercial evidence contract must also preserve calibration, contact, action semantics, privacy, and evaluation exclusion. [S‑02] [S‑03]
Five budgets. No single winner.
The buyer’s shortlist changes with the job: outsource collection, buy a rig, manage logs, simulate worlds, or keep building internally. The company must position against each substitution set.
Tier 1A — managed physical-data programs
| Product | Public surface | Strongest advantage | Opening for the retrofit company | Threat |
|---|---|---|---|---|
| Scale Physical AI | Global factories and collectors, Scale Harness, bimanual demonstrations, annotation, platform and internal validation | Operating scale, enterprise trust and frontier-lab relationships; claims 1,000+ demonstration hours daily | Customer-owned appliance, installed mixed-fleet compatibility, simulator neutrality and independent evaluation | Very high |
| Encord Physical AI | Labs, in-field operators, leader/follower, UMI, multimodal ingestion, curation, annotation and deployment feedback | Strong end-to-end data platform and enterprise deployment options | Deeper controller normalization, signed calibration, cross-simulator delivery and pervasive hardware appliance | Very high |
| Roborax | Teleop, demonstrations, sensor capture, annotation, simulation, evaluation and several operating models | BPO-backed labor and compliance breadth; claims 2.4M+ trajectories | Public product proof and customer outcomes require diligence; win on technical evidence and installed-base depth | High |
| Scaled Motion | Contact-rich, deformable and mobile-bimanual fine-tuning packs with custom capture kits | Correct high-value task wedge | Execution maturity and platform breadth need validation | Medium |
| Defined.ai | Marketplace plus custom multimodal physical-AI collection | Existing data operations, audio assets and enterprise sourcing | Less visible controller-level action and real↔sim specialization | Medium |
| Specialists Tactum · Paddy · Operant | Focused site, vertical, task or regional programs | Speed, specialization and price | Early public proof; potential partners, acquisition targets or wedge competitors | Watch |
Tier 1B — capture and teleoperation
Hardware is getting cheap
- Trossen Mobile AI packages a bimanual leader/follower station, synchronized RGB-D, MCAP and LeRobot/HDF5 outputs.
- TRumi commercializes a portable UMI-style workflow, publicly listing dual kits below $2,200 with cameras.
- ALOHA 2, UMI and GELLO provide open collection designs and strong research baselines.
- Isaac Teleop provides a unified device and retargeting framework across ROS 2, Isaac Sim and Isaac Lab.
- LeRobot connects hardware, datasets, policies, simulation and community distribution.
A recorder is not a moat.
The defensible chain is sensor identity → clock → calibration → controller state → action semantics → outcome → training transform → twin → hidden result.
Tier 2A — data and robot operations
| Product | Owns today | Recommended posture |
|---|---|---|
| Foxglove + MCAP | Open raw container, visualization, ingest, search, governance and evaluation | Embrace MCAP; integrate Foxglove rather than inventing a container/viewer |
| Roboto | Cross-format log ingestion, automated analysis, anomalies, root cause, curation and fleet reliability | Potential analytics partner; differentiate in acquisition, calibration, simulation and capability delivery |
| Rerun | Open visualization plus query, transformation, catalog and lineage | Use as an optional inspection surface |
| Formant / InOrbit | Fleet operations, telemetry, teleoperation, missions, incidents and orchestration | Coexist: they optimize uptime; the foundry certifies learning evidence |
| Viam | Hardware abstraction, modules, connectivity, capture, sync, query, ML and deployment | Closest horizontal edge threat; stay narrow and deeper on action/contact/evaluation integrity |
Tier 2B — simulation and validation
| Platform | Core advantage | Strategic response |
|---|---|---|
| NVIDIA Isaac / Cosmos / GR00T / Data Factory | Broadest integrated real/sim teleop, OpenUSD worlds, physics, synthetic generation, training and evaluation | First-class target, never the only target; own field execution, calibration and neutral evaluation |
| Applied Intuition | Enterprise autonomy data, simulation, reconstruction and V&V | Avoid vehicle/autonomy head-on; focus contact-rich manipulation and installed robot cells |
| Parallel Domain | Photoreal sensor simulation and real-scene replicas | Complement for perception; differentiate in manipulation dynamics and action data |
| Foretellix | Coverage-driven scenario verification for safety-critical autonomy | Potential evaluation partner and methodological model |
| MuJoCo Playground / ManiSkill3 / RoboCasa | Open, fast physics, training, tasks, demonstrations and benchmarks | Use them; sell customer-specific task authoring, system identification and real correlation |
The open seam is evidence continuity.
Managed providers operate programs. Hardware vendors provide rigs. Data platforms organize logs. Simulators create worlds. The company should own the integrity contract across all four.
Build these assets
- Robot/controller/firmware compatibility graph
- Longitudinal clock and calibration ledger
- Native-preserving action and task ontology
- Validated training-view transformation recipes
- Real↔sim dynamics, sensor and latency residuals
- Coverage and data-influence models
- Leakage-controlled private evaluation network
- Task-specific collection operating curve
Do not confuse these for moat
- Generic first-person video hours
- Camera and edge-compute resale
- A ROS bag uploader or new raw format
- Labor arbitrage without technical QA
- Public assets without real calibration
- Normalized actions that erase native dynamics
- A marketplace before rights and transfer value are proven
The data-rights posture should be conservative: the customer owns its raw and task-derived data; the company retains adapter code, generic schemas, de-identified calibration statistics, aggregate quality patterns, generic residual models, and benchmark methods. Cross-customer trajectory reuse should be explicit opt-in. This creates trust without giving away the compounding technical system. [E‑009]
The Physical Data Plane
A rugged edge appliance is the distribution mechanism. The actual product is a certified chain from a physical run to a model result.
FIG. 02 — The product boundary. The universal core owns evidence integrity; hardware-specific control remains behind an entitled safety gateway.
Capture broadly. Control selectively.
The safe product expansion is a ladder. Each rung adds value—and a different class of engineering, safety and liability.
Accessible interfaces, rich manipulation data and strong overlap between learning labs and industry.
Research, medical/assistive relevance, bimanual collection and packaged leader/follower workflows.
Humanoid, legged, mobile and aerial expansion through ROS 2/DDS-compatible surfaces.
Industrial installed base, but controller generations, options, integrators and safety cells demand certification.
| Family | Official interface evidence | Capture | Teleop | Recommended move |
|---|---|---|---|---|
| Universal Robots | RTDE + official ROS 2 | High | High | P0 production reference |
| Franka FR3 / Panda | FCI/libfranka, 1 kHz state and control, ROS integrations | High | High with real-time constraints | P0 calibration reference |
| UFactory xArm | Python/C++ SDK and ROS/ROS 2 packages | High | High | P0 affordable dual-arm/vertical rig |
| Kinova Gen3 | official Kortex ROS 2 | High | High | P0/P1 medical and research |
| Unitree | SDK2 + ROS 2/DDS across several families | High | Medium–high | P1 humanoid reference |
| Clearpath | unified official ROS 2 API | High | High for guarded base velocity | P1 mobile reference |
| FANUC | official ROS 2 driver, high-rate control and I/O | High on supported stack | Medium | P2 with integrator/OEM |
| ABB / KUKA / peers | ABB RWS, ROS-Industrial/vendor-specific options | Medium–high | Controller-specific | Paid-demand certification |
| Closed humanoids | Partnership-oriented access | Low without OEM | Partnership only | Do not promise aftermarket control |
Interface availability is not permission to bypass safety. Real deployments must preserve vendor controllers, safety PLCs, interlocks, e-stops, watchdogs, workspace and force limits, and site risk assessment. [E‑008]
Sell a TaskPack, not a terabyte.
The commercial unit should describe a measurable capability. Recording hours and trajectories remain cost metrics inside the factory.
90-day Capability Retrofit Pilot
- Retrofit two customer robots and one teleoperation modality.
- Instrument one commercially important contact-rich task.
- Collect accepted successes, failures and recoveries.
- Deliver a calibrated simulator task and synthetic expansion.
- Train or support a baseline and run frozen evaluations.
- Report lift, accepted yield, coverage, cost and remaining failure slices.
Positioning
For physical-AI teams that already own robots but cannot turn heterogeneous runs into reliable capability fast enough, the company is the hardware-neutral physical data plane and capability foundry. It retrofits the installed fleet, certifies every observation-action episode, expands scarce interactions in calibrated simulation, and proves model improvement on hidden tests—without replacing the customer’s robots, models, cloud or simulator.
When the company wins
- The customer has several robots or embodiments and repeats integration work.
- Temporal integrity, calibration, contact and outcome semantics affect policy quality.
- Simulation exists but is not demonstrably correlated with the real task.
- Failures and interventions are abundant but not converted into training and evaluation slices.
- Neutrality, customer data ownership and private evaluation matter.
When it loses
- A small lab has one stable robot and can operate an open Trossen/LeRobot workflow itself.
- The buyer primarily needs commodity annotation or thousands of operators immediately.
- The only pain is log visualization, fleet uptime or orchestration.
- The customer is fully standardized on a single vertical stack and already has excellent physical-data operations.
Battlecard shorthand
Against Scale or Encord
Do not claim superior raw scale. Position the appliance as the customer-owned evidence standard across internal and outsourced sources. Win on hardware depth, signed calibration, simulator neutrality and private capability evaluation. Lose when procurement wants one established managed-service vendor.
Against NVIDIA
Isaac is a first-class target, not the product boundary. Win through field operations, non-NVIDIA support, rights-cleared site access, real hardware calibration and independent evaluation. Lose when the buyer is fully standardized on NVIDIA and staffs the last mile internally.
Against Trossen, ALOHA, UMI or LeRobot
Keep them. The company connects those sources to every other robot, applies one evidence contract, and handles enterprise operations, transformations, twins and tests. Lose when one open rig is all the customer needs.
Against Foxglove, Roboto, Rerun, Formant or Viam
They help teams see, search, operate or abstract robots. The foundry makes episodes trainable and links them to simulator and model outcomes. Integrate rather than replace unless a platform expands directly into certified acquisition and evaluation.
Against internal build
Never say “you cannot build it.” Say: keep your models and control stack; stop spending senior roboticist time repeating clock, calibration, schema, conversion and evaluation infrastructure for every robot and site.
Twelve months to prove compounding.
The objective is not broad compatibility on paper. It is repeated deployment with falling marginal integration cost and demonstrated model lift.
Make one physical run trustworthy
Canonical episode and calibration schema; ROS 2/DDS + MCAP edge recorder; LeRobot and RLDS views; UR, Franka and xArm L1 adapters; automated skew, frame, kinematic and outcome QA; one Isaac and one MuJoCo task from the same setup.
Prove the abstraction survives new hardware
Add Kinova and Trossen/ALOHA, leader/follower, UMI/TRumi and XR ingestion; make calibration technician-operable; establish task/failure ontology; close three paid Capability Retrofit Pilots.
Operate a small heterogeneous fleet
Add Unitree and Clearpath; production capture-node health; signed session certificates; customer VPC/on-prem deployment; Foxglove/Rerun/Roboto connectors; real↔sim residual dashboard.
Certify through partners
FANUC and ABB L1 reference adapters with integrators; guarded L2 teleoperation on only two validated arm families; third-party adapter test suite; private evaluation renewal; open a narrow Physical Episode Conformance specification.
Operating metrics
| Metric | What it proves |
|---|---|
| Time to first accepted episode on a new robot | Adapter and deployment leverage |
| Accepted episodes per staffed hour | Hardware, reset, operator and QA productivity |
| Certified sessions / total sessions | Protection from silent clock and calibration defects |
| Cost per accepted novel slice | Coverage value rather than repetition |
| Failure-to-targeted-data cycle time | Closed-loop responsiveness |
| Real↔sim scenario rank correlation | Whether simulation guides selection and evaluation |
| Hidden real-test lift | Whether the TaskPack bought capability |
| Adapter reuse ratio | Whether the services business is becoming a platform |
Design the company to be killable.
The retrofit thesis is attractive, but it can collapse into bespoke services, unsafe control work, or an appliance customers refuse to install. Six design partners should produce decisive evidence.
Main risks and mitigations
| Risk | Mitigation |
|---|---|
| NVIDIA absorbs the workflow | Own the real-world last mile, multi-simulator neutrality, calibration and independent tests |
| Every adapter is bespoke | Conformance levels, adapter SDK, certification harness, explicit support matrix and paid exceptions |
| Control liability dominates | L1 read-only default; isolate L2; preserve vendor safety; use certified integrators |
| Normalization destroys signal | Immutable native streams plus derived views with transformation loss and uncertainty |
| Customers will not share data | Customer ownership; build moat from adapters, aggregate quality metadata and evaluation methods |
| Synthetic data does not transfer | Real seed, paired calibration trials, real holdouts and correlation thresholds for every program |
| Collection becomes low-margin BPO | Price accepted capability evidence; automate QA/resets; partner for labor |
Six-part kill test
- Median new-arm L1 integration still takes more than four engineer-weeks after three reference adapters.
- Fewer than three of six partners will install the node on more than five robots.
- Customers value annotation and search but will not pay for calibration and outcome instrumentation.
- The system cannot find quality defects that materially affect policy performance.
- Real↔sim calibration fails to improve scenario ranking or transfer on two task families.
- Security and procurement consistently reject the appliance while accepting a pure software agent.
If the appliance fails but the evidence layer wins, pivot to an adapter SDK and managed calibration/evaluation service running on customer-owned edge hardware.
Sources, confidence and limits
This publication synthesizes two local research reports totaling 17,958 words and 114 unique public links. The frozen source snapshot records checksums, evidence IDs, privacy and the publication contract. Company scale, performance and customer statements are attributed to vendors, not treated as independently audited. Strategic ratings and all prices are inferences or hypotheses.
- E‑001 / E‑002 — State of Physical-AI Data. Local source:
PHYSICAL_AI_DATA_LANDSCAPE.md, SHA-2565dc791449135edf641455f4022e2d66c9aeb097894fdd0571003e25a5f4fcb0f. - E‑003–E‑010 — Competitive Products and Retrofit Strategy. Local source:
PHYSICAL_AI_COMPETITIVE_PRODUCTS_AND_RETROFIT_STRATEGY.md, SHA-256e059b33b8ef72f91268d2410755c001cddccc3f7aaf52f80759380b1f4e8b3f2. - S‑01 — Synthetic motion generation. NVIDIA, Building a Synthetic Motion Generation Pipeline for Humanoid Robot Learning.
- S‑02 — Training and data ecosystem. Hugging Face, LeRobot documentation.
- S‑03 — Episodic sequential-data structure. Google Research, RLDS ecosystem.
- S‑04 — Integrated data-factory threat. NVIDIA, Physical AI Data Factory Blueprint.
- Direct data providers. Scale, Encord, Roborax, Scaled Motion, Defined.ai, Tactum Labs, Paddy.
- Capture and teleoperation. Trossen Mobile AI, TRumi, ALOHA 2, UMI, GELLO, Isaac Teleop.
- Robot-data infrastructure. Foxglove, Roboto, Rerun, Formant, InOrbit, Viam, Sift.
- Simulation and validation. Isaac Sim, Applied Intuition, Parallel Domain, Foretellix, Bifrost, MuJoCo Playground, ManiSkill3, RoboCasa.