Building a Data Layer for AI Personalization on Robots

Robots that understand people need more than sensors and actuators. They need a memory architecture — a structured data layer that captures who you are, what you prefer, and how you change over time. This document lays out the design.

1. Why Robots Need a Personalization Data Layer

The robotics stack has historically been organized around two problems: perception (what is in the world?) and control (how do I move through it?). Decades of research have produced increasingly capable solutions for both. A modern mobile manipulator can navigate a cluttered kitchen, identify objects on a counter, and execute grasps with sub-centimeter precision.

And yet, most robots deployed in homes and workplaces today are, frankly, not very useful. Not because they can't see or move — but because they don't know you.

This data layer — what we might call the personalization substrate — sits between the foundation-model reasoning layer and the physical execution layer. It provides the memory, preferences, and contextual knowledge that make a robot's actions feel intelligent rather than merely competent.

What "personalization" actually means for robots

Personalization on robots is not a recommendation engine. It spans at least four dimensions:

None of these fit neatly into a SLAM map or an object detector. They require a dedicated data architecture — one designed for the unique constraints of physical agents operating in human environments.

2. Architecture: What the Data Layer Looks Like

The personalization data layer is not a single database. It is a composition of stores, each optimized for a different kind of knowledge, connected through a unified query interface that the robot's reasoning layer can access in real time.

Core data stores

The data layer decomposes into five primary stores:

The unified query interface

The reasoning layer should not need to know which store holds what. Instead, it queries a personalization context API that assembles a coherent context object for any given situation:

// Pseudocode: what the reasoning layer sees
const ctx = await personalization.getContext({
  user: "alice",
  location: "kitchen",
  time: now(),
  task: "prepare_morning_beverage",
  history_window: "7d"
});

// ctx contains:
// - alice.preferences.beverages → { weekday: "oat latte", weekend: "cortado" }
// - alice.behavior.morning_tolerance → "low" (don't be chatty)
// - kitchen.objects.mug_locations → [{ name: "alice_favorite", shelf: 2, slot: 3 }]
// - kitchen.recent_changes → [{ item: "oat_milk", status: "low", detected: "2h ago" }]
// - interaction_history.beverage_requests → last 30 days of what was asked/served

This context object is the primary input to the LLM or policy network that decides what to do. The personalization data layer's job is to make assembling this context fast (sub-100ms), consistent (no stale preference contradicting a recent correction), and privacy-preserving (only surface data the current task is authorized to access).

3. Key Design Decisions

On-device vs. cloud

This is the defining architectural tension. On-device keeps data private and latency low, but limits storage, compute, and fleet-wide learning. Cloud enables richer models and cross-device continuity, but introduces latency and trust concerns.

Privacy by design

Personalization data is inherently sensitive. The architecture must encode privacy as a structural property, not a policy overlay:

• Data minimization: Store derived signals (preference vectors, behavioral summaries) rather than raw sensor feeds. A robot doesn't need to keep video of you making breakfast — it needs to know you make breakfast at 7:15 AM.
• Tiered retention: Raw interaction logs age out aggressively (7–30 days). Derived preferences persist longer. Explicit user settings persist indefinitely.
• Per-user encryption at rest: Each user's personalization data encrypted with a key derived from their identity. Household-shared data uses a shared key. Robot manufacturer cannot read personalization data without user consent.
• Access scoping: Each task execution gets a scoped view. The "make coffee" task can read beverage preferences but not health data. Enforced at the query interface level, not application logic.
• Audit log: Every data access is logged. Users can inspect what the robot knows about them and delete any subset.

Real-time inference vs. batch learning

The data layer supports two temporal modes, and getting the boundary right matters enormously:

• Fast loop (real-time): Immediate corrections ("no, the blue mug") update a preference cache in-place. Uses simple rules or lightweight Bayesian updates. Must complete in <50ms to not break interaction flow.
• Slow loop (batch): Periodically (nightly, or on-device idle), re-trains preference models from the full interaction history. Discovers longer-term patterns: seasonal shifts, evolving routines, preference drift. Produces updated embeddings and model weights that replace the previous versions.

The fast loop is authoritative for recent corrections. The slow loop provides the baseline. Conflicts are resolved with a recency-weighted merge — explicit recent corrections always win over statistical patterns.

Schema design for heterogeneous form factors

A home assistant, a warehouse logistics bot, and a healthcare companion share no physical morphology. But they can share a personalization schema — because personalization is about the human, not the robot.

// Core personalization schema (form-factor agnostic)
PersonalizationRecord {
  user_id:          string
  household_id:     string
  
  // Explicit preferences (user-set)
  explicit_prefs:   Map<domain, Map<key, value>>
  
  // Learned preferences (model-derived)
  learned_prefs:    Map<domain, PreferenceVector>
  
  // Behavioral calibration
  interaction_style: {
    proactivity:    float   // 0 = only when asked, 1 = anticipate needs
    verbosity:      float   // 0 = silent, 1 = chatty
    pace:           float   // 0 = deliberate, 1 = fast
    personal_space: float   // meters — physical proximity comfort
  }
  
  // Context graph edges (portable across robots)
  routines:         Routine[]
  relationships:    Relationship[]    // to other users, pets, objects
  
  // Form-factor-specific extensions
  extensions:       Map<form_factor, any>
}

The key insight: the core schema is about the person and their context, with form-factor-specific capabilities layered as extensions. When a household adds a new robot, it inherits the family's personalization data immediately. Day-one personalization without a cold-start problem.

4. The AI Stack: Foundation Models Meet Personalization

Foundation models — LLMs, vision-language models, multimodal transformers — are increasingly the reasoning backbone of robot systems. The personalization data layer connects to these models in three primary ways:

Embeddings for user intent

Every user interaction is embedded into a shared vector space — not just the words, but the intent in context. "Make it warmer" means different things when said to a thermostat robot vs. a beverage-making robot vs. a robot adjusting a blanket.

The personalization layer maintains per-user intent embeddings that capture how this specific user uses language. Over time, the system learns that when Alice says "the usual," she means a specific sequence of actions that's different from what Bob means.

Retrieval-augmented personalization (RAP)

Just as RAG (retrieval-augmented generation) grounds LLMs in external documents, RAP grounds robot reasoning in personal context. Before the LLM plans an action:

1. The query is embedded and used to retrieve the top-k relevant personalization records from the vector store.
2. Retrieved context is injected into the LLM prompt alongside the task specification.
3. The LLM generates a personalized plan that accounts for the user's specific preferences and history.

// RAP prompt assembly
const relevantContext = await vectorStore.query({
  embedding: embed(userRequest),
  filters: { user_id: "alice", recency: "90d" },
  top_k: 10
});

const plan = await llm.generate({
  system: ROBOT_SYSTEM_PROMPT,
  context: [
    { role: "personalization", content: formatContext(relevantContext) },
    { role: "environment", content: currentSceneDescription },
    { role: "user", content: userRequest }
  ]
});

This pattern is powerful because it separates the personalization data from the reasoning model. You can upgrade the LLM without losing personalization. You can transfer personalization to a new robot without retraining. The data layer is the durable asset; the model is a replaceable reasoning engine.

Few-shot adaptation

For novel preferences that don't have enough history for statistical learning, the system uses few-shot adaptation: a small number of examples (often just one explicit correction) are stored and used as in-context demonstrations for the LLM.

Multimodal grounding

Personalization doesn't only come from language. The data layer ingests signals from multiple modalities:

• Visual: Facial expression → mood estimation → behavioral adjustment (gentler interaction when user appears stressed)
• Spatial: Where the user spends time, movement patterns → routine inference
• Haptic: Force feedback during handovers → grip preference calibration
• Ambient: Time of day, lighting, temperature, noise level → contextual defaults

All signals are fused into the context graph as weighted edges, creating a rich, multimodal representation of the user's current state and long-term patterns.

5. Concrete Examples Across Domains

Home assistant robots

Warehouse & logistics robots

Healthcare companion robots

6. Infrastructure Patterns

Event sourcing for robot interactions

Every interaction is an event. Rather than maintaining mutable state ("Alice's mug preference is green"), the data layer stores the full event stream ("Alice corrected mug choice to green at timestamp T"). This pattern provides:

• Complete audit trail: Every preference can be traced to the interaction that established it.
• Temporal queries: "What did Alice prefer three months ago?" is a first-class query, not a reconstruction.
• Replayability: If preference models are updated, they can be retrained from the full event history.
• Conflict resolution: When a real-time correction conflicts with a batch-learned preference, the event log provides the ground truth.

// Interaction event schema
InteractionEvent {
  event_id:       uuid
  timestamp:      datetime
  user_id:        string
  robot_id:       string
  session_id:     string
  
  event_type:     enum {
    COMMAND, CORRECTION, FEEDBACK_EXPLICIT,
    FEEDBACK_IMPLICIT, OBSERVATION, SYSTEM
  }
  
  domain:         string        // "beverage", "navigation", "social", ...
  payload:        json          // event-type-specific data
  
  context_snapshot: {
    location:     string
    time_of_day:  string
    people_present: string[]
    ambient:      json
  }
  
  // Derived at write time
  embeddings: {
    intent:       float[768]    // semantic embedding of the interaction
    context:      float[384]    // compressed context embedding
  }
}

Vector stores for semantic memory

The interaction history, once embedded, lives in a vector store that supports semantic retrieval — "find interactions similar to this situation" rather than "find interactions matching these exact fields."

This powers several key capabilities:

The vector store runs on-device with a compact index (HNSW with product quantization, typically under 500MB for a year of household interactions). Cloud sync pushes anonymized embeddings (not raw events) for fleet-wide model improvement.

Federated learning for fleet-wide personalization

The hardest infrastructure problem: how do you improve personalization across a fleet of thousands of robots without centralizing raw user data?

The federated approach works in three stages:

1. Local training: Each robot trains (or fine-tunes) a personalization model on its local data. This happens during idle periods — overnight, when the house is empty, during charging.
2. Gradient sharing: Instead of sending raw data, each robot sends model gradients (or parameter deltas) to the aggregation server. These gradients contain statistical patterns but not individual user data. Differential privacy noise is added before upload.
3. Federated averaging: The server averages gradients across the fleet, producing an improved global model that benefits from the collective experience of all robots. This model is pushed back to each robot as a better starting point for local adaptation.

Sync engine for multi-robot households

When a household has more than one robot (or when a user interacts with robots in multiple locations — home, office, car), the personalization data layer needs a conflict-free sync protocol:

• CRDTs for preference state: Last-writer-wins registers for simple preferences. Observed-remove sets for lists (e.g., favorite playlists). Counters for usage statistics.
• Causal ordering for corrections: If Alice corrects a preference on Robot A, Robot B must apply that correction before any subsequent interaction — even if B hasn't synced the full event log yet. Priority: explicit corrections > learned preferences > defaults.
• Selective sync: Not all data syncs everywhere. The kitchen robot's detailed spatial memory of cabinet contents doesn't need to sync to the car. Sync is scoped by domain and relevance.

7. Open Problems & Research Directions

This architecture is a starting point. Several hard problems remain unsolved or under-explored:

Benchmarks we need

The field lacks standard benchmarks for evaluating personalization quality in embodied agents. We need:

• PersonaBench: Standardized user profiles with known preferences, routines, and correction patterns. Evaluate how quickly a system learns and how accurately it personalizes.
• DriftSim: Simulated preference evolution over time — seasonal changes, life events, gradual shifts. Evaluate adaptation and staleness detection.
• MultiUserSim: Household simulation with conflicting preferences, shared resources, and varying interaction frequencies. Evaluate fairness and negotiation.
• PrivacyAudit: Automated tools that verify a personalization system doesn't leak data across user boundaries, across permission scopes, or to external services.

8. The Agent Infrastructure Thesis

Here's the bigger picture: robots are physical agents. And the personalization data layer they need is structurally identical to what software agents need.

Consider what a good AI coding agent maintains about its user: coding style preferences, project context, past decisions, preferred tools, communication style, review feedback history. Now consider what a home robot maintains: interaction style preferences, environmental context, past decisions, preferred routines, communication calibration, correction history.

These are the same primitives.

The implication is that the personalization data layer should not be built as a robotics-specific system. It should be built as a general agent personalization infrastructure that physical agents consume alongside software agents.

This convergence is already happening in practice. Systems like LangMem, Mem0, and Zep provide memory and personalization layers for LLM-based agents. The missing piece is extending these to handle the additional modalities and constraints of embodied agents: spatial data, real-time sensor streams, physical safety constraints, and multi-user environments.

What a unified agent personalization platform looks like

The company that builds this unified layer — agent personalization infrastructure that works for both software and physical agents — is building one of the most important pieces of the AI stack. Every agent, whether it lives in a terminal or a robot body, needs to know its user. The data layer that enables that knowledge is the foundation everything else is built on.