Problem Set

(a) The conditional velocity

The path is linear in τ for a fixed pair (x₀, x₁). Differentiate component-wise, treating the endpoints as constants:

This is the conditional velocity: it is a constant vector (independent of τ) pointing straight from the noise sample to the data sample. The whole probability path from N(0,I) to the data is built by averaging these straight lines over all pairs — the field a network must learn is the expectation of this constant over the pairs that pass through a given (x_τ, τ).

(b) The Conditional Flow Matching loss

We regress a network v_θ onto that target. Sample τ uniformly, noise x₀, a data action x₁ with its observation o, form x_τ, and minimise the squared error:

The key fact (Lipman et al., 2023) is that this conditional objective has the same gradient as the intractable marginal flow-matching objective. Minimising the per-pair regression therefore makes v_θ learn the marginal velocity field that transports the noise distribution onto the data distribution. We never need the marginal velocity in closed form — the magic is that regressing the simple constant target is enough.

(c) Sampling: the probability-flow ODE and an Euler step

At inference, start from noise and integrate the learned field forward in τ:

Discretise with N uniform steps of size Δτ = 1/N (explicit Euler):

After N steps, x_N ≈ a sample from p(action | o). Because the conditional paths are straight lines, the marginal field is unusually well-behaved, so a handful of Euler steps (often N = 5–10) already gives clean samples — that is the speed advantage flow matching has over many-step diffusion.

(d) Why SmolVLA / π0 train exactly this

• The action head must output a distribution over action chunks, not a point — robot demonstrations are multimodal (see Problem 2). Flow matching is a mode-covering generative objective.
• The training target is a plain MSE regression onto (x₁−x₀): stable, no adversarial game, no discrete codebook, plugs straight onto a transformer's hidden state conditioned on the VLM tokens (o = vision + language + proprioception).
• Sampling is a cheap ODE with very few function evaluations, which matters for a real-time control loop on the LeKiwi — fewer NFEs than DDPM for comparable quality.

∎

(a) The concrete two-mode case

Let the predictor output a scalar c. The expected squared error is

Differentiate and set to zero:

Since J″(c) = 2 > 0, c⋆ = 0 is the unique minimum. The optimal point prediction is 0 = 𝔼[a|s] — a value the expert never produced. If +a and −a meant "veer left" / "veer right" around the obstacle, the averaged action drives straight into it.

(b) The general theorem

For any state s, minimise the expected squared error over predictors f. Add and subtract the conditional mean μ(s) := 𝔼[a|s]:

Condition the cross term on s. Given s, the factor μ(s) − f(s) is constant, and 𝔼[a − μ(s) | s] = μ(s) − μ(s) = 0, so by the tower rule the cross term vanishes:

So the objective splits into an irreducible noise term plus a non-negative term that is zeroed only by f = μ:

The least-squares-optimal function is always the conditional mean — full stop. Part (a) is just this theorem evaluated on the symmetric two-point distribution.

(c) Why this dooms unimodal BC, and the fix

• A network trained with MSE (the default for naive behavioral cloning) is provably driven to the conditional mean. When the demonstrations are multimodal — and human teleop almost always is — the mean lands in a low-probability or invalid region.
• Switching to a Gaussian head with learned variance does not save you: a single Gaussian still has one mode and just inflates variance, smearing probability across the bad gap.
• The cure is a head that can represent a multimodal conditional distribution and from which we sample rather than average: CVAE (ACT, Problem 5), diffusion (Problem 6), or flow matching (Problem 1). Each commits to one mode per sample instead of averaging across modes.

∎

(a) Deriving the bound

Introduce a latent style variable z with prior p(z)=N(0,I) and an encoder (recognition model) q(z|a,s). Start from the log-evidence and insert q:

Apply Jensen's inequality (log is concave) to pull the expectation out:

The gap between LHS and RHS is exactly KL(q(z|a,s) ‖ p(z|a,s)) ≥ 0, which is why the inequality holds and is tight when the encoder matches the true posterior.

(b) Reading the two terms + reparameterization

• Reconstruction 𝔼_q[log p(a|z,s)]: decode the action chunk from the sampled style z and the state; under a Gaussian decoder this is the familiar L1/L2 reconstruction loss on the chunk.
• Regularization KL(q(z|a,s)‖p(z)): keeps the encoder's posterior near the standard-normal prior so that, at test time, sampling z ∼ N(0,I) yields coherent chunks. ACT typically down-weights this term with a small β.
• Reparameterization trick: to backprop through the sampling z ∼ q, write z = μ_φ(a,s) + σ_φ(a,s) ⊙ ε with ε ∼ N(0,I). The randomness moves into ε, so gradients flow through μ_φ, σ_φ deterministically.

(c) Chunking shortens the effective horizon

The decoder emits a whole chunk of k actions per latent draw. One policy decision now advances the controller k steps, cutting the number of autoregressive decision points over a horizon T from T to T/k — which directly tempers compounding error (made quantitative in Problem 9). The CVAE handles multimodality; chunking handles horizon. At inference you can discard the encoder entirely and just sample z ∼ p(z).

∎

(a) From the closed-form forward process to the loss

The forward marginal lets us jump to any noise level t in one shot using the reparameterization

The principled objective is the variational bound on log p(a₀|o), which for Gaussian transitions reduces to predicting the posterior mean of each reverse step. Ho et al. (2020) show that, after dropping t-dependent weights, this is equivalent to the simple denoising-score / noise-prediction regression: have the network guess the noise ε that was added.

Because ε determines a₀ given a_t, predicting the noise is equivalent (up to scaling) to predicting the clean action or the score ∇_at log q(a_t). Conditioning on the observation o makes it a conditional generative head for actions.

(b) One reverse denoising step

Given a_t, predict the noise, take a step toward less noise, and inject a little fresh noise (DDPM ancestral sampling):

Iterating from t = T down to 0 walks pure noise back to a clean action chunk. (A DDIM variant drops the stochastic term for a deterministic ODE-like trajectory.)

(c) NFE / cost vs flow matching

Aspect	Diffusion Policy (DDPM)	Flow matching (P1)
Target	noise ε per step	velocity x1−x0
Path geometry	curved (variance schedule)	straight lines
NFE at inference	tens–hundreds (DDIM cuts to ~10–50)	often 5–10 Euler steps
Real-time fit	heavier per action	lighter — why SmolVLA/π0 chose it

Both are mode-covering generative heads that fix the MSE collapse of Problem 2. Flow matching's straight conditional paths make the marginal field easier to integrate, so it usually hits the same sample quality in far fewer function evaluations — decisive for the creature's control loop.

∎

Decision points

If the policy is queried every step, a horizon T contains T independent chances to step off the expert distribution. Predict a chunk of k actions per query and execute the whole chunk open-loop, and the policy is only consulted

Why it tempers compounding error

Connect to Problem 4: BC's worst-case cost scales like the per-decision error rate ε times the square of the number of decision points. Re-derived over the chunked horizon, the relevant count drops from T to T/k, so the compounding penalty shrinks roughly by a factor of k² in the worst case — fewer junctions at which a single mistake can snowball into off-distribution states. Empirically (ACT) this is a big part of why chunking works.

The trade-offs

• Smoothness vs reactivity: long open-loop chunks are smooth but blind — the robot can't react to a disturbance mid-chunk. Short chunks are reactive but reintroduce decision points (and jitter at chunk boundaries). Temporal ensembling / action overlap blends chunk boundaries to keep motion smooth.
• Latency: a fresh chunk is computed only every k control ticks, amortising the (expensive) policy forward pass — relevant to the fast/slow split in Problem 10.

∎

Setup

Suppose BC, evaluated on the expert's own state distribution, makes a mistake (an action the expert would not take) with probability at most ε per step. Assume a per-step cost in [0,1] and that one mistake can cost up to 1 per remaining step.

The quadratic blow-up

The trouble is distribution shift: the moment the learner errs, it visits states the expert never demonstrated, where the error rate is no longer bounded by ε. Bound it by the step at which the first mistake happens. The probability the first mistake lands at step t is ≈ ε, and from there up to T − t further steps can each incur cost ~1:

(Ross & Bagnell, 2010.) The expert, by contrast, stays on its own distribution, so its cost over the horizon is just O(εT) — error accrues linearly, not quadratically. The extra factor of T is the price of compounding: errors push you off-distribution, which raises the future error rate, which pushes you further off.

How DAgger restores ~O(εT)

• Roll out the learner, collecting the states it actually visits (including off-distribution ones).
• Query the expert for the correct action on those states, and aggregate this into the dataset (DAgger = Dataset Aggregation).
• Retrain and repeat. Training and test distributions now coincide, so the ε bound holds on the states actually encountered and the cost collapses back to the near-linear O(εT) regime (no-regret online-learning guarantee).

∎

Grounding "red cup" → actions

A VLA is a conditional policy p(action | image, instruction, proprio). The token sequence "pick up the red cup" enters the same transformer as the image patches and joint states. Cross-attention lets the language tokens select the visual features that co-occur with the action of grasping that object in the demonstrations. Nothing in the network ever materialises a discrete symbol RED_CUP with a typed slot; "red cup" is grounded purely as the statistical hinge that, conditioned on these pixels, shifts the learned action distribution toward the trajectories that reached for the red cup during training. Grounding = a conditioning effect on the output distribution, learned end-to-end.

Why end-to-end skips perception → symbol → plan

• The classical pipeline (detect → build a symbolic world model → plan over symbols → execute) has brittle seams: each stage's errors compound, and the symbol vocabulary has to be designed up front.
• End-to-end imitation collapses all stages into one differentiable map from sensors to actions, optimised jointly. There is no explicit world model to keep consistent — the policy only needs the regularities that the task actually exploits.
• This is Brooks' "Intelligence without Representation": competent behaviour can emerge from a tight sensorimotor coupling without a central reconstructed model of the world. The VLA is the modern, learned incarnation — "the world is its own best model," now grounded in pretrained vision-language features rather than hand-built behaviours.

∎

The recurrent state-space model

A deterministic recurrent state h_t = f(h_t−1, z_t−1, a_t−1) carries history. Two heads read it: the prior p(z_t|h_t) predicts the stochastic latent from history alone; the posterior q(z_t|h_t,x_t) corrects it using the current observation x_t.

The ELBO (per step)

Same Jensen-bound structure as Problem 5: a reconstruction term that forces the latent to retain enough about the observation to predict it, minus a KL term that aligns the observation-free prior with the observation-informed posterior.

Why the KL is the whole point

• Minimising KL(q ‖ p) trains the prior to predict what the posterior would have inferred — i.e. to guess the next latent without seeing the next observation.
• Once the prior is a good stand-in for the posterior, you can drop the observation entirely and roll the dynamics in latent space: sample z_t ∼ p(z_t|h_t), update h_t+1, repeat. That is latent imagination (Dreamer) — generating long synthetic rollouts to train a policy/value with no environment steps.
• If the prior did not match the posterior, imagined trajectories would diverge from what real observations would have produced, and any policy trained on them would fail on the real robot. The KL term is precisely the contract that keeps imagination faithful.

∎

(a) Frame duration

The frame rate is the number of codec frames per second; the frame period is its reciprocal:

So Mimi emits one set of audio tokens every 80 ms, and the language model runs at this same 12.5 Hz token clock.

(b) Theoretical end-to-end latency

Two unavoidable 80 ms quanta dominate the ideal pipeline:

You must accumulate ~one frame of incoming audio before there is anything to encode, and the model needs at least one frame of compute/lookahead before it can emit the first output frame. 160 ms is the architectural floor implied purely by the 12.5 Hz clock.

(c) What adds to it in practice, and the full-duplex win

• Real additions: model forward-pass time (must beat the 80 ms budget to run real-time), audio buffering on capture/playback, network/RTC jitter buffers, OS audio stack, and any extra lookahead frames the codec uses. Real systems land higher than 160 ms.
• Full-duplex is the differentiator: classic voice assistants insert a turn-taking gate — VAD waits for end-of-speech, then ASR → LLM → TTS run in sequence, stacking hundreds of ms of silence-detection and pipeline delay on top of compute. Moshi instead models both streams (user + assistant) at the same 12.5 Hz clock simultaneously, so it can listen and speak at once.
• With no end-of-turn gate, the model can start responding, back-channel ("mm-hm"), or interrupt within roughly one frame of the user's audio. The felt latency approaches the 160 ms floor instead of the multi-second feel of gated cascades — which is exactly what makes the creature feel alive.

∎

(a) RVQ and its bitrate

Residual VQ quantizes the vector in successive refinement stages. Start with r₀ = x; at stage i, pick the nearest entry q_i from codebook i to the current residual, then subtract it:

Each of the L stages emits one index into a codebook of size K, costing log₂K bits. At f frames per second:

(b) Mimi-like numbers

Take L = 8 codebooks, K = 2048 (so log₂K = 11 bits), f = 12.5 Hz:

That is the headline: telephone-quality-and-up speech tokenised at roughly 1 kbps, which is what makes it cheap enough for an LLM to model audio token-by-token at the 80 ms clock. (Exact Mimi config is version-sensitive — confirm L, K against the release you run.)

(c) Semantic vs acoustic tokens

• Semantic token: the first codebook is distilled to align with a self-supervised speech model (e.g. WavLM), so it carries linguistic / content information — what was said. This makes the token stream easy for a language model to reason over.
• Acoustic tokens: the remaining residual codebooks capture how it sounds — speaker timbre, prosody, recording conditions — the detail needed to reconstruct natural-sounding audio.
• Splitting the two lets one model predict meaning and generate faithful audio from the same low-bitrate stream — the foundation of Moshi's full-duplex speech (Problem 3).

∎

Why synchronous coupling is fatal

If the fast loop must wait for each VLA call to return before it can act, the effective tick period of the whole creature becomes the slow loop's period:

Compare against the conversational floor from Problem 3 (~160 ms). A 1–3 s gate is 6–19× over budget: the creature freezes — no voice, no micro-motion — every time it "thinks." Felt latency is dominated by the slowest component.

Async with phase reporting

Decouple the loops onto separate clocks. The fast loop keeps running at 200 ms (voice back-channels, gaze, idle sway) while the VLA computes in the background; when the new action chunk is ready it is swapped in. The user-perceived latency of the interactive channel becomes the fast period, independent of the VLA:

• Phase reporting closes the gap that remains: while the VLA runs, the fast loop emits a "thinking…" cue (a verbal "let me look", a head tilt, a status light). The delay becomes legible rather than dead air — perceived responsiveness rises even further than the raw number.
• Staleness vs responsiveness: the executed action chunk may be up to one slow-loop period old. Keep chunks short enough (Problem 9) that mild staleness is harmless, and let the fast loop handle anything reactive.

∎