Action Models

A · Recap & roadmap

Recall the compounding-error result from page 1: a behavior-cloning policy with per-step error ε drifts off the expert distribution at rate O(ε T2) over a horizon T. But there is a deeper failure even with zero noise: a unimodal head trained by MSE collapses two valid trajectories into their average, which is often invalid. The cure is a head that can represent a distribution over action chunks, not a single mean. Below are the three families we cover, all conditioned on observation o (vision + state, and later language).

Foreshadow: by page 3 we converge on flow matching, because both SmolVLA and π0 attach a flow-matching action expert to a VLM backbone. The math here is the foundation for that.

B · Action chunking & ACT

Maps to ETH Lecture 7 · Sequence Modeling — Zhao et al., "Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware" (ALOHA / ACT)

B.1 · Action chunking — predict k actions at once

Instead of predicting one action per observation and re-querying every step, ACT predicts a chunk of the next k actions in a single forward pass. This is the single most important trick for cheap-hardware imitation learning, and it attacks compounding error directly.

If error compounds like O(ε T2) in the number of decision points, then cutting decision points by a factor of k cuts the compounding term by roughly k2. Chunking also lets the policy commit to a coherent motion (e.g. a full grasp) rather than re-deciding mid-reach, which is where unimodal indecision causes pauses and jitter.

Temporal ensembling

Overlapping chunks predicted at consecutive timesteps give several predictions for the same future action. ACT averages them with an exponential weight, which smooths the executed trajectory without adding latency:

where i indexes how old a prediction is and m sets how fast old predictions decay.

B.2 · ACT is a conditional VAE

A chunk of demonstrated actions is multimodal: the same observation can correspond to several valid motions, and within a chunk there is natural variation in style. ACT models p(achunk | o) with a latent variable z — a conditional VAE — so the decoder can produce different coherent chunks for the same observation by varying z.

The ELBO, derived

We want to maximize the conditional log-likelihood of the demonstrated chunk. It is intractable, so we lower-bound it. Introduce a variational posterior q(z|a,o) and apply Jensen's inequality:

This is the ELBO, the objective ACT maximizes (equivalently, it minimizes the negative ELBO). Two terms:

• Reconstruction  𝔼_q[log p(a|z,o)] — decode z and o back into the chunk. With a Gaussian/Laplace decoder this is just an L1/L2 reconstruction loss on the predicted actions.
• KL regularization  KL(q‖p) — pulls the posterior toward the prior p(z) = N(0,I) so the latent space is smooth and samplable at test time. ACT weights it with a small β (β-VAE).

Reparameterization trick

You cannot backprop through a random sample. So instead of sampling z ∼ N(μ,σ²) directly, write it as a deterministic function of a fixed-noise variable:

Now the randomness lives in ε, and gradients flow cleanly into μ and σ.

At test time

Discard the encoder entirely (it needed the ground-truth chunk, which you don't have at deploy). Either sample z ∼ N(0,I) for stochastic, multimodal chunks, or set z = 0 (the prior mean) for the deterministic "most-likely" chunk — LeRobot's ACT defaults to the latter for stable rollouts. Decode with o to get â1:k, execute the chunk, slide forward, repeat.

How this maps to my build: ACT is the simplest expressive head and LeRobot's most battle-tested for cheap arms — it is my baseline. I'll keep chunk_size≈50, set n_action_steps to match, and default z = 0 for repeatable pick-place rollouts on the LeKiwi.

C · Diffusion Policy

Maps to ETH Lecture 6 · Generative Models — Janner/Du "Diffuser / Planning with Diffusion", Florence et al. "Implicit Behavioral Cloning", Wagenmaker et al. "Steering Diffusion Policies with Latent-Space RL"

Diffusion Policy treats the action chunk as something to be denoised into existence. Start from pure Gaussian noise and iteratively clean it, conditioned on the observation, until a coherent action sequence emerges. Because it samples from the learned distribution instead of regressing a mean, it represents multimodality natively.

C.1 · Forward process (fixed noising)

Take a clean action chunk a0 from the dataset and gradually add Gaussian noise over diffusion steps t = 1…N. The closed form for any step uses the cumulative product ᾱt of the noise schedule:

As t→N, ᾱt→0 and at becomes pure noise. This process is not learned — it just defines the corruption we will reverse.

C.2 · Training objective (predict the noise)

Train a network εθ to predict the noise that was added, given the noisy chunk, the step index, and the observation conditioning o. This is the DDPM simple loss:

• at — the noised chunk built from a0 and ε via the forward formula.
• t — diffusion step (embedded), tells the net how much noise to expect.
• o — observation conditioning (vision + state); the policy never denoises blind.

C.3 · Reverse process (sampling)

To act, sample aN ∼ N(0,I) and walk the chain backward, subtracting predicted noise each step (DDPM form):

After N steps you arrive at a clean action chunk a0. The stochastic term σt z is exactly what lets two runs from the same observation land on different valid modes.

How this maps to my build: Diffusion Policy is my strongest multimodal head and a great quality bar — but N≈50–100 network evals per chunk is heavy for the LeKiwi's async runner. I'll train it as a quality reference, then check whether flow matching gets me the same multimodality at a fraction of the inference cost.

D · Flow Matching heart of the page

This is what SmolVLA & π0 use — page 3

Flow matching keeps diffusion's expressivity but replaces the noisy, many-step stochastic chain with a deterministic transport: learn a velocity field that carries noise to data along a smooth path, then integrate it with a handful of ODE steps. Far fewer function evaluations means far faster inference — which is exactly what our async runner needs.

D.1 · Continuous normalizing flows & probability paths

Imagine a continuous time τ ∈ [0,1]. At τ=0 we have an easy distribution x0 ∼ N(0,I); at τ=1 we want the data distribution x1 (an action chunk). A probability path is a continuum of distributions connecting them, and a velocity field v(x,τ) is the flow that pushes samples along that path. If we know the right velocity field, we can transport any noise sample to a data sample by following it.

D.2 · Conditional Flow Matching with the OT (linear) path

The trick that makes this trainable: instead of matching the intractable marginal velocity, match a conditional one defined per data point. Pick a pair (noise x0, data x1) and connect them with the simplest possible path — a straight line (the optimal-transport interpolant):

Now derive the target velocity. Since xτ is linear in τ, differentiate:

The target velocity is constant along the path — just the straight-line displacement x1−x0. So we train a network vθ to regress it. This is the centerpiece:

Every symbol:

• τ ∼ U[0,1] — random time along the path.
• x0 ∼ N(0,I) — a fresh noise sample (the source).
• x1 — a real action chunk from the dataset (the target).
• xτ = (1−τ)x0 + τ x1 — the interpolated point fed to the net.
• o — observation/conditioning: vision + language + proprioceptive state.
• vθ — the learned velocity field; target is the constant x1−x0.

Note the loss is a plain regression — no KL, no noise schedule, no variational posterior. That is why flow matching is famously stable to train.

D.3 · Sampling — integrate the ODE

At inference, draw noise x0 ∼ N(0,I) and integrate the learned ODE from τ=0 to τ=1. A few Euler steps suffice because the OT paths are nearly straight:

# Flow-matching sampling: noise -> action chunk
x = randn(chunk_shape)            # x0 ~ N(0, I)
o = encode_observation(images, state, lang)
n = 10                            # integration steps — often 4..10
dt = 1.0 / n
for i in range(n):
    tau = i * dt
    v = v_theta(x, tau, o)         # one network eval per step
    x = x + dt * v                  # Euler step along the flow
return x                           # x ~ p(action_chunk | o)

Diffusion typically needs N≈50–100 evals; flow matching needs ~4–10. For our LeKiwi async runner — where the policy must emit a fresh chunk faster than the arm exhausts the last one — that latency gap is decisive.

D.4 · Discrete-token vs continuous-expert action heads

When we bolt an action head onto a VLM (page 3), there are two camps. Discrete-token heads (e.g. RT-2-style) tokenize actions into bins and let the language model autoregress them — simple to plug in, but binning loses precision and decoding is sequential. Continuous-expert heads (SmolVLA, π0) attach a small flow-matching "action expert" that outputs continuous chunks in one parallel pass — precise, multimodal, and fast. Page 3 builds the continuous-expert path on top of exactly the CFM loss above.

How this maps to my build: Flow matching is my shipping action head. Stable regression training, multimodal samples, and ~4–10 evals per chunk — that latency is what lets the LeKiwi's async runner keep the arm fed. On page 3 I attach it to SmolVLA as a continuous action expert.

E · Comparison matrix

Four heads, the same question each: can it represent more than one valid future, and what does it cost?

Lab · Action-head bake-off on the LeKiwi dataset

ETH papers — the source material

Short reads behind this page, with how each one lands in the build.