A day-by-day roadmap to bring ANIMA-Kiwi to life — build each capability in week one, fuse them into one creature in week two. With the gates that say "done," the ETH study spine that grounds the theory, and a rehearsed demo runbook so the live show actually works.
You are at Day 0. Hardware is on the bench, repos are cloned. The strip below is your map; the cards under it are your daily marching orders. Each card is colour-coded by its dominant track and links the page that teaches the how-to.
◀ Week 1 · build each capability
Week 2 · fuse & ship ▶
You are here · Day 0
D1
install·teleop
D2
record·Moshi
D3
finetune·face
D4
deploy·lipsync
D5
barge-in
D6
policy·intent
D7
GATE
D8
orchestrator
D9
navigation
D10
close loop
D11
harden
D12
HRI polish
D13
rehearse
D14
SHIP
The sprint strip — Day 0 (you are here) → Day 14 (ship). Axis colour shows the dominant track per stretch; ◆ D7 is the hard gate, ● D14 is the ship line.
The gate is the whole point
If you are not through Day 7 by Day 7, do not start Week 2. Fusing two unreliable subsystems gives you one very unreliable creature. Spend the buffer on whichever track is below bar.
Three tracks, three completion bars. A track is not "done" because the code runs — it is done when it hits its bar on repeated, honest trials. No cherry-picking the one good take.
Track 1 · Body
Done when…
Language-prompted drive-and-pick ≥ 60% over 10 trials
Prompt varies ("grab the red cup", "pick up the block")
Includes the drive-up phase, not just a pre-positioned grasp
Failures logged with cause, not hidden
Track 2 · Voice
Done when…
Phone holds a sub-~500ms-feel conversation
Interruptible — you can talk over it (barge-in)
Lip-syncing face that moves with the speech
Affect: face emotion matches conversational state
Track 3 · Creature
Done when…
End-to-end: spoken command → expressive ack →
drive → grasp → expressive report
60–70% reliable over repeated runs
Recovers or fails gracefully on the other 30–40%
Honest scope
This is 14-day operational mastery, not a research result. 60–70% end-to-end reliability with a creature that talks and grasps is a genuinely hard, genuinely impressive bar. Resist the urge to claim more than the metrics show.
C · The ETH Study Spine
The theory backbone is Robot Learning: From Fundamentals to Foundation Models (Oier Mees, ETH Zürich, Spring 2026). Twelve lectures, each mapped to where it lands in your build and which course page derives it.
Guest spotlights worth following
Gupta · Danfei Xu · Aviral Kumar · Wagenmaker · Cheng Chi (diffusion policy / UMI) · Ted Xiao · Scott Reed (world models) · Quan Vuong (Physical Intelligence) · Archit Sharma · Lucas Beyer. Their talks and papers are the live edge of everything in the table above.
D · The Demo Runbook demo/
Startup sequence — bring it up in this order
Pi · LeKiwi host — power the robot, start the host process on the Raspberry Pi. Confirm motors respond. T1
GPU · policy_server — launch the VLA policy server on the GPU box. Confirm it loads your frozen Day-6 checkpoint. T1
Moshi + intent watcher — start full-duplex audio and the voice→intent watcher. Confirm a test utterance produces an intent. T2
Face on phone — open the face client; confirm it connects and lip-syncs to TTS. T2
Orchestrator — start it last; it is the conductor and assumes the others are already up. Watch its state log go to IDLE. T3
Capture setupScreen = the face (phone mirrored / screen-recorded). Room cam = the robot driving + grasping. On-screen metrics = current state, intent, success flag, latency. Frame both so the face reaction and the physical action are visible together — that synchrony is what sells the creature.
The rehearsed demo script — storyboard
Run A: a clean command-to-completion. Run B: an interrupt mid-task to prove full-duplex barge-in. Each panel ties a creature state to a face emotion and what it says.
face emotion / speechorchestrator stateinterrupt (barge-in)
Panel 1 · Listen
human speaks
"grab the red cup"
state: LISTENING · attentive
Panel 2 · Think + ack
parses intent
"on it!"
state: THINKING · curious
Panel 3 · Drive
VLA drives to cup
— (silent, focused)
state: DRIVING · focused
Panel 4 · Grasp
closes gripper
— (concentrating)
state: GRASPING · determined
Panel 5 · Report
lifts + returns
"here you go!"
state: REPORTING · happy
Run A — the clean path: command → think/ack → drive → grasp → happy report. Face emotion tracks the orchestrator state at every step.
Run B · 1 · Working
mid-drive to cup
— (driving)
state: DRIVING
Run B · 2 · INTERRUPT
human talks over it
"wait — the blue one!"
barge-in detected · surprised
Run B · 3 · Re-plan
stops, updates intent
"oh — the blue one, got it!"
state: THINKING · re-plans
Run B · 4 · Resume
drives to blue cup
"here's the blue one!"
state: DRIVING→REPORT · happy
Run B — the money shot: interrupt mid-task. Full-duplex barge-in lets the human override; the creature reacts (surprised → re-plan → resume) instead of finishing the wrong command.
E · Risk & Hardware-Gremlins Register
The failures you will actually hit, and what to do about them. Keep this open during rehearsal.
Likely failure
Symptom
Mitigation
Calibration drift
Arm misses where it "thinks" the object is; success decays over a session
Re-run calibration daily and before the demo; log calibration timestamp in RUNBOOK.md
Jerky / unsafe motion
Robot lurches between actions
Usually wrong n_action_steps / chunk size — tune action-step execution to smooth the chunk boundary
Moshi VRAM
OOM on load or mid-conversation crash
Check VRAM headroom; don't co-locate with the VLA on the same GPU; use the right model size
WebSocket races
Face freezes, intent arrives out of order, state machine stalls
Strict startup order (see runbook); add reconnection + sequence numbers on the bus
Grasp false positives
Reports "got it" with an empty gripper
Confirm grasp with current/force or a vision check before transitioning to REPORT
Network latency on the tether
Laggy teleop / control; conversation feels slow
Wired link Pi↔GPU where possible; measure round-trip; keep policy_server close to the host
F · Runbook & Operating Notes
Keep a living RUNBOOK.md
Exact reproduce steps — anyone can bring the demo up from cold
Calibration timestamps + which checkpoint is "the demo one"
Operating discipline
Run long jobs in tmux so a dropped SSH doesn't kill them
Log metrics every run — never trust a single take
Verify lerobot / moshi / pipecat --help before running — these tools are version-sensitive
Ask before you burn time or bandwidth
Multi-hour GPU runs and big downloads are expensive and slow. Confirm before kicking one off — wrong flags discovered three hours in is a lost day on a 14-day clock.
The through-line
Week 1 makes each capability hit its bar (Gate D7). Week 2 fuses them into a creature that talks, drives, and grasps — 60–70% reliable, rehearsed, recorded, and honestly written up by Ship D14.