Planning with imagination

Predict before the reveal: A model predicts ten possible futures. What additional machinery is required before any one of them becomes an action?

ORIGINAL · NARRATED · CAPTIONED · DETERMINISTIC SVG

Deep visual explainer

Planning with imagination

The course preview places the live HTML video above this page. It uses semantic SVG paths, exact narration ownership, captions, and seek-safe motion.

Open player

What the explainer teaches

Prediction needs an objective: A rollout tree contains possibilities, not preferences. Each edge represents an action and each node a predicted state, but nothing yet says which branch should be chosen. A reward, cost, goal distance, value estimate, or constraint assigns decision meaning. The planner then budgets computation among branches rather than treating every possible future equally.
Actor and critic amortize planning: In actor-critic imagination, the actor proposes actions and the learned dynamics generates latent futures. The critic estimates return at every imagined state. Training pushes the actor toward high predicted return. Deployment can then use the actor directly, turning expensive imagined practice into fast action selection.
Search spends computation online: Tree search retains online deliberation. Policy priors focus expansion, predicted rewards score transitions, and values estimate outcomes before every branch reaches a terminal state. Search can improve a decision at test time, but repeated model queries amplify bias. A good search policy must know when deeper simulation stops adding trustworthy information.
Replan before error compounds: Model-predictive control optimizes a short action sequence, executes only the first action, and then observes the world again. This receding horizon creates a correction rhythm: imagine, act, measure, update, and replan. It trades long open-loop confidence for repeated contact with reality.

CONCEPT MAP

The mechanism at a glance

OBJECTIVE 1

Trace how actions, predicted transitions, rewards, and values form a rollout objective

Demonstrate this capability in the guided or transfer task; reading alone marks it only as introduced.

OBJECTIVE 2

Compare actor–critic imagination, Monte Carlo tree search, and model-predictive control

Demonstrate this capability in the guided or transfer task; reading alone marks it only as introduced.

OBJECTIVE 3

Diagnose compounding error, model exploitation, and uncertainty failures

Demonstrate this capability in the guided or transfer task; reading alone marks it only as introduced.

REFERENCE CHAPTER

A rollout is not a decision

Prediction supplies candidate futures. A decision procedure needs an objective: reward, cost, value, constraint satisfaction, goal distance, or a learned preference. It also needs a computation policy that decides which branches deserve deeper simulation. The world model and planner are therefore separate contracts.^{E2, E5–E6}

Actor–critic imagination

An actor proposes actions from latent states. The learned model rolls them forward. A critic estimates return, including beyond the finite imagination horizon. The actor is optimized toward high predicted return. This amortizes planning into a policy: expensive learning produces fast action selection at deployment.

Tree search

Monte Carlo tree search allocates simulations adaptively. Policy priors suggest promising actions; value estimates truncate branches without simulating to the end. Search can improve decisions online, but costs latency and depends on a latent transition that remains reliable under repeated querying.

Model-predictive control

MPC optimizes an action sequence over a finite horizon, executes only the first action, then replans from a new observation. This is a powerful pattern for imperfect models because every real observation resets accumulated error. Sampling-based MPC can work even when dynamics or objectives are not differentiable.

Three uncertainty sources

Aleatoric uncertainty represents genuine multiple futures. Epistemic uncertainty reflects insufficient data or model knowledge. Objective uncertainty means the score itself may not capture what the user or environment values. A planner that models only the first can still confidently optimize a blind spot.

Planning discipline

Use shorter horizons where error grows quickly, penalize uncertain or unsupported states, keep hard constraints outside learned reward when possible, validate imagined advantage in the real environment, and log where the planner queries the model. The query distribution—not average validation loss—determines operational risk.

GUIDED + TRANSFER

Use the model

Guided practice

Given a three-action rollout tree, annotate immediate reward, bootstrap value, uncertainty, and a hard safety constraint. Choose a branch and explain the trade-off.

Transfer

Design receding-horizon planning for an agent that edits code: define horizon, action candidates, model correction signal, and a hard rollback constraint.

Feedback rule: Name the state, action, prediction, objective, uncertainty, and real-world correction signal. If one is missing, the explanation is incomplete.

DIAGNOSTIC CHECKS

Can you reason with it?

CHECK 1 · recall

Why does a critic appear at the end of a finite imagined horizon?

CHECK 2 · application

What distinguishes model-predictive control?

CHECK 3 · transfer

Validation loss is low, but the planner visits unusual states where predictions are optimistic. What should you inspect?

EVIDENCE LEDGER

Sources and limits

“World model” is used inconsistently across the literature. This course uses the anchor paper’s explicit–implicit distinction as an organizing device, while treating hybrids and task-specific operational tests as first-class.

ev-world-model-operation A Tutorial on World Models and Physical AI · Il-Seok Oh / arXiv
A generic latent world model can be decomposed into an encoder E, history update H, latent dynamics F, and optional decoder D. Imagined rollouts alternate policy actions with learned latent transitions instead of repeatedly querying the real environment.
ev-dreamer Mastering diverse control tasks through world models · Hafner et al. / Nature
DreamerV3 uses a recurrent state-space model to predict latent states, rewards, and continuation, while actor and critic networks learn from imagined latent trajectories. The published system reports one configuration across more than 150 tasks.
ev-muzero Mastering Atari, Go, chess and shogi by planning with a learned model · Schrittwieser et al. / Nature
MuZero learns a representation, recurrent dynamics, reward, policy, and value sufficient for Monte Carlo tree search without reconstructing the environment's raw observations. Its latent model is optimized for decision-relevant prediction.
ev-world-model-design A Tutorial on World Models and Physical AI · Il-Seok Oh / arXiv
Important design axes include state representation, deterministic versus stochastic dynamics, how prediction is used, and the training objective. The explicit–implicit distinction is useful but is not a claim that every system fits a perfectly clean binary.