Explicit world models · World Models

Predict before the reveal: Which is more important for planning: predicting the next image accurately, or predicting how actions change reward and value? Defend one answer.

ORIGINAL · NARRATED · CAPTIONED · DETERMINISTIC SVG

Deep visual explainer

Explicit world models

The course preview places the live HTML video above this page. It uses semantic SVG paths, exact narration ownership, captions, and seek-safe motion.

Open player

What the explainer teaches

World Models 2018: The early World Models architecture made a clean modular claim. Compress images into latent codes, learn temporal dynamics over those codes, and train a compact controller using the learned internal world. Its memorable result was policy training inside generated dreams. The deeper lesson is architectural separation: perception, dynamics, and behavior can be learned as related but distinct problems.
Dreamer learns in imagination: Dreamer uses a recurrent state-space model grounded by real replay. From inferred states, an actor chooses actions while the learned dynamics predicts latent futures, rewards, and continuation. A critic estimates long-term return over those imagined futures. Gradients update actor and critic without requiring every policy improvement step to touch the real environment.
MuZero searches a latent tree: MuZero maps observation history to a latent root. For each simulated action, its dynamics predicts a new latent state and reward; its prediction head supplies policy priors and value. Monte Carlo tree search uses those quantities to focus computation on promising branches. The latent world need not look like pixels. It must support the decisions search asks it to make.
Choose the interface by its consumer: Reconstruction serves grounding and inspection. Actor-critic imagination serves differentiable policy learning. Search serves online comparison among structured alternatives. No single interface dominates every task. The correct explicit world model is the smallest predictive contract that remains accurate where the downstream planner, critic, or controller will query it.

CONCEPT MAP

The mechanism at a glance

OBJECTIVE 1

Compare Ha–Schmidhuber, Dreamer, and MuZero by state, dynamics, objective, and decision mechanism

Demonstrate this capability in the guided or transfer task; reading alone marks it only as introduced.

OBJECTIVE 2

Explain the Dreamer actor–critic imagination loop

Demonstrate this capability in the guided or transfer task; reading alone marks it only as introduced.

OBJECTIVE 3

Explain why MuZero predicts decision-relevant quantities instead of reconstructing observations

Demonstrate this capability in the guided or transfer task; reading alone marks it only as introduced.

REFERENCE CHAPTER

One family, three emphases

Ha and Schmidhuber made the decomposition vivid: compress frames, model temporal evolution, and train a small controller inside the learned dream. Dreamer turns this into an end-to-end recurrent state-space system whose actor and critic learn from imagined latent trajectories. MuZero couples a learned latent transition with tree search and predicts reward, policy, and value rather than pixels.^E4–E6

Ha–Schmidhuber: modular dreaming

The original architecture used a variational autoencoder for visual compression, a recurrent mixture-density model for temporal prediction, and a compact controller. Its conceptual contribution is the separation of representation, dynamics, and policy. Its limitation is that a memoryless encoder and separately trained components can leave important uncertainty or task structure poorly integrated.

Dreamer: learn behavior in imagination

Dreamer learns a recurrent state-space model from replayed experience. Starting from posterior states inferred from real sequences, it rolls the dynamics forward under the actor. A critic estimates long-term return on imagined states; the actor is updated to choose actions with high predicted return. Real interaction trains the model, while much policy learning happens in latent imagination.

MuZero: plan without rendering

MuZero's representation function maps observation history to a latent root. Its dynamics function predicts the next latent state and reward for an action. Its prediction function supplies policy priors and values. Monte Carlo tree search uses those outputs to allocate simulations. The model needs to be correct in the ways search consumes—not pixel-identical to the environment.

The decisive choice

Use actor–critic imagination when differentiable latent rollouts and broad continuous-control learning are central. Use search when online deliberation over discrete or structured actions is valuable. Use reconstruction when observation grounding and interpretability justify its cost. These are engineering trade-offs, not a leaderboard ordering.

Model exploitation

A policy may discover trajectories where the learned model is optimistic because the data are sparse. This is model exploitation: high predicted return inside imagination, poor return in reality. Holdout prediction, uncertainty penalties, conservative objectives, short horizons, and continual real-data correction reduce—but do not eliminate—the risk.

GUIDED + TRANSFER

Use the model

Guided practice

Fill a matrix with rows for representation, dynamics output, reconstruction, decision mechanism, and main failure mode; columns are World Models 2018, Dreamer, and MuZero.

Transfer

Choose an architecture for warehouse manipulation with continuous actions and sparse contacts. Explain why search, actor–critic imagination, or a hybrid is appropriate.

Feedback rule: Name the state, action, prediction, objective, uncertainty, and real-world correction signal. If one is missing, the explanation is incomplete.

DIAGNOSTIC CHECKS

Can you reason with it?

CHECK 1 · recall

What does MuZero deliberately avoid requiring?

CHECK 2 · application

Where does Dreamer obtain most actor and critic training trajectories?

CHECK 3 · transfer

A policy finds actions that look excellent only inside the model. What failure is this?

EVIDENCE LEDGER

Sources and limits

“World model” is used inconsistently across the literature. This course uses the anchor paper’s explicit–implicit distinction as an organizing device, while treating hybrids and task-specific operational tests as first-class.

ev-ha-world-models World Models · David Ha and Jürgen Schmidhuber / arXiv
The 2018 World Models system learns a compressed visual representation and temporal model, then trains a compact controller on features from that learned model; the authors also demonstrate training inside generated rollouts and transferring the policy back to the environment.
ev-dreamer Mastering diverse control tasks through world models · Hafner et al. / Nature
DreamerV3 uses a recurrent state-space model to predict latent states, rewards, and continuation, while actor and critic networks learn from imagined latent trajectories. The published system reports one configuration across more than 150 tasks.
ev-muzero Mastering Atari, Go, chess and shogi by planning with a learned model · Schrittwieser et al. / Nature
MuZero learns a representation, recurrent dynamics, reward, policy, and value sufficient for Monte Carlo tree search without reconstructing the environment's raw observations. Its latent model is optimized for decision-relevant prediction.
ev-world-model-design A Tutorial on World Models and Physical AI · Il-Seok Oh / arXiv
Important design axes include state representation, deterministic versus stochastic dynamics, how prediction is used, and the training objective. The explicit–implicit distinction is useful but is not a claim that every system fits a perfectly clean binary.