ORIGINAL · NARRATED · CAPTIONED · DETERMINISTIC SVG
Deep visual explainer
The course preview places the live HTML video above this page. It uses semantic SVG paths, exact narration ownership, captions, and seek-safe motion.
What the explainer teaches
- Prediction can live inside representation: Not every world model exposes a function that can be rolled forward. Predictive regularities can be embedded in representations, policies, language models, or vision-language models. These systems may infer what is likely without offering a clean simulator interface. The operational question is what their representation lets a downstream agent predict, compare, and control.
- Generative models render futures: A generative video world model predicts future observations or latent tokens. The result is inspectable, and action conditioning can turn generation into an interactive environment. But visual realism is not causal proof. A plausible video may still violate object permanence, dynamics, or rare constraints that matter to a planner.
- JEPA predicts an abstract target: A joint-embedding predictive architecture encodes context and target separately. A predictor maps the context representation toward a representation compatible with the hidden or future target. By avoiding pixel reconstruction, the model can ignore unpredictable detail. The risk shifts to representation sufficiency: the abstraction must retain what future planning will need.
- Broad priors, controllable dynamics: A practical hybrid first learns broad temporal and semantic structure from large-scale passive video. It then adds a smaller action-conditioned model using robot interaction data. This division combines scalable implicit knowledge with explicit control. V-JEPA 2 demonstrates the pattern, while generative systems such as Genie emphasize visible interactive futures.
CONCEPT MAP
The mechanism at a glance
OBJECTIVE 1
Distinguish explicit rollouts from predictive structure embedded in representations
Demonstrate this capability in the guided or transfer task; reading alone marks it only as introduced.
OBJECTIVE 2
Compare generative observation prediction with joint-embedding prediction
Demonstrate this capability in the guided or transfer task; reading alone marks it only as introduced.
OBJECTIVE 3
Evaluate whether a language, vision-language, or video model deserves the world-model label for a given task
Demonstrate this capability in the guided or transfer task; reading alone marks it only as introduced.
REFERENCE CHAPTER
Predictive structure without a simulator API
An implicit world model stores regularities in representations or policy parameters without exposing a standalone transition function. A vision-language model may infer a plausible next event from a scene; a video model may encode temporal structure; a JEPA may predict a future-compatible representation. These systems can shape decisions even when the agent cannot freely roll them forward step by step.E1, E3, E7–E8
Generative prediction
A generative model predicts observations or tokens. This makes futures inspectable and can support interactive simulation. Genie infers latent actions from video and learns action-conditioned generation. The advantage is a visible, controllable environment model. The cost is spending capacity on unpredictable surface detail, and visual plausibility can be mistaken for causal correctness.
Joint-embedding prediction
JEPA-style models predict the representation of a missing or future target from context. Instead of reconstructing every pixel, the objective asks for compatibility in representation space. This can discard irrelevant detail and focus learning on stable structure. But the abstraction is only useful if the representation preserves information required by planning.
When is an LLM a world model?
A language model contains substantial world knowledge and can generate plausible continuations. That does not automatically make it an action-conditioned simulator with calibrated state, uncertainty, and intervention semantics. Use a task-specific test: can the model represent the relevant state, condition on actions, predict consequences, support counterfactual comparison, and remain grounded under feedback?
Generation is not understanding
A generated future may look coherent while violating conservation, object permanence, contact, or rare safety constraints. Conversely, a non-generative latent predictor may support useful control despite being impossible to visualize directly. Evaluate downstream consistency, controllability, and transfer—not aesthetic realism alone.
Hybrids are the likely endpoint
Large-scale implicit pretraining can supply broad visual and semantic structure. A smaller action-conditioned module can add controllable dynamics. A planner or policy can consume the resulting state. V-JEPA 2 illustrates this pattern: large-scale action-free video pretraining followed by action-conditioned post-training for robot planning.
CURATED COMPANION VIEWING
Watch with a question
These external videos extend the chapter. Their demonstrations and arguments are contextual material; the primary evidence ledger below remains authoritative for technical claims.
EXTERNAL LECTURE · Harvard CMSA
Yann LeCun | Self-Supervised Learning, JEPA, World Models, and the future of AI
Companion lecture for the JEPA chapter; compare its claims with the explicit rollout view.
Open on YouTube ↗EXTERNAL LECTURE · Google DeepMind
Project Genie | Experimenting with infinite interactive worlds
Official demonstration of interactive generative environments; use as an example, not as evidence of general physical understanding.
Open on YouTube ↗GUIDED + TRANSFER
Use the model
Guided practice
Evaluate an imagined video generator on five axes: state persistence, action controllability, causal consistency, uncertainty, and downstream planning utility.
Transfer
Decide whether a code language model is a world model for repository maintenance. Specify the state, actions, transitions, and feedback it would need.
Feedback rule: Name the state, action, prediction, objective, uncertainty, and real-world correction signal. If one is missing, the explanation is incomplete.
DIAGNOSTIC CHECKS
Can you reason with it?
CHECK 1 · recall
What does a JEPA primarily predict?
CHECK 2 · application
A video looks realistic but ignores the agent's steering input. What capability is missing?
CHECK 3 · transfer
Which is the best test of whether an LLM is a world model for a tool-use task?
EVIDENCE LEDGER
Sources and limits
“World model” is used inconsistently across the literature. This course uses the anchor paper’s explicit–implicit distinction as an organizing device, while treating hybrids and task-specific operational tests as first-class.
- ev-world-model-definition A Tutorial on World Models and Physical AI · Il-Seok Oh / arXiv
World models are organized around predictive structure: explicit models expose learned dynamics for rollouts and planning, while implicit models encode predictive regularities inside learned representations. Both aim to support reasoning beyond reactive control. - ev-world-model-design A Tutorial on World Models and Physical AI · Il-Seok Oh / arXiv
Important design axes include state representation, deterministic versus stochastic dynamics, how prediction is used, and the training objective. The explicit–implicit distinction is useful but is not a claim that every system fits a perfectly clean binary. - ev-genie Genie: Generative Interactive Environments · Bruce et al. / arXiv
Genie learns controllable generative environments from unlabelled Internet video by combining a video tokenizer, latent action model, and dynamics model. It illustrates a generative route to action-conditioned world simulation. - ev-vjepa2 V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning · Assran et al. / arXiv
V-JEPA 2 pretrains an action-free joint-embedding predictive model on large-scale video, then post-trains an action-conditioned latent world model with robot video. The paper demonstrates image-goal planning on robot arms without task-specific reward training in the deployment environments.