Lesson

See the policy loop

Translate a robot task into observations, actions, a policy, and—only when useful—an MDP.

Primary workshop

Lecture 2 frames a policy as the mapping from what the robot observes to what it does, then introduces the Markov property and MDP tuple.

Narrated mechanism

Use this shorter explanation to trace the mechanism, implementation order, and failure checks.

Name the observation, action, and policy in a concrete task

Explain when state differs from observation

Write a small MDP without overcomplicating it

Build the mental model

See the mechanism before memorizing the code.

Every later method changes one part of the same closed loop. If this map is solid, the acronyms stop feeling unrelated. Start with the diagrams, then use the compact implementation as a map rather than code to copy blindly.

Explanatory diagram

The three-part mental model

Read left to right: Observe → decide → act → learn. Each part prepares the next decision.

01Observation is evidence. Camera frames and joint readings are what the policy receives. They may be incomplete or noisy; do not casually call them the full state.
02A policy closes the loop. A policy π(a|o) chooses an action from the current observation. Running it creates the next observation, so errors feed back into future decisions.
03An MDP is a modeling tool. Use (S, A, P, R, γ) when rewards and consequences matter. For imitation learning, demonstrations may be the more useful starting point.

Explanatory diagram

One complete worked trace

Follow one concrete pass before changing the task or adding more machinery.

01Step 1. Task: place a mug on a coaster.
02Step 2. Observe: wrist image + joint angles.
03Step 3. Decide: a short action chunk.
04Step 4. Act: move joints and gripper.
05Step 5. Learn: compare the outcome with the demonstration or reward.

Why it works

Observation is evidence

Camera frames and joint readings are what the policy receives. They may be incomplete or noisy; do not casually call them the full state.

A policy closes the loop

A policy π(a|o) chooses an action from the current observation. Running it creates the next observation, so errors feed back into future decisions.

An MDP is a modeling tool

Use (S, A, P, R, γ) when rewards and consequences matter. For imitation learning, demonstrations may be the more useful starting point.

Implementation

Keep sensing and acting in one loop

Python-like pseudocode

def policy(observation):
    return choose_action(observation)

while not task_done:
    observation = robot.sense()
    action = policy(observation)
    outcome = robot.step(action)
    learner.update(observation, action, outcome)

Build gate

01Choose one task: reach, push, or pick.
02Write one line each for observation, action, and success.
03Draw the five-step loop and mark where uncertainty enters.
04Only then decide whether you need a reward function.

Failure signatures

01“The camera image is the state.” It is an observation. Hidden friction, object mass, and occluded geometry can still affect what happens next.

02Diagnostic warning: The task is no longer an MDP.

03Diagnostic warning: The policy cannot be learned.

Primary evidence

Knowledge check

Make the model stick

1. A camera cannot see that a gripper is slipping. What is the cleanest statement?

2. For a mobile robot asked to “follow me,” name one hidden state variable and one sensor you could add to make it less hidden.

Open video player