Lesson
See the policy loop
Translate a robot task into observations, actions, a policy, and—only when useful—an MDP.
Primary workshop
YouTubeLecture 2 frames a policy as the mapping from what the robot observes to what it does, then introduces the Markov property and MDP tuple.
Narrated mechanism
Use this shorter explanation to trace the mechanism, implementation order, and failure checks.
Name the observation, action, and policy in a concrete task
Explain when state differs from observation
Write a small MDP without overcomplicating it
Build the mental model
See the mechanism before memorizing the code.
Every later method changes one part of the same closed loop. If this map is solid, the acronyms stop feeling unrelated. Start with the diagrams, then use the compact implementation as a map rather than code to copy blindly.
Explanatory diagram
The three-part mental model
Read left to right: Observe → decide → act → learn. Each part prepares the next decision.
- 01Observation is evidence. Camera frames and joint readings are what the policy receives. They may be incomplete or noisy; do not casually call them the full state.
- 02A policy closes the loop. A policy π(a|o) chooses an action from the current observation. Running it creates the next observation, so errors feed back into future decisions.
- 03An MDP is a modeling tool. Use (S, A, P, R, γ) when rewards and consequences matter. For imitation learning, demonstrations may be the more useful starting point.
Explanatory diagram
One complete worked trace
Follow one concrete pass before changing the task or adding more machinery.
- 01Step 1. Task: place a mug on a coaster.
- 02Step 2. Observe: wrist image + joint angles.
- 03Step 3. Decide: a short action chunk.
- 04Step 4. Act: move joints and gripper.
- 05Step 5. Learn: compare the outcome with the demonstration or reward.
Why it works
Observation is evidence
Camera frames and joint readings are what the policy receives. They may be incomplete or noisy; do not casually call them the full state.
A policy closes the loop
A policy π(a|o) chooses an action from the current observation. Running it creates the next observation, so errors feed back into future decisions.
An MDP is a modeling tool
Use (S, A, P, R, γ) when rewards and consequences matter. For imitation learning, demonstrations may be the more useful starting point.
Implementation
Keep sensing and acting in one loop
def policy(observation):
return choose_action(observation)
while not task_done:
observation = robot.sense()
action = policy(observation)
outcome = robot.step(action)
learner.update(observation, action, outcome)Build gate
- 01Choose one task: reach, push, or pick.
- 02Write one line each for observation, action, and success.
- 03Draw the five-step loop and mark where uncertainty enters.
- 04Only then decide whether you need a reward function.
Failure signatures
Primary evidence
Knowledge check
Make the model stick
1. A camera cannot see that a gripper is slipping. What is the cleanest statement?
2. For a mobile robot asked to “follow me,” name one hidden state variable and one sensor you could add to make it less hidden.