Learn a score for every move, then play the best one.
On this page
Value-based methods are one of four RL families. Every family runs the reinforcement learning loop; what differs is what the agent learns to pick its move. A value is an estimate of how much future reward to expect; a policy is the rule that picks the move. Methods that learn, or are handed, a model predicting each move’s result plan with it (model-based); the rest are model-free. Actor-critic methods learn both a policy, the actor, and a value, the critic.
Given one state, a value-based agent outputs a score per move, a policy-based one move probabilities, and an actor-critic probabilities plus the critic’s score of the state.
Tap a square to start there.
A value-based agent keeps a score, a Q-value, for every move in every state: the future reward it expects from that move and good play after. It mostly plays the highest score, explores with a small chance ε (ε-greedy) and updates as in RL Basics. Q-learning (Watkins and Dayan 1992) keeps these scores in a table, one row per state.
In the cliff walk of Sutton and Barto’s RL textbook (2018, chapter 6), a 4 × 12 grid, an episode runs from start S to goal G, the two ends of the bottom row, past the cliff between them. Every step costs −1; the cliff costs −100 and a return to S. One step right along the cliff edge costs −1 and lands two moves from the goal (Figure 4).
The same pull between a short risky route and a safe one faces a robot that learns while it explores next to a drop.
The code runs tabular Q-learning on gymnasium’s CliffWalking.
import random
import numpy as np
import gymnasium as gym
alpha, gamma, eps = 0.5, 1.0, 0.1 # values from the book's cliff code
env = gym.make("CliffWalking-v1")
Q = np.zeros((48, 4)) # 48 squares, 4 moves
def epsilon_greedy(Q, s):
if random.random() < eps:
return random.randrange(4)
return int(Q[s].argmax())
for episode in range(500):
s, info = env.reset()
done = False
while not done:
a = epsilon_greedy(Q, s)
s_next, r, terminated, truncated, info = env.step(a)
best_next = 0.0 if terminated else Q[s_next].max()
Q[s, a] += alpha * (r + gamma * best_next - Q[s, a])
s, done = s_next, terminated or truncated
An off-policy method learns about a policy other than the one making its moves. Q-learning is off-policy because its target scores the best move in s′, whatever the agent plays next, so it learns best-move values while exploring. SARSA, named for the five things one update uses (State, Action, Reward, next State, next Action), changes that one term (the RL textbook, chapter 6). SARSA scores the same step with the move the agent really plays next, slips included (Figure 5).
The SARSA loop picks a_next first and reuses Section 3’s env, Q and epsilon_greedy.
for episode in range(500):
s, info = env.reset()
a = epsilon_greedy(Q, s)
done = False
while not done:
s_next, r, terminated, truncated, info = env.step(a)
a_next = epsilon_greedy(Q, s_next) # really played
q_next = 0.0 if terminated else Q[s_next, a_next]
Q[s, a] += alpha * (r + gamma * q_next - Q[s, a])
s, a = s_next, a_next
done = terminated or truncated
| Runs | once after every real step |
|---|---|
| Carries | the table Q, 48 squares × 4 moves here |
| Cost | one table read and one write per step |
| α | share of the gap each update closes; higher learns faster but noisier, 0 never learns, 1 keeps only the latest target; 0.5 in the book’s cliff code |
| γ | weights the next score in the target; lower favors sooner reward, 0 sees one step, 1 counts all future reward alike; 1 on the cliff, where every episode ends |
| ε | chance of a random move; higher explores more and falls more, 0 never tries anything new, 1 is a random walk; 0.1, over 500 episodes |
| Code | cliff_walking.py, the book’s figures in Python |
| Gotchas | no future after terminated, but keep it after truncated, a time limit (CliffWalking itself never truncates) |
DQN, deep Q-network (Mnih et al. 2015), replaces the table with a neural network that outputs one score per move. Two fixes keep it from oscillating or diverging:
Among noisy scores the highest is usually one that came out too high, so the max makes Q-learning and DQN overestimate (van Hasselt et al. 2016). Double DQN fixes the target. In the banana arena, a square with nothing in reach is worth 0 for every move, yet the guesses are noisy (Figure 8).
The step pays r = 0 and γ is 0.99; Q− is the target network, the frozen copy.
Dueling DQN (Wang et al. 2016) changes the network instead, splitting it into two streams. With a banana ten steps ahead, every move scores about 0.9; the moves differ by less than 0.02 (Figure 10).
Here γ is 0.99. The dueling paper reports the gain on Atari games, where in many frames the moves score about the same.
My banana agent uses both; Deep Q-Networks (DQN) draws its two streams.
One DQN update; for Double DQN, pick the next move with q_net and score it with target_net.
import random
import torch
import torch.nn.functional as F
gamma, batch_size, C = 0.99, 32, 10_000 # DQN paper
def dqn_update(q_net, target_net, optimizer, buffer, n_updates):
batch = random.sample(buffer, batch_size)
s, a, r, s_next, done = map(torch.tensor, zip(*batch))
with torch.no_grad():
best_next = target_net(s_next).max(1).values
target = r + gamma * best_next * (1 - done.float())
q_sa = q_net(s).gather(1, a.unsqueeze(1)).squeeze(1)
loss = F.smooth_l1_loss(q_sa, target) # Huber
optimizer.zero_grad()
loss.backward()
optimizer.step()
if n_updates % C == 0:
target_net.load_state_dict(q_net.state_dict())
| Runs | one update every 4 actions, once the buffer holds 50,000 steps |
|---|---|
| Carries | a buffer of the last 1,000,000 steps, the target network, ε |
| Cost | one pass on 32 steps per update; the buffer dominates memory |
| C | updates between target copies; larger keeps targets steadier but staler, 1 means no frozen copy; 10,000 (the DQN paper) |
| Buffer | how far back replay reaches; larger mixes more old policies, very small brings back look-alike batches; 1,000,000 |
| Minibatch | steps per update; larger is less noisy but costs more compute; 32 |
| Learning rate | step size of RMSProp, an optimizer like Adam; higher learns faster but can diverge, 0 never learns; 0.00025 |
| Code | CleanRL dqn.py, whose CartPole (pole balancing) defaults are smaller (buffer 10,000, batch 128) |
| Gotchas | the paper clips the error to [−1, 1], which is what the Huber loss in the code does |
Two facts usually decide: whether actions are a short list (discrete) or real numbers, and what a real step costs.
PPO and A2C are taught in Policy Gradients: REINFORCE, Actor-Critic and PPO; TD3 and SAC in Continuous-Action Actor-Critic: TD3 and SAC; AlphaZero, MuZero, Dyna and Dreamer in Model-Based RL: Dyna, AlphaZero and MuZero.
| When | Reach for | Because |
|---|---|---|
| Few states, fits in a table | Q-learning, or SARSA if exploring mistakes are costly | SARSA keeps clear of its own slips |
| Short list of moves, many numbers or pixels in | DQN with Double and Dueling | Off-policy, so replay reuses every step |
| Continuous actions, real steps costly | TD3 or SAC | Off-policy with replay, and steadier than DDPG |
| Many cheap copies of a simulator | PPO or A2C | On-policy updates from fresh data; PPO’s clip keeps each small |
| Tuning a language model from human preferences | PPO | Reinforcement learning from human feedback (RLHF) in Aligning Language Models with RL |
| Rules known and exactly simulated | AlphaZero-style search | Looks ahead with the real rules |
| Rules unknown, real steps costly | Dreamer or Dyna | A learned model gives imagined practice |
| Rules unknown, look-ahead matters | MuZero | Searches with a learned model |
A score per move needs a max over a list of moves. What does an agent learn when it outputs the move itself? Policy Gradients: REINFORCE, Actor-Critic and PPO takes up that question, and Continuous-Action Actor-Critic: TD3 and SAC carries it to real-number moves such as a joint torque.
| What breaks | What you see | Fix and where |
|---|---|---|
| Max over noisy scores | Predicted values climb above the returns earned | Double DQN, TD3’s twin critics |
| Targets chase their own updates | Scores oscillate or blow up | Target network |
| Whole-episode returns | Very noisy learning curve | Baseline, critic |
| Policy too certain too early | Entropy, the spread of the move probabilities, falls toward zero and reward stalls | Entropy bonus, SAC |
| Too many steps on one batch | Reward collapses right after an update | PPO clipping |
| Agent exploits a learned model’s flaws | High reward inside the learned model, low in the real world | Check real-world returns; add randomness to the model, as World Models, a learned model of the world (Model-Based RL, section 3), did |