Learn the move itself, judge it with a critic, and keep each update small.
On this page
These methods learn the policy, the rule that picks the move, directly: a network turns the state into move probabilities, or into the action itself, and training shifts the policy toward moves that led to more reward. REINFORCE learns once per episode; actor-critic adds a critic that judges every step; PPO, proximal policy optimization, reuses each batch for several small updates. All three are on-policy: they learn from moves the current policy just made. Moves that are real numbers, such as a robot’s joint torques, are on Continuous-Action Actor-Critic: TD3 and SAC.
A policy-based agent skips the scores. Its network turns the state into a probability for each move, and the agent samples from them. Sampling matters when any fixed move can be beaten, as in rock-paper-scissors, where the best play is each move a third of the time (Prisner 2014).
Take a language model writing a reply to the prompt of Aligning Language Models with RL, “Explain the moon landing to a 6 year old”. It picks the first token of its reply from three candidates; replies starting with “People” score best on average (toy scores, Figure 2). The reward here is a reward model’s score of the finished reply, and γ is 1 inside one reply; a score averaged over replies is Q, one reply’s score is G.
REINFORCE (Williams 1992), the simplest policy-based method, learns once per episode. After one reply, REINFORCE pushes the token it sampled by the reply’s score (Figure 3).
One REINFORCE episode on gymnasium’s CartPole:
import gymnasium as gym
import torch
from torch.distributions import Categorical
gamma = 0.99
env = gym.make("CartPole-v1")
policy = torch.nn.Sequential(
torch.nn.Linear(4, 128), torch.nn.ReLU(),
torch.nn.Linear(128, 2))
optimizer = torch.optim.Adam(policy.parameters(), lr=1e-2)
s, info = env.reset()
log_probs, rewards, done = [], [], False
while not done:
dist = Categorical(logits=policy(torch.as_tensor(s)))
a = dist.sample()
s, r, terminated, truncated, info = env.step(a.item())
log_probs.append(dist.log_prob(a))
rewards.append(r)
done = terminated or truncated
G, returns = 0.0, []
for r in reversed(rewards):
G = r + gamma * G
returns.insert(0, G)
G_t = torch.tensor(returns)
loss = -(torch.stack(log_probs) * G_t).sum()
optimizer.zero_grad()
loss.backward()
optimizer.step()
Its weakness is noise: Gt includes the luck of every later reward, so updates vary a lot between episodes (the RL textbook, chapter 13). Subtract what the state usually returns, here J = 4.3, and the push keeps its direction with far smaller swings (Figure 4).
Each reply’s score G is its token’s Q plus some noise. With no baseline every score here is above 0, so whichever token is drawn gets pushed up. A token drawn a few times early gains chance, gets drawn more and gains more, so over many replies a run (one training from the starting chances) can lock onto a worse token. Subtracting b keeps the average push but sets how far separate runs drift apart: they agree most when b sits near J, the state’s average score before any update, while a b far above J lets them drift almost as far as with no baseline, with almost every push down instead of up (Figure 5).
A policy that puts almost all its chance on one token stops trying the others (Figure 6).
| Runs | once per episode, after it ends, no replay |
|---|---|
| Carries | the policy network only, no value network |
| Cost | one forward pass per step while playing, one backward pass over the episode |
| Learning rate | Adam step size; higher learns faster but can diverge, 0 never learns; 0.01 (PyTorch example) |
| γ | shrinks later rewards in Gt; lower favors near rewards, 0 counts only the next one, near 1 the whole episode; 0.99 (the PyTorch example also rescales returns, which the snippet skips) |
| Entropy β | weights the bonus on the entropy card; higher keeps moves spread out, 0 lets the policy lock in early, too high keeps the policy random; 0.01, the entropy weight in CleanRL’s PPO (Section 3) |
| Code | pytorch/examples reinforce.py |
| Gotchas | never detach the log-probabilities |
Waiting for the end of an episode to judge every move is slow and noisy. An actor-critic agent judges each move one step later, by whether things went better or worse than it expected.
In actor-critic, the actor is the REINFORCE policy. The critic V(s) is a learned baseline that estimates the return from a state. Scoring both the state before and the state after a move lets it judge that move one step later, not at the episode’s end (the RL textbook, chapter 13). After the token The, the critic’s guess for the rest of the reply drops from 4.3 to 3.8; that drop, δ, judges the token now (Figure 7).
no_grad in the code). Together: REINFORCE with a learned per-step baseline. Builds on the baseline card and the update card of RL Basics; PPO in Section 3 uses this A, and Continuous-Action Actor-Critic: TD3 and SAC uses the same actor and critic with a real-number move.One update on the last B steps the current policy played, where no_grad holds V(s′) fixed.
import torch
from torch.distributions import Categorical
gamma = 0.99
def actor_critic_update(actor, critic, optimizer, batch):
s, a, r, s_next, done = batch # tensors, B steps each
v_s = critic(s).squeeze(-1)
with torch.no_grad():
v_next = critic(s_next).squeeze(-1)
delta = r + gamma * v_next * (1 - done) - v_s
A = delta.detach()
log_pi = Categorical(logits=actor(s)).log_prob(a)
actor_loss = -(log_pi * A).mean()
critic_loss = delta.pow(2).mean()
optimizer.zero_grad()
(actor_loss + critic_loss).backward()
optimizer.step()
A3C, asynchronous advantage actor-critic (Mnih et al. 2016), runs many workers on copies of the world, all updating one shared network, and uses an n-step return (Figure 8) with the entropy bonus. A2C, advantage actor-critic, is the synchronous version: all copies step, then make one joint update.
| Runs | each A3C worker updates after 5 actions or at an episode’s end |
|---|---|
| Carries | no replay; workers on CPU threads share one network and the statistics of RMSProp, an Adam-like optimizer |
| Cost | a forward pass of actor and critic per action, one backward pass per 5 actions per worker (the A3C paper) |
| n | real rewards before the critic’s guess takes over; higher is less biased but noisier, 1 is the one-step δ, a whole episode is REINFORCE with a baseline; 5 in A3C and in Stable-Baselines3 A2C |
| Workers | copies of the world feeding one network; more gives more varied experience at more CPU, 1 loses that steadying; 16 (the A3C paper) |
| Learning rate | RMSProp step size; higher learns faster but can diverge, 0 never learns; 0.0007 in Stable-Baselines3 A2C |
| Code | Stable-Baselines3 A2C |
| Gotchas | zero V(s′) at an episode’s end; detach A in the actor’s loss |
A batch of steps only describes the policy that played it, so learning from it several times is safe only while the policy stays close to that one. PPO reuses one batch of replies for several updates, but stops paying once a token’s chance has moved 20% from where the batch was collected (Figure 9).
PPO is actor-critic that reuses each on-policy batch for several updates but stops rewarding any change that moves the policy far from the one that collected it. Without that stop, every extra pass pushes the same moves further using advantages measured under the old policy, the new policy drifts far from it, and performance can collapse (Schulman et al. 2017):
One PPO update over a rollout, the steps just played, EPOCHS passes over shuffled minibatches:
import torch
from torch.distributions import Categorical
EPS_CLIP, EPOCHS, MINIBATCH = 0.2, 4, 128 # CleanRL ppo.py
def ppo_update(actor, optimizer, s, a, old_log_pi, A):
old_log_pi, A = old_log_pi.detach(), A.detach()
for epoch in range(EPOCHS):
for mb in torch.randperm(len(s)).split(MINIBATCH):
dist = Categorical(logits=actor(s[mb]))
ratio = torch.exp(dist.log_prob(a[mb]) - old_log_pi[mb])
clipped = ratio.clamp(1 - EPS_CLIP, 1 + EPS_CLIP)
objective = torch.min(ratio * A[mb], clipped * A[mb])
loss = -objective.mean()
optimizer.zero_grad()
loss.backward()
optimizer.step()
PPO gets A from GAE (Schulman et al. 2016), a weighted blend of the one-step δ over the next steps. The paper’s full loss adds the critic’s loss and an entropy bonus. PPO is also the RL step of RLHF, reinforcement learning from human feedback, in Aligning Language Models with RL.
| Runs | N copies each play T steps, then K epochs of minibatches over those N × T steps |
|---|---|
| Carries | the N × T rollout with old log-probabilities and advantages, dropped after K epochs, plus the critic |
| Cost | K × (N × T / minibatch) gradient steps per rollout, 10 × 32 at MuJoCo (simulated robot) settings (the PPO paper) |
| Clip ε | half-width of the band; wider allows bigger changes per round, 0 freezes the policy, very wide is plain policy gradient; 0.2 MuJoCo, 0.1 lowered linearly to 0 on Atari video games (the PPO paper), 0.2 CleanRL |
| Learning rate | Adam step size for the policy and critic; higher learns faster but can diverge, 0 never learns; 0.0003 MuJoCo, 0.00025 lowered linearly to 0 on Atari (the PPO paper), 0.00025 lowered linearly to 0 in CleanRL |
| K | passes over each batch; more squeezes each batch but drifts further from the policy that collected it, 1 is a single A2C-like pass; 10 MuJoCo, 3 Atari, 4 CleanRL |
| T | steps each copy plays per round; longer gives more data but fewer updates per step played; 2,048 MuJoCo, 128 Atari with 8 copies (the PPO paper), 128 CleanRL with 4 |
| Minibatch | steps per gradient step; larger gives a less noisy gradient but fewer steps per epoch; 64 MuJoCo, 128 CleanRL |
| GAE λ | how fast the blend’s weight on later steps fades; lower is steadier but leans on the critic, 0 is the one-step δ, 1 the full return less V; 0.95 (the PPO paper, CleanRL) |
| Code | CleanRL ppo.py (docs) |
| Gotchas | detach the old log-probabilities and A; CleanRL also rescales A per minibatch and clips the gradient norm at 0.5 |
When the move is a real number, such as a joint torque, there is no list of chances to push. Continuous-Action Actor-Critic: TD3 and SAC carries the actor-critic idea there.