Three stabilizers keep DDPG's two updates from diverging, and one full update shows them working together.
This page builds on Deep Deterministic Policy Gradient (DDPG): Actor and Critic, which teaches the actor, the critic and their two updates.
Run alone, those two updates diverge. The critic chases its own guesses, it learns from runs of nearly identical steps, and the deterministic actor never tries anything new. Three stabilizers each stop one failure (Figure 1): slow target copies hold the critic's target still, a replay buffer mixes old and new steps, and smooth noise makes the actor explore. One training update then runs all five of its parts, and the arm trains.
The running example is the Reacher arm of the RF-DDPG repo, which reads 33 numbers and outputs 4 torques in [−1, 1] every step and earns +0.1 while the hand is on the target. Twenty arms train in parallel, and the task counts as solved when their mean score, averaged over 100 episodes, reaches 30.
Figure 1. Each stabilizer prevents one failure.
The critic update trains the critic toward a target value.
The actor update moves the actor's torques the way the critic scores higher.
On this page
The critic's target is built from its own predictions, so if the critic changes fast, its target changes fast too. Slow target copies of both networks follow the live ones instead.
soft_update applies the card's blend to every weight: t.lerp_(p, TAU) sets t to t + TAU * (p - t), which is the card's line. It runs once for the critic pair and once for the actor pair.
TAU = 1e-3
@torch.no_grad()
def soft_update(tgt, live):
dst = list(tgt.parameters())
src = list(live.parameters())
for t, p in zip(dst, src):
t.lerp_(p, TAU)
soft_update(critic_target, critic)
soft_update(actor_target, actor)Deep Q-Networks (DQN) copied its target network in full every 10,000 updates (Mnih et al., 2015); DDPG makes the copy continuous and small.
Each arm's consecutive steps are nearly identical, so learning from them in order overfits the latest moment; random minibatches mix old and new.
Every transition from every arm, with its episode-end flag done, goes into one memory, and training samples random minibatches from it. Random sampling breaks the link between consecutive steps, each transition is reused many times, and twenty arms fill the buffer twenty times faster than one. Deep Q-Networks (DQN) uses the same buffer.
import random
from collections import deque
BUFFER_SIZE, BATCH = 100_000, 256
buf = deque(maxlen=BUFFER_SIZE)
step = (s, a, r, s_next, done)
buf.append(step)
batch = random.sample(buf, BATCH)A deterministic actor left alone repeats its first guess forever, so during training DDPG adds Ornstein-Uhlenbeck (OU) noise to the torques. Gaussian (white) noise is independent every step, so torques stutter; OU noise remembers its last value and is pulled back toward zero, so it wanders smoothly (Figure 2).
For an arm with inertia (a moving part keeps moving), smooth noise tries coherent strategies ("push left for a while, see what happens") instead of vibrating in place. The DDPG paper chose OU noise for this reason, and the repo keeps it.
One training update has five parts: sample, compute the target, update the critic, update the actor, update the targets. This is off-policy learning: the networks learn from experience gathered by older versions of the actor, which is why the replay buffer can be reused.
Figure 3. Live networks learn every update; target copies follow slowly.
A toy two-joint arm shows what training changes (Figure 4); the real run is the RF-DDPG repo on the Unity ML-Agents Reacher task, with the settings in Figure 5.
Figure 4. A toy 2D arm; as training goes on, the hand settles on the target.
| Setting and repo value | Raising it | Lowering it |
|---|---|---|
| Arms, state, torques 20; 33 in, 4 out | fixed by the Reacher task | |
| Layers actor 256, 128, 64, tanh; critic 256, +4, 128, 64 | more capacity, slower | cheaper, may underfit |
| Replay buffer 100,000 | more varied, adapts slower | forgets faster, batches alike |
| Batch 256 | steadier, more compute | cheaper, noisier |
| Updates 64 per 256 steps | more reuse; critic can overfit | cheaper, slower |
| Adam learning rate 1e-3 for both | faster; the critic can diverge | steadier, slower |
| Discount γ 0.99 | far rewards count more | γ = 0 sees only the next reward |
| Target update τ 1e-3 | faster, less stable; τ = 1 copies | steadier; τ = 0 freezes |
| Critic clip norm 1 | larger steps; huge turns it off | steadier, slower |
| OU pull κ 0.15 | tighter wander | longer drifts |
| OU kick σ 0.2 | wider, noisier torques | narrower; 0 stops exploring |
One arm's score varies a lot from episode to episode. Twenty arms, each with its own exploration noise, average that variation out (Figure 6 plots their mean score) and fill the buffer with more varied experience.
The repo runs one update function 64 times every 256 environment steps, a quarter of an update per step, with the settings in Figure 5; Stable-Baselines3 has a widely used PyTorch DDPG. Each update costs two target forward passes, a critic forward and backward pass, an actor pass through the critic, and two weight blends. Memory is dominated by the buffer: 100,000 transitions of 33 + 4 + 1 + 33 + 1 = 72 numbers each is about 29 MB at 4 bytes per number, while the four networks (two live, two target) hold about 200,000 weights, under 1 MB.
ddpg_update runs one update on a sampled batch, using the actor and critic built in the actor and critic networks section of Deep Deterministic Policy Gradient (DDPG): Actor and Critic, their target copies and their optimizers.
import torch
import torch.nn.functional as F
import torch.nn.utils as U
GAMMA, TAU, CLIP = 0.99, 1e-3, 1.0
pairs = [(actor_target, actor),
(critic_target, critic)]
@torch.no_grad()
def soft_update(tgt, live):
dst = list(tgt.parameters())
src = list(live.parameters())
for t, p in zip(dst, src):
t.lerp_(p, TAU)
@torch.no_grad()
def td_target(r, s_next, done):
a2 = actor_target(s_next)
q2 = critic_target(s_next, a2)
keep = 1 - done
return r + GAMMA * q2 * keep
def ddpg_update(batch):
s, a, r, s_next, done = batch
y = td_target(r, s_next, done)
q = critic(s, a)
critic_loss = F.mse_loss(q, y)
critic_opt.zero_grad()
critic_loss.backward()
U.clip_grad_norm_(
critic.parameters(), CLIP)
critic_opt.step()
q = critic(s, actor(s))
actor_loss = -q.mean()
actor_opt.zero_grad()
actor_loss.backward()
actor_opt.step()
for tgt, live in pairs:
soft_update(tgt, live)soft_update.keep is 0 at the end of an episode (update card in Section 2, line 1).Three mistakes break this update.
done; without it the last step takes value from the next episode.TD3 and SAC each change the target from One DDPG training update (Section 2).
TD3 (Twin Delayed DDPG, Fujimoto et al., 2018) changes DDPG's target in two ways and its schedule in one. It adds a little noise, kept within a small range, to the next torques, and it scores them with two slow target critics and keeps the lower score, so one critic's lucky high guess cannot lift the target. It also updates the actor and the slow copies less often than the critics, so the actor climbs a critic that has had time to settle.
The full TD3 update, worked on one Reacher step, is on Continuous-Action Actor-Critic: TD3 and SAC.
SAC (Soft Actor-Critic, Haarnoja et al., 2018) keeps the policy random and rewards it for spreading its choices (high entropy), so exploration lives in the objective instead of in added OU noise, and the critic's target carries that payment. The same page teaches it in SAC pays for spread.
Multi-agent DDPG is taught on Multi-Agent Reinforcement Learning, where one shared actor and critic drive both rackets of a tennis game (self-play). The MADDPG paper (Lowe et al., 2017) instead gives each agent its own actor and its own centralized critic, which sees every agent's observations and actions.
DDPG fits continuous actions, a cheap simulator, and a controller that should act with no noise at test time, the same torque every time for the same state; for a short list of discrete actions, DQN fits better.