An actor network outputs continuous torques and a critic network scores them.
Deep Q-Networks (DQN) chooses from a short list of actions: its banana-collecting agent had four (forward, backward, turn left, turn right), and DQN scores each one and takes the biggest.
A robot arm needs a continuous action: how hard to twist each joint, any real number from −1 to 1, so there are infinitely many actions to score. The running example is the RF-DDPG repo, a DDPG solution for the Unity ML-Agents Reacher task, where a double-jointed arm keeps its hand on a floating target.
DDPG uses two networks. An actor turns the state into torques, and a critic scores that state and those torques. The critic learns its scores from rewards as DQN does, and the actor moves its torques in the direction the critic scores higher.
The two networks and their two updates are enough to run DDPG; three stabilizers, taught in Training DDPG: Target Networks, Replay and Noise, make it train reliably.
All twenty arms act every step, but learning runs in bursts (Figure 1).
On this page
The Reacher environment is a double-jointed arm bolted to a table, one segment driven by the shoulder and one by the elbow. The job is to keep the hand on a small floating ball for a whole episode, not to touch it once.
Every step the agent gets thirty-three numbers: how bent each joint is, how fast it swings, where the segments are, and where the target sits.
It answers with four torques, two per joint, one for each axis it can turn about, each any real number from −1 to 1. That is a continuous action space: infinitely many settings, recomputed every step.
Episodes run a fixed number of steps. A step with the hand on the target pays +0.1, any other step pays 0, and the episode score is the total.
The repo trains twenty identical arms in parallel in one Unity build; Unity runs the physics and Python runs the networks. env is the repo's Unity environment (UnityEnvironment from the unityagents package), and brain_name, which is env.brain_names[0], is the key of the arms in its output. s is (20, 33), one row per arm, and actions is (20, 4): random torques for now, which the actor of Section 3 replaces in Section 7.
import numpy as np
NUM_ARMS = 20
out = env.reset(train_mode=True)
info = out[brain_name]
s = info.vector_observations
scores = np.zeros(NUM_ARMS)
while True:
actions = np.random.uniform(
-1, 1, (NUM_ARMS, 4))
out = env.step(actions)
info = out[brain_name]
scores += info.rewards
s = info.vector_observations
if any(info.local_done):
breakThree things make it hard. Most steps pay 0, so the agent must work out which of hundreds of past torques earned each +0.1: the credit assignment problem. A slightly wrong torque fails quietly, drifting the hand a millimeter off. And the arm never starts in the same pose twice, so the policy must work from any 33-number state.
Early in training the hand moves in circles and scores under 1; late in training it stays on the ball and scores 35 to 38 (the training curve in Training DDPG: Target Networks, Replay and Noise).
DQN chooses with one rule: take the action with the biggest Q value. That needs a finite list of actions.
The DDPG paper (Lillicrap et al., 2016) keeps DQN's Q-network (here the critic), its replay buffer and its target network, and adds an actor. The replay buffer holds up to 100,000 past steps from all twenty arms, and each update samples a random minibatch of 256 from it. The target network is a slow copy of each network, moved a share τ = 1e-3 of the way toward the live one after every update, so its values hold still. Deep Q-Networks (DQN) teaches the replay buffer and the target network. The actor is trained with the deterministic policy gradient, which means it follows the gradient the critic hands it. Training DDPG: Target Networks, Replay and Noise shows why the replay buffer and the slow target copies are needed and what raising or lowering each setting does.
DDPG is actor-critic and model-free (it learns from rewards alone, with no model of how the world responds). One algorithm, architecture and set of hyperparameters (settings chosen before training) solved more than 20 simulated physics tasks.
Figure 3. Argmax needs a list; the actor outputs a continuous torque directly.
DDPG splits the job in two. The actor is a policy network: state in, four torques out. The critic is a Q-network: a state-action pair in, one score out.
The layer sizes come from the repo. The actor's last layer is a tanh, which squashes every output into [−1, 1], so it cannot ask for an impossible torque. Clipping would have zero gradient outside the range; tanh is smooth, so gradients keep flowing. The repo also applies batch normalization to the input state, and in the actor after the first two hidden layers; the snippets leave it out.
Figure 4. Same layer sizes, opposite jobs: the actor proposes, the critic judges.
import torch
import torch.nn as nn
import torch.nn.functional as F
S, A = 33, 4
L = nn.Linear
actor = nn.Sequential()
actor.append(L(S, 256))
actor.append(nn.ReLU())
actor.append(L(256, 128))
actor.append(nn.ReLU())
actor.append(L(128, 64))
actor.append(nn.ReLU())
actor.append(L(64, A))
actor.append(nn.Tanh())The critic adds the action late. The state goes through a 256-unit layer first, then torch.cat joins the 4 torques to it, giving 260 numbers, and the head turns them into one score.
class Critic(nn.Module):
def __init__(self):
super().__init__()
self.fs = L(S, 256)
self.f1 = L(256 + A, 128)
self.f2 = L(128, 64)
self.out = L(64, 1)
def forward(self, s, a):
h = F.relu(self.fs(s))
h = torch.cat([h, a], 1)
h = F.relu(self.f1(h))
h = F.relu(self.f2(h))
return self.out(h)S, A and L from the actor snippet; the action joins after the first layer.Deterministic means the actor outputs the same torques for the same state, with no sampling. A stochastic policy samples from a distribution and explores without added noise, but it jitters, and an arm should hold still. DDPG adds exploration separately, as noise on the actions.
Each network has roughly 50,000 parameters, just over 100,000 together, plus their target copies.
DDPG runs one loop: the actor turns the state into torques, the world returns a reward and the next state, the critic scores the state and torques, and the critic's gradient goes back to the actor.
Figure 5. One lap of the loop; the buffer and the target critic are left out.
The critic learns like DQN's Q-network, by a Bellman backup: a move's value should equal the reward it just earned plus the discounted value of where it leaves the arm. The discount factor γ (gamma, 0.99 in the repo) shrinks that future part: near 1, far rewards count almost fully; at 0, only the reward just earned counts. One recorded step (state, torques, reward, next state) is a transition, and the next value comes from the target actor and target critic, not from a search.
In PyTorch, the update is one function over a minibatch (a small random set of transitions from the replay buffer); r and done have shape (B, 1) for a batch of B.
@torch.no_grad()
def td_target(r, s_next, done):
a2 = actor_target(s_next)
q2 = critic_target(s_next, a2)
keep = 1 - done
return r + gamma * keep * q2
def critic_update(batch):
s, a, r, s_next, done = batch
y = td_target(r, s_next, done)
q = critic(s, a)
loss = F.mse_loss(q, y)
critic_opt.zero_grad()
loss.backward()
critic_opt.step()a2 and the target critic scores it (critic card, line 1). keep is 0 where an arm's episode ended, so no future value is added there; no_grad keeps y fixed.learn() in Section 7 calls it, and the setup block there creates critic, critic_opt, gamma and the two target copies.The critic never picks an action; it only judges pairs. Choosing is the actor's job.
The actor never sees a target action. The critic is differentiable, so the gradient of Q with respect to the action says which way the torques should move to score higher, and the actor's weights shift that way.
The chain rule turns that into one line, the deterministic policy gradient theorem (Silver et al., 2014):
In code, the actor minimizes minus Q: critic(s, actor(s)) scores the actor's own torques with the critic, and backward multiplies the two factors by the chain rule. Only the actor's optimizer steps, so the critic is not changed here.
def actor_update(batch):
s = batch[0]
q = critic(s, actor(s))
actor_loss = -q.mean()
actor_opt.zero_grad()
actor_loss.backward()
actor_opt.step()critic_update zeroes them before the critic's next step. learn() in Section 7 calls it, and actor_opt is created there.The actor goes through the critic because the Unity physics can be run but not differentiated. The critic is a learned, differentiable score of each action, trained on real rewards.
Early on the critic's scores are close to random, and the actor follows them anyway: a wrong critic teaches the actor a wrong action. The three stabilizers taught in Training DDPG: Target Networks, Replay and Noise limit this.
The code below runs the schedule of Figure 1 with all twenty arms: act and store every step (①), and every 256 steps run 64 updates on random minibatches (②), each followed by a small nudge of the target copies (③). The first block sets up the pieces Section 5 and Section 6 use.
import random
from collections import deque
from copy import deepcopy
import numpy as np
import torch
from torch.optim import Adam
critic = Critic()
actor_target = deepcopy(actor)
critic_target = deepcopy(critic)
pairs = [(actor_target, actor),
(critic_target, critic)]
actor_opt = Adam(
actor.parameters(), lr=1e-3)
critic_opt = Adam(
critic.parameters(), lr=1e-3)
gamma, tau = 0.99, 1e-3
buf = deque(maxlen=100_000)
last100 = deque(maxlen=100)Critic class of Section 3, a slow target copy of each network, pairs to match each target copy with its live network, an Adam optimizer for each (learning rate 1e-3), the discount γ = 0.99, the target share τ = 1e-3, a buffer that keeps the newest 100,000 transitions, and last100, the scores of the last 100 episodes.Three helpers turn rows into tensors, store each arm's transition, and nudge the target copies.
def to_tensor(rows):
arr = np.array(rows)
return torch.tensor(
arr, dtype=torch.float32)
def store(s, act, info):
obs = info.vector_observations
r = info.rewards
done = info.local_done
buf.extend(
zip(s, act, r, obs, done))
return obs
@torch.no_grad()
def soft_update(tgt, live):
dst = list(tgt.parameters())
src = list(live.parameters())
for t, p in zip(dst, src):
t.lerp_(p, tau)obs, episode-end flag) and returns the next states (①).learn() samples one minibatch and runs one update on it.
def learn():
if len(buf) <= 256:
return
rows = random.sample(buf, 256)
cols = [to_tensor(c)
for c in zip(*rows)]
s, a, r, s_next, done = cols
r = r[:, None]
done = done[:, None]
batch = s, a, r, s_next, done
critic_update(batch)
actor_update(batch)
for tgt, live in pairs:
soft_update(tgt, live)r and done become one column so they match the critic's 256 by 1 output.critic_update of Section 5, then actor_update of Section 6, on this minibatch. The repo also clips the critic's gradient norm to 1 (DDPG in practice on Training DDPG: Target Networks, Replay and Noise).The loop is the episode loop of Section 1, with the actor choosing the torques and storing and learning added.
def solved():
if len(last100) < 100:
return False
return np.mean(last100) >= 30
step = 0
while not solved():
out = env.reset(
train_mode=True)
info = out[brain_name]
s = info.vector_observations
scores = np.zeros(NUM_ARMS)
while True:
s_tensor = to_tensor(s)
with torch.no_grad():
act = actor(s_tensor)
act = act.numpy()
out = env.step(act)
info = out[brain_name]
s = store(s, act, info)
scores += info.rewards
step += 1
if step % 256 == 0:
for _ in range(64):
learn()
if any(info.local_done):
break
last100.append(scores.mean())step counts steps across episodes.act before env.step, taught in the noise section of Training DDPG: Target Networks, Replay and Noise.last100.Why does each of the slow copies, the buffer and the exploration noise matter, and what does one full update look like on the real arm? Training DDPG: Target Networks, Replay and Noise answers both.