A robot arm’s move is a set of joint torques, real numbers in a range, like the four torques of my Reacher arm on the DDPG page, so there is no list of moves to take a max over. An actor network outputs the torques and a critic scores them. The actor moves its torque to wherever the critic scores highest, so it finds the places where the critic guesses too high (Figure 1). TD3 and SAC are two ways to stop that; both reuse replay of past steps and slow target copies from Value-Based RL.
Figure 1. One critic’s guess (toy): the actor climbs to where it guesses too high.
1 Why one critic is not enough
DDPG, deep deterministic policy gradient (Lillicrap et al. 2016), trains a deterministic actor μ(s), which outputs one set of torques, through a critic Q(s, a) that scores a state with those torques. It is off-policy, learning from a replay buffer, with slow target copies of both networks. The DDPG page teaches both updates; what keeps it stable is on Training DDPG: Target Networks, Replay and Noise.
Figure 2. The DDPG target on one Reacher step (toy scores).
On one step of my Reacher arm, the slow actor picks a next shoulder torque of 0.60, one slow critic scores it 9.5, and DDPG’s target is 9.505 (Figure 2).
This is the DDPG target: the critic learns toward one slow critic’s score of one exact next move.
target =r + γ · Q−(s′, μ−(s′))
r is the reward, 0.1 here, and s′ the next state. γ is the discount of the return in RL Basics, 0.99 on this step: a score one step later counts 0.99 times as much. μ− and Q− are the slow copies of the actor and critic, with the small minus that Value-Based RL uses for DQN’s target network; the DDPG pages write a prime for the same copies. Together: the Bellman target of RL Basics with the max over moves replaced by the slow actor’s move, 9.505 here. Builds on that target and the DDPG page’s critic update; the TD3 card, next, edits it twice.
2 TD3 trusts the lower of two critics
On the same step, two slow critics score the jittered next torque 9.5 and 9.1, and TD3 believes the lower (Figure 3).
Figure 3. The TD3 target on the same step (toy scores): jitter, two critics, the lower.
This is the TD3 target (Twin Delayed DDPG, Fujimoto et al. 2018): jitter the next torque a little, score it with two critics, keep the lower score. One critic’s lucky high guess, the kind the actor finds in Figure 1, can no longer lift the target.
ã =μ−(s′) + clip(n, −c, c)
ã is the jittered next torque and μ−(s′) the DDPG card’s next move, 0.60. n is Gaussian noise, n ~ N(0, σ2) with standard deviation σ 0.2 in the paper, drawn fresh for each target: 0.06 in Figure 3. clip keeps it within c, 0.5, of zero, so the jitter never moves the torque far: at σ 0.2 the clip acts about 1 time in 81, and a rare draw of size 0.7 or more, about 1 in 2,149, is cut to 0.5. Together: the target is taken at a small cloud of torques around μ−(s′), so a narrow too-high spike in the critic cannot carry it (the TD3 paper, eq. 15); the jittered torque is then kept in the torque range [−1, 1] (the TD3 paper, Appendix D).
qmin =min(Q1−(s′, ã), Q2−(s′, ã))
Q1− and Q2− are the slow copies of two critics trained side by side toward this same target; min keeps the lower score, 9.1 of 9.5 and 9.1. A max over noisy guesses runs high on average (the TD3 paper, section 4), as Value-Based RL shows for DQN; the lower of two cancels much of it (Figure 4). When it errs low instead, that is the safer error: the actor steers away from torques scored too low, so, unlike a too-high score, the error is not passed on through the actor update (the TD3 paper, section 4.2).
target =r + γ · qmin
The DDPG card’s target with Q−(s′, μ−(s′)) replaced by qmin. Together: the same Bellman target with a more careful score of the next move, 9.109 on the Reacher step against DDPG’s 9.505 (Figure 3). Builds on the DDPG target, edited twice, and on Double DQN’s caution; the SAC card keeps line 2.
TD3’s actor moves its torque uphill on the first critic Q1, as in the DDPG actor update. It and the slow copies update once every d = 2 critic updates (after critic updates 2, 4, 6), so the actor climbs a critic that has had time to settle (the TD3 paper, sections 5.2 and 6.1). The lower of two is used only when building the target, while the actor still climbs Q1.
Each critic is the true value plus its own wavy error, of size 0.3 in this page’s toy. The actor picks the torque critic 1 rates highest, where its error points up, which pushes the target up: over 20,000 random critic pairs it lands 0.27 too high. Taking the lower of two critics pushes it down. The two roughly cancel, leaving the lower of two under 0.01 low, the safer error of the TD3 card (toy, illustrative). A critic update moves the critic toward its targets over a batch of replayed steps, and the slow copies carry it into later targets, so a too-high error builds on itself (the TD3 paper, sections 4.1 and 5.1). That is why DDPG’s too-high error piles up over updates while TD3’s stays much closer to the true value (Figure 4).
The TD3 paper saw the same on the simulated robots Hopper (one leg) and Walker2d (two legs): over a million steps, DDPG’s value estimate ran clearly above the true return, and the lower of two greatly reduced the gap (the TD3 paper, section 4.1 and its Figure 1).
At update 20, DDPG runs 0.88 too high,TD3 0.03 too low (toy)
Figure 4.1DDPG’s drift from the true value; 2DDPG’s target error (toy, illustrative).
Figure 4’s toy leaves out the slow copies, the actor delay and the jitter, and its top panel draws one fixed wave shape, so its numbers are illustrative; with only 8 replayed steps per update, TD3’s average target wobbles a little above and below the true value. The code below computes the TD3 target in a few lines and returns the 9.109 of Figure 3.
import random
SIGMA, C = 0.2, 0.5 # TD3 paper
def jitter(next_torque):
n = max(-C, min(C, random.gauss(0.0, SIGMA)))
return max(-1.0, min(1.0, next_torque + n))
def td3_target(r, gamma, q1_next, q2_next):
return r + gamma * min(q1_next, q2_next)
print(round(td3_target(0.1, 0.99, 9.5, 9.1), 3)) # 9.109
1Noise standard deviation σ and clip bound c.
2The jittered torque, the TD3 card, line 1: noise clipped to c, then the torque kept in [−1, 1].
3The lower of two critics and the target, lines 2 and 3.
SAC’s actor draws each torque from a bell curve, its policy π, keeps that curve spread out and pays for it at every step (Figure 5). The log π of a torque is the log of the curve’s height there: high for torques the actor draws often, low for rare ones.
Figure 5. A spread of torques (toy) and what SAC pays for it on the same Reacher step.
This is the soft actor-critic objective (Haarnoja et al. 2018): reward plus a payment for keeping the torques spread. The critic’s target carries the payment, and the actor raises the lower critic’s score while keeping its spread.
objective =Σt E[rt + α · H(π(· | st))]
The return of RL Basics with the spread paid at every step. π(· | st) is the policy, the bell curve of torques the actor draws from in state st. E, the expectation, averages over the torques the policy draws. H is the entropy of the entropy card on Policy Gradients, here for a bell curve: its continuous form is 0.73 nats at standard deviation 0.5, with the natural log. α, the temperature, sets how much spread is worth against reward (the SAC paper, eq. 1); here it is not a learning rate. How α is set is in Section 4. Together: the goal of TD3 with a paid spread.
V(s) =E[min(Q1, Q2)(s, a)
− α · log π(a | s)]
The soft value of a state: for a torque a drawn from the policy, the lower of the two critics’ scores, the TD3 idea, which the SAC paper adopts, plus α times −log π, which averages to the entropy. The soft value in Figure 5 is 9.245. Together: a state’s worth counts the room it leaves to explore.
target =r + γ · V−(s′)
The Bellman target with the soft value. V− is a slowly updated copy of a value network trained toward line 2 (the SAC paper, eq. 8); CleanRL computes it straight from the two critics’ copies instead. Together: the TD3 target with the spread payment inside, 9.253 against the 9.109 of TD3 on the same step (Figure 5; the toy sets V−(s′) to the soft value 9.245).
actor loss =E[α · log π(a | s)
− min(Q1, Q2)(s, a)]
The actor raises the lower critic’s score of the torque it samples while keeping log π low, that is, keeping its torques spread. A torque sampled straight from the bell curve has no gradient. SAC writes it as mean + sd × n, with n drawn once from a standard bell curve: for example 0.2 + 0.5 × 1.0 = 0.7. With n held fixed the torque is an ordinary function of the actor’s mean and sd, so the gradient flows from Q back into the actor (the SAC paper, eq. 11 and 12). For the torque in Figure 5 the loss is −9.245, the soft value with its sign flipped. Together: the actor-critic step of Policy Gradients with a paid spread. Builds on line 2 of the TD3 card, the entropy card and the actor-critic card; it is the last card of the model-free chain.
The soft value and target in a few lines; they return the 9.245 and 9.253 of Figure 5.
2The torque of Figure 5: critics 9.5 and 9.1, log π −0.726.
3The soft value and the soft target, line 3.
4 TD3 and SAC in practice
Runs
one update per real step from replay; TD3’s actor every second update; SAC also trains a value network in the paper; CleanRL instead tunes α
Memory and τ
SAC keeps a replay buffer of 1,000,000 steps and TD3 keeps the whole history of the run (1 million steps); both move their slow copies with τ 0.005, the share each slow copy moves toward its network per update: larger follows the networks faster but steadies the target less, 1 copies them every step. SAC trains with Adam at 0.0003, batch 256, two hidden layers of 256 units (SAC paper, Appendix D; TD3 paper, section 6.1)
TD3 noise and delay
target noise σ 0.2: larger smooths the target more but blurs real differences between torques, 0 is DDPG’s sharp target; clip c 0.5: larger lets rare big draws through, 0 removes the noise; delay d 2: larger lets the critic settle before the actor follows it, 1 is DDPG (the TD3 paper)
SAC α
larger α keeps the torques more spread and explores more; α 0 drops the payment and leaves plain reward, TD3’s goal (the SAC paper, eq. 1). The first SAC paper sets it through the reward scale; a later paper (Haarnoja et al. 2018b) learns it toward an entropy target of minus the number of torques, −4 for the Reacher arm: a narrow bell curve’s entropy is below 0, so the target asks for a narrow spread; CleanRL starts it at 1.0 and fixes it at 0.2 with tuning off
Acting after training
TD3 acts with its actor’s torque, no noise; SAC acts with the mean torque
Every method so far learns only from real steps. What if the agent also learned how the world responds, and practised inside that model? Model-Based RL: Dyna, AlphaZero and MuZero covers that family.
5 References
Lillicrap, T. P., et al. (2016). Continuous Control with Deep Reinforcement Learning. ICLR. arXiv:1509.02971.
Fujimoto, S., et al. (2018). Addressing Function Approximation Error in Actor-Critic Methods. ICML. arXiv:1802.09477.
Haarnoja, T., et al. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML. arXiv:1801.01290.
Haarnoja, T., et al. (2018b). Soft Actor-Critic Algorithms and Applications. arXiv:1812.05905.
Huang, S., et al. (2022). CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms. JMLR, 23(274). GitHub.