← All tutorials

RL Basics

No answers, just a score. The agent figures out the rest.

On this page

Reinforcement learning trains an agent’s strategy for picking moves to collect as much total reward as it can. There are no labels and no examples of correct behavior, just a world, a few possible moves and a reward; the agent tries things, the world returns rewards, and the strategy slowly shifts toward the moves that earn them.

Whenever the right move is unknown but the score is clear, RL is the tool.

1 The reinforcement learning loop

Each turn the agent sees the state, what it senses right now. Its policy, the strategy being learned (often a neural network), picks an action. The world answers with a reward, the score for what just happened, and the next state, the world after the action lands. The goal is the most total reward over time.

2 The banana collector environment

Walk the loop with the banana collector, a Unity ML-Agents environment from my Value-based-methods repo. ML-Agents is the toolkit that connects a Unity game world to training code. Each step the Python side sends one of four moves; Unity steps the physics and sends back 37 numbers, the next state, plus a reward: +1 for a yellow banana, −1 for a blue one, 0 otherwise.

The actions are a list of four, not a continuous range: forward, back, left, right.

This snippet plays one episode, one game from reset until the world reports done, with a random policy that learns nothing. Each pass of the while loop is one turn, and env.step returns the next state and the reward.

import random
from unityagents import UnityEnvironment

env = UnityEnvironment(file_name="Banana.app")
brain = env.brain_names[0]
info = env.reset(train_mode=True)[brain]
state = info.vector_observations[0]   # 37 numbers
score = 0
while True:
    action = random.randrange(4)      # random policy
    info = env.step(action)[brain]
    state = info.vector_observations[0]
    reward = info.rewards[0]          # +1, -1 or 0
    score += reward
    if info.local_done[0]:
        break
print("score", score)

1. 1Starts an episode by resetting the banana world and reading the first state, the 37 numbers in Figure 3.
2. 2One turn of the loop in Figure 2, a random move and its reward.
3. 3Ends the episode when the world reports done, then prints the score.

3 Why copying an expert is not enough

The obvious alternative is behavior cloning: train a network to copy an expert’s moves. It is stage 1 of RLHF (reinforcement learning from human feedback) in Aligning Language Models with RL. The clone only saw perfect play, so once it drifts where the expert never went, its mistakes compound. RL learns from its own failures.

4 Why RL is hard

Three things make RL hard. First, credit assignment. The agent turns left at step 40, drifts forward, and eats the banana at step 57. The +1 lands at 57, but the decision that mattered was the turn at 40. The discount gives earlier steps a share of that reward, smaller for each step of delay. Sparse rewards, long stretches that pay nothing, make every credit trail longer.

One banana at step 57 is worth less the further back you stand (Figure 5).

Second, exploration. You cannot learn what you never try, and trying costs blue bananas, so the agent explores most early on (Section 6).

Third, sample inefficiency. Learning from scratch takes many tries. My banana run played 1,600 episodes for a task with four moves (Value-based-methods repo).

5 Scoring each move with Q-values

A Q-value, Q for quality, is the agent’s guess of the return from one move in one state. This agent plays forward 5 times in 8 and each other move 1 time in 8 (the ε-greedy rule of Section 6 at ε 0.5); the returns of single plays vary, so it keeps averages (Figure 7).

Training aims at a different Q: the return when every later move is the best one. Its guesses sit in a table, and the table’s guess for forward is 0.82; one real step says more (Figure 8).

The step earns reward +1 and uses γ 0.99, my DQN agent’s setting. The best Q in the next square is 0.50, so best play afterwards is worth 0.99 × 0.50 = 0.495, more than this agent’s own 0.4 in Figure 7, and the target is 1 + 0.495 = 1.495.

A table needs one row per state, and 37 continuous numbers never repeat, so a network replaces the table.

6 Explore early, exploit late

The agent’s Q-values say forward has been the best move so far. Always stepping forward means never learning that right might be better; picking at random every turn means never using what it knows.

The standard fix is ε-greedy: with chance ε (epsilon, eps in code) play a random move, otherwise the move with the highest Q-value. With ε 0.5 forward is played half the time plus a quarter of the random half, 5 times in 8, the chances in Figure 7.

import random
import numpy as np

N_MOVES = 4
EPS_FLOOR, EPS_DECAY = 0.01, 0.998   # as in my run
eps = 1.0

def epsilon_greedy(q_values, eps):
    if random.random() < eps:
        return random.randrange(N_MOVES)   # explore
    return int(np.argmax(q_values))       # exploit

q_values = np.array([0.82, 0.10, -0.40, 0.05])
action = epsilon_greedy(q_values, eps)
eps = max(EPS_FLOOR, eps * EPS_DECAY)  # after each episode

1. 1Sets ε to start at 1 and decay to a floor of 0.01, the curve in Figure 10.
2. 2A random move with probability ε, else the move with the top Q-value.
3. 3Picks one move from toy Q-values, then shrinks ε once per episode.

7 One RL step on a toy banana field

8 The trained banana agent

My agent is a Dueling Double DQN (Deep Q-networks in Value-Based RL): dueling splits each score into how good the state is plus how much better each move is. It trains on batches drawn from replay memory, a store of past steps. For the Section 5 target, the learning network picks the next move and the target network, a slowly updated copy, scores it. Double stops the max from picking scores that came out too high (Value-Based RL, section 5).

import torch
import torch.nn.functional as F

GAMMA, TAU = 0.99, 1e-3   # values in my agent.py

def learn(q_local, q_target, optimizer, batch):
    s, a, r, s_next, done = batch   # 64 replayed steps
    with torch.no_grad():
        a_star = q_local(s_next).argmax(1, keepdim=True)   # learning net picks
        best_next = q_target(s_next).gather(1, a_star)     # target net scores
        target = r + GAMMA * best_next * (1 - done)
    q_sa = q_local(s).gather(1, a)   # Q(s, a) of moves played
    loss = F.mse_loss(q_sa, target)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    for p, p_t in zip(q_local.parameters(), q_target.parameters()):
        p_t.data.copy_(TAU * p.data + (1 - TAU) * p_t.data)

1. 1Double DQN target for 64 replayed steps, eq card in Section 5, line 2, with the best next move picked by the learning network.
2. 2Compares Q(s, a) with the target, the TD error, line 3.
3. 3Takes one gradient step on that loss.
4. 4Moves the target network a fraction τ toward the learning network.

The training code (env.py) stops early once the 100-episode average score reaches 17.0. The run never hit that bar, so all 1,600 episodes ran (the DQN page).

9 Reward hacking

The agent does not learn what you meant. It learns what you rewarded. Reward hacking is an agent raising its reward in a way the designer did not intend. An OpenAI agent playing the boat-racing game CoastRunners learned to circle a lagoon knocking over three respawning targets, kept catching fire, and still outscored human players by about 20 percent instead of finishing the race (Clark and Amodei 2016).

So before training, ask "what is the easiest way to score high without doing the task?"

Reward shaping, extra pay for getting closer, carries the same risk (an agent can approach and back away forever), and so do learned rewards, where it is called reward over-optimization in RLHF.

Watch what the agent actually does, then fix the reward, not the agent.

An agent that sits in a corner for 600 steps eats nothing: 0 by the banana count, 60 at 0.10 per step alive (Figure 13).

10 Three algorithms, one loop

Three agents built on this loop live in my Value-based-methods, RF-DDPG and TennisGym repos. The banana agent picks from four moves; a robot arm and a pair of tennis rackets must output real numbers (Figure 14).

The DDPG page shows the actor and critic code.

These three agents are one value-based and two actor-critic methods. What other families are there, and how do value-based methods learn? Value-Based RL maps them all, teaches the value-based ones and ends with a guide to choosing a method.

11 References

1. Sutton, R. S., and Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press. Book site.
2. Unity ML-Agents Toolkit, GitHub.
3. Value-based-methods, my banana DQN repo (env.py runs up to 1,600 episodes and stops early at a 100-episode average of 17.0).
4. TennisGym, my tennis repo (one shared actor-critic for both rackets).
5. RF-DDPG, my DDPG repo (solves the 20-arm Reacher task).
6. Clark, J., and Amodei, D. (2016). Faulty reward functions in the wild. OpenAI. https://openai.com/index/faulty-reward-functions/