RLHF, reinforcement learning from human feedback, turns people's rankings of answers into a reward and trains a language model to earn it.
On this page
A large language model (LLM) is a neural network trained to predict the next token, a word or piece of a word, of web text. People want something else from it: to "follow the user's instructions helpfully and safely". Ouyang et al. 2022 call next-token prediction misaligned with that goal and built InstructGPT by tuning GPT-3 with RLHF.
Explain the theory of gravity to a 6 year old.
Explain the theory of relativity to a 6 year old in a few sentences.
Explain the big bang theory to a 6 year old.
Explain evolution to a 6 year old.
Writing the ideal answer is slow; picking the better of two takes seconds. Copying written answers alone has limits (why copying an expert falls short, on RL Basics), so RLHF also learns from people's rankings of the model's own attempts. The idea comes from Christiano et al. 2017, who trained Atari and simulated-robot agents from people’s picks between two clips.
Supervised fine-tuning (SFT) teaches by example, a reward model (RM) learns a score from people's rankings, and RL with PPO, proximal policy optimization (Schulman et al. 2017), raises that score while a KL penalty charges the model for moving away from the SFT model. The stages run in order, and stages 2 and 3 can repeat on rankings of the newest policy’s answers (the InstructGPT paper).
A model writing an answer runs the reinforcement learning loop, and each token it writes is one move.
The score arrives once per answer, which ends the episode (InstructGPT paper, section 3.5). Sampling a token has no gradient, so RLHF uses a policy-gradient update, the REINFORCE idea that PPO refines in stage 3: raise the log-probability of tokens in better-than-expected answers and lower it in worse ones.
In this code, GPT-2 from Hugging Face writes one answer, one token per loop pass; the reward model would score the finished state once.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MAX_TOKENS = 40
tok = AutoTokenizer.from_pretrained("gpt2")
policy = AutoModelForCausalLM.from_pretrained("gpt2")
END_ID = tok.eos_token_id
state = tok("Explain the moon landing.", return_tensors="pt").input_ids
with torch.no_grad():
for _ in range(MAX_TOKENS):
logits = policy(state).logits[0, -1] # next-token scores
probs = torch.softmax(logits, dim=-1)
token = torch.multinomial(probs, 1) # the move
state = torch.cat([state, token[None]], dim=1)
if token.item() == END_ID:
break
The pretrained model is fine-tuned on answers written by hired labelers, with the usual next-token loss, for 16 epochs.
The reward model, a language model whose last layer outputs one number, learns to score answers the way people rank them. Every pair in a labeler’s ranking becomes a training example. It trains for a single epoch, since more epochs quickly overfit.
Take the prompt of Figure 2 and its two answers, yw the InstructGPT answer, which people prefer, and yl the GPT-3 list, as if these were the only two answers. The reward model scores them 1.5 and 0.3; it must turn that into a chance that people prefer the first (Figure 5).
A toy reward model on 8 random pairs, each answer 16 made-up numbers instead of text.
import torch
import torch.nn.functional as F
N_FEATURES, N_PAIRS = 16, 8
reward_model = torch.nn.Linear(N_FEATURES, 1) # toy RM
chosen = torch.randn(N_PAIRS, N_FEATURES) # y_w; rejected is y_l
rejected = torch.randn(N_PAIRS, N_FEATURES)
r_w = reward_model(chosen).squeeze(-1)
r_l = reward_model(rejected).squeeze(-1)
gap = r_w - r_l
loss = -F.logsigmoid(gap).mean() # stable -log sigmoid
loss.backward()
The policy, a copy of the SFT model, answers a fresh prompt, the frozen reward model scores it, and PPO updates the policy to raise the score. InstructGPT ran about 256,000 such episodes.
Training moved the policy away from the SFT model; one answer’s drift can even be negative, the average cannot; both are measured in nats, the unit the natural log gives (Figure 6).
Toy numbers for one 12-token answer.
import torch
BETA, N_TOKENS = 0.02, 12 # InstructGPT's beta
log_pi = -torch.rand(N_TOKENS) # toy log pi per token
log_pi_sft = -torch.rand(N_TOKENS)
score = torch.tensor(1.3)
drift = log_pi - log_pi_sft
reward = -BETA * drift
reward[-1] += score # score on the last token
objective = reward.sum()
If training finds the best policy, with β 1 it writes yw 77 times in 100, the same split as Figure 5; with β 0.1, almost always.
| Runs | sample a batch of 512 answers, score each once, one PPO epoch in minibatches of 64, repeat |
|---|---|
| Carries | the policy, a critic, the frozen SFT model and the frozen reward model |
| Cost | per round, generate 512 whole answers, run policy, critic, SFT copy and reward model on each, then 8 PPO minibatch steps |
| β | weights KL in the objective; higher stays near SFT, 0 lets the policy exploit the reward model’s flaws, very high leaves SFT unchanged; 0.02 in InstructGPT |
| Code | Hugging Face TRL PPO trainer |
| Gotchas | watch the logged KL (objective/kl in TRL); a rising reward with fast-rising KL is the failure in Section 4 |
Labelers preferred InstructGPT’s answers to GPT-3’s (Figure 8), with the setup below.
| Part | InstructGPT setup |
|---|---|
| People | about 40 hired labelers |
| Prompts | sent by customers to earlier InstructGPT models on the OpenAI API (OpenAI's paid web service), plus some written by the labelers |
| Models | GPT-3 architecture at 1.3B, 6B and 175B parameters (B = billion) |
| Reward model | one 6B reward model for every policy size |
| Judged by | the same labelers, on prompts from customers held out of training |
| Final models | PPO plus a next-token loss on pretraining text (PPO-ptx), used for the results here; the best-policy line in Section 2.4 ignores this extra loss |
Labelers also preferred 1.3B InstructGPT over 175B GPT-3, and held-out labelers, who wrote no training data, agreed about as often.
InstructGPT aligns to one group, mostly the labelers and researchers, and still makes simple mistakes.
A reward model is a learned reward with flaws, and stage 3 finds and exploits them: reward hacking, as with the toy agent on RL Basics. Gao et al. 2022 measured this with a reward model standing in for people:
| Part | Over-optimization setup |
|---|---|
| Stand-in for people | a fixed 6B "gold" reward model in the InstructGPT setting |
| Proxy reward models (what training sees) | 3M to 3B parameters (M = million), trained on the gold model's preferences |
| Policy | 1.2B parameters, trained with PPO against one proxy, no KL penalty in the main runs |
| Measured | gold and proxy reward model scores as the policy drifts from its starting model, drift measured by KL divergence |
A KL penalty did not lift the gold curve: at the same KL distance the gold score came out about the same, so the penalty acted like stopping early.
Answer length is one measured flaw. PPO lengthened answers on all three preference datasets Singhal et al. 2023 tested, among them the WebGPT comparisons, people’s picks between two web-search answers. Judged by OpenAI models standing in for people (AlpacaFarm), a reward for length alone nearly matched standard PPO.
Newer methods drop parts of the RLHF pipeline (Figure 11).
DPO, direct preference optimization (Rafailov et al. 2023), trains the language model directly on preference pairs collected beforehand, sampling nothing during training.
Read the best-policy line backwards and the policy’s own drift is a reward (Figure 12).
One DPO loss on a batch of pairs. Each input is the summed log-probability of a whole answer; the ref_ ones come from the frozen SFT model without gradients.
import torch.nn.functional as F
BETA = 0.1 # the DPO paper's value
def dpo_loss(logp_w, logp_l, ref_logp_w, ref_logp_l):
r_w = BETA * (logp_w - ref_logp_w) # r_DPO(x, y_w)
r_l = BETA * (logp_l - ref_logp_l) # r_DPO(x, y_l)
gap = r_w - r_l
return -F.logsigmoid(gap).mean()
| Runs | ordinary training passes over a fixed set of pairs; batch 64, RMSprop learning rate 0.000001, 150 warmup steps (the paper) |
|---|---|
| Carries | a frozen reference, run on both answers of each pair; TRL can precompute it (precompute_ref_log_probs) |
| Cost | per pair, two forward passes of the policy and two of the reference, one backward pass, no generation |
| β | scales rDPO; higher keeps the policy near the reference, near 0 the policy can move far from the reference; 0.1 in the paper |
| Code | Hugging Face TRL DPO trainer |
| Gotchas | the gap can grow by lowering the rejected answer’s probability; watch rewards/margins in TRL |
GRPO, group relative policy optimization (Shao et al. 2024, the DeepSeekMath paper), is PPO with its clipped step kept and the critic dropped. It samples a group of answers to one question and judges each against the group.
GRPO judges each answer only against its siblings (Figure 13).
One GRPO loss for one question: the group’s rewards become Ai, and the rest is PPO’s clipped objective per token, without the KL term.
import torch
EPS_CLIP = 0.2 # PPO's clip
def grpo_loss(log_pi, old_log_pi, rewards):
# log_pi, old_log_pi: (G, T) per-token log-probs
A = (rewards - rewards.mean()) / (rewards.std() + 1e-8)
A = A[:, None] # same A_i for every token of answer i
ratio = torch.exp(log_pi - old_log_pi.detach())
clipped = ratio.clamp(1 - EPS_CLIP, 1 + EPS_CLIP)
objective = torch.min(ratio * A, clipped * A)
return -objective.mean()
| Runs | sample 64 answers per question up to 1,024 tokens, score them, then a single update per sampling round; batch 1,024 (DeepSeekMath) |
|---|---|
| Carries | no critic; the sampling policy’s log-probabilities and a reference for the KL term |
| Cost | generating 64 answers per question and a reward for each, then one backward pass per round |
| Group size G | answers per question; more gives a steadier group average, 1 gives no comparison at all; 64 in DeepSeekMath |
| KL β | weights the KL term like RLHF’s β; higher stays closer to the reference, 0 turns the term off; 0.04 in DeepSeekMath, 0 by default in TRL |
| Clip ε | as in PPO: wider allows bigger changes per round, 0 freezes the policy; 0.2 in the PPO paper |
| Code | Hugging Face TRL GRPO trainer |
| Gotchas | a group where every answer scores the same gives A = 0 and teaches nothing (TRL logs frac_reward_zero_std); dividing by std can bias toward question difficulty (TRL scale_rewards) |
When an answer can be checked automatically, a rule replaces the reward model. DeepSeek-R1-Zero (DeepSeek-AI 2025) used GRPO and checked a final math answer against the known one, ran code against set test cases, and required reasoning inside marked tags, with no learned reward model, since a learned one can be gamed.
In the on-policy and off-policy terms of Value-Based RL, PPO-based RLHF and GRPO are on-policy, learning from answers the current policy just wrote, while DPO learns offline from a fixed set of pairs (the GRPO paper).