On this page
A language model writes a reply one word at a time. At every step a decoder network like the one in The Transformer Block scores every word, or word piece called a token, in its vocabulary. How text is cut into tokens is in Tokenizers. Decoding is the rule that turns those scores into the one word written next. Softmax first turns the scores into probabilities. The rule then either chooses the likeliest words, which gives the same reply every time, or draws a word at random, which is why the same question can get two different answers. It runs once for every generated token, and only when the model writes text. Training on fixed text never decodes, because the loss reads the true next token from the data. The exception is RL fine-tuning in Aligning Language Models with RL, where the model samples its own replies during training.
Every example uses one toy step. After the cat sat on the, a model scores ten candidate words.
Softmax is the first step of every decoding method. The raw scores are logits, any real numbers, and they do not add to anything. Softmax makes them a probability distribution, positive numbers that add to 1, while keeping their order.
Each word’s chance is its exponentiated logit as a share of the total over all words.
A logit like 1000 makes the exponential overflow to infinity, and infinity over infinity gives NaN, not a number. Subtracting the largest logit first divides the top and bottom by the same number, so every pi stays the same (Goodfellow et al. 2016, section 4.1), and the biggest term becomes 1.
The same softmax picks the weights in Attention and feeds the loss in Cross-Entropy Loss.
Greedy decoding and beam search both look for the likeliest reply, with no randomness. Greedy decoding writes the likeliest word and moves on, so a likely first word can force unlikely words after it. Beam search keeps the best few partial replies at every step, its beam width, and keeps the best reply at the end.
num_beams, default 1, which is greedy. A wider beam compares more replies at that many times the cost, and a narrower one compares fewer; at the extreme, a beam that kept every possible reply would find the reply with the highest score, but there are far too many to keep. Even width 16 still loops on open-ended text (Figure 14).Figure 3. Greedy against beam search with width 2 on a small reply tree.
Greedy picks the top word at each step. Beam search keeps the two best partial replies by summed log probability.
Every word adds a negative log, so the plain sum s favors short replies. The replies in Figure 3 all have two words, so it never shows this bias. To remove it, beam search divides the score by a power of the length.
The corrected score is the summed log probability divided by the reply length raised to a power.
length_penalty in Hugging Face, default 1.0, so its beam search divides by the generated length unless told otherwise. A larger α favors longer replies, and a smaller one favors shorter replies.Speech recognition uses the same two methods over alignments in CTC Loss.
For open-ended writing, the likeliest reply turns out to loop. Holtzman et al. (2020), the paper that introduced nucleus sampling, gave GPT-2 Large, a 774-million-parameter language model from 2019, the opening of a story and let beam search continue it.
Repetition feeds itself, because each repeat makes the next one likelier.
A repetition penalty from CTRL (Keskar et al. 2019), a 1.63-billion-parameter Salesforce language model, makes a repeat less likely by lowering the logit of every word already written.
A used word’s logit is divided by the penalty before the softmax.
repetition_penalty in Hugging Face, default 1.0, which is no penalty. A larger θ pushes harder away from used words, a smaller one pushes less, and a value below 1 would reward repeats.The penalty lowers every used word alike, so it also lowers words that should repeat, such as a name or the.
Sampling draws the next word at random in proportion to its probability, so the output varies from run to run. Pure sampling draws each word with chance pi, straight from the softmax. The catch is the tail, the many words that each have a small chance. In the toy, moon, piano, Tuesday and of make no sense after the cat sat on the. These unlikely words hold about 4% together.
A small risk per word becomes a near certainty over a long reply, and one unlikely word pulls the words after it off topic.
Temperature, τ, is one number that sets how random sampling is, by rescaling every logit.
Each logit is divided by τ before the softmax.
temperature in Hugging Face, default 1.0. Raising it flattens the distribution and lowering it sharpens it. Near 0, sampling acts like greedy. As τ grows, every gap between logits shrinks toward 0, so every word’s chance nears one over the number of words.Neither a low nor a high τ solves the tail problem. A low τ drifts back toward greedy, and a high one gives the tail more probability. The same temperature softens a teacher’s probabilities in Knowledge Distillation, which writes it T, and sets how sharply similarities become probabilities in InfoNCE and CLIP. The Transformer Block has its own temperature slider.
A cut, also called truncation, keeps only the likeliest words, sets the chance of the rest to 0 and rescales the kept ones before the draw. The simplest cut keeps a fixed number of words, and the cuts after it fix its weak spot.
Top-k keeps a fixed number of the likeliest words at every step.
Top-k keeps a word only if it is among the k likeliest.
top_k in Hugging Face, default 50. A smaller k drops good words at flat steps, and k = V, the vocabulary size, keeps every word, which is pure sampling. The nucleus sampling paper tests 40 and 640, and 10 is used for stories.One fixed k cannot fit every step, because some steps have many good words and others have one. The nucleus sampling paper says it with probability mass, meaning total probability: “Flat distributions lead to many moderately probable tokens, while peaked distributions concentrate most probability mass into just a few tokens.”
Top-p, also called nucleus sampling, fixes this by keeping a share of the total probability instead of a count.
Top-p keeps the fewest top words whose chances add up to a set share of the total probability.
top_p in Hugging Face, default 1.0, which keeps every word. A smaller ptop keeps a smaller nucleus and safer text, and near 0 only the top word stays, which is greedy. The nucleus sampling paper uses 0.95.Top-p keeps many words when the distribution is flat and few when it is peaked. The captioner in From BLIP to the Q-Former uses top-p at 90%.
Min-p (Nguyen et al. 2025) adapts in a different way. It keeps the words above a threshold, the bar, that moves with the top word’s probability.
Min-p keeps every word whose chance is at least a set fraction of the top word’s.
min_p in Hugging Face, unset by default. A smaller pbase lowers the bar and keeps more words. 0 keeps every word, which is pure sampling, and 1 keeps only words tied with the top, which is greedy. The docs suggest 0.01 to 0.2. The min-p paper suggests 0.05 to 0.1, and 0.5 to 0.7 for very high τ.All three cuts then renormalize, rescaling the kept words so they add to 1 again.
After any cut, each kept word’s chance is rescaled so the kept words add to 1.
Hugging Face, vLLM (a model-serving library) and the min-p paper apply temperature before the cut, while llama.cpp (a C/C++ library for running models locally) applies it last by default.
Figure 13. Bars show the renormalized p′. Top-k keeps a fixed count, top-p a fixed share of the probability.
A task with one right answer wants the likeliest reply, and open-ended writing wants text that is varied but stays on topic. The nucleus sampling paper measured both sides. It wrote 5,000 passages of up to 200 tokens with each method, each from a short opening of web text the model never trained on. Perplexity per token is how surprised the model is by the text, where higher means less predictable. Loops is the share of passages that end stuck repeating a phrase of two or more tokens at least three times.
In a real system the decoding rule sits inside a generate loop. Each pass runs the model on the text so far, takes the logits at the last position, applies the penalty, temperature and cut, picks one token, appends it and repeats until an end-of-text token or a length limit. A key-value cache stores the attention keys and values of every past token, so each pass computes only the new token instead of the whole text.
The toy sampling step in NumPy, from the ten logits to one drawn word.
import numpy as np
# the ten toy logits, 2.0 down to -2.6
z = np.array([20, 12, 8, 5, 1, -3, -10, -18, -22, -26]) / 10
rng = np.random.default_rng(0)
def softmax(z, tau=1.0):
e = np.exp((z - z.max()) / tau) # max first: no overflow
return e / e.sum()
def top_p(p, p_top=0.9):
order = np.argsort(p)[::-1] # likeliest first
n_keep = np.searchsorted(np.cumsum(p[order]), p_top) + 1
p_prime = np.zeros_like(p)
p_prime[order[:n_keep]] = p[order[:n_keep]]
return p_prime / p_prime.sum()
def top_k(p, k=3):
p_prime = np.where(p >= np.sort(p)[-k], p, 0) # k likeliest
return p_prime / p_prime.sum()
p = softmax(z, tau=1.0)
p_prime = top_p(p)
word_id = rng.choice(len(p), p=p_prime)
# prints [0.43 0.19 0.13] 5 3
print(p[:3].round(2), np.count_nonzero(p_prime), np.count_nonzero(top_k(p)))Greedy and beam search on the tree of Figure 3, where width 1 is greedy.
from math import log
TREE = {(): {"sat": 0.5, "is": 0.4, "ran": 0.1},
("sat",): {"on": 0.4, "down": 0.3, "there": 0.3},
("is",): {"sleeping": 0.9, "here": 0.05, "fat": 0.05},
("ran",): {"away": 0.8, "off": 0.2}}
def beam_search(width, depth):
beams = [(0.0, ())] # (s, prefix)
for _ in range(depth):
pool = []
for s, prefix in beams:
for word, P in TREE[prefix].items():
pool.append((s + log(P), prefix + (word,)))
beams = sorted(pool, reverse=True)[:width]
return beams[0]
s, words = beam_search(width=1, depth=2)
print(round(s, 2), words) # -1.61 ('sat', 'on')
s, words = beam_search(width=2, depth=2)
print(round(s, 2), words) # -1.02 ('is', 'sleeping')A hand-written PyTorch generate loop with temperature and a random draw is in The Transformer Block.
Hugging Face’s model.generate runs the loop of Section 8.1 with a key-value cache, and each call below is one method.
from functools import partial
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
inputs = tok("The cat sat on the", return_tensors="pt")
gen = partial(model.generate, **inputs, max_new_tokens=40)
greedy = gen()
beam = gen(num_beams=4)
sampled = gen(do_sample=True, temperature=0.7, top_k=0, top_p=0.9)
no_loops = gen(repetition_penalty=1.2) # greedy, CTRL penalty
print(tok.decode(sampled[0], skip_special_tokens=True))| Method | Reads | Keeps | Work per token |
|---|---|---|---|
| Greedy | last logits | nothing extra | one argmax over V |
| Beam search, width B | logits of B replies | B replies, their scores, the cache reshuffled to match | B rows per model pass, the top 2B of B × V, the length division at the end |
| Sampling | final probabilities | the random state | one softmax and one draw over V |
| Repetition penalty | logits and all tokens so far, prompt too | nothing extra | a lookup and write-back of their logits |
| Temperature, top-k, top-p, min-p | logits | nothing extra | temperature divides each logit oncetop-k runs a partial sorttop-p runs a full sort and a running summin-p takes a max and compares |
generate is greedy and stops after 20 new tokens. temperature, top_k and top_p do nothing unless do_sample is on, because Hugging Face’s generate applies them only when sampling.top_k defaults to 50. Pass top_k=0 for top-p by itself.The Hugging Face Generation docs list every argument, and the Generation strategies guide shows each method in use.
generation/utils.py and generation/logits_process.py, and it applies the repetition penalty, then temperature, before the top-k, top-p and min-p cuts, multiplies a negative logit by the penalty, and divides a finished beam’s score by its length to the power length_penalty. Argument names and defaults are in Generation, examples in Generation strategies, and output types in Utilities for Generation.LogitsProcessor interface of its V2 model runner states that custom processors run “before temperature, min_p and top-k/top-p”. docs.vllm.ai.--samplers option in its tools’ help text, default order ending in temperature. Debian manual page llama-perplexity(1). manpages.debian.org.