On this page
What does a language model actually read? Not letters. A tokenizer cuts the text into pieces called tokens, looks each one up in a fixed list, the vocabulary, and hands the model one integer per token, its token id. The id picks one row of the model’s embedding table (The Transformer Block). Text to ids is encoding, ids back to text is decoding. V is the number of vocabulary entries and T the number of tokens in one input.
Long pieces give a short T but need a huge V; short pieces need only a small V but give a long T.
One token per word gives the shortest T. But the list of words never ends: names, typos, new words, other languages. A real system caps the list and maps every other word to one unknown token, <unk>, which the model can neither read nor write. Sennrich, Haddow and Birch (2016) note that word vocabularies in neural machine translation, models that translate text, are usually capped somewhere between 30,000 and 50,000 words.
With one token per character or byte, nothing is unknown. UTF-8 writes every character as 1 to 4 bytes, and a byte has only 256 values. The price is a T several times longer, and attention, which compares every token with every other token (Attention), does work that grows with the square of T (Transformer Cost and Efficiency).
a takes 1 byte, é 2 and 誕 3, so the 9-character greeting takes 27 bytes.Subword tokenizers keep frequent words whole and split rare words into frequent pieces, so nothing is unknown and T stays short.
A bigger vocabulary costs parameters and buys a shorter sequence.
The next-token loss is a categorical cross-entropy over the V entries (Cross-Entropy Loss), so V is also the number of classes.
The tokenizer is trained once, before the model, on a large collection of text called a corpus. The result is a fixed file that holds the vocabulary and the rules for splitting text with it (splitting new text with the merge list shows them for BPE). The model’s embedding rows are tied to these exact ids, so a new tokenizer means training the model, or at least its embeddings, again. API use “is priced by token” (Reference 12), so a tokenizer that needs more tokens for a text makes it cost more and fit less.
Each call encodes one text on its own, before the model runs. During generation the tokenizer only decodes each new id. It keeps no state between calls: unless sampled splits are turned on while a model trains (sampling a different split each time), the same text always gives the same ids.
BPE and WordPiece grow a vocabulary by merging pieces (byte-level BPE is BPE run on bytes), Unigram shrinks a large list by pruning, and SentencePiece is a toolkit that runs either kind on raw text.
BPE, WordPiece and Unigram are each built step by step in How Subword Tokenizers Are Built: BPE, WordPiece and Unigram.
| Term | What it means | Example or where used |
|---|---|---|
| merge | BPE joins the adjacent pair of pieces that appears most often into one new token, and repeats; byte-level BPE starts from the 256 byte values | u g becomes ug |
| score | WordPiece merges like BPE but picks the pair that occurs together more often than its parts’ own counts predict: the pair’s count divided by the product of its two parts’ counts | BERT |
| pruning | Unigram starts from a big list of pieces and drops the ones the corpus needs least | T5 |
## | WordPiece’s mark on a piece that continues a word | anti ##dis |
[UNK], <unk> | the unknown token, for text the vocabulary cannot spell; BERT writes [UNK] | a word with no matching piece |
▁ | SentencePiece writes each space as this visible symbol, on the front of the next piece | ▁is |
| byte fallback | a character missing from the vocabulary is written as its UTF-8 bytes instead of the unknown token | 誕 as 3 bytes |
| byte fragment | a byte-level token that holds only part of a character, not text on its own; hatched in Figure 12 | 1 of the 3 bytes of 誕 |
Released vocabularies grew from about 30K entries to about 262K, and byte-level BPE is the most common choice among them. The 2025 releases keep the same families: Qwen3 uses byte-level BPE and Gemma 3 uses SentencePiece (References 18, 19).
tiktoken, OpenAI’s tokenizer library, names OpenAI’s byte-level BPE vocabularies by encoding: r50k_base, the same vocabulary as GPT-2’s, for the original GPT-3, p50k_base for Codex and text-davinci-002 and text-davinci-003, cl100k_base for GPT-3.5 and GPT-4, and o200k_base for GPT-4o (References 12, 13). It maps GPT-5 to o200k_base and the open gpt-oss models to o200k_harmony. o200k_harmony keeps the same merges and adds special tokens, entries that mark structure rather than text (Section 5.6); 1,081 of its 201,088 ids are reserved placeholders (Reference 13).
A vocabulary learned from frequent text decides how numbers, other languages and spaces are split, and the model sees only those splits.
r50k_base and p50k_base learned number tokens by plain BPE, so which numbers get one token is an accident; Singh and Strouse (2024) note that in p50k_base “710 may get a dedicated token, while 711 will not”. cl100k_base and o200k_base cut digit runs into chunks of at most three digits, left to right, with no space in front (Reference 13). Llama 2 and Gemma split every number into single digits (References 6, 8).
The paper gave gpt-3.5-turbo and GPT-4 (March 2023 versions) 90 addition problems with 7 to 9 digit numbers and 8 solved examples in the prompt (8-shot), always took the most likely next token (greedy decoding), and changed only the digit grouping.
A tokenizer trained mostly on English has long tokens for English and short pieces for other scripts. Petrov et al. (2023) counted tokens for the same sentences translated into each language and divided by the English count.
cl100k_base narrows the gap. Data from Reference 15.In GPT tokenizers “spaces are usually grouped with the starts of words” (Reference 12), so the same word is a different token with and without a space in front. SentencePiece does the same with ▁.
r50k_base, 2 (1) and 2 after a space (2) are different tokens; o200k_base gives that space its own token (3).A byte-level BPE vocabulary holds the 256 byte values plus one entry per merge (counting pairs and merging). To encode, the text is first cut into chunks (runs of letters, digits or other symbols, each with the space before it, and endings like 's), and no merge crosses a chunk border. Each chunk starts as single bytes, and the merges are replayed inside it in the order they were learned (splitting new text with the merge list). GPT-2 keeps 50,000 merges; GPT-4o’s tokenizer, about 200,000 entries, splits “tiktoken is great!” into six tokens (Figure 1). Keeping fewer merges gives a smaller V and, usually, a longer T, the trade-off of Figure 3. A merge exists only for byte pairs that were frequent in the training text.
Figure 12. Each added entry joins two tokens, so T (2) never grows with V.
Rumbelow and Watkins (2023) found tokens in the 50,257-token vocabulary shared by GPT-2 and GPT-3 that make models behave strangely. One explanation they suggest is that these tokens were rarely or never seen in training, so their embeddings stayed close to their starting values.
Special tokens are vocabulary entries that never come from text; they mark structure. GPT-2’s <|endoftext|> is id 50256, and BERT wraps every input in [CLS] and [SEP]. The tokenizer must match the model: id 2212 means ␣great only under o200k_base. Chat models are trained on conversations wrapped in control tokens, special tokens that mark who is speaking. A chat template writes them, and with the wrong ones models perform “drastically worse” (Reference 2).
CTC adds a blank token to its vocabulary (CTC Loss), a special token that never comes from text, like the ones here. The Q-Former’s outputs enter the language model as soft prompts, vectors that act like extra tokens without ids, placed before the text token embeddings (From BLIP to the Q-Former).
You load the tokenizer that ships with the model; you train one only when you pretrain a model (training your own vocabulary). The official code lives in tiktoken, Hugging Face tokenizers and SentencePiece.
Code 1 loads GPT-4o mini’s encoding, turns the string of Figure 1 into ids and back, and counts tokens with a named encoding.
import tiktoken
encoding = tiktoken.encoding_for_model("gpt-4o-mini")
ids = encoding.encode("tiktoken is great!")
print(ids) # [83, 8251, 2488, 382, 2212, 0]
print(encoding.decode(ids)) # tiktoken is great!
print([encoding.decode_single_token_bytes(t) for t in ids])
# [b't', b'ikt', b'oken', b' is', b' great', b'!']
def num_tokens_from_string(string, encoding_name):
encoding = tiktoken.get_encoding(encoding_name)
return len(encoding.encode(string))
print(num_tokens_from_string("tiktoken is great!", "o200k_base")) # 6encoding_for_model picks o200k_base for gpt-4o-mini.b' is'. Decoding a single token can be lossy when it is not a whole UTF-8 character, so this call returns bytes.Code 2 loads the tokenizer that ships with a chat model and builds the prompt of Figure 14.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-beta")
messages = [
{"role": "system", "content": "You are a friendly chatbot who always responds in the style of a pirate"},
{"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(prompt)
prompt_ids = tok(prompt, add_special_tokens=False)["input_ids"]add_generation_prompt=True adds the last line.add_special_tokens=False stops them being added twice.Every tokenizer here works from a vocabulary learned once from a corpus. How is that list of pieces chosen, and why do BPE, WordPiece and Unigram split the same word differently? How Subword Tokenizers Are Built: BPE, WordPiece and Unigram builds each one step by step.
bert-base-uncased tokenizer.json, Hugging Face Hub.