Tokenization is the first thing that happens to your prompt and the last thing that happens to a model’s output. Understanding how does tokenization work lets you debug context window limits, estimate costs, and avoid subtle bugs when switching models. This guide walks through the mechanics, the algorithms, and the practical consequences for anyone building on top of LLMs.
What tokenization actually does
A tokenizer maps raw text to a sequence of integers — token IDs — that the model can process. Each integer indexes into the model’s embedding matrix. The mapping is deterministic and reversible (mostly): given a token ID sequence, you can reconstruct the original text, modulo normalization choices like Unicode handling.
The vocabulary size is fixed at training time. GPT-4o uses ~200k tokens. Llama 3 uses 128k. Smaller vocabularies mean longer sequences for the same text; larger vocabularies increase embedding matrix size and softmax computation. Every model ships with its own tokenizer. You cannot mix them.
# Different models, different tokenizations for the same text
text = "tokenization"
# GPT-4o (o200k_base)
# [9468, 42292] -> 2 tokens
# Llama 3 (tokenizer.model)
# [13522, 29892] -> 2 tokens
# But the IDs mean completely different things
# and the boundaries often differThe three main approaches
Tokenization strategies fall into three categories. Modern LLMs almost exclusively use subword methods.
Character-level
Each character becomes a token. Vocabulary is tiny (256 for bytes, ~100k for full Unicode). Sequences become extremely long — English averages ~5-6 characters per word, so a 4k token context window holds only ~800 words. Rarely used for LLM training today, but useful for byte-level fallbacks.
Word-level
Each word gets a unique ID. Vocabulary explodes (hundreds of thousands to millions). Out-of-vocabulary words break the system. Fast to train, but poor generalization to unseen words and massive embedding tables. Used in early NLP (word2vec era), not in modern LLMs.
Subword (BPE, WordPiece, Unigram)
The sweet spot. Frequent words stay single tokens. Rare words split into meaningful pieces. Vocabulary stays manageable (32k-256k). Handles any Unicode text. This is what every production LLM uses.
How BPE works
Byte Pair Encoding (BPE) is the most common subword algorithm. It starts with a character vocabulary, then iteratively merges the most frequent adjacent pair.
# Simplified BPE training loop
from collections import Counter
def train_bpe(corpus: list[str], vocab_size: int) -> dict[tuple[int, ...], int]:
# Start with byte-level vocabulary (256 tokens)
vocab = {bytes([i]): i for i in range(256)}
merges = {}
# Pre-tokenize: split on whitespace, keep punctuation attached
words = []
for text in corpus:
words.extend(text.split())
# Count word frequencies
word_freqs = Counter(words)
# Represent each word as sequence of byte IDs
word_splits = {word: [bytes([b]) for b in word.encode("utf-8")]
for word in word_freqs}
next_id = 256
while next_id < vocab_size:
# Count all adjacent pairs across the corpus
pair_counts = Counter()
for word, freq in word_freqs.items():
split = word_splits[word]
for i in range(len(split) - 1):
pair = (split[i], split[i + 1])
pair_counts[pair] += freq
if not pair_counts:
break
# Merge most frequent pair
best_pair = pair_counts.most_common(1)[0][0]
new_token = best_pair[0] + best_pair[1]
merges[best_pair] = next_id
vocab[new_token] = next_id
next_id += 1
# Update all word splits
for word in word_splits:
split = word_splits[word]
new_split = []
i = 0
while i < len(split):
if i < len(split) - 1 and (split[i], split[i + 1]) == best_pair:
new_split.append(new_token)
i += 2
else:
new_split.append(split[i])
i += 1
word_splits[word] = new_split
return vocab, mergesAt inference time, you apply the learned merges in order:
def encode_bpe(text: str, merges: dict[tuple[bytes, bytes], int]) -> list[int]:
# Start with bytes
tokens = [bytes([b]) for b in text.encode("utf-8")]
# Apply merges in priority order (earlier merges = higher priority)
for (a, b), new_id in sorted(merges.items(), key=lambda x: x[1]):
i = 0
new_tokens = []
while i < len(tokens):
if i < len(tokens) - 1 and tokens[i] == a and tokens[i + 1] == b:
new_tokens.append(a + b)
i += 2
else:
new_tokens.append(tokens[i])
i += 1
tokens = new_tokens
# Convert to IDs (requires full vocab mapping)
return [vocab[token] for token in tokens]Real implementations (tiktoken, Hugging Face tokenizers) optimize this heavily with tries and pre-tokenization regexes. The principle is the same.
WordPiece and Unigram
BPE is greedy — it always merges the most frequent pair. WordPiece (used by BERT) chooses the merge that maximizes likelihood of the training data. Unigram (used by T5, XLNet, Llama) starts with a large vocabulary and prunes tokens that least reduce likelihood. Both produce slightly different segmentations but serve the same purpose.
# Hugging Face tokenizers expose the algorithm choice
from tokenizers import Tokenizer, models, trainers, pre_tokenizers
# BPE (GPT-style)
tokenizer = Tokenizer(models.BPE())
trainer = trainers.BpeTrainer(vocab_size=32000, special_tokens=["<pad>", "<s>", "</s>"])
# WordPiece (BERT-style)
tokenizer = Tokenizer(models.WordPiece(unk_token="[UNK]"))
trainer = trainers.WordPieceTrainer(vocab_size=32000, special_tokens=["[PAD]", "[CLS]", "[SEP]"])
# Unigram (Llama/T5-style)
tokenizer = Tokenizer(models.Unigram())
trainer = trainers.UnigramTrainer(vocab_size=32000, special_tokens=["<pad>", "<s>", "</s>"])Special tokens and chat templates
Every tokenizer reserves IDs for control signals. These are not compressible text — they are protocol.
# Common special tokens across models
special_tokens = {
"pad_token": "<pad>", # Padding for batch alignment
"bos_token": "<s>", # Beginning of sequence
"eos_token": "</s>", # End of sequence
"unk_token": "<unk>", # Unknown (rare with subword)
"mask_token": "<mask>", # For masked language modeling
}
# Chat templates add structure tokens
# Llama 3 chat template (simplified)
"""
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
Hello<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>
"""The chat template is part of the tokenizer configuration, not the model weights. Apply it correctly or the model will hallucinate structure.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "How does tokenization work?"},
]
# This applies the chat template and tokenizes
input_ids = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True, # Adds assistant header for generation
return_tensors="pt"
)Practical implications for engineers
Context window accounting
Token counts determine what fits. A 128k context window means 128k token IDs, not characters or words. English averages ~1.3 tokens per word. Code averages ~1.5-2 tokens per word (more punctuation, identifiers split aggressively).
def estimate_tokens(text: str, model: str = "gpt-4o") -> int:
import tiktoken
encoding = tiktoken.encoding_for_model(model)
return len(encoding.encode(text))
# Rough guidelines
# 1k tokens ≈ 750 English words ≈ 500 words of code
# 4k tokens ≈ 3k English words ≈ 2k words of code
# 128k tokens ≈ 100k English words ≈ 65k words of codeCost estimation
Providers bill by input and output tokens. Know your tokenizer.
# Pricing example (hypothetical)
PRICING = {
"gpt-4o": {"input": 5.00, "output": 15.00}, # per 1M tokens
"gpt-4o-mini": {"input": 0.15, "output": 0.60},
}
def estimate_cost(prompt_tokens: int, completion_tokens: int, model: str) -> float:
rates = PRICING[model]
return (prompt_tokens * rates["input"] + completion_tokens * rates["output"]) / 1_000_000Tokenization drift across models
Switching models changes token counts. A prompt that fits in 4k tokens on GPT-3.5-turbo might exceed 4k on Llama 3 because of different vocabulary choices. Always re-measure.
prompt = "Explain how does tokenization work in LLMs"
import tiktoken
from transformers import AutoTokenizer
gpt4o_enc = tiktoken.encoding_for_model("gpt-4o")
llama3_tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
print(f"GPT-4o: {len(gpt4o_enc.encode(prompt))} tokens")
print(f"Llama 3: {len(llama3_tok.encode(prompt))} tokens")
# Output differs — plan for itCommon pitfalls
Assuming whitespace equals token boundaries
Subword tokenizers split inside words. “tokenization” → [“token”, “ization”] or [“tok”, “en”, “ization”]. Never split on spaces and assume alignment.
# WRONG: assuming word-level alignment
words = text.split()
token_ids = tokenizer.encode(text)
for word, token_id in zip(words, token_ids): # Breaks immediately
...
# RIGHT: use tokenizer's offset mapping
encoding = tokenizer(text, return_offsets_mapping=True)
for (start, end), token_id in zip(encoding.offset_mapping, encoding.input_ids):
span = text[start:end]
...Ignoring the pre-tokenizer
Most tokenizers run a regex pre-tokenization step before BPE. GPT-4o’s o200k_base uses a regex that preserves contractions, handles numbers, and splits on specific punctuation. This affects token boundaries.
# tiktoken exposes the pre-tokenization regex pattern
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
print(enc._pat)
# Regex pattern that determines initial splits before BPE mergesTreating token IDs as portable
Token ID 42 means something completely different in every vocabulary. Never store raw token IDs in databases or logs expecting them to be reusable across models. Store text, or store (model_name, token_ids) pairs.
Forgetting the byte fallback
All modern tokenizers can represent any Unicode text via byte fallback. But the byte-level tokens (IDs 0-255 in many vocabularies) are inefficient — one byte per token. Text in unsupported scripts balloons token count.
# English: ~1.3 tokens/word
# Chinese: ~1.5-2 tokens/character (no spaces, each char ~1 token)
# Emoji: often 2-3 tokens each (byte fallback sequences)
# Code: identifiers split aggressively (snake_case → multiple tokens)Tradeoffs to consider
Vocabulary size vs. sequence length
Larger vocabulary → shorter sequences → more context fits → larger embedding matrix → slower softmax. Llama 3 chose 128k vocab. GPT-4o chose ~200k. There’s no free lunch.
Training data language mix
A tokenizer trained on 90% English will segment English efficiently but fragment other languages. Multilingual models (mT5, XLM-R, Llama 3) use larger vocabularies and balanced training data to mitigate this.
Adding tokens post-training
You can extend a tokenizer’s vocabulary (new special tokens, domain terms) but the model’s embedding matrix must also expand and the new embeddings initialized randomly. This requires continued training. Don’t do this lightly.
# Adding tokens to a Hugging Face tokenizer
tokenizer.add_special_tokens({"additional_special_tokens": ["<CUSTOM>"]})
# Model embeddings must be resized to match
model.resize_token_embeddings(len(tokenizer))
# New embeddings are random — model needs fine-tuningTokenizer versioning
Tokenizers have versions. cl100k_base (GPT-3.5/4) vs o200k_base (GPT-4o) produce different tokenizations for the same text. Pin your tokenizer version in production.
# Explicit version pinning
import tiktoken
enc = tiktoken.get_encoding("cl100k_base") # Not "encoding_for_model"
# Or with transformers
tokenizer = AutoTokenizer.from_pretrained(
"meta-llama/Meta-Llama-3-8B",
revision="main" # Pin to specific commit/hash in production
)Debugging tokenization issues
When something looks wrong — costs spike, context overflows, output quality drops — inspect the tokens.
def debug_tokenization(text: str, tokenizer) -> None:
encoding = tokenizer(text, return_offsets_mapping=True)
print(f"Text: {repr(text)}")
print(f"Token count: {len(encoding.input_ids)}")
print("Tokens:")
for i, (token_id, (start, end)) in enumerate(zip(encoding.input_ids, encoding.offset_mapping)):
token_text = text[start:end] if start < end else "<special>"
print(f" [{i}] ID={token_id:6d} | {repr(token_text):20s} | span=({start}:{end})")
# Example output:
# Text: "tokenization"
# Token count: 3
# Tokens:
# [0] ID= 1234 | 'token' | span=(0:5)
# [1] ID= 5678 | 'ization' | span=(5:12)
# [2] ID= 2 | '</s>' | span=(12:12)Summary
Tokenization is a fixed, deterministic preprocessing step that shapes everything downstream. The algorithm (BPE, WordPiece, Unigram) matters less than the vocabulary and pre-tokenization choices baked into each model. As an engineer, you need to:
- Count tokens accurately for your specific model before sending requests
- Respect special tokens and chat templates — they are part of the model’s protocol
- Never assume portability of token IDs across models
- Monitor token usage in production; it drives cost and latency
- Pin tokenizer versions to avoid silent drift
The tokenizer is the narrow waist of the LLM stack. Everything passes through it. Treat it like the critical infrastructure it is.