What is a token in AI? A plain-English definition

A token is the atomic unit of text that a large language model processes — each token maps to an integer ID in the model’s fixed vocabulary, and the model predicts the next token ID given the sequence so far. Tokenization is the deterministic algorithm that converts raw text into that sequence of IDs, and it happens before the model ever sees the input. Understanding tokens is necessary because every LLM constraint — context window, pricing, latency, and generation quality — is expressed in tokens, not characters or words.

How tokenization works

Tokenization algorithms compress text into a sequence of vocabulary entries. The dominant approach in modern LLMs is byte-pair encoding (BPE) or its variants (SentencePiece, WordPiece). These algorithms build a vocabulary by iteratively merging the most frequent adjacent byte or character pairs in a training corpus until reaching a target vocabulary size — typically 32,000 to 256,000 entries for current models.

# Simplified BPE training loop
def train_bpe(corpus: list[bytes], vocab_size: int) -> dict[bytes, int]:
    # Start with single-byte vocabulary (256 entries)
    vocab = {bytes([i]): i for i in range(256)}
    merges = []
    
    while len(vocab) < vocab_size:
        # Count adjacent pairs
        pair_counts = Counter()
        for word in corpus:
            for i in range(len(word) - 1):
                pair_counts[(word[i], word[i+1])] += 1
        
        if not pair_counts:
            break
            
        # Merge most frequent pair
        best_pair = max(pair_counts, key=pair_counts.get)
        new_token = best_pair[0] + best_pair[1]
        new_id = len(vocab)
        vocab[new_token] = new_id
        merges.append((best_pair, new_token))
        
        # Update corpus with merged token
        corpus = [merge_word(word, best_pair, new_token) for word in corpus]
    
    return vocab, merges

At inference time, the tokenizer applies the learned merges greedily from longest to shortest. The same algorithm must produce identical token sequences for the same input across every deployment — tokenization is deterministic and versioned alongside the model weights.

# Greedy BPE encoding at inference
def encode_bpe(text: str, vocab: dict[bytes, int], merges: list[tuple]) -> list[int]:
    tokens = list(text.encode('utf-8'))
    for (a, b), merged in merges:
        i = 0
        while i < len(tokens) - 1:
            if tokens[i] == a and tokens[i+1] == b:
                tokens[i:i+2] = [merged]
            else:
                i += 1
    return [vocab[token] for token in tokens]

Different models use different tokenizers. GPT-4 uses a 100,277-token vocabulary derived from BPE on a mixed-code corpus. Llama 3 uses a 128,256-token SentencePiece vocabulary optimized for multilingual data. These vocabularies are not interchangeable — feeding GPT-4 tokens into Llama 3 produces garbage.

Why tokens matter for engineers

Context windows are token budgets

Every model has a hard maximum sequence length measured in tokens. GPT-4o supports 128,000 tokens; Llama 3.1 supports 128,000; Claude 3.5 Sonnet supports 200,000. This budget includes both your prompt and the generated completion. If your prompt consumes 120,000 tokens, you have at most 8,000 tokens for the response before the model must truncate or fail.

def estimate_context_usage(prompt: str, max_completion: int, model: str) -> dict:
    encoder = tiktoken.encoding_for_model(model)
    prompt_tokens = len(encoder.encode(prompt))
    available = MODEL_CONTEXTS[model] - prompt_tokens
    return {
        "prompt_tokens": prompt_tokens,
        "max_completion_tokens": min(max_completion, available),
        "will_truncate": available < max_completion
    }

Pricing is per token

Provider APIs bill by input tokens and output tokens, often at different rates. As of this writing, GPT-4o charges $2.50 per 1M input tokens and $10.00 per 1M output tokens. A 10,000-token prompt with a 2,000-token response costs roughly $0.045. Token counting is not optional — it is the unit of account.

{
  "usage": {
    "prompt_tokens": 10243,
    "completion_tokens": 2048,
    "total_tokens": 12291
  }
}

Latency scales with token count

Generation is autoregressive: each output token requires a full forward pass through the model. Doubling output length roughly doubles latency. Streaming mitigates perceived latency by delivering tokens as they are generated, but total time-to-last-token remains linear in output length.

async def stream_completion(prompt: str, max_tokens: int):
    async for chunk in client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        stream=True
    ):
        token = chunk.choices[0].delta.content
        if token:
            yield token  # Deliver immediately, but total time ∝ max_tokens

Tokenization affects retrieval and chunking

When building RAG systems, chunk size must be defined in tokens, not characters. A naive 1,000-character chunk might be 150 tokens in English but 400 tokens in Korean (where each character often becomes its own token). Chunk overlap, embedding input limits, and reranker context windows all operate in token space.

Concrete example: tokenizing a sentence

Consider the sentence: "The quick brown fox jumps over the lazy dog."

Using GPT-4o’s tokenizer (cl100k_base):

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
text = "The quick brown fox jumps over the lazy dog."
tokens = enc.encode(text)
print(tokens)
# [791, 4828, 2636, 4021, 6689, 367, 791, 4186, 3923, 13]
print(enc.decode(tokens))
# The quick brown fox jumps over the lazy dog.

Token breakdown:

Ten tokens for 44 characters — roughly 4.4 characters per token, typical for English prose. But the ratio varies wildly:

samples = {
    "English": "The quick brown fox jumps over the lazy dog.",
    "Code": "def fibonacci(n):\n    if n <= 1:\n        return n\n    return fibonacci(n-1) + fibonacci(n-2)",
    "Korean": "빠른 갈색 여우가 게으른 개를 뛰어넘습니다.",
    "Numbers": "1234567890",
    "Mixed": "User ID: 42, Status: active, Score: 99.5%"
}

for name, text in samples.items():
    tokens = enc.encode(text)
    print(f"{name:8} | {len(text):3} chars | {len(tokens):3} tokens | {len(text)/len(tokens):.1f} chars/token")

Output:

English  |  44 chars |  10 tokens | 4.4 chars/token
Code     |  97 chars |  31 tokens | 3.1 chars/token
Korean   |  48 chars |  29 tokens | 1.7 chars/token
Numbers  |  10 chars |  10 tokens | 1.0 chars/token
Mixed    |  42 chars |  21 tokens | 2.0 chars/token

Code tokenizes less efficiently because indentation, punctuation, and identifiers fragment across tokens. Korean and other non-Latin scripts often approach one token per character. Numbers frequently tokenize digit-by-digit.

Common misconceptions

“A token is a word”

False. Tokens are vocabulary entries. Common words (the, and, is) are single tokens. Rare words fragment: tokenization → token + ization (two tokens). Proper nouns often split: OpenAI → Open + AI (two tokens). The average English word is ~1.3 tokens.

“Token count equals word count times 1.3”

Only roughly true for clean English prose. Code, logs, JSON, multilingual text, and formatted documents deviate significantly. Always count programmatically with the model’s actual tokenizer.

# Never estimate — count
def count_tokens(text: str, model: str = "gpt-4o") -> int:
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))

“The tokenizer doesn’t affect model behavior”

The tokenizer is part of the model. Changing tokenization changes the input distribution the model sees during training. A model trained on a 32K vocabulary cannot use a 128K vocabulary without retraining. Tokenizer version mismatches between training and inference degrade quality silently.

“Special tokens are just more vocabulary”

Special tokens (<|endoftext|>, <|im_start|>, <|im_end|>, <|fim_prefix|>) control generation behavior. The end-of-sequence token tells the model to stop. Chat template tokens delimit roles. Fill-in-the-middle tokens enable infilling. Treating them as ordinary tokens breaks chat formatting and stop conditions.

# Chat template example (simplified)
def apply_chat_template(messages: list[dict]) -> str:
    parts = ["<|im_start|>system\nYou are a helpful assistant.<|im_end|>"]
    for msg in messages:
        parts.append(f"<|im_start|>{msg['role']}\n{msg['content']}<|im_end|>")
    parts.append("<|im_start|>assistant\n")
    return "".join(parts)

“Token limits are soft”

They are hard. Exceeding the context window returns an error or triggers automatic truncation (often dropping the middle of your prompt). Some providers offer “context window extension” via sliding windows or retrieval, but the model’s native attention window remains fixed.

Practical token hygiene

Count before you send. Use the model’s tokenizer locally to validate prompt size. The tiktoken library for OpenAI models, transformers tokenizers for Hugging Face models, and vendor SDKs all expose exact counting.

Reserve output budget. Never consume 100% of the context window with your prompt. Leave headroom for the completion plus a safety margin.

MAX_CONTEXT = 128000
SAFETY_MARGIN = 1000

def build_prompt(system: str, context: str, query: str, max_output: int) -> str:
    enc = tiktoken.encoding_for_model("gpt-4o")
    system_tokens = len(enc.encode(system))
    query_tokens = len(enc.encode(query))
    budget = MAX_CONTEXT - system_tokens - query_tokens - max_output - SAFETY_MARGIN
    context_tokens = enc.encode(context)[:budget]
    return system + enc.decode(context_tokens) + query

Stream long completions. For outputs over ~500 tokens, stream to avoid request timeouts and improve perceived latency.

Log token usage. Record prompt_tokens, completion_tokens, and total_tokens for every request. This enables cost tracking, anomaly detection, and optimization.

# Structured logging for token observability
import structlog

logger = structlog.get_logger()

async def tracked_completion(request: CompletionRequest) -> CompletionResponse:
    response = await client.chat.completions.create(**request.model_dump())
    logger.info(
        "llm_completion",
        model=request.model,
        prompt_tokens=response.usage.prompt_tokens,
        completion_tokens=response.usage.completion_tokens,
        total_tokens=response.usage.total_tokens,
        latency_ms=response.response_ms,
        finish_reason=response.choices[0].finish_reason
    )
    return response

Handle tokenizer drift. When a provider updates a tokenizer (rare but possible), token counts for identical inputs change. Pin tokenizer versions in your counting logic and monitor for discrepancies.

Tokenization across the stack

Tokenization is not just a preprocessing step — it propagates through your entire LLM pipeline:

Mismatched tokenizers between components cause silent failures. A retriever using a 512-token embedding model fed 1,000-token chunks from a 4K-token generator will truncate unpredictably.

Summary

Tokens are the currency of LLM engineering. Every constraint — context, cost, latency, quality — is denominated in tokens. Tokenization is deterministic, model-specific, and versioned. Count tokens programmatically with the correct tokenizer. Reserve output budget. Stream long generations. Log usage. Treat token limits as hard boundaries, not suggestions. Engineers who internalize token economics build systems that scale; those who ignore them hit invisible walls.