Tokens vs words: the real difference, with examples

If you’ve ever been surprised by an API bill or hit a context window limit you thought you had room for, you’ve run into the tokens vs words mismatch. LLMs don’t process words — they process tokens, and the conversion ratio varies wildly by language, model, and even whitespace handling. This post breaks down what that means for your code, your costs, and your architecture.

What a token actually is

A token is an integer ID from a fixed vocabulary that the model was trained on. The tokenizer (usually BPE, WordPiece, or Unigram) splits text into subword units. Common words stay whole (“the” → one token). Rare or compound words fragment (“tokenization” → “token”, “ization” → two tokens). Whitespace, punctuation, and case all affect the count.

# tiktoken example — GPT-4o tokenizer
import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")
text = "tokenization"
print(enc.encode(text))          # [9468, 4758] → 2 tokens
print(enc.encode(" tokenization"))  # [291, 9468, 4758] → 3 tokens (leading space)
print(enc.encode("TOKENIZATION"))   # [73317, 4758] → 2 tokens (different split)

The same logical word can be 1, 2, 3, or more tokens depending on context. This is not a bug — it’s how subword tokenization works.

Words are a human abstraction; tokens are the model’s reality

A “word” usually means a whitespace-delimited string. That’s fine for UI, but meaningless to the model. The tokenizer runs before the model sees anything. Your prompt template, your RAG chunks, your function call arguments — all get tokenized first.

# Same semantic content, different token counts
prompts = {
    "compact": "Summarize: The quick brown fox jumps over the lazy dog.",
    "verbose": "Please provide a concise summary of the following sentence: The quick brown fox jumps over the lazy dog.",
}

for name, p in prompts.items():
    print(f"{name}: {len(enc.encode(p))} tokens")
# compact: 17 tokens
# verbose: 31 tokens

Verbosity costs real money and context space. Every token in your prompt is a token you can’t use for completion.

Billing: you pay per token, not per word

Every major provider (OpenAI, Anthropic, Google, Mistral, Cohere) bills by token. The tokens vs words ratio determines your effective price per word. For English, the rule of thumb is ~1.3 tokens/word. For German, ~1.8. For Chinese, ~1.5–2.5 depending on the tokenizer. For code, it’s often 2–3 tokens/word because of symbols and indentation.

// Hypothetical pricing illustration (check current provider pages)
{
  "gpt-4o": { "input_per_1M": 2.50, "output_per_1M": 10.00 },
  "claude-3-5-sonnet": { "input_per_1M": 3.00, "output_per_1M": 15.00 }
}

If you send 100K English words (~130K tokens) to GPT-4o input, that’s $0.33. The same word count in German (~180K tokens) is $0.45. At scale, language choice becomes a line item.

Context windows are token budgets

A 128K context window means 128K tokens, not words. That 100-page PDF you want to stuff in? If it’s English prose, maybe 30K words fit. If it’s Korean legal text, maybe 12K. If it’s minified JSON, maybe 40K.

# Estimating fit before you send
def estimate_fit(text: str, model: str = "gpt-4o", reserve: int = 4096) -> dict:
    enc = tiktoken.encoding_for_model(model)
    tokens = len(enc.encode(text))
    limit = 128_000 if "gpt-4o" in model else 200_000  # adjust per model
    return {
        "tokens": tokens,
        "limit": limit,
        "available_for_completion": limit - tokens - reserve,
        "fits": tokens + reserve < limit
    }

print(estimate_fit("word " * 50_000))  # ~50K words
# {'tokens': 65001, 'limit': 128000, 'available_for_completion': 62999, 'fits': True}
print(estimate_fit("tokenization " * 50_000))  # same word count, more tokens
# {'tokens': 100001, 'limit': 128000, 'available_for_completion': 27999, 'fits': True}

Always count tokens, not words, before constructing a request. The tiktoken library (OpenAI models) or transformers tokenizers (open models) give exact counts.

Latency and throughput scale with token count

Generation is autoregressive: each output token requires a forward pass. 500 output tokens ≈ 500 forward passes. Input tokens are processed in parallel (prefill), but prefill time still scales linearly with input length.

# Rough mental model (varies by hardware, batching, KV cache)
# Prefill: ~0.5–2 ms per 1K input tokens
# Decode:  ~30–100 ms per output token (single request, no batching)

If your p99 latency budget is 2 seconds and you’re generating 500 tokens, you’re already at the edge. Streaming helps perceived latency but not total time. Token efficiency is latency efficiency.

Tokenization variance across models

Different models use different tokenizers. GPT-4o uses o200k_base (200K vocab). Llama 3 uses a 128K vocab BPE. Gemma uses 256K SentencePiece. The same string produces different token counts and different token IDs.

from transformers import AutoTokenizer

llama3 = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
gpt4o = tiktoken.encoding_for_model("gpt-4o")

text = "The quick brown fox jumps over the lazy dog."
print(f"Llama 3: {len(llama3.encode(text))} tokens")   # 16
print(f"GPT-4o:  {len(gpt4o.encode(text))} tokens")    # 17

If you’re routing requests across providers (as n4n.ai does with its unified endpoint), you cannot assume token parity. Count per model, or use the provider’s usage field in the response for ground truth.

Multilingual: the token tax is real

Non-Latin scripts often pay a heavy token tax. Chinese characters may map 1:1 or fragment into radicals. Korean jamo decomposition can explode counts. Arabic diacritics add tokens. Emoji are often 2–4 tokens each.

samples = {
    "english": "Hello, how are you today?",
    "chinese": "你好，今天过得怎么样？",
    "korean": "안녕하세요, 오늘 하루 어땠어요?",
    "arabic": "مرحباً، كيف حالك اليوم؟",
    "emoji": "🎉🚀💯",
}

for lang, text in samples.items():
    print(f"{lang:8s}: {len(gpt4o.encode(text)):2d} tokens  |  {text}")
# english  :  8 tokens  |  Hello, how are you today?
# chinese  : 11 tokens  |  你好，今天过得怎么样？
# korean   : 18 tokens  |  안녕하세요, 오늘 하루 어땠어요?
# arabic   : 14 tokens  |  مرحباً، كيف حالك اليوم؟
# emoji    :  9 tokens  |  🎉🚀💯

If you’re building a multilingual product, budget 1.5–3× the English token count for the same semantic content.

Prompt engineering ergonomics

Tokens change how you write prompts. Few-shot examples consume budget fast. A 5-shot prompt with 200-token examples = 1K tokens before your actual task. Chain-of-thought reasoning burns output tokens you pay for.

# Token-aware few-shot construction
def build_few_shot_prompt(examples: list[dict], task: str, max_tokens: int = 3000) -> str:
    enc = tiktoken.encoding_for_model("gpt-4o")
    header = "Examples:\n"
    footer = f"\nTask:\n{task}\nAnswer:"
    budget = max_tokens - len(enc.encode(header + footer))
    
    selected = []
    used = 0
    for ex in examples:
        ex_text = f"Q: {ex['q']}\nA: {ex['a']}\n"
        ex_tokens = len(enc.encode(ex_text))
        if used + ex_tokens > budget:
            break
        selected.append(ex_text)
        used += ex_tokens
    
    return header + "".join(selected) + footer

Design prompts with a token budget, not a word budget. Trim examples, compress instructions, use structured output formats (JSON, not prose) to reduce output tokens.

Tooling ecosystem

Always verify with the actual tokenizer. Heuristics are for capacity planning, not billing reconciliation.

Comparison table: tokens vs words across dimensions

Which to choose: verdict by use case

Capacity planning & cost modeling → Think in tokens. Build a token counter into your CI/CD. Estimate spend per 1K requests using actual tokenizer counts on representative payloads. Words are for talking to product; tokens are for talking to finance.

Prompt engineering & RAG chunking → Think in tokens. Chunk by token count (e.g., 512 tokens with 50 overlap), not word count. Trim few-shot examples to fit a token budget. Use the model’s tokenizer, not a heuristic.

UI/UX & user-facing copy → Think in words. Users understand “500-word summary.” Convert to tokens internally, show words externally. Never expose token counts to end users unless they’re developers.

Cross-provider routing → Think in tokens per model. A request routed to GPT-4o vs Claude 3.5 Sonnet vs Llama 3 70B will have different input token counts for the same text. Your routing layer must count per target model or accept provider-reported usage as ground truth.

Multilingual products → Think in tokens with language-specific multipliers. Build a lookup table: English 1.0×, Spanish 1.2×, German 1.5×, Chinese 1.8×, Korean 2.2×, Arabic 1.6×. Update when tokenizers change.

Debugging & observability → Log both. Token counts explain latency, cost, and truncation. Word counts explain user intent. Correlate them to spot anomalies (e.g., sudden token inflation = prompt injection or tokenizer change).

The model only sees tokens. Your wallet only feels tokens. Your users only know words. Engineer the translation layer between them — that’s where the leverage lives.