Why token counts differ between LLM providers

Why token counts differ between providers is a question every engineer faces when reconciling LLM bills across backends. The same prompt sent to OpenAI, Anthropic, and Meta models can return usage numbers that vary by 20–40%, and the gap widens with chat formatting, tools, and reasoning steps. This analysis breaks down the root causes and gives a concrete strategy for building cost monitoring that doesn’t lie.

Tokenization is provider-specific by design

Every lab trains its own tokenizer. OpenAI uses a BPE variant with a closed vocabulary of roughly 100k tokens. Anthropic uses a custom BPE tuned for English and code. Meta’s Llama models ship with SentencePiece (Unigram for Llama-2, BPE for Llama-3). Google’s Gemini uses a proprietary tokenizer. The same sentence splits into different subword units, so counts diverge.

import tiktoken
from transformers import AutoTokenizer

text = "Function calling requires careful schema design."

# OpenAI GPT-4 tokenizer
oai_tok = tiktoken.encoding_for_model("gpt-4")
oai_count = len(oai_tok.encode(text))

# Llama-2 BPE via HuggingFace
llama_tok = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
llama_count = len(llama_tok.encode(text))

print(oai_count, llama_count)  # e.g., 9 vs 11

The difference is not a bug. Tokenizers optimize for compression on training data; a tokenizer trained mostly on English Wikipedia behaves differently from one trained on GitHub and multilingual text. Vocabulary size matters: a 32k vocabulary must split rare words more aggressively than a 100k vocabulary, inflating token counts for the same string.

Whitespace and unicode normalization

Some tokenizers strip leading spaces, others keep them as separate tokens. Unicode normalization (NFC vs NFKC) changes accented characters. A prompt with café may be one token in one backend and three in another. Numeric strings are worse: 1234567890 is a single token in GPT-4 but multiple tokens in Llama-2 because of differing digit grouping.

print(len(oai_tok.encode("1234567890")))   # 1
print(len(llama_tok.encode("1234567890"))) # 10

These micro-decisions compound across a long system prompt.

Chat templates and control tokens

Raw text is rarely what you send. Providers wrap your messages in a chat template that injects role markers, separators, and sometimes a system preamble.

{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Summarize this."}
  ]
}

OpenAI’s chat model adds tokens for <|im_start|>system and <|im_end|>. Anthropic’s Claude adds \n\nHuman: and \n\nAssistant:. These markers are counted in prompt_tokens. If you compute tokens locally with a bare tokenizer, you will undercount by 10–30 tokens per turn.

A minimal illustration of template overhead:

def claude_template(messages):
    out = ""
    for m in messages:
        role = "Human" if m["role"] == "user" else "Assistant"
        out += f"\n\n{role}: {m['content']}"
    return out

template_text = claude_template(messages)
# This string is what gets tokenized, not the raw content.

Mistral and OpenAI-compatible open models use yet another template ([INST] ... [/INST]). Switching models without adjusting your token estimator breaks budgets.

Hidden tokens: reasoning, caching, and tool schemas

Modern APIs report usage that includes tokens you never saw.

Reasoning models

OpenAI’s o1 family generates internal reasoning tokens. The API returns completion_tokens that may include them, or separates reasoning_tokens in some versions. Anthropic’s extended thinking adds thinking_tokens. If you only count output characters, you miss 30–50% of the cost.

Cache control

Anthropic lets you mark a prefix with cache_control. The first call bills full prompt tokens; subsequent calls bill cache_read_input_tokens at a discount. OpenAI supports prompt_tokens_details.cached_tokens. A gateway that honors client routing directives and forwards provider cache-control hints—such as n4n.ai—will surface these fields, but the underlying token math is still provider-native.

// Anthropic usage example
{
  "usage": {
    "input_tokens": 1200,
    "cache_creation_input_tokens": 1000,
    "cache_read_input_tokens": 0,
    "output_tokens": 300
  }
}

// OpenAI usage example
{
  "usage": {
    "prompt_tokens": 1200,
    "completion_tokens": 300,
    "prompt_tokens_details": { "cached_tokens": 1000 }
  }
}

Tool and function definitions

Function schemas are injected into the prompt. A complex JSON schema can add 200–500 tokens per call. Providers differ in how they serialize the schema (compact vs pretty), changing counts. If you use the same tool spec across providers, expect different prompt_tokens.

Vision and multimodal accounting

Image inputs are tokenized by tiling. OpenAI’s vision models use a grid of 512×512 patches, each costing a fixed token block. Claude computes tokens based on image dimensions and detail level. The same PNG can be 85 tokens in one, 200 in another. Audio and PDF inputs have similar provider-specific rules. Any cost monitor that assumes tokens = chars/4 will fail on multimodal payloads.

Why this matters for cost monitoring

If you aggregate spend across models, naive summation of “tokens” is meaningless. A token from Llama-3-70B is not equivalent to a token from GPT-4o in compute or price. The only common denominator is cost, and even that fluctuates with provider pricing tiers.

A gateway like n4n.ai exposes per-token usage metering across 240+ models behind one OpenAI-compatible endpoint, but the raw counts still reflect each provider’s tokenizer. Your billing layer must record provider, model, prompt_tokens, completion_tokens, and cache_details separately.

interface UsageRecord {
  provider: string;
  model: string;
  promptTokens: number;
  completionTokens: number;
  cachedTokens?: number;
  costUsd: number; // computed from provider price sheet
}

Store the exact JSON the API returns. Do not preprocess it into a single integer.

Tradeoffs of normalizing token counts

You could force every prompt through a single reference tokenizer (e.g., GPT-4’s) to get a “normalized token”. This is tempting for dashboards but flawed:

• It adds a blocking preprocessing step on the critical path.
• It cannot account for hidden reasoning or cache tokens.
• It diverges further on non-English text and images.

Better approach: store raw provider usage, then map to cost using a price table updated per model. For capacity planning, track chars_per_token ratio per provider as a heuristic, not a contract.

# Heuristic only: characters to provider tokens
CHARS_PER_TOKEN = {
    "openai": 4.0,
    "anthropic": 3.6,
    "meta": 4.2,
}

def estimate_cost(text, provider, usd_per_1k):
    return len(text) / CHARS_PER_TOKEN[provider] / 1000 * usd_per_1k

These ratios are observational, not guaranteed. They help with quick sizing, not billing.

Decisive takeaway

Treat token counts as provider-local accounting units, not absolute measures of text length. Log the exact usage object from each API response, persist provider and model metadata, and compute cost downstream. Build your monitoring to tolerate missing fields like reasoning_tokens and cache_read_input_tokens. When you need cross-model comparison, convert to dollars or relative cost per request, never raw tokens.

Engineers who internalize why token counts differ between providers stop trusting client-side estimates and start trusting metered API responses. That shift is what makes LLM cost monitoring accurate.