The token count openai vs anthropic mismatch is not a billing glitch or a rounding error. It is the direct result of two independent tokenizer implementations that split text by different rules, train on different corpora, and optimize for different model architectures. If you build routing logic that assumes a token is a token regardless of provider, you will silently truncate context or misallocate budget.
The root cause: different tokenization algorithms
OpenAI’s modern models use byte-pair encoding (BPE) with fixed vocabularies. GPT-3.5 and GPT-4 use cl100k_base; the o-series and gpt-4o use o200k_base. BPE merges frequent byte pairs iteratively, so common English words stay intact while rare strings fragment into subword pieces.
Anthropic does not publish the exact algorithm for Claude, but its tokenizer is a separate trained model, not BPE in the OpenAI sense. It behaves more like a unigram/SentencePiece style model with a vocabulary tuned for multilingual text and code. The practical effect: the same string produces a different token sequence and a different length.
Whitespace and control characters
BPE treats the leading space as part of the token: " hello" becomes a single token if frequent. Anthropic’s tokenizer often splits on whitespace more aggressively and may encode spaces as explicit markers. This shows up in JSON and Python indentation, where a single trailing newline can shift counts by several tokens.
Vocabulary size and character coverage
OpenAI’s cl100k vocab is ~100k entries. Anthropic’s is larger and more tolerant of Unicode. For CJK text, the token count openai vs anthropic gap widens: OpenAI may use 1–2 tokens per character; Anthropic can pack more characters per token, reducing total count.
Concrete examples
Take a benign English sentence.
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
text = "The quick brown fox jumps over the lazy dog."
print(len(enc.encode(text))) # 10 tokens on cl100k/o200kThe same text sent to Claude via a minimal call:
import anthropic
client = anthropic.Anthropic()
resp = client.messages.create(
model="claude-3-5-sonnet-20240620",
max_tokens=1,
messages=[{"role": "user", "content": text}]
)
print(resp.usage.input_tokens) # typically 9–11, varies by revisionThe absolute difference is small here, but scale it to a 50KB technical document and the delta becomes hundreds of tokens.
Code blocks
def add(a, b):
return a + bOpenAI BPE encodes def, add, (, a, ,, b, ), : as separate pieces. Anthropic may merge def add into one token due to code frequency. For a 1k-line module, the token count openai vs anthropic can differ by 15–20%, directly changing whether you fit inside a 128k window.
Non-Latin scripts
A paragraph of Japanese can be 30% cheaper on Anthropic by token count, while OpenAI fragments kanji more. If your app routes based on cost-per-token without measuring actual counts, you will miscalculate.
Why the difference breaks your pipeline
Context window truncation
You check len(text) / 4 and decide it fits in 100k. Wrong. If the provider is Anthropic and your estimate is based on OpenAI’s BPE, you might be off by thousands of tokens. The model rejects the request or you silently drop the tail of a retrieval context.
Cost estimation errors
Per-token pricing assumes you know the tokens. If you cache the OpenAI count and reuse it for Anthropic, your margin math is wrong. The token count openai vs anthropic divergence makes a single global counter unsafe.
Cache key mismatches
Both providers support prompt caching via cache-control hints. If you compute a cache key from a normalized token stream, the provider’s own tokenization differs, and your cache never hits. Forward the provider’s native cache-control and let them hash the real tokens.
Measuring accurately
Do not guess. Count per provider at ingest time.
def count_openai(text, model="gpt-4o"):
import tiktoken
enc = tiktoken.encoding_for_model(model)
return len(enc.encode(text))
def count_anthropic(text, model="claude-3-5-sonnet-20240620"):
import anthropic
client = anthropic.Anthropic()
resp = client.messages.create(
model=model, max_tokens=1,
messages=[{"role": "user", "content": text}]
)
return resp.usage.input_tokensThe Anthropic call costs a tiny amount of tokens but is authoritative. Cache the result keyed by (provider, model, text_hash).
If you route through a gateway such as n4n.ai, the per-token usage metering returns both the provider’s counted tokens and honors your routing directives, so you can compare the token count openai vs anthropic from one OpenAI-compatible response without maintaining two SDK calls.
Tokenization is model-version locked
OpenAI pins the encoding to the model name. gpt-4o uses o200k_base across its lifetime. Anthropic does not expose the vocab but each Claude minor revision can adjust tokenization. This means caching counts must include model version string, not just provider.
System prompts and structured input
When you send a system prompt with JSON schema, OpenAI’s BPE may tokenize the schema keys as separate tokens, while Anthropic’s tokenizer compresses repeated structural characters. A 2k-char schema might be 300 tokens on OpenAI and 250 on Anthropic. For multi-turn conversations with repeated tool definitions, the token count openai vs anthropic delta compounds.
{
"type": "object",
"properties": {
"location": {"type": "string"},
"unit": {"type": "string", "enum": ["c", "f"]}
}
}Count both before sending. A shared middleware that strips whitespace to “save tokens” can backfire: it changes the tokenization boundary and may increase count on one provider while decreasing on another.
Output tokens diverge too
Tokenization applies to completions. A response of “The temperature is 72°F.” may be 7 tokens on one, 8 on another. If you cap max_tokens based on one provider’s count, you may truncate on the other. Always set max_tokens per provider based on their own output estimates.
Debugging workflow
When a request fails with context length exceeded, do not trust your middleware count. Log the provider’s usage.input_tokens from the error or response. Diff against your local tiktoken count. The gap reveals which tokenizer rule bit you.
A practical log line:
logger.warning(
"token mismatch",
local=len(enc.encode(payload)),
provider=resp.usage.input_tokens,
provider_name="openai"
)If provider - local is consistently positive for Anthropic, your estimate function is biased.
Tradeoffs of normalizing
You could adopt a heuristic: characters divided by 4 for English, 2 for CJK. This is fast and dependency-free but inaccurate at boundaries. For a chat app with short messages, the error is acceptable. For a RAG pipeline with 200-page PDFs, the error means lost context.
Training a custom shared tokenizer is overkill. The pragmatic tradeoff: count natively at the edge, store both counts, and use the larger when enforcing a hard limit. That guarantees you never exceed either provider’s context window.
Decisive takeaway
Stop treating tokens as a provider-agnostic unit. The token count openai vs anthropic gap is structural, not cosmetic. Instrument your ingestion to count per target model, enforce limits using the provider-specific number, and route on real usage metered from the API response. If you do only one thing: replace every len(text)/4 in your codebase with a provider-aware tokenizer call today.