The token count differences gpt-4o claude produce on the same input routinely surprise engineers building multi-model pipelines. GPT-4o ships a BPE tokenizer with a ~200k token vocabulary, while Claude uses a Unigram language model tokenizer with different merge priorities and no public local implementation. Those design choices cause the same English paragraph, JSON blob, or code snippet to split into different token sequences, which directly shifts both cost and context-window math.
The root cause: two unrelated tokenization schemes
BPE vs Unigram
GPT-4o inherits OpenAI’s byte-pair encoding (BPE) lineage. It starts from bytes, merges frequent pairs iteratively, and builds a vocabulary where common substrings become single tokens. Claude’s tokenizer is a Unigram model: it begins with a large candidate vocabulary and prunes to maximize likelihood under a language model. The result is a different preference for token boundaries. BPE is greedy bottom-up; Unigram is top-down with probabilistic scoring.
This is not a tuning difference; it is a different algorithm family. A sequence like abcdef might be one token in Claude’s Unigram if that string is common, but three tokens in GPT-4o’s BPE if no merge rule covers it.
Vocabulary and normalization
GPT-4o’s o200k_base encoding uses NFKC normalization and treats whitespace as part of tokens (the Ġ prefix in tiktoken internals). Claude applies its own normalization (roughly NFC-like, but not identical) and handles spaces without a special prefix. For non-ASCII text, the two tokenizers map bytes to Unicode codepoints differently. Emoji, CJK ideographs, and combining marks land on different token counts.
A practical consequence: the token count differences gpt-4o claude show are largest on mixed-script input. Pure ASCII English prose stays within ~10% variance. Switch to Chinese or Ruby-annotated Japanese and the gap can exceed 30%.
Emoji and byte fallback
When a character is not in the base vocabulary, GPT-4o’s BPE falls back to byte tokens (each byte becomes a token, so an emoji can cost 4–8 tokens). Claude’s Unigram may have dedicated tokens for frequent emoji, making it cheaper on 😀 but not on rare Unicode. Never assume parity on any payload containing non-text symbols.
What this means for your prompt engineering
Whitespace and indentation
Code is where teams get burned. Python indentation, YAML spaces, and JSON formatting are tokenized as explicit characters in GPT-4o (each space often merges with the preceding token). Claude’s Unigram frequently absorbs indentation into surrounding tokens. A 50-line YAML config can be 20% cheaper on Claude simply because runs of two-space indents collapse.
If you trim whitespace to save tokens on GPT-4o, you may be micro-optimizing for the wrong model. Write a normalization pass only after measuring both tokenizers on your real payloads.
Non-Latin scripts
CJK text is the classic divergence. GPT-4o’s BPE splits Mandarin into sub-character byte sequences when the exact character isn’t in vocab; Claude’s Unigram often maps common characters to single tokens. The token count differences gpt-4o claude for a 1,000-character Chinese article can be 1,100 vs 800. That shifts both cost and whether you fit in a 128k window.
Code and structured data
JSON keys repeated across many objects favor BPE’s pair merges: "id": becomes one token after frequent observation. Claude’s Unigram may keep "id" and : separate but compress long string values differently. Benchmark your actual API schemas, not synthetic text.
Measuring tokens without guessing
Local libraries give exact counts for GPT-4o. For Claude, Anthropic exposes a counting endpoint because the tokenizer is not open-sourced.
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
payload = '{"user_id": 42, "events": [{"type": "click"}, {"type": "view"}]}'
print("gpt-4o:", len(enc.encode(payload)))import anthropic
client = anthropic.Anthropic()
res = client.messages.count_tokens(
model="claude-3-5-sonnet-20241022",
messages=[{"role": "user", "content": payload}]
)
print("claude:", res.input_tokens)The Claude call costs a tiny fraction of a cent and returns the authoritative number. Do not ship a heuristic that multiplies GPT-4o’s count by 0.9 and calls it “Claude tokens.” The error compounds on large prompts.
If you route through a gateway such as n4n.ai, the per-token usage metering comes from the provider response, so your billing records reflect the actual token count differences gpt-4o claude incurred rather than a local estimate.
You can also count Claude tokens over HTTP without the SDK:
curl https://api.anthropic.com/v1/messages/count_tokens \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "content-type: application/json" \
-d '{"model":"claude-3-5-sonnet-20241022","messages":[{"role":"user","content":"Hello world"}]}'Cost estimation and the fallback trap
Engineers building multi-model failover often write a pre-flight estimator to decide which model to call. Suppose you estimate cost with a single tokenizer and then trigger automatic fallback when a provider is rate-limited. If your estimator assumes GPT-4o token counts and the request falls back to Claude, your budget guardrail is wrong by the tokenizer delta.
Worse, context-limit checks fail silently. A prompt estimated at 120k GPT-4o tokens might be 140k Claude tokens, exceeding the window after fallback. The request errors mid-stream, wasting the first call’s cost.
The fix is to treat token count as a model-specific property. Store counts per model, or query the provider-native counter at request time when the payload is large enough that the extra latency matters less than misbilling.
Building a token-aware abstraction
A minimal TypeScript interface keeps your code honest:
interface TokenCounter {
count(model: string, text: string): Promise<number> | number;
}
class GPT4oCounter implements TokenCounter {
// node binding to tiktoken, or precomputed cache
count(_model: string, text: string): number {
return tiktokenEncode(text).length;
}
}
class ClaudeCounter implements TokenCounter {
constructor(private client: Anthropic) {}
async count(model: string, text: string): Promise<number> {
const res = await this.client.messages.count_tokens({
model,
messages: [{ role: "user", content: text }],
});
return res.input_tokens;
}
}Cache counts for static fragments (system prompts, few-shot examples) per model. Only dynamically count the variable user slice. This avoids per-request latency for high-throughput small calls while keeping large payloads accurate.
Tradeoffs of native counting
Calling count_tokens adds a round-trip. For high-throughput small requests, that overhead may dominate. In those cases, cache the token count for fixed template fragments measured once per model, and only dynamically count the variable user slice.
Local approximations for Claude exist (community WASM ports of an older tokenizer), but they drift from the production tokenizer on new model revisions. If you use them, pin the model version and validate against the API monthly.
A decisive takeaway
Stop treating tokens as a universal unit. The token count differences gpt-4o claude exhibit are structural, not bugs. For any system that spans both families, compute counts with each provider’s own tokenizer, meter from provider responses, and never let a single global estimate drive cost or context limits. Build your abstraction around tokens_per_model maps, and your billing and truncation logic will survive the next model swap.