Token counting libraries for Claude, GPT-4o, and Gemini

Estimating cost before sending a request is table stakes when you’re shipping LLM features. The right token counting libraries by model save you from surprise bills and truncated context windows. Below we break down the practical options for Claude, GPT-4o, and Gemini, with code you can drop into a service today.

1. tiktoken for GPT-4o and OpenAI models

tiktoken is the reference tokenizer released by OpenAI. GPT-4o uses the o200k_base encoding, which you load directly by name or via encoding_for_model("gpt-4o"). It’s a Rust binary with Python and Node bindings, so counting a 100k-token document takes single-digit milliseconds on a laptop.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
# or enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode("Your prompt for GPT-4o")
print(len(tokens))

In a Node service, install the tiktoken npm package and call getEncoding("o200k_base"). The API is nearly identical. This library does not support Claude or Gemini; forcing it on non-OpenAI text will undercount because vocabularies differ.

A subtle point: tiktoken counts the raw string. OpenAI’s chat completions endpoint adds invisible overhead for role markers and the conversation template. For GPT-4o, budget roughly 3 extra tokens per message and 3 per reply start. If you need exact figures, log the usage field from the API response and calibrate.

The token counting libraries by model conversation starts here because OpenAI is the only vendor that open-sourced its production tokenizer. Use it unconditionally for any GPT-4o path.

2. @anthropic-ai/tokenizer for Claude

Anthropic maintains an official tokenizer as a Rust crate with WASM bindings. The @anthropic-ai/tokenizer npm package is the most accurate client-side counter for Claude text. It mirrors the server’s SentencePiece-style vocabulary, including the quirks around whitespace and code fences.

import { countTokens } from "@anthropic-ai/tokenizer";

const text = "Hello Claude, count me accurately.";
const count = countTokens(text);
console.log(count);

For Python shops, the Anthropic SDK added a messages.count_tokens RPC. It is an API call, not a local computation, but it guarantees parity with the billing system.

from anthropic import Anthropic
client = Anthropic()
resp = client.messages.count_tokens(
    model="claude-3-5-sonnet-20240620",
    messages=[{"role": "user", "content": "Hello"}]
)
print(resp.input_tokens)

Note that Claude bills image tokens based on resolution and placement; the text tokenizer ignores images. If your request mixes modalities, call the API counter rather than the local lib. The token counting libraries by model fragment exactly at this boundary: text-only local, multimodal remote.

3. google-generativeai for Gemini

Gemini’s tokenizer is proprietary, but Google exposes a countTokens method in the official SDKs. The Python google-generativeai package wraps the REST RPC and returns a total_tokens field without consuming generation quota.

import google.generativeai as genai
model = genai.GenerativeModel("gemini-1.5-pro")
result = model.count_tokens("Explain token counting.")
print(result.total_tokens)

The JS SDK @google/generative-ai provides the same:

import { GoogleGenerativeAI } from "@google/generative-ai";
const genai = new GoogleGenerativeAI(process.env.API_KEY!);
const model = genai.getGenerativeModel({ model: "gemini-1.5-pro" });
const { totalTokens } = await model.countTokens("Explain token counting.");
console.log(totalTokens);

These calls hit Google’s servers, so they are not offline. For pure local approximation, load the Gemma 2B SentencePiece tokenizer from HuggingFace; Gemini shares architectural lineage, but expect drift on non-English and rare symbols. The token counting libraries by model for Gemini are therefore “exact via API, approximate via Gemma.”

4. HuggingFace transformers as a fallback

When you cannot reach a vendor API and need a rough estimate, transformers with an open SentencePiece model is the only game in town. For Gemini-adjacent work, google/gemma-2b is the closest public proxy.

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("google/gemma-2b")
ids = tok("Approximate Gemini tokens", add_special_tokens=False)
print(len(ids))

For Claude there is no sanctioned open tokenizer; community replicas exist but diverge on whitespace handling and code tokens. Treat their output as ±5% at best. This is the gray area of token counting libraries by model: vendor-exact versus open-approximate.

If you route through a gateway like n4n.ai, per-token usage metering arrives in the response headers, so post-hoc counts are authoritative. Local libraries still earn their keep by letting you reject oversized prompts before they traverse the network.

5. A unified dispatcher wrapper

No single package covers all three vendors, so production code usually hides them behind a small function. Cache the encoding objects, branch on model prefix, and surface a single integer.

def count_tokens(model: str, text: str) -> int:
    if model.startswith("gpt-4o") or model.startswith("gpt-"):
        import tiktoken
        return len(tiktoken.get_encoding("o200k_base").encode(text))
    elif model.startswith("claude"):
        from anthropic import Anthropic
        return Anthropic().messages.count_tokens(
            model=model, messages=[{"role": "user", "content": text}]
        ).input_tokens
    elif model.startswith("gemini"):
        import google.generativeai as genai
        return genai.GenerativeModel(model).count_tokens(text).total_tokens
    else:
        raise ValueError(f"no counter for {model}")

Add @lru_cache on the tiktoken encoding and reuse a singleton Anthropic/Google client. The token counting libraries by model become swappable adapters, and your business logic stays clean.

Summary table

Pick vendor-native counters where you can; reserve approximations for dashboards and logs. The fragmentation is annoying but manageable with a thin wrapper.