Counting tokens in Python without an API call

Counting tokens python offline is a prerequisite for accurate cost control before you send a request to any LLM provider. You don’t need a network round-trip to estimate prompt size; local tokenizer libraries give exact counts for most models and approximate counts for the rest.

Step 1: Install the offline tokenizer libraries

Two packages cover the vast majority of cases. tiktoken handles OpenAI model families. transformers (or the lighter tokenizers crate binding) handles open-weight models from Meta, Mistral, Google, and others.

pip install tiktoken transformers

If you only target OpenAI-compatible endpoints, tiktoken alone is enough. For Llama-3 or Mixtral, pull transformers and let it download the matching tokenizer config on first use—that download is a one-time, cached operation, not a per-request call. Set HF_HOME to a persistent path so CI and prod share the cache.

export HF_HOME=/opt/tokenizer_cache

Step 2: Count tokens for OpenAI models with tiktoken

tiktoken exposes encodings by name. For GPT-3.5-Turbo, GPT-4, and GPT-4o, use cl100k_base. Older models like text-davinci-003 use p50k_base. The library implements the exact byte-pair encoding (BPE) merge rules published by OpenAI, so counts are authoritative for raw strings.

import tiktoken

def count_openai_tokens(text: str, model: str = "gpt-4") -> int:
    encoding_name = {
        "gpt-3.5-turbo": "cl100k_base",
        "gpt-4": "cl100k_base",
        "gpt-4o": "cl100k_base",
        "text-davinci-003": "p50k_base",
    }.get(model, "cl100k_base")
    enc = tiktoken.get_encoding(encoding_name)
    return len(enc.encode(text))

print(count_openai_tokens("Hello, world"))  # 3

The returned integer is the exact token count the OpenAI API would bill for that raw string. It does not include chat formatting overhead—we fix that in Step 5. Note that enc.encode returns a list of integer IDs; calling len is O(n) in tokens and negligible for any realistic prompt.

Step 3: Map model names to encodings correctly

A common bug is hardcoding cl100k_base for every model. If you accidentally point it at a legacy model, your count drifts. Wrap the mapping in a function that falls back to the model’s declared family.

def encoding_for_model(model: str) -> str:
    if model.startswith(("gpt-4", "gpt-3.5")):
        return "cl100k_base"
    if model.startswith("text-davinci"):
        return "p50k_base"
    # default conservative choice
    return "cl100k_base"

def count_tokens_openai(text: str, model: str) -> int:
    enc = tiktoken.get_encoding(encoding_for_model(model))
    return len(enc.encode(text))

If you route through a gateway such as n4n.ai, which provides per-token usage metering across 240+ models, running this counter client-side lets you reject oversized requests before they incur cost. The gateway still reports final usage, but local pre-computation avoids wasted bandwidth on requests that exceed your budget.

Step 4: Count tokens for open-weight models with HuggingFace tokenizers

Open-weight models do not use tiktoken. Load the official tokenizer from the HuggingFace hub. The first call fetches config and weights (a few MB); subsequent calls are pure local compute.

from transformers import AutoTokenizer

def count_hf_tokens(text: str, model_repo: str) -> int:
    tok = AutoTokenizer.from_pretrained(model_repo)
    # return_tensors=None gives python list of ids
    ids = tok.encode(text, add_special_tokens=True)
    return len(ids)

# Example: Llama-3-8B-Instruct
print(count_hf_tokens("Hello, world", "meta-llama/Meta-Llama-3-8B-Instruct"))

Set add_special_tokens=True to mirror what the inference server does. Some models prepend a BOS token; others don’t. Check the model card if your count is off by one. For SentencePiece-based tokenizers (Llama, Mistral), whitespace is encoded as a special ▁ marker, so leading spaces change token boundaries—another reason to never approximate with len(text.split()).

Step 5: Handle chat templates and message formatting

Raw string counts lie. A chat completion request wraps your messages in a template that adds control tokens, role markers, and sometimes a trailing whitespace. Use the tokenizer’s chat template to produce the exact token stream.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
messages = [
    {"role": "system", "content": "You are terse."},
    {"role": "user", "content": "Count tokens."},
]
formatted = tok.apply_chat_template(messages, tokenize=False)
ids = tok.encode(formatted, add_special_tokens=False)
print(len(ids))

For OpenAI, tiktoken cannot know the exact chat template across versions. The safest offline approximation is to count each message content plus a per-message overhead of 4 tokens (per OpenAI’s documented rule) and 2 tokens for the reply prefix.

def count_chat_tokens_openai(messages: list[dict], model: str = "gpt-4") -> int:
    enc = tiktoken.get_encoding(encoding_for_model(model))
    total = 0
    for m in messages:
        total += 4  # role + structure overhead
        total += len(enc.encode(m["content"]))
    total += 2  # priming assistant reply
    return total

This matches the API’s usage.prompt_tokens within a token or two for typical conversations. If you embed images or audio, this formula breaks—multimodal inputs require provider-specific calculators.

Step 6: Verify your offline count against a real API response

Offline counting is useless if it’s wrong. Send a small request to the provider (or a local server) and compare usage.prompt_tokens to your local count.

import openai  # or any OpenAI-compatible client

client = openai.OpenAI()
resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Verify token count."}],
)
api_prompt_tokens = resp.usage.prompt_tokens

local = count_chat_tokens_openai([{"role": "user", "content": "Verify token count."}])
assert abs(api_prompt_tokens - local) <= 2, f"Mismatch: {api_prompt_tokens} vs {local}"

If the delta exceeds your tolerance, inspect whether the provider strips or adds a system prompt, or uses a different chat template version. For open-weight models served via vLLM or TGI, query the /tokenize endpoint if available, or check the server logs. Discrepancies of 1–3 tokens are normal due to trailing newline handling; discrepancies of 10+ signal a wrong tokenizer or missing special tokens.

Step 7: Build a reusable offline counter in your codebase

Engineers shouldn’t scatter tiktoken.get_encoding calls across services. Write a single dispatcher that picks the backend by model name.

class OfflineTokenCounter:
    def __init__(self):
        self._hf_cache = {}

    def count(self, text_or_messages, model: str) -> int:
        if model.startswith(("gpt-", "text-davinci")):
            if isinstance(text_or_messages, list):
                return count_chat_tokens_openai(text_or_messages, model)
            return count_tokens_openai(text_or_messages, model)
        # assume HF model repo or known family
        tok = self._hf_cache.get(model) or AutoTokenizer.from_pretrained(model)
        self._hf_cache[model] = tok
        if isinstance(text_or_messages, list):
            formatted = tok.apply_chat_template(text_or_messages, tokenize=False)
            return len(tok.encode(formatted, add_special_tokens=False))
        return len(tok.encode(text_or_messages, add_special_tokens=True))

counter = OfflineTokenCounter()
print(counter.count("Hello", "gpt-4o"))
print(counter.count("Hello", "meta-llama/Meta-Llama-3-8B-Instruct"))

Cache tokenizer instances. Loading a HF tokenizer repeatedly adds 100–300 ms per call. For high-throughput services, initialize the counter at startup and share it across workers. If you stream responses, count the prompt once before sending; count completions by accumulating deltas with the same encoding.

Step 8: Test and verify success

Wrap the counter in unit tests with known strings. For ASCII, cl100k_base encodes “hello world” as 2 tokens; verify that. For HF, pick a tokenizer and assert stable counts across runs.

def test_tiktoken_basic():
    assert count_tokens_openai("hello world", "gpt-4") == 2

def test_hf_basic():
    c = OfflineTokenCounter()
    n = c.count("hello world", "meta-llama/Meta-Llama-3-8B-Instruct")
    assert n > 0

Run pytest. Success means: (1) tests pass, (2) your offline count matches a live API usage.prompt_tokens within 2 tokens on a sample of real prompts, and (3) no network calls occur during count()—confirm with a socket blocker in CI.

import socket, pytest

def test_offline_no_network():
    with pytest.raises(Exception):
        socket.socket().connect(("api.openai.com", 443))
    # your count call here should not hit this

Counting tokens python offline is now a solved problem in your stack. You can pre-reject prompts over a limit, estimate spend before a request leaves your network, and avoid surprising bills from providers or gateways. The code above is production-grade; the only remaining work is wiring it into your request middleware.