Every token you send to an LLM costs money and consumes context window. To avoid surprises, you should count tokens with tiktoken before you send a request, especially when you batch conversations or embed long system prompts.
tiktoken is OpenAI’s open-source BPE tokenizer. It runs locally, has no network dependency, and is fast enough to call on every request in a hot path. The counts it produces match what OpenAI and OpenAI-compatible endpoints report for the major model families, which makes it the right tool for pre-flight checks.
Step 1: Install and import tiktoken
Install the package from PyPI. It has minimal dependencies and compiles a Rust extension, so build time is short.
pip install tiktokenImport it in your Python module. If you are on an older environment, pin to a recent version (>= 0.5.0) to get the latest model aliases.
import tiktoken
print(tiktoken.__version__)You now have the library available. No API key is required for counting.
Step 2: Load the correct encoding for your model
The token count depends on the encoding, not just the text. GPT-3.5-Turbo, GPT-4, and GPT-4-Turbo use cl100k_base. GPT-4o and newer OpenAI models use o200k_base. Other providers may ship different tokenizers, but any OpenAI-compatible endpoint that proxies those model names will honor the same counting rules.
The safest way to count tokens with tiktoken for a known model is to let the library resolve the alias:
def get_encoding(model: str):
try:
return tiktoken.encoding_for_model(model)
except KeyError:
# Fallback for unknown model names; cl100k_base covers most chat models
return tiktoken.get_encoding("cl100k_base")
enc = get_encoding("gpt-4o")
print(enc.name) # o200k_baseIf you pass a model string that the library does not recognize, encoding_for_model raises KeyError. In that case, default to cl100k_base unless you have evidence the target model uses a different vocab. Guessing wrong biases your estimate by a few percent, not an order of magnitude.
Step 3: Count tokens in a raw string
For a single prompt string, encoding and measuring length is one line.
text = "Summarize the following RFC in three bullet points: "
tokens = enc.encode(text)
print(len(tokens), "tokens")encode returns a list of integer token IDs. The length of that list is your token count. If you need the string forms for debugging, call enc.decode_single_token_bytes or enc.decode on slices, but never decode untrusted token sequences back into text in production—it is unnecessary work.
A common mistake is using len(text.split()) as a proxy. English prose averages ~1.3 tokens per word, but code, JSON, and non-Latin scripts diverge sharply. Count tokens with tiktoken instead of approximating.
Step 4: Count tokens in a chat conversation
Chat completions are not just concatenated strings. OpenAI’s token accounting adds overhead per message and per conversation. The well-known heuristic from the OpenAI cookbook still holds for the cl100k_base and o200k_base models:
- 3 tokens of overhead per reply (priming the model to generate)
- 1 token per message for the role delimiter
- 2 tokens per message for the
content/role framing
Implement it explicitly so you can tune it:
def num_tokens_from_messages(messages, model="gpt-4o"):
enc = get_encoding(model)
# token overheads are stable across cl100k and o200k
tokens_per_message = 3
tokens_per_name = 1
num_tokens = 0
for message in messages:
num_tokens += tokens_per_message
for key, value in message.items():
num_tokens += len(enc.encode(value))
if key == "name":
num_tokens += tokens_per_name
num_tokens += 3 # reply priming
return num_tokens
messages = [
{"role": "system", "content": "You are a terse API assistant."},
{"role": "user", "content": "Count tokens with tiktoken for this list: [1,2,3]"},
]
print(num_tokens_from_messages(messages, "gpt-4o"))This function does not account for tool definitions or function calls. If you send tools, add roughly 5 tokens per tool plus the serialized JSON of each schema. Measure once with a real API call and cache the delta if you use tools heavily.
Step 5: Estimate cost and enforce a pre-send budget
Once you have a token count, cost is a multiplication. Do not hardcode prices in business logic; load them from config so you can react to provider changes.
PRICE_PER_1K_PROMPT = {
"gpt-4o": 0.005, # example only; replace with your current rate
}
def estimate_cost(token_count, model):
rate = PRICE_PER_1K_PROMPT.get(model, 0.001)
return (token_count / 1000) * rate
prompt_tokens = num_tokens_from_messages(messages, "gpt-4o")
cost = estimate_cost(prompt_tokens, "gpt-4o")
print(f"~{prompt_tokens} prompt tokens, ${cost:.4f} estimated")Set a hard ceiling in your service. If prompt_tokens exceeds, say, 90% of the model’s context window, truncate the oldest non-system messages before sending. This guard prevents 400 errors and runaway bills.
MAX_CTX = 128_000 # gpt-4o context window
def truncate_to_fit(messages, model, max_ctx=MAX_CTX):
while num_tokens_from_messages(messages, model) > max_ctx - 1000:
# never drop the system message
if len(messages) > 1 and messages[0]["role"] == "system":
messages.pop(1)
else:
messages.pop(0)
return messagesStep 6: Send the request and verify the count
Call your LLM provider and compare its reported prompt_tokens to your local count. The two should match exactly for string and chat inputs that do not use special server-side templating.
from openai import OpenAI
client = OpenAI() # or any OpenAI-compatible base_url
resp = client.chat.completions.create(
model="gpt-4o",
messages=messages,
)
print("API prompt_tokens:", resp.usage.prompt_tokens)
print("Local count:", num_tokens_from_messages(messages, "gpt-4o"))If you route through a gateway such as n4n.ai, its per-token usage metering will return the same prompt_tokens field on the response, letting you reconcile your pre-send estimate against the billed amount without custom instrumentation.
A mismatch of more than a few tokens usually means you forgot message overhead, added tools, or the provider injected a system prompt you did not count. Log the difference and adjust your heuristic.
Step 7: Wrap it in a pre-send middleware
Engineers often bolt counting on after the fact. Instead, make it part of the client call site:
def guarded_chat(client, model, messages, max_ctx=MAX_CTX):
enc_model = model
messages = truncate_to_fit(messages, enc_model, max_ctx)
local_tokens = num_tokens_from_messages(messages, enc_model)
if local_tokens > max_ctx - 1000:
raise ValueError("Truncation failed to fit context")
resp = client.chat.completions.create(model=model, messages=messages)
assert resp.usage.prompt_tokens == local_tokens, (
f"Token mismatch: api={resp.usage.prompt_tokens} local={local_tokens}"
)
return resp
# usage
resp = guarded_chat(client, "gpt-4o", messages)This pattern makes the count visible in logs, blocks oversized requests, and fails loudly when the provider’s tokenizer diverges from tiktoken. In high-throughput services, cache the encoding object at module load—encoding_for_model is cheap but not free on every call.
Verify success
You have a working pipeline when:
len(enc.encode(text))matches the provider’sprompt_tokensfor a single-string completion.num_tokens_from_messagesmatchesusage.prompt_tokensfor a multi-turn chat within zero tolerance (no tools).- Your cost estimate before send is within rounding of the metered charge on the invoice.
- Oversized inputs are truncated or rejected before the network call, not after a 400 response.
Run a unit test with fixed strings and known token counts from the tiktoken repo to lock the behavior. For example, "hello world" is 2 tokens in cl100k_base. If your test passes, your counting layer is correct.
Token counting is not glamorous, but it is the difference between a demo and a system that survives contact with real traffic. Count tokens with tiktoken at the edge of your request path, and you will never wake up to a ten-thousand-dollar prompt loop again.