How to count tokens before sending an API request

Counting tokens before you hit an LLM API saves money, prevents silent truncation, and lets you fit the maximum useful context into every request. Most providers bill by the token and enforce hard context limits, but they only tell you the count after the fact — or not at all. This guide shows you how to count accurately on the client side, handle model-specific tokenizers, and verify your counts match what the API actually sees.

Step 1: Pick the right tokenizer for your model

Tokenization is model-specific. GPT-4o uses o200k_base, Claude uses a different vocabulary, and open models like Llama 3 use their own. Using the wrong tokenizer gives you counts that are directionally correct but numerically wrong — sometimes by 15-20%.

For OpenAI-compatible models, use tiktoken. For Anthropic, use their anthropic-tokenizer package. For open models, load the tokenizer from Hugging Face transformers.

# tiktoken for OpenAI and compatible models
import tiktoken

def get_openai_encoding(model: str) -> tiktoken.Encoding:
    """Return the correct tiktoken encoding for a given model name."""
    try:
        return tiktoken.encoding_for_model(model)
    except KeyError:
        # Fallback for newer models not yet in tiktoken's registry
        # gpt-4o, gpt-4o-mini, o1 series all use o200k_base
        if any(m in model for m in ("gpt-4o", "o1", "o3")):
            return tiktoken.get_encoding("o200k_base")
        # Older default
        return tiktoken.get_encoding("cl100k_base")

For Anthropic models, the tokenizer isn’t publicly exposed with the same fidelity, but their SDK provides a counting endpoint. For local open models:

# transformers for Llama, Mistral, Qwen, etc.
from transformers import AutoTokenizer

def get_hf_tokenizer(model_id: str) -> AutoTokenizer:
    """Load a tokenizer from Hugging Face Hub."""
    return AutoTokenizer.from_pretrained(model_id, use_fast=True)

Verification tip: Send a known string to the API and compare the returned usage.prompt_tokens against your local count. They should match exactly for OpenAI models. For Anthropic, expect ±1-2 token variance due to internal formatting.

Step 2: Count tokens in a simple string

The baseline operation is counting a single string. This is useful for estimating costs before you build a full request.

def count_tokens(text: str, encoding: tiktoken.Encoding) -> int:
    """Count tokens in a plain text string."""
    return len(encoding.encode(text))

# Example
enc = get_openai_encoding("gpt-4o")
prompt = "Summarize the following document in three bullet points."
print(f"Token count: {count_tokens(prompt, enc)}")  # ~12 tokens

Verification: Run the same string through the OpenAI tokenizer web tool (platform.openai.com/tokenizer) and confirm the count matches.

Step 3: Count tokens in a chat completion request

Real requests use the chat format with roles, message boundaries, and special tokens. OpenAI’s chat format adds overhead: each message has a header (<|im_start|>role), content, and footer (<|im_end|>), plus a final assistant primer. The exact formula varies by model family.

def count_chat_tokens(
    messages: list[dict[str, str]],
    encoding: tiktoken.Encoding,
    model: str = "gpt-4o"
) -> int:
    """
    Count tokens for a chat completion request.
    Based on OpenAI's published counting rules for o200k_base models.
    """
    # Every reply is primed with <|im_start|>assistant
    token_count = 3  # <|im_start|>assistant<|im_sep|>
    
    for msg in messages:
        role = msg["role"]
        content = msg.get("content", "")
        
        # Message header: <|im_start|>{role}<|im_sep|>
        token_count += 4
        # Content tokens
        token_count += len(encoding.encode(content))
        # Message footer: <|im_end|>
        token_count += 1
    
    return token_count

# Example
messages = [
    {"role": "system", "content": "You are a concise summarizer."},
    {"role": "user", "content": "Summarize this: " + "x" * 5000},
]
enc = get_openai_encoding("gpt-4o")
print(f"Estimated prompt tokens: {count_chat_tokens(messages, enc)}")

Verification: Make a real API call with max_tokens=1 and check usage.prompt_tokens in the response. Adjust the overhead constants if your model family differs (e.g., gpt-3.5-turbo uses slightly different framing).

Step 4: Handle tool/function definitions

If you use function calling, the schema definitions consume tokens too. OpenAI injects a JSON schema representation into the system prompt. Count it by serializing your tool definitions the same way the API does.

import json

def count_tool_tokens(tools: list[dict], encoding: tiktoken.Encoding) -> int:
    """
    Approximate token count for tool definitions.
    OpenAI serializes the function schema as a JSON string in the system prompt.
    """
    # The API injects something like:
    # "namespace functions { type FunctionName = (_: { ... }) => any; }"
    # We approximate by encoding the JSON schema plus overhead.
    tool_json = json.dumps({"tools": tools}, separators=(",", ":"))
    # Rough overhead for the namespace wrapper and type definitions
    overhead = 50  
    return len(encoding.encode(tool_json)) + overhead

tools = [{
    "type": "function",
    "function": {
        "name": "search_web",
        "description": "Search the web for current information",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {"type": "string", "description": "Search query"}
            },
            "required": ["query"]
        }
    }
}]
print(f"Tool tokens: {count_tool_tokens(tools, enc)}")

Verification: Compare against a real request with tools attached. The variance is typically under 5% for simple schemas, higher for deeply nested ones.

Step 5: Account for images (vision models)

Vision tokens depend on image resolution and the model’s patch size. GPT-4o uses a variable token count: a base cost plus per-tile tokens. The formula is documented but easier to implement as a lookup.

def count_image_tokens(width: int, height: int, model: str = "gpt-4o") -> int:
    """
    Estimate vision tokens for GPT-4o family.
    Based on OpenAI's documented pricing: 85 tokens base + 170 per 512x512 tile.
    """
    if "gpt-4o" not in model:
        raise ValueError("Formula validated for gpt-4o family only")
    
    # Resize logic: max dimension 2048, min 768, then tile to 512
    max_dim = max(width, height)
    min_dim = min(width, height)
    
    # Scale down if needed
    if max_dim > 2048:
        scale = 2048 / max_dim
        width = int(width * scale)
        height = int(height * scale)
    elif min_dim < 768:
        scale = 768 / min_dim
        width = int(width * scale)
        height = int(height * scale)
    
    # Tile count
    tiles_x = (width + 511) // 512
    tiles_y = (height + 511) // 512
    tiles = tiles_x * tiles_y
    
    return 85 + 170 * tiles

# 1024x1024 image = 4 tiles = 85 + 680 = 765 tokens
print(f"1024x1024 image tokens: {count_image_tokens(1024, 1024)}")

Verification: Send a single image with a minimal prompt ("What is this?") and check usage.prompt_tokens. Subtract the text token count; the remainder should match your calculation.

Step 6: Build a pre-flight check function

Combine the pieces into a single function you call before every request. This is where you enforce context limits, estimate cost, and decide whether to truncate.

from dataclasses import dataclass
from typing import Optional

@dataclass
class TokenEstimate:
    prompt_tokens: int
    max_completion_tokens: int
    estimated_total: int
    context_limit: int
    will_fit: bool
    estimated_cost_usd: float

MODEL_LIMITS = {
    "gpt-4o": 128_000,
    "gpt-4o-mini": 128_000,
    "o1-preview": 128_000,
    "o1-mini": 128_000,
    "gpt-4-turbo": 128_000,
    "gpt-3.5-turbo": 16_384,
}

# Rough pricing per 1M tokens (input, output) — update from provider pricing page
MODEL_PRICING = {
    "gpt-4o": (2.50, 10.00),
    "gpt-4o-mini": (0.15, 0.60),
    "o1-preview": (15.00, 60.00),
    "o1-mini": (3.00, 12.00),
}

def estimate_request(
    messages: list[dict],
    model: str,
    max_completion_tokens: int,
    tools: Optional[list[dict]] = None,
    images: Optional[list[tuple[int, int]]] = None,  # (width, height)
) -> TokenEstimate:
    """Full pre-flight token and cost estimate."""
    encoding = get_openai_encoding(model)
    context_limit = MODEL_LIMITS.get(model, 128_000)
    
    # Count messages
    prompt_tokens = count_chat_tokens(messages, encoding, model)
    
    # Add tools
    if tools:
        prompt_tokens += count_tool_tokens(tools, encoding)
    
    # Add images
    if images:
        for w, h in images:
            prompt_tokens += count_image_tokens(w, h, model)
    
    estimated_total = prompt_tokens + max_completion_tokens
    will_fit = estimated_total <= context_limit
    
    # Cost estimate
    input_price, output_price = MODEL_PRICING.get(model, (0, 0))
    estimated_cost = (prompt_tokens * input_price + max_completion_tokens * output_price) / 1_000_000
    
    return TokenEstimate(
        prompt_tokens=prompt_tokens,
        max_completion_tokens=max_completion_tokens,
        estimated_total=estimated_total,
        context_limit=context_limit,
        will_fit=will_fit,
        estimated_cost_usd=estimated_cost,
    )

# Usage
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing in simple terms."},
]
estimate = estimate_request(
    messages=messages,
    model="gpt-4o-mini",
    max_completion_tokens=500,
)
print(f"Prompt: {estimate.prompt_tokens} tokens")
print(f"Total: {estimate.estimated_total} / {estimate.context_limit}")
print(f"Fits: {estimate.will_fit}")
print(f"Est. cost: ${estimate.estimated_cost_usd:.6f}")

Verification: Log the estimate alongside the actual usage object from every API response. Build a dashboard or alert on divergence >2%.

Step 7: Truncate intelligently when you exceed the limit

Counting is only useful if you act on it. When will_fit is false, truncate the conversation history — not the system prompt, not the current user message. Preserve the most recent context first.

def truncate_messages(
    messages: list[dict],
    encoding: tiktoken.Encoding,
    model: str,
    max_prompt_tokens: int,
    reserve_tokens: int = 100,  # buffer for tools, images, formatting
) -> list[dict]:
    """
    Truncate message history to fit within max_prompt_tokens.
    Keeps system prompt and most recent messages.
    """
    system_msgs = [m for m in messages if m["role"] == "system"]
    other_msgs = [m for m in messages if m["role"] != "system"]
    
    # Count system tokens (usually small, keep all)
    system_tokens = sum(
        len(encoding.encode(m.get("content", ""))) + 4  # header + footer
        for m in system_msgs
    )
    
    available = max_prompt_tokens - system_tokens - reserve_tokens
    if available <= 0:
        raise ValueError("System prompt alone exceeds budget")
    
    # Add messages from most recent backwards
    kept = []
    used = 0
    for msg in reversed(other_msgs):
        msg_tokens = len(encoding.encode(msg.get("content", ""))) + 5  # header + footer + role
        if used + msg_tokens > available:
            break
        kept.insert(0, msg)
        used += msg_tokens
    
    return system_msgs + kept

# Example: force a truncation
long_history = [
    {"role": "system", "content": "You are a helpful assistant."},
] + [
    {"role": "user" if i % 2 == 0 else "assistant", "content": f"Message {i} " + "x" * 2000}
    for i in range(20)
]

enc = get_openai_encoding("gpt-4o-mini")
truncated = truncate_messages(long_history, enc, "gpt-4o-mini", max_prompt_tokens=8000)
print(f"Original: {len(long_history)} messages, Truncated: {len(truncated)} messages")

Verification: After truncation, run count_chat_tokens on the result and confirm it’s under your budget. Run the actual API call and verify no 400 context length exceeded errors.

Step 8: Handle streaming and reasoning models

Streaming responses don’t change prompt token counting, but reasoning models (o1, o3) add a hidden reasoning_tokens bucket in usage.completion_tokens_details. You can’t predict this locally, but you must budget for it in max_completion_tokens.

def estimate_with_reasoning_budget(
    messages: list[dict],
    model: str,
    max_completion_tokens: int,
    reasoning_budget_ratio: float = 0.3,
) -> TokenEstimate:
    """
    For reasoning models, reserve a portion of max_completion_tokens for hidden reasoning.
    """
    estimate = estimate_request(messages, model, max_completion_tokens)
    
    if "o1" not in model and "o3" not in model
    else int(max_completion_tokens * (1 - reasoning_budget_ratio))
    
    # Adjust: the visible output tokens will be lower
    visible_output = int(max_completion_tokens * (1 - reasoning_budget_ratio))
    reasoning_tokens = max_completion_tokens - visible_output
    
    return TokenEstimate(
        prompt_tokens=estimate.prompt_tokens,
        max_completion_tokens=visible_output,
        estimated_total=estimate.prompt_tokens + max_completion_tokens,
        context_limit=estimate.context_limit,
        will_fit=estimate.will_fit,
        estimated_cost_usd=estimate.estimated_cost_usd,  # pricing includes reasoning
    )

Verification: After an o1/o3 call, check usage.completion_tokens_details.reasoning_tokens. Track the ratio over time and adjust reasoning_budget_ratio per use case.

Step 9: Integrate into your request pipeline

Wrap the estimate in a middleware or decorator so every outbound request gets counted automatically. This prevents accidental overspend and gives you centralized logging.

import functools
from typing import Callable, Any

def with_token_accounting(model: str, max_completion_tokens: int):
    """Decorator that logs token estimates and enforces limits."""
    def decorator(func: Callable[..., Any]):
        @functools.wraps(func)
        def wrapper(messages: list[dict], **kwargs):
            estimate = estimate_request(
                messages=messages,
                model=model,
                max_completion_tokens=max_completion_tokens,
                tools=kwargs.get("tools"),
                images=kwargs.get("images"),
            )
            
            # Log for observability
            print(f"[token-accounting] model={model} prompt={estimate.prompt_tokens} "
                  f"max_completion={estimate.max_completion_tokens} "
                  f"total={estimate.estimated_total}/{estimate.context_limit} "
                  f"cost≈${estimate.estimated_cost_usd:.6f}")
            
            if not estimate.will_fit:
                # Auto-truncate or raise
                encoding = get_openai_encoding(model)
                messages = truncate_messages(
                    messages, encoding, model, 
                    max_prompt_tokens=estimate.context_limit - max_completion_tokens - 100
                )
                # Re-estimate after truncation
                estimate = estimate_request(messages, model, max_completion_tokens, 
                                            kwargs.get("tools"), kwargs.get("images"))
                print(f"[token-accounting] Truncated to {estimate.prompt_tokens} prompt tokens")
            
            return func(messages, **kwargs)
        return wrapper
    return decorator

# Usage
@with_token_accounting(model="gpt-4o-mini", max_completion_tokens=1000)
def call_llm(messages: list[dict], **kwargs):
    # Your actual API call here
    pass

Verification: Ship this to staging. Tail logs for a day and confirm:

• No context length exceeded errors
• Estimated vs actual prompt tokens match within 1-2 tokens
• Cost estimates align with billing reports at month end

Step 10: Test edge cases systematically

Token counting breaks in predictable ways. Add these to your test suite.

import pytest

class TestTokenCounting:
    def test_empty_message(self):
        enc = get_openai_encoding("gpt-4o")
        assert count_chat_tokens([{"role": "user", "content": ""}], enc) > 0
    
    def test_unicode_emoji(self):
        enc = get_openai_encoding("gpt-4o")
        # Emoji often tokenize to multiple tokens
        count = count_tokens("🎉🚀💯", enc)
        assert count >= 3  # at least one per emoji
    
    def test_very_long_single_token_word(self):
        enc = get_openai_encoding("gpt-4o")
        # "antidisestablishmentarianism" is one token in cl100k_base
        count = count_tokens("antidisestablishmentarianism", enc)
        assert count == 1
    
    def test_truncation_preserves_system(self):
        enc = get_openai_encoding("gpt-4o-mini")
        msgs = [
            {"role": "system", "content": "x" * 5000},
            {"role": "user", "content": "hello"},
        ]
        truncated = truncate_messages(msgs, enc, "gpt-4o-mini", max_prompt_tokens=100)
        assert truncated[0]["role"] == "system"
    
    def test_tool_counting_matches_api(self):
        """Requires network — run in CI nightly."""
        pass  # Implement with real API call and compare

if __name__ == "__main__":
    pytest.main([__file__, "-v"])

Run these on every PR. The unicode and truncation tests catch the most common production bugs.

Counting tokens locally is table stakes for any production LLM integration. The code above covers the 95% case: OpenAI-compatible chat completions with optional tools and vision. For Anthropic, Bedrock, or Vertex, the principle is identical — swap the tokenizer and the framing constants. The key discipline is verifying your counts against actual API responses continuously, not once at launch.