What is a token and how LLM pricing is calculated

A token is the atomic unit of text that a large language model processes — typically a word fragment, whole word, or punctuation mark — and LLM pricing is calculated by counting the tokens in your prompt (input) and the model’s response (output), then multiplying by the provider’s per-token rate. Understanding tokenization is essential because it directly determines your inference bill, context window limits, and latency profile.

How tokenization works

Tokenization converts raw text into a sequence of integer IDs that the model can process. Each model family uses its own tokenizer with a fixed vocabulary. GPT-4 and GPT-3.5 use cl100k_base (100,277 tokens). Llama 3 uses a 128,000-token BPE vocabulary. These vocabularies are not interchangeable — the same string produces different token counts and different IDs across models.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
text = "Hello, world!"
tokens = enc.encode(text)
print(tokens)        # [9906, 11, 1917, 0]
print(len(tokens))   # 4 tokens

The tokenizer applies byte-pair encoding (BPE) or a variant: it starts with bytes, then iteratively merges the most frequent adjacent pairs until reaching the vocabulary size. This means common words like “the” or “ing” often become single tokens, while rare words split into multiple pieces.

Input vs output token accounting

Providers charge separately for input (prompt) tokens and output (completion) tokens. Output tokens are typically 2–5× more expensive because generation requires sequential forward passes — each new token depends on all previous ones — while prompt processing can be parallelized.

{
  "model": "gpt-4o",
  "usage": {
    "prompt_tokens": 150,
    "completion_tokens": 80,
    "total_tokens": 230
  }
}

A typical pricing table (illustrative, not current):

Model	Input / 1M tokens	Output / 1M tokens
GPT-4o	$2.50	$10.00
GPT-4o-mini	$0.15	$0.60
Claude 3.5 Sonnet	$3.00	$15.00
Llama 3.1 70B (via provider)	$0.80	$0.80

Multiply your token counts by the per-million rate. For the example above with GPT-4o: (150 × $2.50 + 80 × $10.00) / 1,000,000 = $0.001175.

Context window and token budgets

Every model has a maximum context window — the total tokens (prompt + completion) it can accept in a single request. Exceeding it returns an error. You must budget tokens across system prompt, conversation history, user message, and reserved space for the response.

MAX_CONTEXT = 128_000  # GPT-4o
RESERVED_OUTPUT = 4_000

def truncate_history(messages, system_prompt, user_message):
    enc = tiktoken.encoding_for_model("gpt-4o")
    system_tokens = len(enc.encode(system_prompt))
    user_tokens = len(enc.encode(user_message))
    available = MAX_CONTEXT - RESERVED_OUTPUT - system_tokens - user_tokens
    
    history = []
    for msg in reversed(messages):
        msg_tokens = len(enc.encode(msg["content"]))
        if msg_tokens > available:
            break
        history.insert(0, msg)
        available -= msg_tokens
    return history

This sliding-window approach is standard. More sophisticated strategies summarize older turns or use retrieval-augmented generation to stay within budget.

Why token counts differ from character or word counts

Rough heuristics: English text averages ~4 characters per token, ~0.75 words per token. But variance is high:

Code: denser, more tokens per character (lots of symbols, indentation)
Non-Latin scripts: often 1–2 characters per token (each character may be a token)
Numbers: each digit often tokenizes separately (“2024” → 4 tokens)
Whitespace and newlines: count as tokens

samples = {
    "english": "The quick brown fox jumps over the lazy dog.",
    "code": "def fibonacci(n):\n    if n <= 1:\n        return n\n    return fibonacci(n-1) + fibonacci(n-2)",
    "japanese": "こんにちは、世界！",
    "numbers": "Invoice #2024-001234 total: $1,234.56"
}

enc = tiktoken.get_encoding("cl100k_base")
for name, text in samples.items():
    print(f"{name:10} {len(text):3} chars → {len(enc.encode(text)):3} tokens")

Output:

english       44 chars →  10 tokens
code          97 chars →  31 tokens
japanese      10 chars →  10 tokens
numbers       38 chars →  22 tokens

This variance matters when estimating costs for multilingual or code-heavy workloads.

Cached input tokens and prefix caching

Some providers (Anthropic, Google, and others) discount repeated prompt prefixes — system prompts, few-shot examples, or long documents sent across multiple requests. The cache key is typically the exact token sequence. If your prompt prefix matches a recent request, you pay a fraction of the input cost.

# Conceptual: provider returns cache metadata
response = client.chat.completions.create(
    model="claude-3-5-sonnet-20241022",
    messages=[{"role": "user", "content": long_prompt + question}],
    extra_headers={"anthropic-beta": "prompt-caching-2024-07-31"}
)

print(response.usage)
# {
#   "input_tokens": 15000,
#   "cache_creation_input_tokens": 12000,  # new prefix
#   "cache_read_input_tokens": 3000,       # cached prefix
#   "output_tokens": 500
# }

Design your prompts so the stable prefix (system prompt, context documents) comes first and remains identical across requests. Even whitespace changes invalidate the cache.

Streaming and token counting

When streaming responses, you receive tokens incrementally. The final usage object arrives at stream end. If you need real-time cost estimation, count tokens client-side as they arrive using the same tokenizer.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")
output_tokens = 0

for chunk in stream:
    if chunk.choices[0].delta.content:
        content = chunk.choices[0].delta.content
        output_tokens += len(enc.encode(content))
        print(content, end="", flush=True)

print(f"\nEstimated output tokens: {output_tokens}")

Note: the tokenizer must match the model exactly. Using cl100k_base for GPT-4o is correct; using it for a Llama model would produce wrong counts.

Common misconceptions

“Tokens equal words”

False. As shown above, the ratio varies by language, domain, and tokenizer. Budgeting by word count will misestimate costs by 30–50% for code or non-English text.

“All providers count tokens the same way”

False. Each provider uses its own tokenizer. Anthropic’s Claude models use a different vocabulary than OpenAI’s. A 1,000-token prompt for GPT-4o may be 1,200 tokens for Claude 3.5 Sonnet. Always count with the target model’s tokenizer.

“Output tokens cost the same as input tokens”

Rarely true. Output is almost always more expensive. Some open-model providers (e.g., Together, Fireworks) charge symmetric rates, but major closed-model APIs do not.

“The context window is the only limit”

There are also per-request output token limits (often 4,096 or 8,192) separate from the context window. You can have a 128k context window but only generate 4k tokens per request. Plan for multi-turn generation if you need longer outputs.

“Streaming saves tokens”

Streaming does not reduce token count — it only changes delivery. You pay for the same tokens. It can improve perceived latency and allow early termination, which does save tokens if you stop generation early.

Estimating costs before production

Build a token counter into your development workflow. Log prompt and completion tokens per request, aggregate by model and feature, and project monthly spend.

from dataclasses import dataclass
from collections import defaultdict

@dataclass
class TokenUsage:
    model: str
    prompt_tokens: int
    completion_tokens: int
    timestamp: float

# In your request wrapper
def track_usage(model, prompt_tokens, completion_tokens):
    usage = TokenUsage(model, prompt_tokens, completion_tokens, time.time())
    REDIS.lpush("token_usage_log", json.dumps(asdict(usage)))

# Daily aggregation job
def estimate_monthly_cost():
    rates = {
        "gpt-4o": (2.50, 10.00),
        "gpt-4o-mini": (0.15, 0.60),
    }
    daily = defaultdict(lambda: {"in": 0, "out": 0})
    for entry in REDIS.lrange("token_usage_log", 0, -1):
        u = json.loads(entry)
        daily[u["model"]]["in"] += u["prompt_tokens"]
        daily[u["model"]]["out"] += u["completion_tokens"]
    
    for model, tokens in daily.items():
        in_rate, out_rate = rates.get(model, (0, 0))
        daily_cost = (tokens["in"] * in_rate + tokens["out"] * out_rate) / 1_000_000
        print(f"{model}: ${daily_cost:.4f}/day → ${daily_cost * 30:.2f}/month")

This instrumentation pays for itself quickly. Without it, you’re guessing.

Routing and fallback considerations

When you route requests across multiple providers — for cost, latency, or availability — token counting becomes a coordination problem. Each provider returns usage in its own format. Normalize to a canonical schema before logging or billing.

# Normalized usage envelope
class NormalizedUsage:
    model: str
    provider: str
    prompt_tokens: int
    completion_tokens: int
    cached_prompt_tokens: int = 0
    latency_ms: int
    timestamp: float

If you use a gateway that handles automatic fallback (e.g., when a primary provider is rate-limited), the fallback model may have a different tokenizer and pricing. Your cost projections must account for the possible models, not just the primary.

Summary checklist

Count with the correct tokenizer for each model you use. Cache the tokenizer instance.
Separate input and output budgets — output costs more and has its own limit.
Design for prefix caching: keep stable context at the prompt start, byte-identical across requests.
Log usage per request with model, provider, and timestamp. Aggregate daily.
Test with real data: your domain’s token/word ratio will differ from generic benchmarks.
Plan for fallback models — know their tokenizers and rates before you need them.

Token accounting is not glamorous, but it is the difference between predictable inference costs and surprise five-figure bills. Treat it like any other production metric: instrument, alert, and review.