Input tokens vs output tokens: why pricing differs

Most engineers notice the pricing asymmetry immediately: input tokens cost a fraction of output tokens on every major provider. GPT-4o charges $2.50 per million input tokens versus $10 per million output tokens. Claude 3.5 Sonnet runs $3 versus $15. The ratio varies — typically 1:4 to 1:5 — but the pattern holds. This isn’t arbitrary markup. The compute, memory, and scheduling characteristics of generation differ fundamentally from prompt processing, and the pricing reflects that reality.

The compute asymmetry

Processing input tokens is a parallel operation. The entire prompt — whether 500 tokens or 100,000 — feeds through the model in a single forward pass (or a handful for very long contexts with chunked attention). The KV cache gets built once. Matrix multiplications batch efficiently across the sequence dimension. GPU utilization stays high.

Generation is sequential. Each output token requires a full forward pass conditioned on all previous tokens. The KV cache grows with every step, consuming memory bandwidth. You cannot batch across the sequence dimension because token n+1 depends on token n. This forces the model into a memory-bound regime where arithmetic intensity drops and hardware sits underutilized.

# Simplified view of the difference
def process_prompt(prompt_tokens):          # Parallelizable
    kv_cache = model.prefill(prompt_tokens) # One large matmul
    return kv_cache

def generate_next_token(kv_cache):          # Sequential, memory-bound
    logits = model.decode_step(kv_cache)    # Small matmul, large memory read
    next_token = sample(logits)
    kv_cache.append(next_token)
    return next_token, kv_cache

The prefill phase saturates compute. The decode phase saturates memory bandwidth. NVIDIA H100s deliver ~2000 TFLOPS of BF16 compute but only ~3 TB/s of HBM bandwidth. During decode, you hit the bandwidth wall long before you hit the compute wall.

Memory pressure and KV cache economics

The KV cache scales linearly with context length and batch size. For a 70B parameter model with 80 layers, 128-head attention, and 128-dimension heads, each token consumes roughly 1.6 MB of FP16 KV cache (2 * layers * heads * head_dim * 2 bytes). A 100K context window ties up ~160 MB per sequence. At batch size 32, that’s 5 GB just for KV cache — before model weights, activations, or scheduler overhead.

Providers must provision enough HBM to hold the model weights plus the maximum concurrent KV cache. Output tokens extend the KV cache. Input tokens only populate it once. This makes output tokens more expensive to serve at scale: they increase peak memory pressure and reduce the maximum concurrent requests a GPU can handle.

{
  "model": "llama-3-70b",
  "kv_cache_per_token_mb": 1.6,
  "context_100k_mb": 160,
  "batch_32_gb": 5.1,
  "model_weights_gb": 140,
  "total_per_gpu_gb": 145
}

When you pay for output tokens, you’re partly paying for the memory reservation that generation holds hostage for the duration of the request.

Scheduling and batching constraints

Prefill requests can be batched arbitrarily. A server handling 100 concurrent prefill requests processes them as one large batch, maximizing throughput. Decode requests cannot merge this way — each request is at a different position in its generation sequence. Continuous batching (also called iteration-level scheduling) mitigates this by evicting finished sequences and admitting new prefill work into the same batch, but the decode portion remains fundamentally serial per request.

This scheduling reality means output tokens consume more GPU-seconds per token. A 1000-token completion might occupy a GPU slot for 2-3 seconds. The same 1000 tokens as input processes in ~50ms. Providers price accordingly.

Pricing comparison across providers

Ratios cluster around 1:4 for frontier models and 1:3-1:4 for smaller models. The consistency suggests the ratio reflects hardware economics more than provider strategy.

Caching changes the equation

Prompt caching — where providers store and reuse the KV cache for repeated prefixes — shifts cost toward input tokens for workloads with high prefix overlap. Anthropic charges ~$0.30/M for cached input tokens on Claude 3.5 Sonnet (90% discount). OpenAI’s cached input pricing on GPT-4o is ~$1.25/M (50% discount). Google’s context caching on Gemini 1.5 Pro runs $0.3125/M (75% discount).

# Example: 10K system prompt, 1K user query, 500 token response
# Without caching (per request):
#   Input: 11,000 tokens × $3/M = $0.033
#   Output: 500 tokens × $15/M = $0.0075
#   Total: $0.0405

# With 90% cache hit on system prompt (per request after first):
#   Cached input: 10,000 × $0.30/M = $0.003
#   Fresh input: 1,000 × $3/M = $0.003
#   Output: 500 × $15/M = $0.0075
#   Total: $0.0135  (67% reduction)

Caching makes input-heavy workloads cheaper but doesn’t affect output pricing. If your workload generates long completions (code generation, long-form writing, reasoning traces), output tokens still dominate the bill.

Output token optimization strategies

Constrain generation length

The single highest-leverage knob is max_tokens (or max_completion_tokens). Set it to the minimum viable length. Many applications set generous defaults (4096, 8192) and pay for tokens that get truncated or discarded downstream.

# Bad: generous default
response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    max_tokens=8192  # Often wastes $0.08+/request
)

# Better: calibrated to task
response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    max_tokens=500   # Typical answer fits here
)

Use structured outputs to avoid retry loops

Function calling and JSON mode reduce the need for “please output valid JSON” prompt engineering and the retries when the model fails. Each retry doubles your output token cost for that request.

# Structured output eliminates parsing retries
response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    response_format={"type": "json_object"},
    max_tokens=1000
)
# One pass, valid JSON guaranteed

Prefer smaller models for high-volume generation

If you generate millions of tokens daily, the model tier decision compounds. A 1:4 input:output ratio on GPT-4o-mini ($0.15/$0.60) versus GPT-4o ($2.50/$10) means output tokens cost 16x less on the smaller model. For classification, extraction, or templated generation tasks, the quality gap is often negligible.

# Route by task complexity
def route_model(task_type: str) -> str:
    if task_type in ("classification", "extraction", "formatting"):
        return "gpt-4o-mini"
    elif task_type in ("coding", "reasoning", "creative"):
        return "gpt-4o"
    return "gpt-4o-mini"

Stream and truncate early

Streaming lets you detect runaway generations and cut them off. If a model starts hallucinating or looping, you can close the stream before hitting max_tokens.

async def generate_with_guardrails(messages, max_tokens=2000, timeout_s=30):
    collected = []
    async for chunk in client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        max_tokens=max_tokens,
        stream=True
    ):
        collected.append(chunk)
        if len(collected) > max_tokens * 0.9:
            # Approaching limit, check quality
            if detect_loop(collected):
                break
    return collected

Input token optimization strategies

Compress context before sending

For RAG workloads, retrieve fewer chunks or summarize them before inclusion. A 20-chunk retrieval at 500 tokens each = 10,000 input tokens ($0.03 on Sonnet). Summarizing to 3 chunks at 200 tokens each = 600 tokens ($0.0018). The summarization call costs output tokens but pays for itself in 2-3 queries.

def compress_context(chunks, query, budget_tokens=2000):
    if estimate_tokens(chunks) <= budget_tokens:
        return chunks
    # Summarize each chunk to ~150 tokens
    summaries = [summarize_chunk(c, query, max_tokens=150) for c in chunks]
    return summaries[:budget_tokens // 150]

Leverage prompt caching for repeated prefixes

System prompts, few-shot examples, and static documentation should go at the beginning of the message array. Providers cache prefix matches. Putting variable content first breaks the cache.

# Good: static prefix first
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},      # Cached
    {"role": "user", "content": FEW_SHOT_EXAMPLES},    # Cached
    {"role": "user", "content": dynamic_context},      # Fresh
    {"role": "user", "content": user_query}            # Fresh
]

# Bad: dynamic content first breaks prefix caching
messages = [
    {"role": "user", "content": user_query},           # Fresh
    {"role": "system", "content": SYSTEM_PROMPT},      # Not cached effectively
]

Avoid redundant context in multi-turn conversations

Passing the full conversation history every turn grows input tokens linearly with turn count. Use conversation summarization or sliding window truncation.

def build_messages(history, new_query, max_history_tokens=8000):
    # Keep system prompt + recent turns within budget
    system_tokens = count_tokens(SYSTEM_PROMPT)
    budget = max_history_tokens - system_tokens
    
    recent = []
    for msg in reversed(history):
        msg_tokens = count_tokens(msg["content"])
        if budget - msg_tokens < 0:
            break
        recent.insert(0, msg)
        budget -= msg_tokens
    
    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        *recent,
        {"role": "user", "content": new_query}
    ]

The hidden cost: reasoning tokens

Models like o1 and o1-mini introduce “reasoning tokens” — intermediate chain-of-thought tokens billed at output rates but not visible in the final response. A single o1 query might generate 5,000 reasoning tokens and 500 visible tokens. You pay for all 5,500 at output pricing ($60/M on o1).

{
  "usage": {
    "prompt_tokens": 1200,
    "completion_tokens": 500,
    "reasoning_tokens": 5000,
    "total_tokens": 6700
  }
}

This makes reasoning models 5-10x more expensive per query than their base pricing suggests. Budget accordingly.

Which to choose

High-volume, short completions (classification, tagging, extraction) → Use the cheapest model that meets quality (GPT-4o-mini, Gemini Flash, Haiku). Output token volume dominates; 16x price difference between tiers compounds fast. Set aggressive max_tokens (100-300).

Long-form generation (articles, code, reports) → Output tokens are your primary cost. Use the smallest model that produces acceptable quality. Consider splitting: outline with a strong model, draft sections with a cheaper model, stitch together.

RAG / document QA with repeated queries → Invest in prompt caching. Structure prompts with static prefixes first. Compress retrieved context. The 50-90% cache discount on input tokens often makes the effective input cost negligible.

Multi-turn chat / agents → Conversation history grows input tokens. Implement sliding window or summarization. Output tokens per turn are usually small, but 20 turns × 500 output tokens = 10K output tokens per session.

Reasoning-heavy tasks (math, logic, planning) → o1-class models. Accept the reasoning token tax. No current workaround — the intermediate tokens are the product. Consider whether a cheaper model with explicit chain-of-thought prompting achieves similar results for your task.

Batch / async workloads → Many providers offer 50% discounts for batch APIs (24-hour turnaround). Both input and output tokens get the discount. If latency isn’t critical, this halves your bill across the board.

The input/output pricing split isn’t going away — it reflects the physics of transformer inference. The engineers who understand the asymmetry and architect for it will spend 3-10x less than those who treat all tokens as equal.