LLM cost per token vs speed for long-context tasks

The debate over cost per token vs speed long-context tasks usually collapses into two separate leaderboards: cheapest per token and fastest time-to-first-token. That framing fails engineers who need to process hundreds of thousands of tokens repeatedly, because the real constraint is the product of those two variables under sustained load. A model that is cheap but slow can cost more in wasted compute replicas than a pricier model that finishes in a quarter of the time.

The false dichotomy of cost and speed

Picking a model by lowest input price ignores the hours your pipeline spends waiting for generation. Picking by lowest latency ignores the bill when you scale to millions of documents. For long-context workloads, the two couple together through throughput: how many context tokens you can chew per second per dollar.

A 7B model self-hosted might cost tenths of a cent per million input tokens but take 30 seconds to ingest a 50k-token prompt. A frontier API might cost orders of magnitude more per token but return the first chunk in 500ms and stream the rest at high tokens/sec. Which is better depends on whether your bottleneck is human wait time or batch compute budget.

Output tokens matter more than input in many long-context jobs. Summarization produces short output, but extraction, rewriting, or agentic loops with tool calls can emit tens of thousands of tokens per document. Output pricing is typically 2–5x input pricing across proprietary APIs. The cost per token vs speed long-context tasks equation must weight output generation speed, not just prefill.

What long context actually changes

Long context is not just “more tokens.” It changes the shape of the computation.

KV cache and memory bandwidth

Attention scales with sequence length. Beyond a certain point, the KV cache dominates GPU memory, forcing smaller batch sizes or offloading. Throughput per GPU drops as context grows, so the cost curve is non-linear. A model that is cheap at 8k context can become expensive at 128k because you need more replicas to sustain the same QPS.

Sparse attention or sliding-window variants mitigate memory but can silently drop recall on facts buried early in the context. Benchmarks show some models advertised at 200k window degrade in retrieval accuracy past 32k–64k. Speed gains from those architectures are real, but you pay in quality risk.

Batching and queueing

Inference servers batch requests to amortize weights over many sequences. Long sequences reduce max batch size, increasing queue delay under load. Your p50 latency might look fine in isolation but degrade 10x when ten 100k-token jobs arrive concurrently. Single-shot benchmarks lie; you must test with your concurrency profile.

Tokenization variance

Context length is measured in model tokens, not characters. Code, JSON, and non-English text can tokenize 2–4x longer on some tokenizers. A “100k character” document might be 40k tokens on one model and 120k on another. This directly shifts both cost and prefill time. Always measure prompt_tokens from the API response, not your local strlen.

Measuring the right metric: cost per processed token-second

Define effective cost as:

effective_cost = (input_tokens * in_price + output_tokens * out_price) / (total_tokens / latency)

Or simpler: dollars per thousand tokens processed per second. This normalizes both axes and exposes the true loser in a head-to-head.

A minimal benchmarking harness

import time, openai

def bench(model, prompt, client, trials=5):
    samples = []
    for _ in range(trials):
        t0 = time.time()
        r = client.chat.completions.create(
            model=model,
            messages=[{"role":"user","content":prompt}],
            max_tokens=256
        )
        dt = time.time() - t0
        u = r.usage
        total = u.prompt_tokens + u.completion_tokens
        samples.append((dt, total))
    # median
    samples.sort(key=lambda x: x[0])
    dt, total = samples[len(samples)//2]
    return total/dt  # tokens per second

Run this across candidate models with a fixed long prompt. Collect median over trials. Do not trust vendor marketing numbers; your document mix and sampling params shift results. Add a concurrency flag to spawn N parallel calls if you run a service.

Concrete scenario: summarize a 100k-token document

Suppose you summarize legal contracts at 100k tokens each, 1,000 docs/day.

A small open model: input price low, but latency 20s per doc, throughput 5k tok/s. Frontier model: input price 10x higher, latency 4s, throughput 25k tok/s.

If you run one worker, small model finishes batch in 5.5 hours, frontier in 1.1 hours. Add parallel workers: small model needs 5x more replicas to match wall-clock. Those replicas cost compute too. The cost per token vs speed long-context tasks tradeoff flips when labor or SLA penalties exceed hardware.

If the task is asynchronous (nightly batch), cheap slow wins. If user waits on UI, fast expensive wins.

Now add retrieval-augmented generation: a 8k system prompt with cached knowledge plus 2k user query. Prefix caching makes the static prompt a fraction of input price on supporting providers. The cheap model’s advantage shrinks because both pay similar cache-hit rates; the speed gap remains. Suddenly the frontier model’s higher base cost is offset by 90% input savings.

Caching and fallback alter the equation

Prefix caching lets you pay once for static system prompts or retrieved context. Provider cache hits often cost a fraction of full input price. Forwarding cache-control hints from your client to the provider is critical; a gateway that strips them forces recomputation.

If you route through a gateway that honors client routing directives and forwards provider cache-control hints—n4n.ai does this on a single OpenAI-compatible endpoint covering 240+ models—you can shift traffic between a cheap slow model and a pricey fast one without code changes when a provider degrades. Automatic fallback when a provider is rate-limited keeps your pipeline green without manual retry logic.

Example routing directive

{
  "model": "auto",
  "route": {
    "prefer": ["meta/llama-3-70b", "openai/gpt-4o"],
    "fallback_on": ["rate_limit", "timeout"]
  },
  "cache_control": {"type": "ephemeral", "ttl": 3600}
}

This is not standard OpenAI, but a gateway can translate it to backend-specific calls while keeping your client code stable.

Decision framework

1. Measure at target length. Pull prompt_tokens from real responses. Benchmark prefill and decode separately if the API exposes them.
2. Compute dollars per processed token-second. Include output price and expected output length.
3. Map to SLA. If human-facing, set max latency budget; pick fastest within 2x cost of cheapest. If batch, invert: pick cheapest within 2x latency of fastest.
4. Exploit caching. Static context, few-shot examples, and schema instructions belong in a cached prefix. Reuse across requests.
5. Plan for degradation. Abstract provider behind a router with fallback and per-token metering. Track which model actually served the request.
6. Re-evaluate quarterly. New model releases shift the curve constantly; a model that was premium last quarter is mid-tier now.

Takeaway

For long-context workloads, stop ranking models by cost per token or speed alone. The metric that matters is cost per processed token-second under your batch size and concurrency. Cheap slow models win for offline bulk processing; fast premium models win for interactive latency. Use prefix caching to flatten the cost curve, and route through a fallback-aware gateway to keep throughput stable. Engineer the tradeoff, don’t admire it.