n4nAI
Open console →
Benchmarks & performanceComparisonLong-Context Latency Benchmarks

Latency at 128k tokens: GPT-4o vs Claude vs Gemini

A practical 128k token context latency benchmark comparing GPT-4o, Claude, and Gemini on cost, speed, and ergonomics for long-context LLM apps.

n4n Team4 min read961 words

Audio narration

Coming soon — every post will get a voice note here.

Processing a 128k-token prompt is not the same as processing a 2k one. A practical 128k token context latency benchmark across GPT-4o, Claude, and Gemini shows that prefill cost, not decode speed, dictates user-perceived responsiveness, and each provider trades off differently. Below we break down the three flagships on the dimensions that matter when you ship.

The Contenders

GPT-4o

OpenAI’s omni-model handles text, vision, and audio with a 128k token context window. It is the default for teams already on the OpenAI stack.

Claude

Anthropic’s Claude 3.5 Sonnet (the current long-context workhorse) supports 200k tokens, optimized for document reasoning and code. The API is REST-native with explicit cache control.

Gemini

Google’s Gemini 1.5 Pro ships a 1M-token window, multimodal input, and aggressive context caching. At 128k it behaves like a smaller model would elsewhere.

Test Methodology

We synthesized 128k tokens of valid English prose, then requested a 512-token summary. Using a gateway such as n4n.ai, which exposes one OpenAI-compatible endpoint for 240+ models and forwards provider cache-control hints, we kept the client harness identical across providers and measured time-to-first-token (TTFT) and wall-clock completion. No provider-specific tuning was applied beyond enabling cache hints where the API allowed.

We deliberately avoided publishing millisecond figures. Hardware region, batching, and provider load shift absolute numbers week to week. The relative ordering and the architectural causes are stable.

import time, openai

client = openai.OpenAI()  # or gateway base_url
t0 = time.time()
stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": LONG_TEXT}],
    max_tokens=512,
    stream=True,
)
first = None
for chunk in stream:
    if chunk.choices[0].delta.content:
        first = time.time()
        break
print("TTFT seconds:", first - t0)

Capabilities

GPT-4o is the most balanced: vision, audio, and solid reasoning. Claude excels at extracting structure from long legal or code repos, and its JSON mode is reliable. Gemini natively ingests images alongside the 128k text and can reason across both without stitching.

For pure text summarization at 128k, all three produce coherent output. The gap appears in tool use: GPT-4o and Claude have mature function-calling; Gemini’s function ecosystem is younger but improving.

Price and Cost Model

Input tokens dominate at 128k. Public list pricing (per million tokens):

  • GPT-4o: $5 input / $15 output
  • Claude 3.5 Sonnet: $3 input / $15 output
  • Gemini 1.5 Pro: $1.25 input / $5 output for prompts ≤128k (rates step up beyond)

A single 128k-token request costs roughly 0.64 cents in input on Gemini, 1.6 cents on Claude, and 2.6 cents on GPT-4o. Output is negligible at 512 tokens. If you cache the prefix, Gemini and Claude discount repeat input heavily; OpenAI applies automatic prompt caching on long prefixes.

Latency and Throughput

Prefill—the pass that attends to all 128k input tokens—is the bottleneck. Decode (generation) runs at 30–100 tokens/sec across all three, but that only starts after prefill finishes.

GPT-4o shows moderate TTFT: its attention kernel is optimized, but 128k still takes a few seconds. Decode is fast, so short answers feel snappy.

Claude’s TTFT is typically higher but predictable; Anthropic streams prefill progress internally, and throughput stays flat under load. For batch jobs this is fine.

Gemini’s TTFT drops sharply when the context prefix is cached (via context cache or repeated content). Without cache, it lands between GPT-4o and Claude. Its decode throughput is competitive.

{
  "model": "claude-3-5-sonnet-20240620",
  "messages": [
    {"role": "user", "content": LONG_TEXT, "cache_control": {"type": "ephemeral"}}
  ],
  "max_tokens": 512
}

Ergonomics

GPT-4o speaks the OpenAI shape: messages, stream, tools. If your codebase already uses the SDK, nothing changes.

Claude uses a similar messages array but requires explicit cache_control blocks to persist prefixes. Streaming is SSE, and the SDK handles it.

Gemini uses a contents array with parts, and caching is a separate API call that returns a cache_id. More moving parts, but the payoff at 128k is real.

Ecosystem

GPT-4o plugs into the largest plugin and assistant ecosystem. Claude has first-class support in Anthropic’s console and growing third-party tooling. Gemini is native to Vertex AI and Google Workspace, which matters if you already live there.

Limits

  • GPT-4o: hard 128k context cap; rate limits tighten on long prompts.
  • Claude: 200k window, but very long single requests can hit organizational tier limits.
  • Gemini: 1M window, but cached prefixes expire after an hour and cost storage per token-hour.

Head-to-Head Table

Dimension GPT-4o Claude (3.5 Sonnet) Gemini 1.5 Pro
Capabilities Multimodal, general Long-doc, code, JSON 1M ctx, multimodal
Price (in/out per 1M) $5 / $15 $3 / $15 $1.25 / $5 (≤128k)
Latency profile Moderate TTFT, fast decode Higher TTFT, stable Low TTFT w/ cache
Ergonomics OpenAI-compatible Native REST + cache REST + cache API
Ecosystem Largest Anthropic native Vertex/Google
Limits 128k cap 200k cap 1M, cache expiry

Cutting Latency at 128k

Cache the static prefix. In Claude, mark the document with cache_control. In Gemini, create a context cache and reference it. In OpenAI, repeated prefixes auto-cache after 128k tokens (or 1024 for fine-tuned). This turns prefill from O(n) per request to O(1) for repeats.

Trim aggressively. A 128k prompt often contains boilerplate—headers, repeated disclaimers—that can move to system instructions or be deduplicated before send.

If you must stream, set max_tokens tight and stop early. The user sees first token faster than full completion.

Which to Choose

Real-time chat over long docs

Pick GPT-4o. Moderate TTFT and fast decode keep interactive summarization feeling live, and the OpenAI client needs no rewrite.

Overnight batch extraction from 200k repos

Claude 3.5 Sonnet. The higher TTFT is irrelevant offline, and its document reasoning beats the field. Use cache_control to avoid re-prefilling the same repo.

Massive multimodal corpus (text + images) up to 1M

Gemini 1.5 Pro. The 1M window and low cached TTFT make it the only option that doesn’t require chunking. Cost is also lowest.

Cost-constrained startup at 128k

Gemini again. At $1.25/M input it is 4x cheaper than GPT-4o for the same prompt size.

Existing OpenAI stack, no new SDKs

GPT-4o by default. The latency penalty versus cached Gemini is acceptable if you value operational simplicity.

The 128k token context latency benchmark confirms one thing: match the model to the request pattern, not the leaderboard. Cache what you can, measure TTFT in your own region, and ship.

Tagslong-context128k-tokensgpt-4oclaudegemini

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceHow context length affects time to first tokenAnalyze how prompt size drives time to first token in LLM inference, why it's sublinear in practice, and how to benchmark and reduce long-context TTFT.Benchmarks & performanceHow much does a 100k-token prompt add to response timeAnalyzing how a 100k token prompt affects LLM response time: prefill cost, time-to-first-token, caching, and practical latency tradeoffs for engineers.Benchmarks & performanceLlama 4 Scout long-context latency at 10M tokensAnalysis of Llama 4 Scout 10M context latency: why KV-cache footprint and prefill scheduling dominate, and how to serve open-weight long-context models in production.Benchmarks & performanceLong-context latency: Claude Opus 4.5 at 200k tokensAnalyzing Claude Opus 4.5 200k context latency: prefill dominates TTFT, prefix caching helps, and full context isn't always worth the cost.Benchmarks & performanceMeasuring latency cost of large system promptsStep-by-step method to measure system prompt latency cost on LLM endpoints, isolating prefill overhead with streaming and controlled prompt sizes across models.Benchmarks & performanceWhy long-context requests slow down time to first tokenAnalyzes why long context time to first token slowdown occurs, breaking down prefill cost, attention scaling, and scheduling, with practical mitigations.

All long-context latency benchmarks posts →