n4nAI
Open console →
Benchmarks & performanceAnalysisReasoning Model Latency Overhead

Extended thinking budgets and time to first token

Analyze how extended thinking budgets affect time to first token for reasoning models, with measurement code and tradeoff guidance for production LLM systems.

n4n Team4 min read946 words

Audio narration

Coming soon — every post will get a voice note here.

The extended thinking budget time to first token relationship is the single most misunderstood latency knob in reasoning-model deployments. Teams set a thinking token cap expecting a linear delay, then watch their p95 TTFT blow up by an order of magnitude because the model spends the entire budget on a trivial prompt. This article argues you should treat the thinking budget as a latency SLO lever, not a quality dial you can ignore.

What an extended thinking budget actually buys you

Reasoning models from Anthropic, OpenAI, and DeepSeek expose a parameter that limits the number of hidden “thinking” tokens the model may emit before producing the visible answer. The budget is a ceiling, not a directive to think harder. If the prompt is easy, the model exits thinking early. If the prompt is hard, it may burn the full cap and then force an answer, sometimes with truncated reasoning.

Anthropic’s API makes this explicit:

{
  "model": "claude-3-7-sonnet-20250219",
  "thinking": { "budget_tokens": 16000 },
  "messages": [{"role": "user", "content": "Prove sqrt(2) is irrational"}]
}

OpenAI’s o-series uses a reasoning_effort enum rather than a token budget, but the same ceiling effect applies. The key fact: thinking tokens are generated sequentially, and the first answer token cannot appear until the thinking phase ends. That makes the extended thinking budget time to first token curve fundamentally different from a normal completion.

How thinking tokens inflate time to first token

The non-linear curve

In a standard chat model, TTFT is prompt prefill plus one decode step. In a thinking model, the model must decode the entire thought trace (or hit the cap) before the answer begins. Providers differ on whether thinking tokens are streamed. When they are, your client sees a first token quickly. When they are not, even raw TTFT includes the whole think phase.

The extended thinking budget time to first token relationship is non-linear because reasoning effort tracks task complexity, not budget size. Observations from public eval sets show:

  • Budgets of 1–2k tokens cover the vast majority of routing, summarization, and extraction queries with no measurable accuracy loss.
  • Budgets of 8–16k tokens unlock gains on math word problems, competitive programming, and multi-step planning.
  • Budgets beyond 32k rarely move accuracy on benchmarks like GPQA but guarantee multi-minute stalls.

Decode throughput for thinking tokens is identical to output tokens—roughly 20–80 tokens/sec on current accelerators. A 16k-token think phase therefore adds 200–800 seconds of answer latency if run to completion. That is not a typo: minutes.

Streaming semantics and perceived latency

Most OpenAI-compatible gateways surface thinking as a distinct chunk type. A typical stream looks like:

{"type": "thinking", "delta": {"thinking": "Let me enumerate cases..."}}

followed much later by:

{"type": "content", "delta": {"text": "The answer is 42"}}

If you alert on TTFT using the first chunk, you will report 400 ms while your user stares at a spinner for six minutes. Instrument both raw TTFT and time to first answer token (TFAT). The latter is the metric that predicts churn.

Measuring extended thinking budget time to first token in practice

Below is a minimal Python script using the OpenAI client against an endpoint that forwards the thinking budget via extra_body. It records both timestamps.

import time, openai

client = openai.OpenAI(base_url="https://api.n4n.ai/v1", api_key="KEY")
# n4n.ai honors client routing directives and forwards provider cache-control hints,
# so the budget reaches the upstream model unchanged.

start = time.time()
first_token = None
first_answer = None
stream = client.chat.completions.create(
    model="anthropic/claude-3-7-sonnet",
    messages=[{"role": "user", "content": "Solve TSP for 10 cities optimally"}],
    stream=True,
    extra_body={"thinking": {"budget_tokens": 12000}}
)
for chunk in stream:
    if not chunk.choices:
        continue
    delta = chunk.choices[0].delta
    now = time.time() - start
    if first_token is None and (getattr(delta, "thinking", None) or getattr(delta, "content", None)):
        first_token = now
    if first_answer is None and getattr(delta, "content", None):
        first_answer = now
        break
print(f"TTFT (any): {first_token:.2f}s, Answer TTFT: {first_answer:.2f}s")

Run this across budgets of 2k, 8k, 16k, and 32k on a fixed prompt set. Plot the percentiles. The extended thinking budget time to first token slope stays flat until the task crosses the model’s intrinsic complexity threshold, then climbs vertically.

Tradeoffs: quality vs. latency vs. cost

Thinking tokens are billed as output tokens. A 16k thinking phase costs the same as 16k generated answer tokens, plus the answer itself. The budget is therefore a cost lever as much as a latency lever.

When to cap the budget low

For conversational assistants, agent tool routing, or any workflow where a mistake is correctable by a follow-up turn, set the budget to 1–2k. The model still performs lightweight reasoning. You avoid hangs that trigger client timeouts. If a user explicitly requests deep analysis, upgrade the budget on that request only.

When to let it run

For autonomous code generation, formal verification, or financial modeling where a single error is expensive, allow 16–32k. But you must stream thinking chunks to a progress UI. Never block the interface on answer TTFT without feedback; a thinking stream that says “exploring approach A” keeps users patient.

Dynamic budget selection

A static global budget is lazy engineering. Route by task class:

def select_budget(prompt: str, user_tier: str) -> int:
    if user_tier == "premium" and "prove" in prompt:
        return 32000
    if any(k in prompt for k in ("code", "math", "why")):
        return 12000
    return 2000

This heuristic cuts median answer TTFT by skipping deep thinking on trivial turns while preserving accuracy where it matters.

Routing and fallback considerations

If you front models with an inference gateway, provider degradation interacts badly with thinking budgets. A rate-limited provider that drops a stream mid-thinking wastes the entire budget. A gateway that automatically falls back to a healthy provider preserves the request only if it replays the prompt; most do not replay thinking. n4n.ai forwards provider cache-control hints, so a cached prefill on the fallback route reduces the prefill portion of TTFT, but the decode of thinking still restarts from zero. Build idempotent retries at the app layer and cap total think time with a wall-clock timeout.

Why TTFT misreporting hides outages

Many dashboards plot TTFT from the first SSE chunk. With thinking models, that metric becomes useless for detecting slow providers. An upstream that throttles thinking decode to 5 tokens/sec still emits the first thinking token instantly, keeping your TTFT green while answer TTFT rots. Split the two signals. Alert on answer TTFT p95 with a separate threshold.

Takeaway

Treat the extended thinking budget time to first token tradeoff as a first-class SLO. Measure answer TTFT, not just raw TTFT. Set budgets per task class, stream thinking chunks to maintain perceived responsiveness, and cap hard at the point where accuracy gains flatten. Leaving the budget unbounded in production is not a neutral default—it is a latency and cost outage waiting to happen.

Tagsextended-thinkingreasoning-modeltime-to-first-tokenlatency-benchmark

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceClaude 3.7 Sonnet extended thinking: latency vs token budgetAnalyzing Claude 3.7 Sonnet extended thinking latency versus token budget: how reasoning budgets affect TTFT, cost, and quality, with practical tuning advice.Benchmarks & performanceClaude 3.7 Sonnet: latency with thinking on vs thinking offClaude 3.7 Sonnet thinking on vs off latency compared across cost, throughput, and quality, with a head-to-head table and per-use-case verdicts for engineers.Benchmarks & performanceGemini 2.0 Flash Thinking: latency benchmarksEngineering analysis of Gemini 2.0 Flash Thinking latency: measure reasoning overhead, tune thinking budget, and decide when the tradeoff pays off in production.Benchmarks & performanceHow chain-of-thought length predicts response latencyAnalysis of how chain-of-thought length drives LLM response latency in reasoning models, with measurement code and latency budgeting tradeoffs.Benchmarks & performanceo3-mini reasoning effort settings: latency at low vs highAnalyzing o3-mini reasoning effort latency at low vs high settings: how hidden reasoning tokens drive TTFB and total time, with code and tradeoffs.Benchmarks & performanceQwQ-32B: how much does reasoning add to latency?We break down QwQ-32B reasoning latency overhead: why chain-of-thought multiplies tail latency, how to measure it, and when the accuracy tradeoff is worth it.

All reasoning model latency overhead posts →