Time to first token for code completion: 2026 benchmark

Time to first token code completion is the latency metric that decides whether an inline suggestion feels like pair programming or a hanging pause. In this analysis we break down 2026 benchmark methodology, show why raw TTFT numbers mislead, and argue that prefix caching and routing policy matter more than model size for most dev tools.

Why TTFT owns the code completion experience

Inline completion triggers on keypress. The user is mid-thought; a 400 ms delay is perceptible, a 1 s delay breaks flow. Unlike chat, where the user commits a request and waits, code completion is speculative. The model must return something before the developer types past the suggestion.

That makes time to first token code completion the primary UX gate. Throughput—tokens per second after the first—matters less because completions are short (often <20 tokens). If the first token arrives fast, the rest streams in unnoticed.

What a 2026 benchmark actually measures

Most teams still benchmark by timing curl to a /v1/completions endpoint. That conflates DNS, TLS, load balancer, tokenization, prefill, and network egress. To isolate model prefill, measure the gap between sending the last byte of the request and receiving the first streamed chunk.

import time, openai

client = openai.OpenAI(base_url="https://api.example.com/v1", api_key="sk-...")
prompt = "def fib(n):\n    "

start = time.perf_counter()
stream = client.completions.create(
    model="starcoder2-15b",
    prompt=prompt,
    max_tokens=16,
    stream=True,
)
first_token_ts = None
for chunk in stream:
    if chunk.choices[0].text:
        first_token_ts = time.perf_counter()
        break
ttft = first_token_ts - start
print(f"TTFT: {ttft*1000:.1f} ms")

This script ignores client-side tokenization overhead (the OpenAI SDK sends raw prompt text; the server tokenizes). For fair cross-model comparison, send pre-tokenized input if the API supports it, or accept that tokenization cost is part of the user-perceived latency.

A gateway that fronts many models simplifies this. Using an OpenAI-compatible endpoint that addresses 240+ models, we ran the same prompt across local and remote models without changing client code.

Hidden variables that invalidate naive comparisons

Prefix caching

Providers cache the KV state of repeated prefixes. If your dev tool sends the same system prompt and file header on every keystroke, the second request hits cache and TTFT drops by 5–10x. Benchmarking cold vs warm cache produces different orders of magnitude.

Forward cache-control hints explicitly:

{
  "model": "gpt-oss-120b",
  "prompt": "<|file|>/src/app.ts\n...",
  "stream": true,
  "extra_headers": {
    "x-cache-control": "ephemeral"
  }
}

n4n.ai forwards provider cache-control hints and honors client routing directives, so the same benchmark script can test cache behavior across backends.

Context assembly cost

Real code completion embeds the current file, imports, and sometimes retrieved repo context. Assembling that context in the client or a proxy adds latency before the model sees tokens. Measure end-to-end from keypress, not just server TTFT.

Routing and fallback

If a provider is rate-limited, a gateway with automatic fallback shifts the request to a secondary model. That changes TTFT distribution—usually higher but non-zero. A benchmark that assumes a single healthy provider hides tail latency that users actually experience.

Small models vs frontier: the tradeoff is not just speed

In 2026, a 7B–15B open-weight model on a single L4 GPU returns first token in tens of milliseconds for short contexts. Frontier mixtures behind public APIs often land in the 200–600 ms range median, but produce more correct multi-line edits.

The decision is contextual:

• Local/edge model: Best TTFT, worst long-range reasoning. Fine for single-line fills.
• Frontier API: Higher TTFT, better suggestion quality. Use for complex refactors triggered manually.
• Mid-size hosted model: Compromise; often 100–250 ms with acceptable quality.

Optimizing time to first token code completion by shrinking the model ignores that a wrong suggestion costs the developer more time than a slightly slower correct one. Quality decay at small scale shows up as hallucinated APIs or broken indentation, which the user must inspect and delete.

Methodology pitfalls we hit

Cold starts. Serverless inference scales to zero. First request after idle spins up a container; TTFT spikes to seconds. Always warm the endpoint with N requests before sampling.

Tokenization drift. Different tokenizers split code differently. A prompt that is 50 tokens in one model may be 70 in another, changing prefill cost. Report TTFT per input token if you want normalized numbers.

Streaming artifacts. Some providers buffer the first chunk. The “first token” event may actually contain multiple tokens, understating true TTFT. Inspect chunk lengths.

Client clock skew. Measuring from client machine to remote API includes WAN jitter. Run the client in the same region, or subtract measured RTT baseline.

Cache TTL ambiguity. One provider caches prefixes for 5 minutes, another for 1 hour. Your benchmark’s warm number is only valid for your request pattern. Document the inter-request gap.

A realistic benchmark shape

We recommend a benchmark that records:

1. Context size (tokens)
2. Cache state (cold/warm)
3. Region/network RTT
4. Model class
5. TTFT at p50, p90, p99

Example result table (qualitative, based on observed ranges):

No fake precise numbers; these reflect commonly observed ranges across public and self-hosted infrastructure.

How teams fake a good TTFT

We have seen internal dashboards that report 80 ms TTFT by measuring from the proxy to the model, excluding the client’s context-building step. Others benchmark only the cached happy path and ship a feature that falls back to cold frontier calls on every new file. Both lie to the builder.

If you want an honest time to first token code completion number, instrument the exact code path in your editor extension. Wrap the keystroke handler:

const t0 = performance.now();
editor.completionProvider.request(prefix).then(() => {
  const ttft = performance.now() - t0;
  telemetry.report("completion_ttft", ttft, { model, cached });
});

That captures WAN, assembly, and model prefill together.

Decisive takeaway

Stop publishing single-number time to first token code completion benchmarks. They are meaningless without cache state, context length, and routing conditions. For dev tools, target warm-cache TTFT under 150 ms for inline suggestions by using a small model or cached prefix, and reserve larger models for explicit commands where users tolerate higher latency. Measure with the same client path your product uses, including fallback, and you will ship a completion feature that feels instant rather than one that looks fast in a spreadsheet.