What is time-to-first-token in LLM inference?

Time-to-first-token (TTFT) is the elapsed time between sending an inference request and receiving the first token of the model’s response. It captures the full cold-start latency of the generation pipeline: request routing, queue wait, prompt processing (prefill), and the first decoding step. For streaming interfaces, TTFT determines how fast users perceive the system as “alive.”

How TTFT works in the inference pipeline

When a request hits an LLM serving stack, it passes through several stages before the first token emerges. Understanding each stage helps you isolate where latency lives.

Request routing and queueing

The request first hits a load balancer or gateway. If all model replicas are busy, the request sits in a queue. Queue time is often the largest and most variable component of TTFT, especially under burst traffic. A well-designed gateway reports queue time separately so you can distinguish infrastructure saturation from model latency.

# Simplified request flow with timing hooks
async def handle_request(request: InferenceRequest) -> AsyncIterator[Token]:
    t0 = time.perf_counter()
    
    # Gateway routing + queue wait
    replica = await router.acquire_replica(request.model)
    t_queue = time.perf_counter()
    
    # Prefill: process entire prompt in parallel
    prefill_output = await replica.prefill(request.prompt)
    t_prefill = time.perf_counter()
    
    # First decode step
    first_token = await replica.decode_step(prefill_output)
    t_first = time.perf_counter()
    
    yield TimingBreakdown(
        queue_ms=(t_queue - t0) * 1000,
        prefill_ms=(t_prefill - t_queue) * 1000,
        first_decode_ms=(t_first - t_prefill) * 1000,
    )
    yield first_token
    # ... streaming continues

Prefill (prompt processing)

During prefill, the model processes all input tokens simultaneously. The compute scales quadratically with sequence length for standard attention, though flash attention and paged attention reduce the constant factor. Prefill is memory-bandwidth bound on modern GPUs — you’re reading the full KV cache into compute units.

Key insight: prefill time grows with prompt length, but not linearly. A 4k token prompt doesn’t take 4x a 1k prompt because the GPU saturates memory bandwidth. However, very long prompts (32k+) can dominate TTFT entirely.

First decode step

After prefill, the model generates the first output token. This step is compute-light but latency-sensitive: it requires a full forward pass through all layers for a single token. The KV cache from prefill is reused, so memory reads dominate. On H100-class GPUs, this step typically takes 1–3 ms for 7B–70B models; on smaller GPUs or quantized models, it can reach 10–20 ms.

Why TTFT matters for product experience

TTFT directly maps to user-perceived latency in streaming interfaces. Users judge responsiveness by “time to first character,” not throughput.

Streaming UX thresholds

TTFT range	User perception
< 200 ms	Instantaneous
200–500 ms	Noticeable but acceptable
500–1000 ms	Feels sluggish
> 1 s	Broken; users retry or abandon

These thresholds hold across chat, code completion, and agentic workflows. A code assistant with 800 ms TTFT feels broken even if it streams at 100 tokens/sec afterward.

TTFT vs. throughput trade-offs

Batching improves throughput (tokens/sec/GPU) but hurts TTFT. Continuous batching — where new requests join in-flight batches — reduces queue time but adds scheduler overhead. Static batching maximizes throughput but creates head-of-line blocking: a short prompt waits behind a long one.

# Continuous batching scheduler pseudocode
class ContinuousBatcher:
    def __init__(self, max_batch_tokens: int, max_wait_ms: int):
        self.max_batch_tokens = max_batch_tokens
        self.max_wait_ms = max_wait_ms
        self.waiting: list[Request] = []
        self.running: list[Request] = []
    
    async def schedule(self) -> Batch:
        deadline = time.monotonic() + self.max_wait_ms / 1000
        batch_tokens = 0
        batch = []
        
        while self.waiting and batch_tokens < self.max_batch_tokens:
            req = self.waiting[0]
            if batch_tokens + req.prompt_tokens > self.max_batch_tokens:
                break
            if time.monotonic() > deadline and batch:
                break  # Don't wait forever for a full batch
            batch.append(self.waiting.pop(0))
            batch_tokens += req.prompt_tokens
        
        return Batch(requests=batch)

The max_wait_ms parameter directly controls the TTFT/throughput frontier. Lower values improve p50 TTFT but reduce GPU utilization.

Concrete example: measuring TTFT in production

Here’s how to instrument TTFT correctly in a real serving stack. The key is measuring from the client’s perspective, not just server-side.

Client-side measurement

import asyncio
import time
import httpx

async def measure_ttft(
    endpoint: str,
    prompt: str,
    model: str,
    headers: dict | None = None,
) -> dict:
    """Measure TTFT from client perspective including network."""
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "stream": True,
        "max_tokens": 1,
    }
    
    t_start = time.perf_counter()
    async with httpx.AsyncClient(timeout=30.0) as client:
        async with client.stream("POST", endpoint, json=payload, headers=headers) as resp:
            resp.raise_for_status()
            async for chunk in resp.aiter_bytes():
                if chunk.strip():
                    t_first = time.perf_counter()
                    return {
                        "ttft_ms": (t_first - t_start) * 1000,
                        "status": resp.status_code,
                        "first_chunk": chunk[:200],
                    }
    return {"ttft_ms": None, "error": "No response"}

# Run multiple times for percentile distribution
async def benchmark_ttft(endpoint: str, prompt: str, model: str, n: int = 50):
    results = await asyncio.gather(*[
        measure_ttft(endpoint, prompt, model) for _ in range(n)
    ])
    ttfts = [r["ttft_ms"] for r in results if r["ttft_ms"] is not None]
    ttfts.sort()
    return {
        "p50": ttfts[len(ttfts) // 2],
        "p90": ttfts[int(len(ttfts) * 0.9)],
        "p99": ttfts[int(len(ttfts) * 0.99)],
        "samples": len(ttfts),
    }

Server-side breakdown

Instrument each stage separately. This reveals whether you’re queue-bound, prefill-bound, or decode-bound.

# Server-side timing middleware (FastAPI example)
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
import time

class TTFTMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        if not request.url.path.endswith("/chat/completions"):
            return await call_next(request)
        
        t_gateway = time.perf_counter()
        body = await request.body()
        # Parse stream=True, model, prompt tokens...
        
        # Queue acquisition
        t_queue_start = time.perf_counter()
        replica = await self.router.acquire(request.model)
        t_queue_end = time.perf_counter()
        
        # Prefill
        t_prefill_start = time.perf_counter()
        kv_cache = await replica.prefill(parsed_prompt)
        t_prefill_end = time.perf_counter()
        
        # First decode
        t_decode_start = time.perf_counter()
        first_token = await replica.decode_step(kv_cache)
        t_decode_end = time.perf_counter()
        
        # Inject headers for observability
        response = StreamingResponse(
            self.stream_with_timing(replica, kv_cache, first_token),
            media_type="text/event-stream",
        )
        response.headers["X-TTFT-Queue-Ms"] = f"{(t_queue_end - t_queue_start) * 1000:.1f}"
        response.headers["X-TTFT-Prefill-Ms"] = f"{(t_prefill_end - t_prefill_start) * 1000:.1f}"
        response.headers["X-TTFT-Decode-Ms"] = f"{(t_decode_end - t_decode_start) * 1000:.1f}"
        response.headers["X-TTFT-Total-Ms"] = f"{(t_decode_end - t_gateway) * 1000:.1f}"
        return response

With this instrumentation, a typical breakdown for a 2k prompt on a 70B model might look like:

X-TTFT-Queue-Ms: 45.2
X-TTFT-Prefill-Ms: 180.5
X-TTFT-Decode-Ms: 2.1
X-TTFT-Total-Ms: 228.7

Here prefill dominates. Optimization focus: faster attention kernels, prompt caching, or smaller models for high-volume workloads.

Common misconceptions

“TTFT is just network latency”

Network latency (client → gateway → replica) is typically 5–50 ms in the same region. TTFT is usually 100–500 ms. Network is rarely the bottleneck unless you’re crossing continents or using high-latency load balancers. Measure server-side breakdown before optimizing network paths.

“Lower TTFT always means better UX”

Aggressively minimizing TTFT can hurt throughput and cost. A system tuned for 100 ms p50 TTFT might run at 30% GPU utilization, while a 300 ms p50 system runs at 85%. The right target depends on your product: chat needs < 300 ms; batch document processing doesn’t care about TTFT at all.

“TTFT and inter-token latency are the same thing”

TTFT includes prefill; inter-token latency (ITL) is the steady-state decode speed. They have different bottlenecks:

TTFT: prefill compute, queue time, KV cache initialization
ITL: memory bandwidth for KV cache reads, decode kernel efficiency

A system can have excellent TTFT (fast prefill, low queue) but poor ITL (slow memory subsystem), or vice versa. Measure both.

“Speculative decoding fixes TTFT”

Speculative decoding (draft model + verification) accelerates ITL, not TTFT. The first token still requires a full prefill pass on the target model. Speculative decoding helps after the first few tokens. For TTFT, look at prompt caching, prefix caching, or smaller draft models for the prefill phase itself.

“Batch size doesn’t affect TTFT if I use continuous batching”

Continuous batching reduces but doesn’t eliminate batch-induced TTFT variance. A request arriving just after a batch forms waits for the next scheduling cycle. The max_wait_ms parameter creates a hard floor on queue time. At high load, batches fill instantly and queue time approaches zero; at low load, requests wait up to max_wait_ms for batchmates.

Optimizing TTFT: where to start

1. Eliminate queue time at low load

Set max_wait_ms low (10–20 ms) for latency-sensitive workloads. Accept lower GPU utilization during off-peak. Use replica autoscaling with scale-to-zero for sporadic traffic.

2. Prompt caching for repeated prefixes

System prompts, few-shot examples, and document contexts often repeat. Cache the KV cache after prefill and reuse it.

# Prefix cache keyed by prompt hash
class PrefixCache:
    def __init__(self, max_entries: int, ttl_seconds: int):
        self.cache: dict[str, tuple[KVCache, float]] = {}
        self.max_entries = max_entries
        self.ttl = ttl_seconds
    
    def get(self, prompt_hash: str) -> KVCache | None:
        if prompt_hash in self.cache:
            kv, ts = self.cache[prompt_hash]
            if time.time() - ts < self.ttl:
                return kv
            del self.cache[prompt_hash]
        return None
    
    def put(self, prompt_hash: str, kv: KVCache):
        if len(self.cache) >= self.max_entries:
            # Evict oldest
            oldest = min(self.cache.items(), key=lambda x: x[1][1])[0]
            del self.cache[oldest]
        self.cache[prompt_hash] = (kv, time.time())

This can reduce prefill from 200 ms to < 10 ms for cached prefixes.

3. Smaller models for high-volume paths

Route simple queries (classification, extraction, short QA) to 7B–13B models. Reserve 70B+ for complex reasoning. A 7B model on H100 can achieve 50–80 ms TTFT for 1k prompts; 70B takes 200–400 ms.

4. Quantization with care

INT4/GPTQ/AWQ reduce model size and memory bandwidth pressure, improving both TTFT and ITL. But aggressive quantization can degrade quality on complex tasks. Benchmark your specific workload — don’t assume 4-bit is free.

5. Prefill-chunking for very long prompts

For prompts > 16k tokens, chunk prefill across multiple forward passes to avoid OOM and reduce tail latency. This trades peak memory for slightly higher total prefill time but smoother latency distribution.

TTFT in multi-model routing

When a gateway routes requests across multiple providers or model variants, TTFT becomes a routing signal. A request routed to a degraded provider might see 5x TTFT. Smart gateways measure per-provider TTFT in real time and route around degradation.

# Routing policy using live TTFT metrics
class LatencyAwareRouter:
    def __init__(self, providers: list[Provider]):
        self.providers = providers
        self.ttft_ewma: dict[str, float] = {p.name: 200.0 for p in providers}
    
    async def select(self, request: Request) -> Provider:
        candidates = [p for p in self.providers if p.supports(request.model)]
        # Prefer providers with lower recent TTFT, but explore occasionally
        scored = []
        for p in candidates:
            base = self.ttft_ewma[p.name]
            exploration = random.uniform(0, 50)  # 0-50ms exploration bonus
            scored.append((base + exploration, p))
        scored.sort(key=lambda x: x[0])
        return scored[0][1]
    
    def record_ttft(self, provider: str, ttft_ms: float):
        alpha = 0.1
        self.ttft_ewma[provider] = (
            alpha * ttft_ms + (1 - alpha) * self.ttft_ewma[provider]
        )

This approach automatically shifts traffic away from providers experiencing queue buildup or GPU issues, without hardcoding provider priorities.

Summary

Time-to-first-token is the end-to-end latency from request send to first token received. It decomposes into queue time, prefill, and first decode — each with different optimization levers. For streaming products, p50 TTFT under 300 ms and p99 under 1 second are reasonable targets. Measure client-side, break down server-side, and optimize the dominant component. Don’t confuse TTFT with throughput or inter-token latency; they require different optimizations.