Debugging slow time-to-first-token on streaming endpoints

Slow time-to-first-token (TTFT) quietly ruins the perceived quality of any LLM feature. If you’re debugging slow time-to-first-token on a streaming endpoint, you need a method that isolates whether the bottleneck is network, provider queue, or your own client code.

Step 1: Measure TTFT correctly

TTFT is the wall-clock interval between the moment you dispatch the HTTP request and the moment the first response byte arrives that contains a token. Most teams mislabel it by including client-side prompt construction or DNS resolution. Start from the send call, not from process startup.

Use a minimal Python script with the OpenAI SDK (works against any OpenAI-compatible endpoint):

import time
from openai import OpenAI

client = OpenAI(base_url="https://api.example.com/v1", api_key="sk-...")

start = time.perf_counter()
stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Say hi."}],
    stream=True,
)
first_token_ts = None
for chunk in stream:
    if chunk.choices[0].delta.content:
        first_token_ts = time.perf_counter()
        break
ttft_ms = (first_token_ts - start) * 1000
print(f"TTFT: {ttft_ms:.1f} ms")

Run this three times. If numbers vary by more than 2x, you have a queuing or cold-start problem, not a code problem. Debugging slow time-to-first-token starts with trusting this number.

Verify success: You have a reproducible TTFT measurement printed to stderr/stdout that excludes local setup time.

Step 2: Separate network latency from model latency

A high TTFT can be pure TCP/TLS overhead or provider-side queueing. Use curl to get the time to first byte (TTFB) for the same streaming request without any SDK overhead:

curl -o /dev/null -s -w "time_starttransfer: %{time_starttransfer}\n" \
  -X POST https://api.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hi."}],"stream":true}'

time_starttransfer is the moment the first byte arrives—functionally identical to TTFT for a stream. Compare it to the Python number from Step 1.

• If curl TTFB is 80 ms but Python TTFT is 400 ms, your client code (proxy, middleware, async loop) is adding latency.
• If both are 900 ms, the delay is upstream.

This split is non-negotiable. You cannot fix provider queueing by rewriting your for loop.

Step 3: Inspect request payload and caching hints

Large system prompts or long conversation histories force the provider to process more tokens before emitting the first one. Even with streaming, the model must ingest the prefix. If your gateway or provider supports prompt caching, send cache-control hints to skip reprocessing.

Anthropic-style ephemeral cache control can be forwarded through OpenAI-compatible bodies via extra_body:

stream = client.chat.completions.create(
    model="claude-3-5-sonnet",
    messages=[
        {"role": "system", "content": "You are a terse debugger."},
        {"role": "user", "content": "Explain TTFT."}
    ],
    stream=True,
    extra_body={"cache_control": {"type": "ephemeral"}},
)

If you are on a gateway that forwards provider cache-control hints, this reduces repeat-call TTFT dramatically. Without caching, every call pays the full prefix cost.

Also check max_tokens. A requested high limit does not usually affect TTFT, but some providers pre-allocate. Keep it small (e.g., 32) during diagnosis.

Step 4: Check provider routing and fallback behavior

When you call through an inference gateway, automatic fallback can mask degradation. A request routed to a secondary provider because the primary was rate-limited will show higher TTFT with no error.

A gateway such as n4n.ai honors client routing directives and forwards provider cache-control hints, so pinning a route during debugging removes fallback variability. If your gateway supports a routing header, set it explicitly:

client = OpenAI(
    base_url="https://api.example.com/v1",
    api_key="sk-...",
    default_headers={"x-route": "primary"},
)

Replace x-route with whatever your platform documents. The goal is to eliminate the variable of silent re-routing. Run Step 1 again with the pin active. If TTFT drops, fallback was your culprit.

Also confirm the model string actually resolves to the provider you think. A typo like gpt-4o vs gpt-4o-mini can route to a heavier model with slower intake.

Step 5: Profile your client streaming loop

Many “slow TTFT” reports are actually slow first render. The token arrived at 120 ms, but the UI updated at 800 ms because of buffering.

In Python, ensure you flush synchronously:

import sys
for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    if delta:
        sys.stdout.write(delta)
        sys.stdout.flush()

If you use an async framework, don’t await asyncio.sleep(0) in the hot path, and don’t aggregate chunks before sending to the websocket. Measure the delta between chunk_received_ts and bytes_sent_to_client_ts.

For Node.ts, avoid res.write() without disabling compression during debugging—some proxies buffer until a flush threshold:

res.setHeader("Content-Type", "text/plain; charset=utf-8");
res.setHeader("Cache-Control", "no-transform");
// stream raw chunks

A 50 ms TTFT with a 700 ms render delay is a client bug, not a model bug.

Step 6: Reproduce with a minimal payload and verify

Strip the request to the bone: no system prompt, one user token, stream: true, max_tokens: 1. Run ten iterations against the pinned route.

start = time.perf_counter()
stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hi"}],
    stream=True,
    max_tokens=1,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(f"Minimal TTFT: {(time.perf_counter()-start)*1000:.1f} ms")
        break

If minimal TTFT is consistently under your SLO (say 300 ms for a small model on a warm route) but your real payload is 2 s, the issue is payload size or cache misses—go back to Step 3. If minimal TTFT is also 2 s, the provider or network path is the problem—open a support ticket or switch regions.

How to verify success end to end:

• Step 1 script prints stable TTFT across runs.
• curl TTFB matches SDK TTFT within 10%.
• Pinned route removes fallback variance.
• Client flush latency is <20 ms from chunk receipt.
• Minimal payload TTFT meets your latency budget.

Debugging slow time-to-first-token is finished when you can name the layer (network, gateway, provider, or client) and the number moves only when you change that layer. Anything else is guessing.