Debugging slow first-token latency in LangChain streams

First-token latency is the metric that determines whether your LLM integration feels snappy or broken. In LangChain streaming chains, a 500 ms delay before the first chunk arrives often traces to prompt construction, provider cold starts, or callback overhead — not the model itself. This guide walks through instrumenting each layer, isolating the bottleneck, and applying targeted fixes you can verify in production.

Step 1: Establish a baseline with minimal instrumentation

Before adding complexity, measure the raw provider latency. Strip LangChain down to a single ChatOpenAI (or your provider of choice) call with streaming=True and no callbacks, no prompt templates, no output parsers.

# baseline.py
import asyncio
import time
from langchain_openai import ChatOpenAI

async def measure_first_token():
    llm = ChatOpenAI(
        model="gpt-4o-mini",
        streaming=True,
        temperature=0,
        max_tokens=100,
    )
    
    messages = [("human", "Say 'ready' and nothing else.")]
    
    start = time.perf_counter()
    first_token_time = None
    
    async for chunk in llm.astream(messages):
        if first_token_time is None:
            first_token_time = time.perf_counter()
            print(f"First token at: {(first_token_time - start) * 1000:.1f} ms")
        print(chunk.content, end="", flush=True)
    
    total = time.perf_counter() - start
    print(f"\nTotal: {total * 1000:.1f} ms")

if __name__ == "__main__":
    asyncio.run(measure_first_token())

Run this 10 times and record the median. On a warm connection to OpenAI, expect 200–400 ms for the first token. If you see 800 ms+, the provider or network is the issue — stop here and check provider status, region, or try a different model. If baseline looks good, proceed.

Verify success: Median first-token latency under 500 ms on your target provider with a trivial prompt.

Step 2: Isolate prompt construction overhead

Prompt templates, few-shot examples, and dynamic context injection add measurable latency before the request leaves your process. Wrap your prompt-building logic in a timer.

# prompt_timing.py
import time
from langchain_core.prompts import ChatPromptTemplate, FewShotChatMessagePromptTemplate

examples = [
    {"input": "2+2", "output": "4"},
    {"input": "3*3", "output": "9"},
]

example_prompt = ChatPromptTemplate.from_messages([
    ("human", "{input}"),
    ("ai", "{output}"),
])

few_shot = FewShotChatMessagePromptTemplate(
    example_prompt=example_prompt,
    examples=examples,
)

final_prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a math tutor."),
    few_shot,
    ("human", "{question}"),
])

def time_prompt_render(question: str) -> float:
    start = time.perf_counter()
    messages = final_prompt.format_messages(question=question)
    elapsed = time.perf_counter() - start
    print(f"Prompt render: {elapsed * 1000:.2f} ms, {len(messages)} messages")
    return elapsed

# Test with increasing context
for q in ["What is 5+5?", "Explain calculus in 50 words", "x" * 5000]:
    time_prompt_render(q)

Typical render times: 0.5–2 ms for simple prompts, 5–15 ms with few-shot, 50+ ms if you’re stuffing large retrieved contexts into the template. If prompt rendering exceeds 20 ms, consider pre-compiling templates, caching formatted few-shot blocks, or moving context injection out of the hot path.

Verify success: Prompt rendering under 10 ms for your production prompt shapes.

Step 3: Measure callback and handler overhead

LangChain callbacks fire on every token. A synchronous callback that does I/O, logging, or heavy computation will block the event loop and inflate perceived latency. Instrument the callback chain.

# callback_timing.py
import asyncio
import time
from typing import Any, Dict, List
from langchain_core.callbacks import AsyncCallbackHandler
from langchain_openai import ChatOpenAI

class TimingCallback(AsyncCallbackHandler):
    def __init__(self):
        self.first_token_received = False
        self.first_token_time = 0
        self.start_time = 0
        self.token_count = 0
    
    async def on_llm_start(self, serialized: Dict[str, Any], prompts: List[str], **kwargs):
        self.start_time = time.perf_counter()
        self.first_token_received = False
    
    async def on_llm_new_token(self, token: str, **kwargs):
        if not self.first_token_received:
            self.first_token_time = time.perf_counter()
            self.first_token_received = True
            print(f"Callback saw first token at: {(self.first_token_time - self.start_time) * 1000:.1f} ms")
        self.token_count += 1
        # Simulate slow callback work
        # await asyncio.sleep(0.001)  # Uncomment to see impact
    
    async def on_llm_end(self, response, **kwargs):
        total = time.perf_counter() - self.start_time
        print(f"Stream complete: {self.token_count} tokens in {total * 1000:.1f} ms")

async def test_with_callback():
    callback = TimingCallback()
    llm = ChatOpenAI(model="gpt-4o-mini", streaming=True, callbacks=[callback])
    
    await llm.ainvoke("Count to 10.")

if __name__ == "__main__":
    asyncio.run(test_with_callback())

Run with and without the sleep simulation. Each 1 ms of callback work per token adds up fast — 100 tokens × 1 ms = 100 ms of serialized delay. Move all I/O (database writes, HTTP calls, file appends) to a background queue. Keep callbacks synchronous and allocation-free.

Verify success: Callback overhead under 0.1 ms per token; first-token timestamp in callback matches raw provider latency from Step 1 within 5 ms.

Step 4: Check for connection reuse and HTTP client configuration

LangChain’s ChatOpenAI uses httpx under the hood. By default, it creates a new client per request unless you pass a shared AsyncClient. Cold TCP/TLS handshakes add 50–200 ms.

# client_reuse.py
import asyncio
import httpx
import time
from langchain_openai import ChatOpenAI

# Shared client with connection pooling
shared_client = httpx.AsyncClient(
    limits=httpx.Limits(max_connections=100, max_keepalive_connections=20),
    timeout=httpx.Timeout(30.0, connect=5.0),
)

async def test_with_shared_client():
    llm = ChatOpenAI(
        model="gpt-4o-mini",
        streaming=True,
        http_async_client=shared_client,
    )
    
    # Warm-up request
    await llm.ainvoke("warmup")
    
    # Measured request
    start = time.perf_counter()
    async for _ in llm.astream("Say hello"):
        if start:
            print(f"First token: {(time.perf_counter() - start) * 1000:.1f} ms")
            start = None

async def test_without_shared_client():
    llm = ChatOpenAI(model="gpt-4o-mini", streaming=True)
    start = time.perf_counter()
    async for _ in llm.astream("Say hello"):
        if start:
            print(f"First token (new client): {(time.perf_counter() - start) * 1000:.1f} ms")
            start = None

async def main():
    print("With shared client:")
    await test_with_shared_client()
    print("\nWithout shared client:")
    await test_without_shared_client()
    await shared_client.aclose()

if __name__ == "__main__":
    asyncio.run(main())

In production, create the AsyncClient at application startup and pass it to every ChatOpenAI instance. Set max_keepalive_connections to match your concurrency target. If you’re using a gateway like n4n.ai that fronts multiple providers, the same principle applies — reuse the client to the gateway endpoint.

Verify success: Second and subsequent requests show 50–150 ms lower first-token latency than the first request.

Step 5: Diagnose provider-side queuing and cold starts

Even with a warm connection, the provider may queue your request or spin up a model instance. Distinguish this from client-side latency by examining response headers and timing the gap between request send and first byte received.

# provider_timing.py
import asyncio
import time
import httpx
from langchain_openai import ChatOpenAI

async def trace_request():
    client = httpx.AsyncClient(timeout=httpx.Timeout(60.0))
    
    llm = ChatOpenAI(
        model="gpt-4o-mini",
        streaming=True,
        http_async_client=client,
    )
    
    # Patch to capture timings
    original_send = client.send
    
    request_sent_time = 0
    first_byte_time = 0
    
    async def tracing_send(request, **kwargs):
        nonlocal request_sent_time, first_byte_time
        request_sent_time = time.perf_counter()
        response = await original_send(request, **kwargs)
        # For streaming, first byte arrives during iteration
        return response
    
    client.send = tracing_send
    
    start = time.perf_counter()
    async for chunk in llm.astream("Hello"):
        if first_byte_time == 0:
            first_byte_time = time.perf_counter()
            print(f"Time to first byte: {(first_byte_time - request_sent_time) * 1000:.1f} ms")
            print(f"Client overhead (send - start): {(request_sent_time - start) * 1000:.1f} ms")
        print(chunk.content, end="")
    
    await client.aclose()

if __name__ == "__main__":
    asyncio.run(trace_request())

If “Time to first byte” exceeds 1 second consistently, the provider is the bottleneck. Check for:

• Rate limit headers (x-ratelimit-remaining, retry-after)
• Model-specific cold starts (larger models, fine-tunes)
• Geographic latency (request routing to distant regions)

Some gateways expose provider health signals. If you’re routing through a layer that honors cache-control hints and provides automatic fallback, you can shift traffic away from degraded providers without code changes.

Verify success: Time-to-first-byte under 800 ms for your primary provider on warm requests.

Step 6: Eliminate output parser and chain overhead

RunnableSequence and output parsers add function call overhead per chunk. For latency-critical paths, stream raw chunks and parse incrementally or post-process.

# parser_overhead.py
import asyncio
import time
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI

async def with_parser():
    llm = ChatOpenAI(model="gpt-4o-mini", streaming=True)
    chain = llm | StrOutputParser()
    
    start = time.perf_counter()
    first = True
    async for chunk in chain.astream("Count to 5."):
        if first:
            print(f"With parser - first chunk: {(time.perf_counter() - start) * 1000:.1f} ms")
            first = False

async def without_parser():
    llm = ChatOpenAI(model="gpt-4o-mini", streaming=True)
    
    start = time.perf_counter()
    first = True
    async for chunk in llm.astream("Count to 5."):
        if first:
            print(f"Raw - first chunk: {(time.perf_counter() - start) * 1000:.1f} ms")
            first = False

async def main():
    await with_parser()
    await without_parser()

if __name__ == "__main__":
    asyncio.run(main())

The difference is usually 1–3 ms per chunk — negligible for first token but cumulative. If you need structured output, consider:

• Streaming JSON parsers that yield partial objects (json5, pydantic with partial=True)
• Parsing only the final accumulated response
• Using provider-native structured output (OpenAI response_format, Anthropic tool_use) to avoid post-processing entirely

Verify success: First-token latency within 5 ms of raw streaming baseline.

Step 7: Add production observability

You’ve isolated the components. Now wire permanent instrumentation so regressions trigger alerts.

# observability.py
import time
from functools import wraps
from typing import Callable, Any
import structlog

logger = structlog.get_logger()

def measure_first_token(operation: str):
    def decorator(func: Callable) -> Callable:
        @wraps(func)
        async def wrapper(*args, **kwargs):
            start = time.perf_counter()
            first_token_time = None
            token_count = 0
            
            async def track_first_token(chunk):
                nonlocal first_token_time, token_count
                if first_token_time is None:
                    first_token_time = time.perf_counter()
                    logger.info(
                        "first_token_latency",
                        operation=operation,
                        latency_ms=(first_token_time - start) * 1000,
                    )
                token_count += 1
                return chunk
            
            # Assumes the function returns an async iterator
            async for chunk in func(*args, **kwargs):
                yield await track_first_token(chunk)
            
            total = time.perf_counter() - start
            logger.info(
                "stream_complete",
                operation=operation,
                total_ms=total * 1000,
                tokens=token_count,
                tokens_per_sec=token_count / total if total > 0 else 0,
            )
        return wrapper
    return decorator

# Usage
@measure_first_token("chat_completion")
async def stream_chat(messages: list):
    llm = ChatOpenAI(model="gpt-4o-mini", streaming=True)
    async for chunk in llm.astream(messages):
        yield chunk

Emit structured logs to your observability stack (Datadog, Honeycomb, Grafana). Alert on p95 first-token latency exceeding your SLO. Tag by model, provider, prompt template version, and deployment environment.

Verify success: Dashboards show per-operation first-token latency percentiles; alerts fire on regression within 5 minutes.

Step 8: Apply targeted fixes and verify end-to-end

Combine the fixes that applied to your stack. A typical optimized configuration:

# optimized_config.py
import httpx
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

# 1. Shared client with connection pooling
http_client = httpx.AsyncClient(
    limits=httpx.Limits(max_connections=200, max_keepalive_connections=50),
    timeout=httpx.Timeout(30.0, connect=5.0),
    http2=True,  # Enable HTTP/2 if provider supports it
)

# 2. Pre-compiled prompt (module-level, not per-request)
SYSTEM_PROMPT = "You are a concise assistant."
PROMPT_TEMPLATE = ChatPromptTemplate.from_messages([
    ("system", SYSTEM_PROMPT),
    ("human", "{question}"),
])

# 3. LLM instance reused across requests
llm = ChatOpenAI(
    model="gpt-4o-mini",
    streaming=True,
    temperature=0,
    max_tokens=500,
    http_async_client=http_client,
    # Disable callbacks you don't need
    callbacks=[],
)

# 4. Minimal chain — no parser for streaming
chain = PROMPT_TEMPLATE | llm

async def handle_request(question: str):
    """Production handler with all optimizations."""
    messages = await PROMPT_TEMPLATE.ainvoke({"question": question})
    async for chunk in llm.astream(messages):
        yield chunk.content

Run your load test against this configuration. Compare p50/p95/p99 first-token latency against the baseline from Step 1. Typical improvement: 30–60% reduction in p95 first-token latency for chains that previously had prompt overhead, callback blocking, or connection churn.

Verify success: End-to-end p95 first-token latency meets your SLO (e.g., < 600 ms) under production-like load.

Common pitfalls to avoid

Measuring wall time instead of component time. time.perf_counter() around the whole chain tells you nothing about where the delay lives. Instrument each boundary.

Assuming the model is slow. GPT-4o-mini first token is typically 200–400 ms. If you see 2 seconds, it’s your code, not the model.

Adding async callbacks that block. async def on_llm_new_token with await db.write() serializes the stream. Use asyncio.create_task() or a queue.

Ignoring provider headers. x-ratelimit-reset, retry-after, and cache-control hints tell you exactly when the provider is the bottleneck.

Premature optimization. If baseline is 300 ms and your SLO is 1 second, stop. Ship features instead.

Summary checklist

• Baseline raw provider latency (Step 1)
• Prompt render time < 10 ms (Step 2)
• Callback overhead < 0.1 ms/token (Step 3)
• Shared HTTP client with keep-alive (Step 4)
• Provider time-to-first-byte < 800 ms (Step 5)
• Parser/chain overhead < 5 ms (Step 6)
• Production observability emitting percentiles (Step 7)
• End-to-end p95 meets SLO (Step 8)

First-token latency is a system property, not a model property. The model is usually the fastest part of your pipeline. Measure each layer, fix the slowest one, and repeat until the numbers are boring.