LangChain streaming vs batch latency isn’t a theoretical question — it’s a production decision that affects your p99, your infrastructure costs, and whether users perceive your app as responsive. The short answer: streaming reduces time-to-first-token dramatically but adds callback overhead that can increase total latency for short completions. For responses under ~200 tokens, batch often wins on wall-clock time. For anything longer, streaming wins on perceived latency and lets you parallelize downstream work.
What streaming actually does in LangChain
When you call astream() or stream() on a LangChain runnable, you’re not just getting tokens earlier. You’re opting into a different execution model: the runnable yields AIMessageChunk objects through an async generator, each chunk triggering any attached callbacks. The LLM provider still generates tokens sequentially, but your application receives them incrementally instead of waiting for the full completion.
from langchain_openai import ChatOpenAI
from langchain_core.callbacks import AsyncCallbackHandler
llm = ChatOpenAI(model="gpt-4o-mini", streaming=True)
# Batch: single request, single response
batch_result = await llm.ainvoke("Write a haiku about distributed systems")
# Streaming: async generator yielding chunks
async for chunk in llm.astream("Write a haiku about distributed systems"):
print(chunk.content, end="", flush=True)The streaming=True parameter on the model constructor is the switch. Without it, astream() falls back to invoking the model in batch mode and yielding a single chunk — defeating the purpose. With it, the underlying HTTP request uses server-sent events (SSE) or the provider’s native streaming protocol, and LangChain’s callback system fans each chunk out to registered handlers.
Latency breakdown: first token vs total time
The latency profile differs fundamentally between the two modes. Batch latency is straightforward: one request, one response, total time = network RTT + provider queue time + generation time. Streaming splits this into time-to-first-token (TTFT) and inter-token latency.
Batch: [====== 800ms total ======]
Stream: [TTFT 120ms][token][token][token]...[total 950ms]TTFT for streaming is typically 2-5x faster than batch total time because the provider starts streaming before generation completes. But total streaming time often exceeds batch by 5-15% due to callback dispatch overhead per chunk. Here’s a realistic measurement harness:
import asyncio
import time
from statistics import mean, median
async def measure_batch(llm, prompt, runs=20):
latencies = []
for _ in range(runs):
start = time.perf_counter()
await llm.ainvoke(prompt)
latencies.append(time.perf_counter() - start)
return latencies
async def measure_stream(llm, prompt, runs=20):
latencies = []
ttfts = []
for _ in range(runs):
start = time.perf_counter()
first = True
async for chunk in llm.astream(prompt):
if first:
ttfts.append(time.perf_counter() - start)
first = False
latencies.append(time.perf_counter() - start)
return latencies, ttfts
# Typical results on gpt-4o-mini, 150-token completion:
# Batch: median 820ms, p99 1.1s
# Stream: median TTFT 180ms, median total 940ms, p99 1.3sRun this against your actual provider and prompt lengths. The crossover point where streaming total time exceeds batch varies by model, provider, and network path — but it’s real and measurable.
When streaming helps (and when it doesn’t)
Streaming wins when:
- User-facing chat: TTFT under 200ms makes the app feel instant. Users read while tokens arrive.
- Long completions: Summarization, code generation, or multi-step reasoning where total tokens exceed 300. The TTFT advantage compounds.
- Downstream parallelism: You can start parsing, rendering, or feeding tokens to another model before generation finishes.
# Streaming enables pipeline parallelism
async def stream_and_process(llm, prompt):
buffer = []
async for chunk in llm.astream(prompt):
buffer.append(chunk.content)
# Process incrementally: syntax highlight, safety check, partial render
if len(buffer) >= 50: # flush every ~50 chars
await render_partial("".join(buffer))
buffer.clear()
if buffer:
await render_partial("".join(buffer))Streaming loses when:
- Short completions: Classification, extraction, or yes/no answers under 50 tokens. Callback overhead dominates.
- High-throughput batch jobs: Processing 10k documents where you control the pipeline. Batch maximizes provider throughput.
- Structured output with validation: If you need the full JSON to validate before proceeding, streaming adds complexity without benefit.
# Don't stream this — batch is faster and simpler
classification_llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
result = await classification_llm.ainvoke([
{"role": "system", "content": "Classify as SPAM or HAM. Respond with only the label."},
{"role": "user", "content": email_body}
])
# result.content is "SPAM" or "HAM" — one round trip, no parsing partial tokensCallback overhead and hidden costs
LangChain’s callback system is the primary source of streaming overhead. Every chunk dispatches to every registered AsyncCallbackHandler.on_llm_new_token. With the default StdOutCallbackHandler or custom logging handlers, this adds 0.1-0.5ms per chunk. For a 1000-token response, that’s 100-500ms of pure Python dispatch overhead.
# This adds measurable latency per chunk
class LoggingHandler(AsyncCallbackHandler):
async def on_llm_new_token(self, token: str, **kwargs):
await self.log_to_datadog(token) # network call per token = disaster
# Better: batch callbacks or use a no-op handler for production streaming
class BatchedCallbackHandler(AsyncCallbackHandler):
def __init__(self, flush_every=50):
self.buffer = []
self.flush_every = flush_every
async def on_llm_new_token(self, token: str, **kwargs):
self.buffer.append(token)
if len(self.buffer) >= self.flush_every:
await self.flush()
async def flush(self):
if self.buffer:
await self.log_batch("".join(self.buffer))
self.buffer.clear()The streaming=True model parameter also changes connection behavior. Most providers keep the HTTP connection open for the full generation, which consumes a connection pool slot longer. Under high concurrency, this can exhaust client-side connection pools faster than short-lived batch requests. Configure your HTTP client accordingly:
import httpx
from langchain_openai import ChatOpenAI
# Tune for streaming: larger pool, longer keepalive
http_client = httpx.AsyncClient(
limits=httpx.Limits(max_connections=100, max_keepalive_connections=50),
timeout=httpx.Timeout(connect=10.0, read=300.0) # read timeout covers full stream
)
llm = ChatOpenAI(
model="gpt-4o-mini",
streaming=True,
http_client=http_client
)Memory and backpressure considerations
Streaming shifts memory pressure from the response buffer to your application. A batch response holds the full completion in memory once. Streaming holds chunks only as long as your code retains them — but if you buffer the entire stream before processing (a common mistake), you’ve gained nothing and added callback overhead.
# Bad: defeats streaming memory advantage
async def bad_stream(llm, prompt):
chunks = []
async for chunk in llm.astream(prompt):
chunks.append(chunk.content)
return "".join(chunks) # same memory as batch, slower
# Good: process incrementally, bounded memory
async def good_stream(llm, prompt, handler):
async for chunk in llm.astream(prompt):
await handler.process(chunk.content) # write to socket, file, or next stageBackpressure matters when downstream is slower than token generation. If you’re streaming to a WebSocket client on a slow mobile connection, the provider’s TCP buffer fills, the provider blocks, and your generator stalls. LangChain’s astream() doesn’t expose flow control — you’re at the mercy of the underlying HTTP client. For production WebSocket streaming, consider a bounded queue between the LLM stream and the WebSocket write loop:
import asyncio
async def stream_to_websocket(llm, prompt, websocket, queue_size=100):
queue = asyncio.Queue(maxsize=queue_size)
async def producer():
try:
async for chunk in llm.astream(prompt):
await queue.put(chunk.content)
finally:
await queue.put(None) # sentinel
async def consumer():
while True:
token = await queue.get()
if token is None:
break
await websocket.send_text(token)
queue.task_done()
await asyncio.gather(producer(), consumer())This prevents unbounded memory growth when the client can’t keep up.
Production patterns
Hybrid approach: stream long, batch short
Route based on expected token count. Use a cheap classifier or heuristic (prompt length, task type) to decide:
async def smart_invoke(llm, prompt, max_batch_tokens=150):
# Heuristic: if prompt suggests short answer, use batch
if estimated_output_tokens(prompt) < max_batch_tokens:
return await llm.ainvoke(prompt)
# Otherwise stream with optimized callbacks
llm.streaming = True
handler = BatchedCallbackHandler(flush_every=100)
return await llm.astream(prompt, config={"callbacks": [handler]})Structured output with streaming
If you need validated JSON but want streaming UX, stream into a partial parser:
from pydantic import BaseModel
import json
class Extraction(BaseModel):
entities: list[str]
sentiment: str
async def stream_structured(llm, prompt):
buffer = ""
parser = IncrementalJsonParser(Extraction) # hypothetical incremental parser
async for chunk in llm.astream(prompt):
buffer += chunk.content
partial = parser.parse_partial(buffer)
if partial:
yield partial # emit validated partial results
final = parser.parse_complete(buffer)
yield finalNote: LangChain’s JsonOutputParser doesn’t support incremental parsing natively. You’ll need a streaming JSON parser like ijson or a custom implementation.
Fallback and retry strategy
Streaming requests fail differently than batch. A mid-stream network error leaves you with a partial response. Design your retry logic accordingly:
async def stream_with_retry(llm, prompt, max_retries=2):
for attempt in range(max_retries + 1):
try:
async for chunk in llm.astream(prompt):
yield chunk
return # success
except httpx.StreamError as e:
if attempt == max_retries:
raise
# Exponential backoff, but don't retry on client errors
if e.response and 400 <= e.response.status_code < 500:
raise
await asyncio.sleep(2 ** attempt)The decisive takeaway
Default to batch. Use streaming only when you have a measured TTFT requirement or a downstream pipeline that benefits from incremental processing. The callback overhead, connection pool pressure, and error-handling complexity are real costs that don’t pay off for short completions.
Measure your actual crossover point. Run the latency harness against your provider, your prompt templates, your token lengths. If batch p99 is acceptable for your use case, stay with batch. If users complain about “slow to start” on long generations, enable streaming for those specific chains — not globally.
And if you’re routing across multiple providers where some degrade or rate-limit, a gateway that handles fallback transparently (like n4n.ai) keeps your streaming contracts intact without littering your application code with provider-specific retry logic. The streaming vs batch decision should live in your application layer, not your infrastructure layer.