The tradeoff between streaming vs batch latency determines whether your agent feels responsive or runs as an overnight job. Streaming emits tokens as the model generates them, letting your UI paint words or trigger tools mid-response. Batch submits many requests at once and returns completed outputs after a provider-side queue drains, optimizing for throughput over interactivity.
Capabilities
What streaming unlocks
Streaming gives you incremental output. For a chat agent, that means time-to-first-token (TTFT) drives perceived speed. For a tool-calling agent, some models stream partial JSON so you can validate a function signature before the response finishes. You can also abort a generation early if the user hits stop, saving tokens.
The code shape is simple with any OpenAI-compatible client. Point your existing SDK at a unified endpoint like n4n.ai’s single OpenAI-compatible route that addresses 240+ models and the streaming loop stays identical:
from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="KEY")
stream = client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[{"role": "user", "content": "Summarize this trace"}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="")What batch unlocks
Batch trades interactivity for scale. You upload a file of requests (usually JSONL), create a batch job, and poll for completion. There is no mid-response hook. This fits offline summarization, bulk classification, eval runs, or backfilling embeddings. You cannot cancel individual requests once queued.
{"custom_id":"req-1","method":"POST","url":"/v1/chat/completions",
"body":{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Classify: X"}]}}curl -X POST https://api.provider.com/v1/batches \
-H "Authorization: Bearer $KEY" \
-d '{"input_file_id":"file-abc","endpoint":"/v1/chat/completions"}'Price / cost model
Streaming does not alter token pricing on most providers. You pay the same per-input and per-output token rate as a non-streaming call. Batch often qualifies for discounts because providers schedule it on spare capacity. OpenAI’s batch API, for instance, cuts price by 50% with a 24-hour turnaround window—a publicly documented offer. Other providers have similar offline tiers.
When you route through a gateway, cost attribution should not change with delivery style. n4n.ai meters per-token usage uniformly across both modes, so finance sees identical line items whether you stream or batch; the discount originates upstream.
Latency / throughput
Streaming vs batch latency is not a single number. For streaming, measure TTFT and tokens-per-second (TPS). A user perceives latency as TTFT plus decode time, but partial rendering makes it feel faster than the wall clock suggests. Batch has no TTFT; instead you measure job queue delay and total completion time. A 10k-request batch might sit for minutes before the first output, but throughput per GPU-hour is far higher because the provider packs requests efficiently.
Streaming holds one connection per active request. At 1k concurrent agents, that is 1k persistent connections with idle keep-alive cost. Batch collapses 1k requests into one job object, eliminating per-request connection overhead at the cost of waiting for the slowest item in the batch.
Ergonomics
Streaming forces async handling. You iterate chunks, buffer partial tool-call JSON, and manage backpressure if your downstream (WebSocket, TUI) is slower than the model. Disconnects mid-stream require retry logic that may duplicate partial work.
async def stream_to_ws(ws, prompt):
stream = await client.chat.completions.create(stream=True, ...)
async for chunk in stream:
await ws.send(chunk.choices[0].delta.content or "")Batch demands job orchestration. You must upload input, track job ID, poll status, handle per-item errors, and reconcile output files. Idempotency keys matter: a failed batch retry should not double-charge completed requests.
# poll batch status
curl https://api.provider.com/v1/batches/batch_123 \
-H "Authorization: Bearer $KEY"Ecosystem
Streaming is first-class in OpenAI, Anthropic, Gemini, and most open-weight servers (vLLM, TGI). Batch is fragmentary: OpenAI has a formal Batch API; Anthropic and others expose message-batch endpoints with different shapes. If your agent switches models per task, a unified OpenAI-compatible streaming interface saves code, but batch still means per-provider request formatting.
Limits
Streaming limits are connection-oriented: proxies may cap response duration at 60–120 seconds, and mobile networks drop idle SSE. Batch limits are job-oriented: max file size (e.g., 100 MB), max requests per batch, and SLA windows (often 24h). You cannot do interactive correction; bad prompts fail silently per row.
Comparison table
| Dimension | Streaming | Batch |
|---|---|---|
| Capabilities | Incremental output, mid-response tool calls | Bulk processing, no interactivity |
| Cost model | Standard per-token rate | Often discounted (e.g., 50% off) |
| Latency | Low TTFT, real-time feel | Queue delay, high total throughput |
| Throughput | 1 req/connection, limited concurrency | Thousands req/job, provider-packed |
| Ergonomics | Async SSE, backpressure, retry partials | Job polling, idempotency, file I/O |
| Ecosystem | Universal SDK support | Provider-specific endpoints |
| Limits | Connection timeouts, network drops | File size caps, SLA windows |
Which to choose
Real-time agent UX
Use streaming. If a human is waiting on a chat reply, a coding copilot, or a voice agent, the streaming vs batch latency gap is decisive. Paint tokens as they arrive.
Agentic loops with tool calls
Use streaming with partial JSON parsing. When the model emits a function call, you can validate and execute before the full response completes, shrinking round-trips in multi-step plans.
Bulk evaluation or data extraction
Use batch. Running 50k test cases against a prompt variant, extracting structured fields from documents, or backfilling a vector store does not need interactivity. The discount and throughput win.
Hybrid systems
Most production agents need both. Stream for the interactive front door; batch for nightly re-indexing, offline evals, and cost-sensitive backfills. Keep a single client abstraction so swapping modes is a flag, not a rewrite.
High-concurrency serving on a budget
If you must serve many users but cannot afford streaming infrastructure, batch-like micro-batching at the gateway (not the provider batch API) can reduce cost while keeping latency acceptable. That is an infrastructure choice, not a provider feature—measure TTFT before committing.