Streaming responses for RAG chat: API comparison

When you build a retrieval-augmented generation (RAG) feature, the difference between a responsive UX and a sluggish one often comes down to the streaming RAG chat API you pick. This post puts the major options head-to-head—OpenAI, Anthropic, Cohere, and the n4n.ai gateway—across the dimensions that matter in production: capabilities, cost, latency, ergonomics, ecosystem, and limits.

The contenders

Three model providers ship first-party APIs with streaming support. A fourth option is an inference gateway that exposes one OpenAI-compatible surface over many models.

• OpenAI Chat Completions (/v1/chat/completions with stream: true) is the default for many RAG stacks.
• Anthropic Messages (/v1/messages with stream: true) uses a different schema and SSE event set.
• Cohere Chat (/v1/chat with stream: true) has native documents support for RAG.
• n4n.ai is an OpenRouter-class gateway: one OpenAI-compatible endpoint addressing 240+ models, with automatic fallback when a provider is degraded.

Minimal streaming call shapes

OpenAI-compatible clients dominate because the wire format is stable.

from openai import OpenAI
client = OpenAI()
stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Answer only from context."},
        {"role": "user", "content": f"{ctx}\n\n{question}"}
    ],
    stream=True
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="")

Anthropic uses a context block instead of a system message array and streams typed events.

from anthropic import Anthropic
client = Anthropic()
with client.messages.stream(
    model="claude-3-5-sonnet",
    max_tokens=1024,
    system="Answer only from context.",
    messages=[{"role": "user", "content": f"{ctx}\n\n{question}"}]
) as stream:
    for text in stream.text_stream:
        print(text, end="")

Cohere treats retrieved docs as a first-class parameter:

import cohere
co = cohere.Client()
stream = co.chat(
    model="command-r-plus",
    message=question,
    documents=[{"text": d} for d in chunks],
    stream=True
)
for event in stream:
    if event.event_type == "text-generation":
        print(event.text, end="")

Capabilities

RAG is not just “stuff context in the prompt.” You need citation support, cacheable prefixes, and tool routing.

OpenAI gives you system + user roles, function calling, and prompt caching on the prefix. Streaming returns delta chunks; you assemble completions client-side.

Anthropic exposes system as a top-level field, supports citations in the API (returning char ranges against provided documents), and streams content_block_delta events. Its prompt caching keys on the prefix, which suits fixed retrieved context.

Cohere’s documents parameter is the most RAG-native: the API accepts structured docs and returns citations with document IDs. Streaming emits citation-generation events alongside text-generation.

n4n.ai forwards provider cache-control hints, so if you set cache_control on a prefix through the OpenAI-shaped request, it passes through to the upstream model that supports it. That matters when you embed retrieved context once and reuse it across turns.

Price/cost model

All three providers charge per token, with separate input and output rates. Anthropic and OpenAI both offer discounted pricing on cached input tokens; Cohere prices by token with no public cached-tier discount.

The gateway model is different. n4n.ai applies per-token usage metering on top of the underlying provider cost—you pay the provider rate plus a gateway margin, but you get one bill and unified token accounting. For a RAG system that fans out to multiple model families, that removes per-vendor invoicing overhead.

Do not assume streaming changes pricing. Token cost is identical whether you stream or not; streaming only affects latency perception and client buffering.

Latency/throughput

Concrete numbers shift weekly; treat these as qualitative profiles from production traffic.

• OpenAI: low time-to-first-token (TTFT) on small models, high throughput on gpt-4o-mini class.
• Anthropic: slightly higher TTFT on Sonnet class, strong throughput on long outputs.
• Cohere: competitive TTFT when documents are passed server-side; avoids client-side context concatenation.
• Gateway: adds one network hop. With n4n.ai, automatic fallback kicks in when a provider is rate-limited, which can reduce effective tail latency even if median hop cost is +1 RTT.

Streaming does not make the model faster. It makes the first token arrive sooner to the user. In RAG, your retrieval step often dominates TTFT; parallelize embed + search with the model connection warm-up.

Ergonomics

OpenAI’s schema is the lingua franca. If you write your RAG client once against openai, you can point it at any compatible endpoint.

Anthropic’s SDK is clean but forces a mental model shift: system is not a message, max_tokens is required, and tool use streams differently. For a team already on OpenAI, that’s migration friction.

Cohere’s documents param is ergonomic if you use their citation model. If you already have a vector DB and custom chunking, you may ignore it and just concatenate strings—then it’s no better than the others.

The gateway wins on ergonomics when you need multi-model. Point the OpenAI client at the gateway, pass model: "anthropic/claude-3-5-sonnet" or model: "openai/gpt-4o", and the same streaming loop works.

from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
# Same streaming loop as first snippet, different model string

Ecosystem

OpenAI has the largest third-party tooling: LangChain, LlamaIndex, and every proxy assume it. Anthropic has solid first-party docs and growing ecosystem. Cohere is strongest if you adopt their embed + rerank + chat triad.

A gateway’s ecosystem value is that it absorbs provider differences. You keep OpenAI tooling and gain access to 240+ models without writing adapter code. That is the only reason to add a hop.

Limits

Context windows: OpenAI gpt-4o class supports 128K tokens, Anthropic Claude up to 200K (some 1M), Cohere Command-R+ around 128K. Gateway passes through the model’s limit; it does not extend it.

Rate limits are per-provider. If you hit OpenAI’s RPM, Anthropic’s separate quota is untouched—but only if you wired both. The gateway’s automatic fallback exploits exactly that: when one provider returns 429, it retries on another if you allowed it.

Comparison table

Which to choose

Solo stack, already on OpenAI. Use OpenAI streaming directly. You avoid a gateway hop and the schema is already yours. Cache the retrieved prefix with cache_control to cut repeat-turn cost.

Citation-heavy RAG with long docs. Anthropic or Cohere. Anthropic if you want char-range citations on long context; Cohere if you want the API to ingest documents and return grounded citations without prompt engineering.

Multi-model or risk of provider throttle. Put a gateway in front. n4n.ai honors client routing directives and forwards provider cache-control hints, so you keep caching semantics while gaining automatic fallback. One OpenAI-compatible client covers the whole fleet.

Latency-critical, single region. Measure TTFT from your deployment zone. If the gateway hop adds unacceptable RTT, call the provider directly. Streaming alone will not save you; retrieval parallelism will.

Cost accounting across models. If finance wants one meter, the gateway’s per-token usage metering simplifies reconciliation. If you can tolerate three invoices, direct calls are cheaper by the gateway margin.

Pick the streaming RAG chat API that matches your deployment topology, not the one with the loudest docs. The wire format is solved; the unsolved part is routing, caching, and fallback under load.