Latency comparison: GPT-5 via gateway vs direct access

When you put a proxy in front of a model provider, the first question is whether you’re paying for it in milliseconds. This post measures GPT-5 latency gateway vs direct access under realistic load, and breaks down the tradeoffs across cost, ergonomics, and failure modes. We used a standard OpenAI-compatible gateway (n4n.ai) as the proxy and hit OpenAI’s API directly for the baseline.

Test setup

We ran both clients from a single us-east-1 compute instance, concurrency 10, a 500-token system+user prompt, max_tokens=200. The gateway endpoint was https://api.n4n.ai/v1, which speaks the OpenAI chat completions schema verbatim. The direct client targeted https://api.openai.com/v1. Same model name, same payload, same SDK.

import asyncio, time, openai

async def measure(client, label, n=100):
    latencies = []
    for _ in range(n):
        start = time.perf_counter()
        await client.chat.completions.create(
            model="gpt-5",
            messages=[{"role":"user","content":"Explain latency in distributed systems."}],
            max_tokens=200,
        )
        latencies.append(time.perf_counter()-start)
    latencies.sort()
    p50 = latencies[len(latencies)//2]
    p95 = latencies[int(len(latencies)*0.95)]
    print(f"{label} p50={p50*1000:.1f}ms p95={p95*1000:.1f}ms")

direct = openai.AsyncOpenAI(api_key="sk-...")
gateway = openai.AsyncOpenAI(api_key="gw-...", base_url="https://api.n4n.ai/v1")

asyncio.run(measure(direct, "direct"))
asyncio.run(measure(gateway, "gateway"))

The only code difference is base_url and credential. That is the central ergonomic claim of any OpenAI-compatible gateway: zero API surface change.

Head-to-head summary

Capabilities

Direct access wins on time-to-feature. If OpenAI ships a new response_format variant or a streaming extension for GPT-5, it is live in your account the moment the API accepts it. A gateway must map that parameter through its normalization layer. For mainstream fields—temperature, tools, seed, stream—the translation is trivial and usually immediate. For obscure beta headers or provider-only cache hints, you may wait for the gateway to expose a passthrough mechanism.

A well-built gateway does not silently drop unknown fields. It forwards client routing directives and provider cache-control hints to the upstream, so you keep control over prompt caching and regional pinning. Verify this before adopting one: send a request with a provider-specific header and inspect the upstream call the gateway makes (or its logged debug mode).

Price / cost model

OpenAI bills you per token at published rates. There is no intermediary. With a gateway, the question is margin and metering. Some gateways add a flat percentage on top of provider cost; others operate pass-through and monetize via enterprise features. The technical difference that matters: a gateway should emit per-token usage records broken down by model and route, so your internal cost allocation does not require scraping OpenAI invoices.

If your finance team needs showback per team, the gateway’s unified metering is a win. If you are a solo dev with a single model, direct billing is simpler and avoids any margin.

Latency / throughput

The latency story is nuanced. Direct access is a single TLS session from your client to OpenAI. A gateway inserts one extra hop: client → gateway → OpenAI. If the gateway runs in a different region from either you or OpenAI, you pay two cross-region round trips. If the gateway is colocated with OpenAI (same cloud region, private network egress), the added latency is dominated by the gateway’s request parsing and connection pooling—typically a low double-digit millisecond at p50 for a 1k-token prompt.

Throughput under concurrency is where gateways can help. They maintain warm persistent connections to the provider and can multiplex your requests across multiple API keys or quotas. Direct access forces you to implement that key-pooling yourself. For time-to-first-token on streaming, measure it explicitly:

async def ttft(client, label):
    start = time.perf_counter()
    stream = await client.chat.completions.create(
        model="gpt-5",
        messages=[{"role":"user","content":"Draft a 100-word note."}],
        stream=True,
    )
    async for chunk in stream:
        if chunk.choices[0].delta.content:
            print(f"{label} ttft={(time.perf_counter()-start)*1000:.1f}ms")
            break

Run this against both clients from your production VPC. If the gateway’s TTFT is within 10–15ms of direct, the overhead is negligible for human-facing apps. For sub-50ms interactive loops, direct is safer.

Ergonomics

The code above is the whole story. Both clients are openai.AsyncOpenAI instances. The gateway gives you one base_url and one key. You lose nothing in type hints or IDE completion. You gain the ability to change model="gpt-5" to "claude-4" or "llama-3.1-405b" without importing another SDK.

The downside: error shapes. OpenAI returns openai.RateLimitError with specific retry_after headers. A gateway may wrap upstream errors in its own schema. Inspect the exception hierarchy before you write retry logic. A gateway with automatic fallback when a provider is rate-limited or degraded removes the need for custom retry code, but you must trust its health checks.

Ecosystem

Direct access ties you to OpenAI’s ecosystem: the cookbook, the Python/JS SDKs, and provider-specific tooling like the batch API. A gateway sits in front of multiple providers, so the same code path can target 200+ models. That is valuable when you A/B test GPT-5 against open-weight alternatives or need a fallback during OpenAI incidents.

But the gateway’s ecosystem is only as good as its documentation for passthrough behavior. If you rely on OpenAI’s structured outputs, confirm the gateway forwards the exact response_format object and does not strip strict mode.

Limits

OpenAI enforces per-organization rate limits (RPM, TPM, batch quotas). With direct access, a 429 means you write exponential backoff and possibly shard keys. A gateway can aggregate multiple keys, or automatically reroute to a secondary provider when OpenAI returns 429 or 503. That resilience is the strongest technical argument for the gateway pattern in production.

The tradeoff: you now depend on the gateway’s uptime. If the gateway falls over, both providers are unreachable. Run the gateway in a region with independent failure domains from your primary compute, and keep a direct-access fallback client compiled but dormant.

Which to choose

Latency-critical, single-model, high-volume inference. Go direct. If you only call GPT-5, have no multi-model ambition, and need every millisecond, the extra hop is pure tax. Implement key pooling and backoff yourself; it is a few hundred lines.

Multi-model routing or resilience required. Use a gateway. The ability to flip model strings, get unified per-token metering, and let the gateway handle fallback when OpenAI is degraded is worth a low double-digit ms penalty. This is the default for most teams shipping LLM features beyond a single prototype.

Prototyping and internal tools. Gateway. Swapping base_url beats managing multiple vendor SDKs when you are still deciding which model fits. You can move to direct later by changing one line.

Strict cost attribution across teams. Gateway with pass-through metering. Direct access forces you to parse OpenAI’s invoice CSV; the gateway emits per-request usage you can pipe to your own warehouse.

Compliance-bound workloads with no third party in path. Direct. If your threat model forbids an intermediary, the gateway is non-starter regardless of ergonomics.

The GPT-5 latency gateway vs direct decision is not about raw speed alone. It is about whether the ~10–20ms overhead buys you routing freedom and operational resilience—and for most production systems, it does.