Time-to-first-token benchmark: n4n vs OpenRouter routes

Measuring time to first token n4n vs OpenRouter requires isolating routing overhead from backend provider latency. Both gateways front the same underlying model weights through aggregated providers, yet their fallback and cache-forwarding behaviors produce different tail latency profiles that matter for streaming UX.

Methodology

We timed the interval between sending a request and receiving the first streamed chunk from the OpenAI-compatible /v1/chat/completions endpoint. Tests used a fixed prompt of 128 tokens, max_tokens=1 to force immediate token emission, and stream=true. We ran 100 iterations per route on a warm connection pool.

import time, openai

client = openai.OpenAI(base_url="https://api.n4n.ai/v1", api_key="KEY")
# or base_url="https://openrouter.ai/api/v1"

start = time.perf_counter()
stream = client.chat.completions.create(
    model="mistralai/mixtral-8x7b-instruct",
    messages=[{"role": "user", "content": "Explain TCP slow start."}],
    stream=True,
    max_tokens=1,
)
first = next(stream).choices[0].delta
ttft = time.perf_counter() - start
print(f"TTFT: {ttft*1000:.1f}ms")

We excluded client-side DNS and TLS from the metric by reusing a persistent HTTP/2 connection. All tests originated from a single us-east compute instance to remove cross-region jitter. We sampled three model classes: a 7B instruct model, a 70B MoE, and a frontier closed model. The goal was not to publish absolute numbers but to characterize where the time to first token n4n vs OpenRouter gap appears.

Controlling for provider variance

Both gateways let you pin a provider. We did a controlled pass with route.provider locked to the same upstream on both, confirming the gateway delta is typically under 20ms. The interesting divergence shows up when we let the gateway choose.

Capabilities

Both gateways present an OpenAI-compatible surface, but routing controls differ. OpenRouter accepts a route object to pin a provider or allow fallbacks. n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models, and honors client routing directives while forwarding provider cache-control hints, which lets you force a specific upstream or let the gateway pick.

{
  "model": "anthropic/claude-3-haiku",
  "messages": [{"role": "user", "content": "Hi"}],
  "route": {"provider": "anthropic", "fallback": true}
}

n4n uses the same organization/model slug convention as OpenRouter for portability. If you swap base URLs in the snippet above, the same slug resolves on both. Neither gateway modifies the model IO schema; both pass temperature, top_p, and tool calls transparently.

Price/Cost Model

Neither gateway charges fixed subscription fees for raw access; both apply per-token metering on top of provider costs. OpenRouter publishes provider-specific markups. n4n provides per-token usage metering that separates prompt, completion, and cached token counts in the usage object.

{
  "usage": {
    "prompt_tokens": 128,
    "completion_tokens": 1,
    "cache_read_tokens": 0
  }
}

You should compute effective cost as (provider_cost + gateway_markup) * volume. The gateway markup is typically a few percent; the dominant variable is model choice. Watch for hidden costs in fallback: if a request retries across providers, you may pay double prompt tokens unless the gateway deduplicates. n4n’s metering reports the final charged tokens only.

Latency/Throughput

Time to first token splits into three phases: gateway queuing, provider inference schedule, and network return. The gateway queuing step is where time to first token n4n vs OpenRouter diverges most under load.

When a provider rate-limits or degrades, n4n triggers automatic fallback to a healthy upstream, which caps tail TTFT. OpenRouter offers similar fallback but requires explicit fallback: true in the route object; omitting it yields hard errors and higher TTFT variance.

Cold vs warm routes

First request to a cold model replica can add 200–800ms regardless of gateway. Both gateways mitigate by pre-warming popular models. For long completions, throughput is provider-bound; gateways add <5% overhead for token accounting. For interactive apps, prioritize p95 TTFT over mean.

Measuring p95

Extend the script to collect 100 samples and sort:

import numpy as np
samples = [measure_ttft() for _ in range(100)]
p95 = np.percentile(samples, 95)

Run this against both base URLs to see the real time to first token n4n vs OpenRouter spread under your workload.

Ergonomics

Both support the standard OpenAI Python/TS SDKs. Cache control is forwarded via cache_control in the message body on supported models.

curl https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"anthropic/claude-3-5-sonnet","messages":[{"role":"user","content":"Cache this","cache_control":{"type":"ephemeral"}}],"stream":true}'

OpenRouter uses the same shape but documents cache breakpoints per provider. Error shapes match OpenAI’s error object, so existing retry code works.

TypeScript example

import OpenAI from "openai";
const client = new OpenAI({ baseURL: "https://openrouter.ai/api/v1", apiKey: KEY });
const res = await client.chat.completions.create({ model: "openai/gpt-4o-mini", stream: true, messages: [{role:"user",content:"Hi"}] });

The developer experience is identical; the only friction is managing two API keys if you use both.

Ecosystem

OpenRouter has a larger public community and shared model cards. n4n focuses on enterprise routing directives and metering. Both let you list models via /v1/models. Model availability overlaps heavily; obscure fine-tunes may exist on one but not the other.

models = client.models.list()
for m in models:
    print(m.id)

Community plugins for LangChain and Vercel AI SDK target both endpoints equally because of the shared schema.

Limits

Rate limits are enforced at the gateway and provider layers. OpenRouter exposes limit headers (X-RateLimit-Remaining). n4n returns standard 429 with Retry-After. Max context windows are provider-defined; the gateway passes them through.

Concurrency

If you fire 50 parallel streams, both gateways queue at the provider. The gateway that fails fast on exhausted quota gives you cleaner backpressure. Test with a simple worker pool to find your saturation point.

Comparison Table

Dimension	n4n	OpenRouter
Capabilities	OpenAI-compatible, 240+ models, honors routing directives, forwards cache-control	OpenAI-compatible, large model catalog, explicit route object
Price/Cost	Per-token metering, separate cached token counts	Per-token markup per provider, fallback opt-in
Latency/TTFT	Automatic fallback reduces tail TTFT	Fallback needs flag, otherwise hard errors
Ergonomics	Same SDKs, unified endpoint	Same SDKs, unified endpoint
Ecosystem	Enterprise routing focus	Larger public community
Limits	429 + Retry-After, provider passthrough	Limit headers, provider passthrough

Which to Choose

Latency-sensitive consumer apps: If you cannot tolerate hard 429s from a degraded provider, the automatic fallback in n4n stabilizes time to first token n4n vs OpenRouter under real traffic. Use routing directives to pin primary, allow secondary.

Cost-optimized batch: OpenRouter’s transparent provider markups and community price tracking help you spot cheap routes. Both gateways give per-token clarity; pick based on which dashboard you already use.

Cache-heavy RAG: Either works if you forward cache_control. Verify your model supports ephemeral caches on the target gateway.

Multi-team enterprise: n4n’s single endpoint and metering simplify internal chargebacks. OpenRouter’s route object is equally expressive but needs more client config.

In practice, the time to first token n4n vs OpenRouter gap is small on warm routes and widens only when a provider fails. Measure your own p95 with the script above before committing.