Setting latency-based routing thresholds is the only way to keep LLM features responsive when upstream providers wobble. In a production LLM gateway, you need explicit rules for when to reroute a request based on time-to-first-token and total latency, not just blind retries. This how-to gives you an end-to-end pattern you can drop into your client and gateway config.
Step 1: Instrument streaming calls to capture real latency
You cannot set useful latency-based routing thresholds without measured numbers. Wrap your completion calls in a streaming client and record two metrics: time to first token (TTFT) and wall-clock completion time. TTFT drives perceived responsiveness; total latency bounds cost and timeout risk.
import time
from openai import OpenAI
client = OpenAI(base_url="https://your-gateway.example/v1", api_key="sk-...")
def timed_completion(model, messages):
start = time.perf_counter()
first_token_ts = None
chunks = []
stream = client.chat.completions.create(
model=model,
messages=messages,
stream=True,
)
for chunk in stream:
if not chunk.choices:
continue
delta = chunk.choices[0].delta
if delta and delta.content:
if first_token_ts is None:
first_token_ts = time.perf_counter()
chunks.append(delta.content)
end = time.perf_counter()
ttft_ms = (first_token_ts - start) * 1000 if first_token_ts else None
total_ms = (end - start) * 1000
return "".join(chunks), ttft_ms, total_ms
text, ttft, total = timed_completion(
"gpt-4o-mini",
[{"role": "user", "content": "Summarize latency routing."}]
)
print(f"TTFT: {ttft:.1f}ms, Total: {total:.1f}ms")Run this against each model tier you use under a realistic concurrency profile. Collect p50 and p95 over a few hundred requests to understand normal behavior. A single synchronous loop understates tail latency; fire at least 10 concurrent streams per model.
Step 2: Define latency-based routing thresholds in config
With baselines in hand, encode thresholds as data. Separate thresholds for TTFT and total latency, and set different tiers for cheap vs. premium models. Streaming UIs tolerate higher total latency if TTFT is low; batch jobs care about total.
{
"routes": {
"fast": {
"model": "gpt-4o-mini",
"max_ttft_ms": 600,
"max_total_ms": 2500
},
"balanced": {
"model": "gpt-4o",
"max_ttft_ms": 1200,
"max_total_ms": 6000
},
"premium": {
"model": "claude-3-5-sonnet",
"max_ttft_ms": 2000,
"max_total_ms": 12000
}
},
"fallback_order": ["fast", "balanced", "premium"]
}These latency-based routing thresholds are the contract your client and gateway enforce. Keep them in environment-specific files; staging can have looser limits because CI runners add jitter. Revisit the numbers when you add a model or change providers.
Step 3: Implement client-side rerouting on breach
The client should abort a slow stream and switch to the next route. Use the thresholds from config to decide. The code below enforces TTFT mid-stream and falls back on timeout.
import json
import time
from openai import OpenAI
with open("routing.json") as f:
cfg = json.load(f)
client = OpenAI(base_url="https://your-gateway.example/v1", api_key="sk-...")
def complete_with_fallback(messages):
for route_name in cfg["fallback_order"]:
route = cfg["routes"][route_name]
start = time.perf_counter()
first_ts = None
try:
stream = client.chat.completions.create(
model=route["model"],
messages=messages,
stream=True,
timeout=route["max_total_ms"] / 1000,
)
collected = []
for chunk in stream:
if not chunk.choices:
continue
if first_ts is None:
first_ts = time.perf_counter()
delta = chunk.choices[0].delta
if delta and delta.content:
collected.append(delta.content)
# enforce TTFT threshold mid-stream
if first_ts is None and (time.perf_counter() - start) * 1000 > route["max_ttft_ms"]:
raise TimeoutError("TTFT exceeded")
return "".join(collected), route_name
except (TimeoutError, Exception) as e:
# automatic fallback to next route
continue
raise RuntimeError("All routes failed")
text, used = complete_with_fallback([{"role": "user", "content": "Go."}])
print(f"Served by {used}")This pattern puts latency-based routing thresholds directly in the request path. It is crude but effective for single-shot calls. For async services, port the same logic to asyncio with async_openai and cancel the task on breach instead of raising.
Step 4: Push thresholds to the gateway with routing directives
Client-side fallback adds a full round-trip of latency on failure. A gateway that honors client routing directives can execute the same logic server-side, closer to the providers. n4n.ai honors client routing directives and forwards provider cache-control hints, so you can declare fallback preferences once and let the gateway handle degraded providers without custom retry code.
When your gateway supports it, send the threshold profile alongside the request. The exact schema depends on the vendor, but the principle is to attach your latency-based routing thresholds as a directive rather than hand-rolling retries:
# Illustrative directive shape – adapt to your gateway's documented field
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
stream=True,
extra_body={
"routing": {
"fallback_order": ["gpt-4o-mini", "gpt-4o", "claude-3-5-sonnet"],
"max_ttft_ms": 600,
"max_total_ms": 2500
}
}
)Because the gateway also provides automatic fallback when a provider is rate-limited or degraded, your directive only needs to express preferences, not exhaustive error handling. That removes the abort-and-retry jumps from your client and shrinks tail latency.
Step 5: Verify with a simulated load test
Set up a loop that fires requests and injects artificial delay or uses a fault-injecting proxy to confirm thresholds trigger. You want to see the route switch without manual intervention.
#!/bin/bash
# Simple latency verification: force a slow upstream via proxy sleep
for i in {1..20}; do
curl -s -o /dev/null -w "%{http_code} %{time_starttransfer} %{time_total}\n" \
-H "Authorization: Bearer $KEY" \
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"ping"}],"stream":true}' \
https://your-gateway.example/v1/chat/completions
doneParse time_starttransfer (TTFT proxy) and time_total. If your gateway or client reroutes, you should see the model field in responses shift to the next tier when the primary breaches thresholds. For a stricter test, add tc netem delay on the gateway egress or use a mock provider that sleeps.
Success criteria
You have working latency-based routing thresholds when:
- p95 TTFT on the primary route stays under the configured
max_ttft_msfor 95% of traffic. - When you artificially delay the primary, logs show a fallback to the next route within one
max_ttft_mswindow. - No request runs past
max_total_mswithout being terminated or rerouted. - Token usage metering (if your gateway provides it) shows fallback routes consuming expected share, not 100% premium.
Operational notes
Latency thresholds are not static. Provider performance shifts daily. Review your p95 numbers weekly and adjust the JSON config. If you meter per-token usage, correlate token cost with latency tiers to avoid routing everything to premium models just because they are fast.
Set alerts on threshold breaches, not just on errors. A gateway that exposes per-token usage metering lets you attribute slowness to specific model/provider pairs. Use that to tune the fallback order.
Finally, streaming changes the math. A low TTFT with a slow token drip is still a bad user experience. Track inter-token latency too, and add a max_inter_token_ms field to your config if your gateway supports it.
That is the full loop: measure, configure, code fallback, delegate to gateway, verify. Build the instrumentation first; everything else is just editing numbers.