When you migrate from OpenRouter to n4n testing, the objective is to prove behavioral parity before any user-facing traffic shifts. Both gateways speak the OpenAI chat completions API, but routing semantics, model aliases, and fallback behavior differ in ways that break naive swaps. This article lays out a concrete rehearsal sequence: capture real traffic, shadow it against the new gateway, diff responses, and validate failure modes before flipping the switch.
Step 1: Capture a representative request corpus
You cannot test what you don’t record. OpenRouter’s own logs are insufficient because they omit the exact payload shapes your client emits, including custom parameters and message ordering. Instrument your existing client to mirror outbound requests to a local file or queue for a defined window—aim for at least 500 diverse conversations covering your top models, max_tokens settings, temperature ranges, and any response_format constraints.
import json
import time
from openai import OpenAI
class RecordingClient:
def __init__(self, api_key, log_path="corpus.jsonl"):
self.client = OpenAI(base_url="https://openrouter.ai/api/v1", api_key=api_key)
self.log_path = log_path
def chat(self, model, messages, **kwargs):
resp = self.client.chat.completions.create(model=model, messages=messages, **kwargs)
with open(self.log_path, "a") as f:
f.write(json.dumps({
"ts": time.time(),
"model": model,
"messages": messages,
"params": kwargs,
"completion": resp.model_dump()
}) + "\n")
return respRedact any PII in messages before writing to disk if your traffic isn’t already synthetic. Verification: the corpus file grows and contains both request and response fields. Spot-check that model strings match your OpenRouter model IDs (e.g., openai/gpt-4o). This set becomes the fixture for all subsequent steps.
Step 2: Stand up a shadow client for n4n
Point a second client at the new gateway. n4n.ai provides one OpenAI-compatible endpoint that fronts 240+ models, so you keep the same SDK; only the base URL and key change. Use environment variables to avoid hardcoding secrets.
import os
from openai import OpenAI
shadow = OpenAI(
base_url="https://api.n4n.ai/v1",
api_key=os.environ["N4N_API_KEY"]
)Map OpenRouter model IDs to n4n model IDs. Most providers use the same provider/model slug, but verify against the n4n model list. If you relied on OpenRouter’s route=fallback query param, note that n4n honors client routing directives via request headers instead—consult the gateway docs for the exact header name rather than assuming query parity.
MODEL_MAP = {
"openai/gpt-4o": "openai/gpt-4o",
"anthropic/claude-3.5-sonnet": "anthropic/claude-3.5-sonnet",
}Verification: a single manual call returns 200 with a valid completion. Check the usage object includes prompt_tokens and completion_tokens for later metering comparison.
Step 3: Replay and diff responses
Replaying asynchronously exposes latency and content deviations. Write a harness that reads the corpus, sends each request to the shadow client, and records discrepancies. For deterministic prompts (temperature=0), exact-match the first 50 characters. For stochastic outputs, compute embedding cosine similarity locally.
import json
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
def replay(corpus_path, shadow_client, model_map):
results = []
with open(corpus_path) as f:
for line in f:
rec = json.loads(line)
mapped = model_map.get(rec["model"], rec["model"])
try:
resp = shadow_client.chat.completions.create(
model=mapped, messages=rec["messages"], **rec["params"]
)
orig = rec["completion"]["choices"][0]["message"]["content"]
new = resp.choices[0].message.content
sim = model.similarity(model.encode(orig), model.encode(new))
results.append({
"id": rec["ts"], "status": "ok",
"token_delta": resp.usage.completion_tokens - rec["completion"]["usage"]["completion_tokens"],
"sim": float(sim)
})
except Exception as e:
results.append({"id": rec["ts"], "status": "error", "error": str(e)})
return resultsDiff criteria should be pragmatic: HTTP status, token count delta under 5%, and similarity above 0.95 for non-deterministic calls. Verification: produce a summary JSON with error rate and diff rate. A pass threshold might be <1% errors and <2% semantic diffs on your corpus.
Step 4: Validate routing directives and cache hints
OpenRouter uses cache query params; n4n forwards provider cache-control hints via standard HTTP headers. If your app sets Cache-Control: max-age=3600 to leverage provider prompt caching, confirm the gateway passes it through.
resp = shadow.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[{"role": "user", "content": "Repeat: hello"}],
extra_headers={"Cache-Control": "max-age=3600"}
)
print(resp.usage)Inspect the usage object for cache_creation_tokens or cache_read_tokens. If those appear, the hint was honored and you’ll be billed at reduced rates. Also test explicit routing directives (e.g., prefer cheapest) via the documented header; the response model field should reflect the selected provider variant.
Verification: run with and without cache header; observe token cost difference in usage. If cached tokens are reported, the hint worked.
Step 5: Test automatic fallback behavior
A core reason to migrate from OpenRouter to n4n testing is resilience. n4n performs automatic fallback when a provider is rate-limited or degraded. Simulate degradation by placing a local proxy that returns 429 for the primary model, then point the shadow client at the proxy-fronted n4n endpoint.
# mitmproxy example to force 429 on a model
def request(flow):
if "gpt-4o" in flow.request.path:
flow.response = http.HTTPResponse.make(429, b"rate limited")In real fallback, you won’t see the error—n4n retries another provider and returns a completion. To verify, monitor the usage object and confirm you were billed by a backup provider (different model slug in response). If your fault injection is bypassed because n4n already fell back, the client still receives a 200.
Verification: inject a fault; confirm the shadow client receives a 200 from n4n with a different model field than requested, proving fallback occurred.
Step 6: Canary cutover with configuration flag
Do not flip DNS. Use a runtime toggle that sends a small percentage of live traffic to n4n while logging both paths.
import random
def get_client(weight=0.05, prod=None, shadow=None):
if random.random() < weight:
return shadow # n4n
return prod # OpenRouterRun this for 24 hours. Compare error rates, p95 latency, and token cost per request. n4n’s per-token usage metering lets you attribute spend precisely via the usage object. Verification: dashboards show n4n error rate within 0.1% of OpenRouter and latency p95 not worse by more than 20ms. If metrics hold, increase weight to 50%, then 100%.
Step 7: Final cutover and decommission
Once canary passes, change the default base URL in your deployment config. Remove the OpenRouter client code only after a soak period of one billing cycle to catch delayed invoice discrepancies.
kubectl set env deployment/llm-gateway OPENAI_BASE_URL=https://api.n4n.ai/v1
kubectl set env deployment/llm-gateway OPENAI_API_KEY=$N4N_API_KEYVerification: production traffic flows exclusively through n4n. Confirm via access logs that zero requests hit openrouter.ai. Review the first post-cutover invoice for token counts matching your metering expectations.
Caveats engineers miss
Model aliases drift. OpenRouter’s openai/gpt-4o-2024-05-13 may be a dated snapshot; n4n might alias to the latest. Lock versions explicitly if reproducibility matters.
Streaming differs subtly. Test SSE parsing: n4n forwards provider stream chunks, but finish_reason timing can vary. Replay a streaming corpus with stream=True and assert your client handles partial JSON without crashing.
Rate limit headers. OpenRouter returns X-RateLimit-Remaining; n4n uses standard Retry-After on 429. Update your backoff logic accordingly or you’ll throttle yourself unnecessarily.
When you migrate from OpenRouter to n4n testing, the work is not the API swap—it’s the evidence that nothing regressed. The steps above produce that evidence.