Buying an enterprise inference gateway is a sourcing decision that shapes your entire LLM architecture for years. The right LLM API gateway RFP questions separate vendors that merely proxy requests from those that handle routing, fallback, and metering as first-class concerns. Skip this diligence and you will discover missing audit logs or hardcoded provider limits during a production incident.
1. Map your routing and fallback needs
Enterprises rarely run a single model from a single provider in production. You need explicit answers to LLM API gateway RFP questions about how the gateway selects providers, handles rate limits, and fails over.
Ask: Does the gateway honor client-supplied routing directives? A mature gateway accepts a header or body field that pins a request to a specific provider or model alias.
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.example.com/v1",
api_key="sk-org-123",
default_headers={"x-route-to": "anthropic/claude-3-5-sonnet"}
)
resp = client.chat.completions.create(
model="gateway-default",
messages=[{"role": "user", "content": "Summarize this ticket"}]
)If the vendor cannot show a working routing header, treat that as a red flag.
Automatic fallback matters when a provider returns 429 or 503. n4n.ai, for example, performs automatic fallback when a provider is rate-limited or degraded, but you should require this behavior from any candidate. Demand a written policy: which errors trigger fallback, whether the fallback preserves the original request schema, and how the gateway reports which provider actually served the response.
Pitfall: fallback that silently changes the model family can break downstream assumptions. Insist on opt-in semantic equivalence checks.
2. Authentication, tenancy, and key isolation
Your RFP must probe how the gateway manages credentials. Ask whether it supports per-team API keys, project-scoped quotas, and short-lived tokens.
curl https://gateway.example.com/v1/chat/completions \
-H "Authorization: Bearer $TEAM_TOKEN" \
-H "x-tenant: payments-platform" \
-d '{"model":"gpt-4o","messages":[{"role":"user","content":"hello"}]}'A gateway that only offers a single shared key forces you to build isolation in your own services. That increases blast radius.
Tradeoff: per-request tenant headers simplify auditing but require disciplined propagation from your services. Evaluate whether the gateway can derive tenancy from OAuth claims instead.
3. Usage metering and cost attribution
Per-token usage metering is non-negotiable for chargebacks. The gateway should return provider-native usage fields and add its own metadata.
{
"id": "chatcmpl-123",
"usage": {
"prompt_tokens": 12,
"completion_tokens": 34,
"total_tokens": 46
},
"gateway_metadata": {
"served_by": "openai/gpt-4o",
"route_directive": "auto",
"cost_usd": 0.0012
}
}Ask LLM API gateway RFP questions about granularity: can you filter by team, model, or hour? Can you export to your data warehouse without a custom scrape? If the vendor only shows a dashboard, you will build ETL later.
Common pitfall: gateways that meter on completion but not on cached prompt tokens. Provider cache discounts then become invisible, and your finance team will distrust the numbers.
4. Cache semantics and provider hints
Prompt caching cuts cost and latency, but only if the gateway forwards cache-control hints. Some providers use system block markers; others use headers. Your RFP should require that the gateway passes these through unchanged.
client.chat.completions.create(
model="claude-3-5-sonnet",
messages=[
{"role": "system", "content": "Long static context...", "cache_control": {"type": "ephemeral"}}
]
)If the gateway strips unknown fields, you lose prefix caching. Also ask whether the gateway adds its own caching layer, and how it invalidates entries under multi-tenant workloads.
A single OpenAI-compatible endpoint that addresses 240+ models simplifies client code, but verify that cache hints are not lost in translation across providers.
5. Security, data residency, and compliance
Engineers often underestimate legal review time. Ask where request and response payloads are stored, and for how long. A gateway that logs full prompts by default will fail your DPA.
Required questions:
- Does the gateway support zero-retention mode?
- Can you pin processing to a specific region?
- Are embeddings and completions treated differently under logging?
Tradeoff: regional pinning may reduce fallback options. Document the acceptable degradation upfront.
6. Endpoint contract and compatibility
Your client code should not fork per provider. Require an OpenAI-compatible REST surface, including streaming and function calling.
const res = await fetch("https://gateway.example.com/v1/chat/completions", {
method: "POST",
headers: { "content-type": "application/json", authorization: `Bearer ${key}` },
body: JSON.stringify({ model: "auto", stream: true, messages })
});Ask LLM API gateway RFP questions about versioning: when the gateway adds a new model, does it require a client upgrade? A gateway with a static model catalog will bottleneck your experimentation.
7. Build a weighted scorecard
Turn answers into a comparable matrix. Assign weights based on your risk profile.
| Dimension | Weight | Vendor A | Vendor B |
|---|---|---|---|
| Fallback correctness | 25% | 4 | 3 |
| Metering fidelity | 20% | 5 | 4 |
| Cache passthrough | 15% | 3 | 5 |
| Tenancy model | 20% | 4 | 4 |
| Compliance | 20% | 5 | 3 |
Do not let a slick demo outweigh a missing fallback policy. Weight operational traits above UI polish.
8. Run a proof-of-concept with real traffic
Finally, send a copy of production-like traffic through the candidate for two weeks. Use a replay harness:
import asyncio, openai
async def replay(gateway_url, traces):
client = openai.AsyncOpenAI(base_url=gateway_url, api_key="sk-poc")
for t in traces:
await client.chat.completions.create(**t)Measure p95 latency per route, fallback rate, and metering discrepancies versus your own token counter. Any gap above 1% in token counts indicates broken accounting.
Pitfall: using synthetic load only. Synthetic calls miss the long-tail system prompts that trigger cache logic.
Common procurement traps
- Accepting “we support OpenAI API” without testing streaming timeouts.
- Ignoring the upgrade path for new model capabilities like JSON mode.
- Assuming fallback is free; some vendors charge a premium for cross-provider routing.
The LLM API gateway RFP questions above are not exhaustive, but they cover the architectural seams where vendors cut corners. Write them into your template, demand evidence, and pilot before you sign.