An LLM API gateway is a proxy layer that presents a single API surface to multiple model providers behind it. How gateways choose which models to add is a curation decision driven by observed request patterns, provider reliability, and operational cost—not a naive scrape of every published checkpoint. For engineers building on these gateways, that decision shapes which model strings will resolve at 3am when a provider degrades.
What “adding a model” actually means
Adding a model is not a configuration toggle. It is the act of registering a routing entry that maps a logical model ID to a provider endpoint, then maintaining the translation code that reconciles request and response schemas.
The gateway must handle authentication to the provider, stream normalization, token counting, and error mapping. If the model supports function calling, the gateway must translate your tool schema into the provider’s dialect.
{
"model": "anthropic/claude-3-5-sonnet",
"messages": [{"role": "user", "content": "Explain Raft."}],
"max_tokens": 512,
"tools": [{"type": "function", "function": {"name": "get_logs"}}]
}That request looks uniform. Behind it, the gateway converts tools to Anthropic’s tools format or OpenAI’s functions legacy, depending on the target.
Core criteria gateways use to decide
Demand signals from existing traffic
Gateways are pull-based systems. They watch which model identifiers clients request. A steady stream of misses for google/gemini-1.5-pro tells the curator that users want it.
class CurationMonitor:
def record_miss(self, model_id: str):
self.miss_counts[model_id] += 1
if self.miss_counts[model_id] > self.threshold:
self.queue_for_review(model_id)This is how gateways choose which models to add when organic demand crosses an internal bar.
Provider API stability and SLAs
A gateway stakes its own uptime on the providers it proxies. If a provider’s API returns 500s intermittently or lacks documented rate limits, adding its models transfers that risk to your calls.
How gateways choose which models to add includes a review of the provider’s status page history and contract. A provider with no SLA but a great open-weight model may still be added if self-hosted.
Licensing and commercial terms
Some models prohibit indirect access or require revenue share. The gateway’s legal layer blocks those unless cleared. Permissive licenses (MIT, Apache 2.0) for weights are straightforward; hosted proprietary APIs need acceptable-use alignment.
Capability coverage and differentiation
A gateway curates for capability spread: a fast cheap classifier, a 200k-context reasoner, a vision model. Adding three near-identical 7B Instruct fine-tunes fragments the test matrix.
Cost and price volatility
Provider prices change. A model that costs a predictable amount per million tokens may shift abruptly. Gateways track per-token economics and may avoid models with unstable pricing that would make metering misleading.
Why catalog breadth is a double-edged sword
A large catalog means one integration point. But each model is a liability: credential rotation, schema drift, price monitoring.
How gateways choose which models to add weighs marginal utility against ongoing ops cost. A gateway that exposes one OpenAI-compatible endpoint addressing 240+ models (as n4n.ai does) only sustains that because redundant entries are pruned.
Broad catalogs also complicate caching. Provider cache-control hints must be forwarded correctly; a model that ignores cache_control breaks prefix caching and inflates bills.
A concrete example: routing around a degraded provider
Your service sends a chat completion to openai/gpt-4o. The provider starts returning 429s due to a regional outage. A curated gateway has fallback entries:
{
"model": "openai/gpt-4o",
"route": {
"fallback": ["anthropic/claude-3-5-sonnet", "meta/llama-3-70b-instruct"]
}
}The gateway honors your directive, forwards provider cache-control hints, and meters per-token usage on the fallback. Your code sees a successful response, maybe with a different model field echoed back.
This works only because the gateway previously decided those fallbacks were worth the maintenance. That decision is the output of how gateways choose which models to add.
Common misconceptions about model curation
“More models is always better”
A gateway with 500 model strings often has 100 aliases to the same underlying weights. Overlap reduces test coverage and increases the chance a random pick fails.
“Gateways add every new release instantly”
When a lab publishes a new checkpoint, gateways wait for stable API docs, evaluate demand, and clear licensing. How gateways choose which models to add includes a deliberate lag.
“Curation is only about availability”
Availability is table stakes. The model must respect temperature, stream correctly, and return usage metadata. A model that breaks JSON mode is a liability even if it’s online.
“Fallbacks are automatic magic”
Fallbacks require the gateway to have added and validated the secondary models. If the curator skipped a model, your fallback directive silently fails.
How to evaluate a gateway’s catalog strategy
When comparing gateways, don’t count model strings. Ask:
- What is the criteria for adding a model?
- How are deprecated models removed?
- Do they forward provider cache-control?
- Is per-token metering accurate across fallbacks?
A gateway that answers these with a documented curation pipeline is safer than one advertising “every model ever”.
Internal curation pipeline sketch
If you run your own gateway, model the decision:
def should_add(model_candidate):
if not legal.clear(model_candidate.license):
return False
if provider_sla(model_candidate.provider) < 0.99:
return False
if demand_rate(model_candidate.id) < MIN_REQUESTS:
return False
if overlaps_existing(model_candidate):
return False
return TrueThis mirrors how gateways choose which models to add in production.
Impact on your application code
Write calls that tolerate substitution. Pass routing hints, not hard dependencies:
const res = await client.chat.completions.create({
model: "anthropic/claude-3-5-sonnet",
messages,
route: { fallback: ["openai/gpt-4o-mini"] }
});
console.log(res.model); // may differ from requestTreat the model field as a preference. The gateway’s catalog will shift; your code should absorb it.
Monitoring catalog changes
Subscribe to the gateway’s catalog diff:
curl -s https://gateway.internal/v1/catalog/changes?since=2024-01-01 \
| jq '.added[] | .model'This exposes how gateways choose which models to add in your own stack—as observable data rather than mystery.
Bottom line
Curation is engineering, not a popularity contest. How gateways choose which models to add determines whether your single API call reaches a reliable backend or triggers a retry storm. Design for that reality.