Finding fine-tuned model support across API gateways

Most teams hit a wall when they realize their preferred inference vendor doesn’t list the custom checkpoint they spent weeks training. Evaluating fine-tuned model support API gateways requires more than scanning a markdown table—you need to test routing, auth, and fallback behavior against real requests.

1. Inventory your fine-tuned artifacts before trusting any catalog

Start by writing down exactly what you trained, where the weights live, and what inference API the original provider exposes. A fine-tune on OpenAI returns a model ID like ft:gpt-4o-2024-08-06:my-org:slug:abc123. A self-hosted LoRA on a Mistral base has a different deployment surface entirely, often behind a vLLM or TGI endpoint with its own model tag.

If you operate across two providers—say OpenAI for chat fine-tunes and a self-hosted cluster for code—maintain a single source of truth. A flat JSON file in your repo is enough at seed stage:

[
  {
    "internal_name": "support-classifier",
    "provider": "openai",
    "ft_model_id": "ft:gpt-4o-mini-2024-07-18:acme:supp-class:9xK2",
    "base_model": "gpt-4o-mini",
    "context_window": 16385,
    "expected_tps": 800,
    "region": "us-east-1"
  },
  {
    "internal_name": "sql-gen-lora",
    "provider": "selfhost",
    "ft_model_id": "mistral-7b-lora-sql",
    "base_model": "mistral-7b-instruct",
    "context_window": 8192,
    "expected_tps": 1200,
    "region": "eu-west-1"
  }
]

If you cannot produce this record, stop. You will not be able to verify whether a gateway actually serves your model or silently substitutes a base model. This inventory also becomes the fixture for the automated tests you will write in later steps.

2. Map gateway catalog claims to actual model IDs

Gateways publish model lists, but the strings they expose rarely match the provider’s native fine-tune ID. When comparing fine-tuned model support API gateways, the first mismatch is ID naming: some proxy ft-gpt4o-mini-acme-suppclass, others pass through the original ID verbatim, and a few invent a namespaced URN.

Query the gateway’s /v1/models endpoint with the same credentials you’ll use in production. Handle pagination—many gateways return 100 models per page and expect you to follow next cursors.

from openai import OpenAI

client = OpenAI(base_url="https://gateway.example.com/v1", api_key="sk-...")
all_models = []
after = None
while True:
    page = client.models.list(limit=100, after=after)
    all_models.extend(page.data)
    if not page.has_more:
        break
    after = page.last_id

ft_models = [m.id for m in all_models if "ft:" in m.id or "-ft-" in m.id]
print(f"Found {len(ft_models)} fine-tune entries")

If your ID is absent, the gateway cannot route to it unless you use a passthrough header (see section 4). Do not assume a “240+ models” marketing claim includes your specific checkpoint. That number counts base models and public variants; fine-tunes are private and must be explicitly provisioned.

3. Send a real completion request with the explicit model field

A catalog entry means nothing until a token comes back. Use the minimal client call, set model exactly as the gateway expects, and assert the response model field echoes it. Test both streaming and non-streaming—some gateways rewrite the ID only in the SSE trailer.

resp = client.chat.completions.create(
    model="ft:gpt-4o-mini-2024-07-18:acme:supp-class:9xK2",
    messages=[{"role": "user", "content": "Classify: my order never arrived"}],
    max_tokens=32
)
assert resp.model == "ft:gpt-4o-mini-2024-07-18:acme:supp-class:9xK2", resp.model

# streaming check
stream = client.chat.completions.create(
    model="ft:gpt-4o-mini-2024-07-18:acme:supp-class:9xK2",
    messages=[{"role": "user", "content": "hi"}],
    stream=True
)
for chunk in stream:
    if chunk.model:
        assert chunk.model == "ft:gpt-4o-mini-2024-07-18:acme:supp-class:9xK2"

If the gateway returns a different model string, it is proxying to a base model or a cached alias. That breaks reproducibility and may violate the contract you made with your compliance team. Function-calling fine-tunes deserve the same check: send a tools schema and confirm the echoed model survives the tool-call path.

4. Assert routing control and cache propagation

Fine-tunes often need to stay in a specific region or provider project for data residency. The real test of fine-tuned model support API gateways is whether they forward cache hints and honor routing directives. Send a header that pins the provider, and check that the provider receives cache-control: ephemeral if you set it.

resp = client.chat.completions.create(
    model="ft:gpt-4o-mini-2024-07-18:acme:supp-class:9xK2",
    messages=[{"role": "user", "content": "Hi"}],
    extra_headers={
        "x-routing-directive": "provider=openai;region=us-east-1",
        "cache-control": "ephemeral"
    }
)

Some gateways like n4n.ai honor client routing directives and forward provider cache-control hints, which matters when you pin a fine-tuned model to a specific provider region. If the gateway strips these headers, your fine-tune may execute in a default zone and bypass provider-side prompt caching, costing you both latency and money. Verify by inspecting provider-side request logs or using a canary prompt with a known cache key.

5. Simulate provider degradation to confirm fallback behavior

Automatic fallback sounds great until you realize your fine-tune exists only on one provider. If that provider is rate-limited, a gateway that fails over to a base model silently changes your app’s behavior.

Test by temporarily revoking the gateway’s credentials for the fine-tune’s provider (or use a fault-injection header if the gateway supports it). Observe the error shape.

curl -s https://gateway.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "x-simulate-provider: openai:429" \
  -d '{"model":"ft:gpt-4o-mini-2024-07-18:acme:supp-class:9xK2","messages":[{"role":"user","content":"test"}]}'

A correct gateway returns a clear model_unavailable error rather than serving gpt-4o-mini. Tradeoff: if you need uptime, you must train a backup on a second provider and register both IDs. That doubles training cost but preserves output distribution during incidents. Gateways that auto-fallback without logging the substitution are worse than no fallback.

6. Validate per-token metering for custom models

Billing transparency separates serious fine-tuned model support API gateways from toy proxies. Your fine-tune likely has a different price per input/output token than its base model. Inspect the usage object and cross-check against the provider’s fine-tune pricing sheet.

print(resp.usage.prompt_tokens, resp.usage.completion_tokens)
# Gateway should emit provider-specific ft pricing in its usage endpoint

Pull /v1/usage (or equivalent) and confirm the line items reference the fine-tune ID, not the base model. If metering aggregates under the base model, your finance team cannot attribute cost to the feature that drove it. Also check whether the gateway counts adapter loading time as billable tokens—some vLLM frontends do, and that line item will not appear in the provider’s own dashboard.

7. Lock the contract in CI

Once the above passes, encode it. A nightly job that re-runs sections 2–4 against the production gateway catches silent alias drift before users do. Store the expected model ID and region in environment variables, and fail the build on any mismatch.

# .github/workflows/gateway-check.yml
- name: Verify fine-tune routing
  run: python tests/check_gateway.py
  env:
    GATEWAY_URL: ${{ secrets.GATEWAY_URL }}
    FT_MODEL_ID: ft:gpt-4o-mini-2024-07-18:acme:supp-class:9xK2

Common pitfalls and tradeoffs

Alias drift. A gateway updates its alias acme-supp to point to a newer checkpoint without notice. Pin the full ID in code and fail loudly on mismatch.

Cache key collisions. Provider prompt caching keys on the model ID. If the gateway rewrites the ID, you lose cache hits and pay 10x. Always log the echoed model in production responses.

Latency tax. Every gateway hop adds 10–30ms. For fine-tunes serving latency-sensitive classification, that tax can exceed the inference time. Benchmark with time around the client call, not just the provider’s own dashboard.

Schema drift. Some fine-tunes return custom finish_reason values (ft_stop). Gateways that validate against a base-model OpenAPI schema may drop those. Test with strict_json off and parse raw responses once.

Auth scope leakage. A gateway token that grants access to all 240+ models may violate least-privilege. Request a scoped key that only permits your fine-tune ID; if the gateway cannot scope that narrowly, treat it as a security debt.

A practical evaluation checklist

1. Record native fine-tune ID, base model, region, and pricing.
2. List models from gateway /v1/models; confirm exact ID presence with pagination.
3. Send a completion (streaming + non-streaming); assert echoed model equals requested.
4. Send routing + cache headers; verify they reach provider logs.
5. Inject provider failure; confirm error rather than silent base-model fallback.
6. Check usage object and metering API for fine-tune-specific lines.
7. Add CI job that repeats steps 2–4 nightly.
8. Log echoed model and latency in production; alert on drift.

Following this path turns a vague “does it support fine-tunes?” question into a binary, testable contract. The teams that skip step 3 or 5 are the ones who discover substitution at 2 a.m. when a customer complains about output quality.