A checklist for migrating LLM providers safely

Switching the backend that serves your generative features is not a drop-in replacement. This LLM provider migration checklist distills the steps we use to move workloads between vendors without breaking production traffic, losing eval quality, or blowing up latency. Each item below is a concrete engineering task, not a slide.

1. Audit your current prompt and response contracts

Before writing any migration code, capture exactly what your existing provider receives and returns. Pull the system prompts, few-shot examples, stop sequences, temperature, top_p, and any response-format constraints (JSON mode, function calls) from your live config or logs.

Treat this as a forensic snapshot. If you use OpenAI’s response_format: { type: "json_object" }, note that the new provider may express the same intent via a different field or a grammar constraint.

{
  "model": "gpt-4o-mini",
  "messages": [{"role": "system", "content": "You output JSON."}],
  "temperature": 0.2,
  "response_format": {"type": "json_object"}
}

Store these artifacts in version control. A missing stop token can silently corrupt downstream parsing after cutover.

2. Build an abstraction layer over the chat API

Never call a provider SDK directly from business logic if you plan to migrate. Define a narrow interface—complete(messages, params) -> Completion—and implement one adapter per vendor. This isolates diffs in auth, base URL, and parameter naming.

from abc import ABC, abstractmethod
from dataclasses import dataclass

@dataclass
class Completion:
    text: str
    model: str
    usage: dict

class LLMClient(ABC):
    @abstractmethod
    def complete(self, messages: list, **params) -> Completion: ...

class OpenAIAdapter(LLMClient):
    def complete(self, messages, **params):
        # call SDK, normalize response
        ...

The abstraction pays off the moment you need to A/B two providers or route by latency.

3. Map model capabilities and context windows

Not all “equivalent” models are equivalent. Build a lookup table that records max context, max output tokens, supported modalities, and known quirks. A 200K context model from vendor A may only accept 32K on vendor B for the same tier.

{
  "anthropic.claude-3-5-sonnet": {"ctx": 200000, "out": 8192, "json_mode": false},
  "openai.gpt-4o": {"ctx": 128000, "out": 16384, "json_mode": true}
}

This mapping should drive validation in your abstraction layer: reject requests that exceed the target model’s limits before they hit the wire.

4. Normalize provider-specific parameters

Vendors diverge on parameter names and semantics. OpenAI uses max_tokens for output; Anthropic uses max_tokens too but historically required it; some gateways split max_completion_tokens. Seed parameters, logit_bias, and stop arrays vary in type.

Define a canonical request schema and translate at the adapter boundary.

interface CanonicalRequest {
  messages: {role: string; content: string}[];
  maxOutputTokens: number;
  temperature: number;
  stop?: string[];
}
// Inside adapter: { max_tokens: req.maxOutputTokens, stop: req.stop ?? undefined }

Ignoring this step produces silent truncations or 400 errors under load.

5. Implement explicit fallback and routing logic

Degraded providers are a matter of when, not if. Your migration must define what happens if the primary target returns 429 or 503. If you sit behind a gateway such as n4n.ai that provides automatic fallback when a provider is degraded, you still must test how your code reacts to the fallback model’s output shape and latency.

Client-side routing directives can also be forwarded. For example, force a specific provider via header:

curl https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "x-routing: provider=anthropic" \
  -d '{"model":"claude-3-5-sonnet","messages":[{"role":"user","content":"hi"}]}'

Log which model actually served the request; metering without attribution is useless.

6. Validate output schemas and error envelopes

Provider B may return the same JSON but wrap errors differently. Write contract tests that assert your parser handles both error.message and error.error.message paths. If you depend on structured output, run a schema validator on a sample of real responses.

from pydantic import BaseModel, ValidationError

class Ticket(BaseModel):
    id: str
    priority: int

def parse(completion_text: str) -> Ticket:
    try:
        return Ticket.model_validate_json(completion_text)
    except ValidationError as e:
        raise RuntimeError(f"Schema drift: {e}")

Do this in CI against recorded fixtures from the new provider before flagging the migration live.

7. Run shadow traffic and diff evaluations

Stand up the new provider in parallel and mirror a percentage of production requests. Compare outputs on a fixed eval set: exact-match for code, embedding cosine for long-form, or LLM-as-judge for subjective tasks.

for case in eval_set:
    old = old_client.complete(case.messages)
    new = new_client.complete(case.messages)
    score = judge(old.text, new.text)
    if score < 0.9:
        flagged.append(case.id)

A safe LLM provider migration checklist includes a quantitative bar—e.g., <2% regression on critical intents—not just “it runs.”

8. Meter usage and cost per token before cutover

Per-token metering is non-negotiable. Capture usage.prompt_tokens and usage.completion_tokens from each response and tag by model and route. Discrepancies in tokenization (cl100k vs Claude tokenizer) change cost by 20–40% on the same text.

{"model":"gpt-4o","usage":{"prompt_tokens":182,"completion_tokens":44}}

Aggregate daily. If the new provider costs 3x on your dominant prompt shape, that is a finding before traffic shifts, not after.

9. Stress test rate limits and latency SLAs

Provider docs lie about real throughput. Run a sustained load test at 2x your peak QPS against the new endpoint and observe 429 behavior, backoff headers, and p99 latency.

for i in $(seq 1 1000); do
  curl -s -o /dev/null -w "%{http_code} %{time_total}\n" \
    https://api.target.com/v1/chat/completions -d @payload.json &
done

Your fallback logic from step 5 must engage cleanly under this pressure. If it cascades, you have a retry storm, not a migration.

10. Pin versions and document rollback

Always pin model versions (gpt-4o-2024-08-06, claude-3-5-sonnet-20241022). Unpinned aliases shift underneath you. Keep a one-command rollback: flip a config key back to the old adapter and deploy.

llm:
  primary: anthropic.claude-3-5-sonnet-20241022
  fallback: openai.gpt-4o-2024-08-06
  rollback_to: openai.gpt-4o-mini-2024-07-18

Following this LLM provider migration checklist ends with a tested, reversible cutover—not a hope and a pager.

Summary