A pre-migration checklist for LLM model upgrades

Model deprecations hit without mercy, and a disciplined llm model migration checklist is the only thing standing between your pipeline and a 3 a.m. page. Before you flip a single environment variable, you need to know exactly what runs where, how it behaves, and what breaks when the version string changes.

1. Inventory every model reference across your stack

A migration fails when a hardcoded "gpt-4-0613" survives in a forgotten worker. Pull a full inventory of model identifiers from source, config files, database rows, and prompt templates. Treat any string that hits an inference endpoint as a liability.

Run a recursive grep with a tight regex, then verify against your gateway logs:

grep -rEi '"model"\s*:\s*"[a-z0-9\-]+"' ./services --include=*.py --include=*.ts

For dynamic routing, trace where the model name is constructed. A central ModelRegistry class beats scattered constants. If you use an OpenRouter-class gateway, confirm which aliases resolve to the deprecated version so you can catch indirect references.

2. Diff the API contract and default behaviors

Version bumps change more than the name. Temperature defaults, max token ceilings, and response schema fields shift silently. Build a diff of the request and response shapes between old and new model docs, focusing on fields your code actually reads.

{
  "old": { "model": "claude-2.1", "max_tokens": 4096, "stop": ["\n"] },
  "new": { "model": "claude-3-5-sonnet", "max_tokens": 8192, "stop_sequences": ["\n"] }
}

Note renamed parameters (stop → stop_sequences) and dropped fields. If your parser assumes a key exists, it will throw on the new version. The llm model migration checklist must include a contract test that fails on missing keys.

3. Stand up an evaluation harness before touching production

Do not trust provider claims of drop-in compatibility. Build a static eval set of 50–200 representative inputs with golden outputs or scoring functions. Run it against the old model, snapshot metrics, then run the same set against the candidate.

def eval_batch(client, model, cases):
    results = []
    for case in cases:
        resp = client.chat.completions.create(
            model=model, messages=case["messages"], temperature=0
        )
        results.append(score(case["expected"], resp.choices[0].message.content))
    return sum(results) / len(results)

baseline = eval_batch(client, "old-model", cases)
candidate = eval_batch(client, "new-model", cases)
assert candidate >= baseline - 0.02, "regression beyond tolerance"

If the candidate drops below threshold, you have data to push back on the migration or to justify prompt retuning. This step converts guesswork into a go/no-go signal.

4. Map fallback and routing directives explicitly

Providers degrade. Your client must declare what happens when the primary model is unavailable. If you sit behind an inference gateway such as n4n.ai, it will auto-fallback on rate limits, but you still need to encode routing hints and cache-control forwarding in your request layer.

await client.chat.completions.create({
  model: "provider/new-model",
  messages,
  // honor gateway cache hints
  headers: { "x-cache-control": "max-age=3600" },
  // explicit fallback order if gateway allows directive
  routing: { fallback: ["provider/old-model", "provider/alt-model"] }
});

Document which model groups share embedding spaces or fine-tune lineage. A fallback to a different family changes output distribution; your eval harness from step 3 should cover those paths too.

5. Audit prompt templates and structured output schemas

Prompts tuned for one model’s quirks often break on another. Check system prompts that mention the model name, few-shot examples using old formatting, and JSON mode schemas. Newer models may enforce stricter schema validation or refuse previously tolerated inputs.

Extract all template files and lint for version-specific tokens:

grep -rn "gpt-4\|claude-2\|text-davinci" ./prompts | wc -l

If you use function calling, verify argument names and required parameters. A model that reorders tool inputs will silently corrupt downstream calls. The llm model migration checklist should include a schema fuzz test that sends malformed but plausible tool args to both versions.

6. Reconcile token metering and cost attribution

Per-token usage metering breaks when model names change but your billing tags don’t. Ensure your usage middleware captures the exact model string returned in the response, not the one you sent. Providers sometimes rewrite aliases.

usage = resp.usage
log_metric(
    model=resp.model,  # actual returned id
    prompt_tokens=usage.prompt_tokens,
    completion_tokens=usage.completion_tokens
)

Mismatched tags make cost spikes invisible until the invoice arrives. If you route through a gateway with per-token metering, confirm its usage records align with your internal ledger within a 1% margin on a canary sample.

7. Ship a canary with mirrored traffic

Never cut over 100% of traffic. Mirror a small percentage (1–5%) to the new model while serving production from the old. Compare latency, error rates, and eval scores on live inputs.

# route 5% via header injected at proxy
curl -H "x-traffic-split: new-model=0.05" https://api.internal/completions

Watch for increased 429s or malformed streams. The llm model migration checklist is incomplete without a rollback trigger: if canary error rate exceeds baseline by 2x for five minutes, auto-revert.

8. Document rollback and deprecation deadlines

Write the rollback command in the runbook before you migrate, not after. Note the provider’s hard shutdown date for the old version and set an internal deadline a week earlier to absorb delays.

Keep a table of who owns each service using the model and their sign-off:

A clear llm model migration checklist turns a chaotic upgrade into a sequenced engineering task. Inventory first, prove behavior with evals, route with intent, canary with metrics, and keep rollback one command away.