Debugging prompt differences between local and prod models

You shipped a feature that passes every local test, then watched it produce off-toned replies in production because of prompt differences local vs prod model behavior. The gap rarely comes from your business logic; it comes from undeclared system messages, sampling parameter drift, or provider-specific tokenization that your mock never reproduced.

Step 1: Stand up a local OpenAI-compatible mock

Your app should hit one endpoint shape in dev and prod. Build a minimal mock that accepts the same /v1/chat/completions payload and returns a stubbed choice, but also echoes the received body so you can inspect what your client actually sends.

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

app = FastAPI()

@app.post("/v1/chat/completions")
async def mock_completion(req: Request):
    body = await req.json()
    # persist for later diffing
    with open("local_sent.jsonl", "a") as f:
        f.write(__import__("json").dumps(body) + "\n")
    return JSONResponse({
        "id": "mock",
        "object": "chat.completion",
        "model": body.get("model", "gpt-4o-mini"),
        "choices": [{
            "index": 0,
            "message": {"role": "assistant", "content": "OK"},
            "finish_reason": "stop"
        }],
        "usage": {"prompt_tokens": 0, "completion_tokens": 1, "total_tokens": 1}
    })

Run it with uvicorn mock:app --port 4000. Point your client’s base_url to http://localhost:4000. Verification: curl -X POST localhost:4000/v1/chat/completions -d '{"model":"x","messages":[]}' returns the stub and appends a line to local_sent.jsonl.

A mock that only returns canned text hides the request. You need the request captured to debug prompt differences local vs prod model mismatches.

Step 2: Capture real production requests

You cannot debug prompt differences local vs prod model mismatches without the actual prod payload. Log the exact request body sent to the provider, redacting PII. In a Node/Express service:

import fs from 'fs';
app.use('/v1/chat/completions', (req, res, next) => {
  const { messages, temperature, max_tokens, model, seed } = req.body;
  const redacted = messages.map(m => ({
    role: m.role,
    content: m.content.slice(0, 50) // truncate for log safety
  }));
  fs.appendFileSync('prod_calls.jsonl',
    JSON.stringify({ ts: Date.now(), model, redacted, temperature, max_tokens, seed }) + '\n');
  next();
});

Keep these captures. They are your ground truth for what the app actually sent under real load. If you use a gateway, ensure it forwards provider cache-control hints so your redaction layer does not strip fields that affect caching.

Verification: after a day of traffic, wc -l prod_calls.jsonl shows entries, and a sample line contains the full message array with parameters.

Step 3: Replay logical inputs through your local client

Write a test that drives your application’s prompt-builder with the same input that generated a prod capture, but routes to the local mock. Assert the emitted request body equals the captured one.

import json, contextlib

def mock_endpoint(url):
    # monkeypatch your client base_url
    return contextlib.nullcontext()

def test_prompt_parity():
    inp = json.load(open("fixtures/user_123.json"))
    with mock_endpoint("http://localhost:4000"):
        app.handle(inp)  # your code sends to local mock
    sent = json.loads(open("local_sent.jsonl").readlines()[-1])
    prod = json.loads(open("prod_calls.jsonl").readline())
    assert sent["messages"] == prod["messages"], "message mismatch"
    assert sent.get("temperature") == prod.get("temperature"), "temp drift"
    assert sent.get("seed") == prod.get("seed"), "seed drift"

If this fails, you have found a prompt differences local vs prod model bug: maybe dev injects a debug system message, or prod sets temperature=0.2 while local uses 0. The test output will point at the exact field.

Step 4: Diff sampling parameters and message order

Most silent breaks come from parameters, not text. Extract and compare:

{
  "prod":  {"temperature": 0.7, "max_tokens": 512, "seed": 42, "model": "gpt-4o"},
  "local": {"temperature": 0,   "max_tokens": 512, "seed": null, "model": "gpt-4o-mini"}
}

A missing seed or different temperature changes output distribution. Model name drift is its own trap: if local uses a mini model and prod uses full, prompt differences local vs prod model quality will appear even with identical text. Fix your client config to read from one shared constants file.

Verification: a jsondiff run on the two dicts reports {}.

Also check message order. Some providers reject consecutive user messages; your local mock may not. Reorder logic that runs only in prod will surface here.

Step 5: Emulate provider tokenization and context limits

Local mock accepts any length, but prod models truncate. Use tiktoken to count tokens exactly as the OpenAI models do:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
def count(msg_list):
    return sum(len(enc.encode(m["content"])) for m in msg_list)

prompt_tokens = count(prod["messages"])
assert prompt_tokens <= 8000, f"prod would truncate: {prompt_tokens}"

If local builds a 9k-token prompt but prod model caps at 8k, you will see max_tokens errors only in prod. Gate this in tests.

If you route through a gateway such as n4n.ai, the same OpenAI-compatible endpoint can address 240+ models with automatic fallback when a provider is degraded, which removes one class of prompt differences local vs prod model mismatches caused by endpoint switching. Your local tests hit the gateway’s sandbox model; prod hits the real one with identical request shape.

Verification: test fails when count(messages) > 8000 and passes when within limit.

Step 6: Lock parity in CI

Add a job that replays a fixed set of captures and diffs them. A minimal bash gate:

pytest tests/prompt_parity.py || exit 1
python check_tokens.py prod_calls.jsonl || exit 1

In GitHub Actions:

jobs:
  parity:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install fastapi uvicorn tiktoken pytest
      - run: uvicorn mock:app --port 4000 &
      - run: bash ci_parity.sh

Run it on every PR. This catches environment-conditional prompt building before merge.

Verification: a deliberately injected dev-only system message turns the build red; removing it returns green.

Step 7: Watch for post-deploy drift

Even with CI, someone may change prod config via env vars. Ship a weekly replay of recent prod captures against the current local build. If the diff grows, alert.

for line in open("recent_prod.jsonl"):
    prod = json.loads(line)
    local = build_from_same_input(prod["input_ref"])
    if diff(prod["messages"], local["messages"]):
        send_alert(f"prompt differences local vs prod model drift: {prod['ts']}")

This keeps prompt differences local vs prod model gaps from silently returning after a config hotfix.

Step 8: Verify end-to-end output quality

Parity in request shape is necessary but not sufficient. Run a small golden set through both local mock (with a recorded prod response baked in) and prod, then score with a deterministic metric like exact-match on structured fields.

def extract_json(resp):
    return json.loads(resp["choices"][0]["message"]["content"])

assert extract_json(local_resp) == extract_json(prod_resp)

For free-form text, compute cosine similarity of embeddings to catch tone drift. When this passes and the earlier diffs are clean, you have closed the loop on prompt differences local vs prod model debugging.

Step 9: Document the contract

Write a short PROMPT_CONTRACT.md listing required parameters, system message template, and max token budget. New engineers will otherwise reintroduce the drift you just fixed. Treat the local mock and prod capture suite as the executable version of that doc.

Verification: a teammate spins up the mock, runs the parity test, and gets a green check without your help. That is the signal you are done.