Version pinning prompts to a specific model snapshot

Model behavior shifts silently when providers retrain or swap weights. Pinning prompts to model snapshots eliminates that variance by locking a prompt version to an exact model build like gpt-4-0613 or claude-2.0-20230905, so your test suite validates the same system you ship.

Why snapshot pinning is non-negotiable for production

A prompt that scores 95% accuracy on Tuesday can drop to 80% on Thursday because the underlying model changed. Most teams blame their prompt. The real culprit is an unpinned model reference like gpt-4 that resolves to a moving target.

If you ship LLM features to paying users, you need three things: a known prompt, a known model snapshot, and a record that ties them together. Without pinning prompts to model snapshots, you cannot reproduce a bug, roll back a regression, or prove compliance with an eval you ran last month.

Snapshot identifiers are provider-specific but universally necessary. OpenAI publishes dated snapshots (gpt-3.5-turbo-0125). Anthropic uses a anthropic-version header plus model ID. Open-weight models have HuggingFace commit hashes or tag names (mistralai/Mistral-7B-Instruct-v0.2). Treat these strings as part of your source code.

Step 1: Resolve the exact snapshot identifier

Do not guess the snapshot name. Query the provider’s model list or read its versioning docs. For OpenAI-compatible endpoints, hit the /v1/models route.

curl -s https://api.openai.com/v1/models \
  -H "Authorization: Bearer $OPENAI_API_KEY" | jq '.data[] | .id' | grep '0613'

If you use an OpenRouter-class gateway, the same call works against its base URL. The response includes exact IDs you can pin.

from openai import OpenAI

client = OpenAI()  # uses OPENAI_API_KEY
models = client.models.list()
snapshots = [m.id for m in models if "gpt-4" in m.id and "0613" in m.id]
print(snapshots)  # ['gpt-4-0613', 'gpt-4-32k-0613']

Pick the snapshot that matches your latency, cost, and quality needs. Write it down exactly as returned—case and date matter.

Step 2: Store prompt and snapshot in a Git-tracked manifest

Create a directory structure that keeps prompts and their pins together:

prompts/
  summarize/
    manifest.yaml
    prompt.md

The manifest records the snapshot, not just the family. Use a strict schema.

# prompts/summarize/manifest.yaml
name: summarize
model_snapshot: gpt-4-0613
temperature: 0.2
max_tokens: 256
version: 1
prompt_file: prompt.md

prompt.md holds the actual template:

# Summarize the following support thread in one sentence.

{{thread}}

Commit these files. They are now version-controlled artifacts. Anyone who checks out this commit gets the exact prompt and the exact model snapshot.

Step 3: Build an inference call that respects the pin

Write a thin loader that reads the manifest and calls the model with the pinned ID. Do not hardcode the snapshot in your application code—read it from the manifest so Git remains the source of truth.

import yaml
from openai import OpenAI

def load_manifest(path: str) -> dict:
    with open(path) as f:
        return yaml.safe_load(f)

def run_prompt(manifest_path: str, thread: str) -> str:
    m = load_manifest(manifest_path)
    with open(m["prompt_file"]) as f:
        template = f.read()
    messages = [{"role": "user", "content": template.replace("{{thread}}", thread)}]
    client = OpenAI()
    resp = client.chat.completions.create(
        model=m["model_snapshot"],  # pin enforced here
        messages=messages,
        temperature=m["temperature"],
        max_tokens=m["max_tokens"],
    )
    return resp.choices[0].message.content

# If you route through n4n.ai, its OpenAI-compatible endpoint honors client
# routing directives, so the snapshot string in `model` is passed through
# verbatim and per-token usage is metered against that exact build.

The critical line is model=m["model_snapshot"]. If the manifest says gpt-4-0613, the API call requests that build. No alias resolution happens in your code.

Step 4: Commit, tag, and document the pair

Treat the prompt+snapshot pair as a release. After verifying locally, commit and tag:

git add prompts/summarize
git commit -m "pin summarize prompt to gpt-4-0613"
git tag -a prompt-summarize-v1 -m "Stable summarize behavior on gpt-4-0613"
git push origin main --tags

In your README or internal wiki, note the tag and the eval score it achieved. When a bug report mentions “summarize broke”, you check out prompt-summarize-v1 and reproduce against the same snapshot.

Step 5: Write a verification test

A pin is only useful if you assert it is honored. Add a test that calls the API and checks the returned model field matches the manifest. Most OpenAI-compatible responses echo the model ID.

import os
import yaml
from openai import OpenAI

def test_snapshot_pinned():
    m = yaml.safe_load(open("prompts/summarize/manifest.yaml"))
    client = OpenAI()
    resp = client.chat.completions.create(
        model=m["model_snapshot"],
        messages=[{"role": "user", "content": "ping"}],
        max_tokens=1,
    )
    assert resp.model == m["model_snapshot"], (
        f"Expected {m['model_snapshot']}, got {resp.model}"
    )

Run it in CI. If a provider deprecates the snapshot or your gateway silently rewrites the ID, the test fails loudly. You can extend the test with a golden-output check: store a known response for a fixed input and assert similarity above a threshold.

Step 6: Plan upgrades without breaking the pin

Pinning does not mean freezing forever. Models get cheaper, faster, or safer. The workflow is:

1. Copy the manifest to manifest.next.yaml.
2. Change model_snapshot to the new build (e.g., gpt-4-0613 → gpt-4-1106).
3. Run your eval suite against both snapshots in shadow mode.
4. If the new snapshot passes, commit, tag prompt-summarize-v2, and flip production to it.

Keep the old tag alive. Rollback is git checkout prompt-summarize-v1 and deploy.

For open-weight models, pin the HuggingFace commit hash in the manifest:

model_snapshot: mistralai/Mistral-7B-Instruct-v0.2@a1b2c3d

This survives tag reassignments.

Verifying success

After deployment, confirm the pin works by inspecting the response metadata. For OpenAI-compatible calls, log resp.model and resp.usage on every request. If you see gpt-4-0613 in the model field and your metering shows tokens attributed to that ID, the pin is active.

A quick manual check:

resp = client.chat.completions.create(model="gpt-4-0613", messages=[{"role":"user","content":"hi"}])
print(resp.model)  # must print gpt-4-0613

If the printed model differs, your client or gateway is rewriting the request—fix the manifest or the routing config before trusting any eval. With the snapshot locked and the manifest in Git, you have a reproducible LLM component that behaves identically across machines, days, and incidents.