How to diff two prompt versions and know what changed

Tracking changes to LLM prompts is messy until you treat them like code. Diffing prompt versions with standard git tools reveals exactly what text, variables, and structure shifted between iterations, and gives you a reviewable history.

Step 1: Store prompts as structured files in Git

Don’t keep prompts in a notebook, a Slack message, or a CMS. Write each prompt as a file in a repo. Use a templating format that separates static text from variables. Jinja2 is a solid choice because it parses cleanly and lets you extract variable names programmatically. Avoid storing prompts as Word docs or Notion exports; they don’t diff. If a non-engineer must edit in a CMS, export to flat files in CI before commit.

Create a directory layout that ties a prompt to a logical owner:

mkdir -p prompts/summarize
touch prompts/summarize/v1.jinja

A minimal template looks like this:

You are a {{ role }}.
Summarize the following {{ doc_type }} in {{ max_sentences }} sentences.

TEXT:
{{ input_text }}

Commit it:

git add prompts/summarize/v1.jinja
git commit -m "add baseline summarize prompt v1"

Branch per experiment if you want parallel variants:

git checkout -b prompt/summarize-tone

Step 2: Tag the baseline so you can reference it later

Tags give you stable anchors for diffing prompt versions without remembering commit hashes. They also make it easy to roll back a bad prompt in production.

git tag prompt-summarize-v1

When you ship a new variant, tag that too. This makes git diff prompt-summarize-v1 prompt-summarize-v2 trivial and reviewable in any PR.

Step 3: Edit the prompt and commit the new version

Copy the template, change it, and save as v2.jinja. Suppose you added a tone directive and renamed role to persona:

You are a {{ persona }} with a concise, technical tone.
Summarize the following {{ doc_type }} in {{ max_sentences }} sentences.

TEXT:
{{ input_text }}

Commit with a message that states intent, not just “updated prompt”:

git add prompts/summarize/v2.jinja
git commit -m "summarize v2: add tone, rename role->persona"

Step 4: Run a textual diff with Git

The fastest way to see what changed is git diff between the two files. Use --word-diff to avoid noise from line wraps, and configure .gitattributes so *.jinja is treated as text:

echo "*.jinja text" >> .gitattributes
git diff --word-diff prompts/summarize/v1.jinja prompts/summarize/v2.jinja

Output highlights inserted and deleted spans. You’ll immediately see the added “with a concise, technical tone” and the variable rename. If your prompts live in JSON or YAML, pretty-print them before diffing so the diff is structural, not whitespace noise:

jq . prompts/summarize/v1.json > /tmp/v1.pp.json
jq . prompts/summarize/v2.json > /tmp/v2.pp.json
git diff --no-index /tmp/v1.pp.json /tmp/v2.pp.json

Line-based diffs lie when a single long line changes by one word. Word diff is the correct default for prose-heavy prompts.

Step 5: Normalize before diffing to cut false positives

Prompts edited in IDEs often acquire trailing whitespace or reordered keys. Write a small normalization step in your pre-commit hook or CI:

import json, sys, yaml

def normalize(path):
    if path.endswith(".json"):
        return json.dumps(json.load(open(path)), indent=2, sort_keys=True)
    if path.endswith((".yml", ".yaml")):
        return yaml.safe_dump(yaml.safe_load(open(path)), sort_keys=True)
    # for jinja, strip trailing spaces per line
    return "\n".join(line.rstrip() for line in open(path))

print(normalize(sys.argv[1]))

Run both versions through this and diff the normalized output. You’ll only see semantic changes. This step turns diffing prompt versions from a whitespace archaeology project into a clean review.

Step 6: Diff variables and structure programmatically

Text diffs miss the fact that a variable was renamed or deleted. Extraction is cheap with the jinja2 meta API:

from jinja2 import Environment, meta

def vars_in(path):
    env = Environment()
    src = open(path).read()
    return set(meta.find_undeclared_variables(env.parse(src)))

v1 = vars_in("prompts/summarize/v1.jinja")
v2 = vars_in("prompts/summarize/v2.jinja")

print("Removed:", v1 - v2)
print("Added:", v2 - v1)
print("Common:", v1 & v2)

For the example above, this prints Removed: {'role'} and Added: {'persona'}. That’s the kind of change that breaks a caller even if the prose looks fine. Wire this script into a pytest check so PRs fail when a variable disappears without a corresponding migration. If you use Python f-strings instead of Jinja, parse with ast and walk JoinedStr nodes—same principle.

Step 7: Measure behavior change by running both prompts

A diff tells you what changed in text; it doesn’t tell you if the model output shifted. Run both versions through a model and compare. Use any OpenAI-compatible client. If you route through n4n.ai, per-token usage metering lets you quantify cost delta directly from the response, and automatic fallback covers rate limits during batch tests.

from openai import OpenAI
from jinja2 import Template

client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")

def run(prompt_path, **vars):
    template = open(prompt_path).read()
    rendered = Template(template).render(**vars)
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": rendered}],
        max_tokens=200
    )
    return resp.choices[0].message.content, resp.usage.total_tokens

out1, tok1 = run("prompts/summarize/v1.jinja", role="engineer", doc_type="report", max_sentences=3, input_text="Long text...")
out2, tok2 = run("prompts/summarize/v2.jinja", persona="engineer", doc_type="report", max_sentences=3, input_text="Long text...")

import difflib
similarity = difflib.SequenceMatcher(None, out1, out2).ratio()
print(f"v1 tokens: {tok1}, v2 tokens: {tok2}, output similarity: {similarity:.2f}")

Compare the outputs manually or with a similarity score. If the token count jumps 30% because you added a system preamble, that’s a cost change worth recording in the commit message. Diffing prompt versions without this step leaves you blind to regressions in latency or spend.

Step 8: Verify success

You’ve completed diffing prompt versions when all of the following hold:

1. git diff between the two tagged versions shows only the intended textual edits, with no whitespace noise.
2. The variable extraction script reports exactly the added/removed variables you expected (none silently dropped).
3. The normalization step produces no spurious diffs on unchanged content.
4. The model run script returns both outputs and token counts; the delta matches your intent (e.g., tone changed, length stable, similarity score within expected band).

Add a CI job that runs steps 5–7 on every prompt PR. A minimal GitHub Actions step:

- name: Diff prompt versions
  run: |
    python normalize.py prompts/summarize/v1.jinja > /tmp/v1
    python normalize.py prompts/summarize/v2.jinja > /tmp/v2
    diff /tmp/v1 /tmp/v2
    python var_diff.py prompts/summarize/v1.jinja prompts/summarize/v2.jinja

If the diff is empty when you expected changes, your baseline tag is wrong. If the variable script crashes, the template has a syntax error. Both are cheaper to catch in CI than in production.

Step 9: Keep a changelog per prompt

Git history is the source of truth, but a short CHANGELOG.md in the prompt directory helps reviewers who don’t want to read raw diffs. Append one line per version:

## v2
- Added technical tone directive
- Renamed `role` to `persona`
- Measured +12% tokens vs v1

This turns diffing prompt versions from a forensic exercise into a routine code review. Treat prompts as modules: versioned, tested, and diffable. The next time a stakeholder asks “what changed in the summarizer,” you answer with a commit link, not a guess.