Prompts are the closest thing we have to source code for LLM behavior, yet most teams still paste them into dashboards and hope for the best. A git branching workflow for prompt experiments treats every meaningful change as a branch, giving you diffs, code review, and rollback without leaving your existing tooling. This guide lays out a concrete pattern you can adopt this afternoon.
Why prompts need version control
LLM outputs are sensitive to phrasing, ordering, and even trailing whitespace. Changing “You are a helpful assistant” to “You are a concise assistant” can cut average completion tokens by 30% and alter task success in ways that are invisible until you measure. Unlike traditional code, the runtime (the model) is externally updated and partially opaque, so you need a clean history to bisect regressions.
Prompts also interact with few-shot examples, tool schemas, and decoding parameters. A git history lets you reproduce the exact input sent to the model months later, which is essential for debugging compliance incidents or sudden quality drops. If you cannot answer “what did we change in the summarizer prompt last Tuesday?”, you are flying blind.
The branch-per-prompt pattern
The core idea: one branch per discrete prompt experiment. Not per feature, not per sprint—per hypothesis about wording, structure, or persona.
Steps
- Start from
main, which holds prompts currently in production. - Create a branch:
git checkout -b prompt/summarizer-compress-20240521. - Edit the prompt file under
prompts/. - Commit incrementally:
git commit -m "drop redundant instruction line". - Run your eval harness against the branch.
- Open a PR with eval delta attached.
- Merge to
mainor delete the branch.
Naming convention
Use prompt/<area>-<hypothesis>-<date>. The date avoids collisions when two engineers test the same idea. Example: prompt/classifier-strict-refusal-20240522.
When to branch vs commit directly
Branch when the outcome is uncertain: new persona, reordered instructions, added constraints. Commit directly to main for typos, punctuation, or variable name fixes where eval is unlikely to move. Ceremony should match risk.
Repository layout
Keep prompts in a dedicated directory at repo root. Separate them from application logic so they can be reviewed independently.
prompts/
summarizer/
v1.md
experiment.md
classifier/
system.txt
eval/
run.py
cases.jsonl
results/Use plain text or Markdown with frontmatter for metadata:
---
model: gpt-4o
temperature: 0.2
cache: true
---
You are a terse summarizer. Compress the input to three bullet points.The frontmatter lets your loader pick parameters without hardcoding them. Gateways such as n4n.ai forward provider cache-control hints, so marking your static prefix as cacheable in frontmatter cuts eval cost on repeated runs against the same model.
Making prompts evaluable
A prompt change is only useful if you can measure its effect. Structure prompts as templates with explicit variables:
from string import Template
prompt_tpl = Template(open("prompts/summarizer/experiment.md").read())
filled = prompt_tpl.safe_substitute(text=document)Your eval script should call the model and score the response. If you route through a gateway like n4n.ai, which exposes one OpenAI-compatible endpoint covering 240+ models with automatic fallback when a provider is degraded, you can swap model in the frontmatter and run the same branch against Claude, Llama, or Mixtral without editing code.
import openai, json, os
client = openai.OpenAI(base_url="https://api.n4n.ai/v1", api_key=os.environ["KEY"])
def run_case(case):
filled = prompt_tpl.safe_substitute(text=case["input"])
resp = client.chat.completions.create(
model=frontmatter["model"],
temperature=frontmatter["temperature"],
messages=[{"role": "system", "content": filled}],
)
return resp.choices[0].message.content
cases = [json.loads(l) for l in open("eval/cases.jsonl")]
scores = [1.0 if c["expect"] in run_case(c) else 0.0 for c in cases]
print(sum(scores)/len(scores))That is the only place the gateway appears; the rest is standard OpenAI client usage.
Running isolated experiments
Check out the branch and point your eval at its prompt path:
git checkout prompt/summarizer-compress-20240521
PYTHONPATH=. python eval/run.py --prompt prompts/summarizer/experiment.mdStore outputs in a branch-specific directory: eval/results/summarizer-compress-20240521.jsonl. This makes PR review trivial: attach the before/after score deltas.
To run multiple experiments in parallel without stashing, use git worktrees:
git worktree add ../exp-compress prompt/summarizer-compress-20240521
cd ../exp-compress && python ../repo/eval/run.py --prompt prompts/summarizer/experiment.mdAvoid mutating shared eval cases mid-experiment. If you need new test cases, add them on main first and rebase.
Reviewing prompt diffs
GitHub and GitLab render Markdown diffs well. A good prompt PR includes:
- The diff of the
.mdfile. - Eval score delta (accuracy, latency, cost per 1k cases).
- A few qualitative examples.
Example diff:
- You are a terse summarizer.
+ You are a terse summarizer. Ignore greetings and meta-commentary.Pitfall: large rewrites. If you change the persona, the instruction order, and the output format simultaneously, you cannot tell which move helped. Keep branches narrow. Want to test persona? Branch prompt/summarizer-persona-curious. Want to test format? Separate branch.
Merging and tagging
When a branch beats baseline on main, merge it. Immediately tag the prompts directory:
git tag -a prompt-summarizer-v1.3 -m "compress experiment merged"
git push origin prompt-summarizer-v1.3Tags give you an immutable reference for rollback. If a model provider pushes a silent update that breaks things, you can restore the old prompt:
git checkout prompt-summarizer-v1.2 -- prompts/summarizer/Then re-run eval to confirm whether the prompt or the model caused the regression.
Common pitfalls and tradeoffs
Long-lived branches rot. Models change underneath you. A branch opened 30 days ago may fail eval not because the prompt is bad, but because the default model version shifted. Rebase weekly or kill stale branches.
Prompt-only versioning ignores model version. Git tracks your words, not the weights. Pin model and model_version in frontmatter where the API allows. Otherwise your history is incomplete.
Overhead for trivial tweaks. If you are fixing a typo, a branch is ceremony. For sub-line changes, commit directly to main with a clear message and rely on CI eval to catch issues. Branch-per-prompt is for experiments with uncertain outcome.
Eval nondeterminism. Sampling introduces noise. Set seed if the endpoint supports it, and run each case multiple times to get a confidence interval. A 2% score bump on 50 cases is not signal.
Secrets and PII. Never embed API keys or user data in prompt files. Use template vars and inject at runtime.
Branch explosion. Ten engineers running ten experiments generate noise in the repo. Enforce a stale-branch bot that comments after 14 days of inactivity.
Automating with CI
Add a workflow that runs eval on any branch matching prompt/*:
name: prompt-eval
on:
pull_request:
branches: [main]
jobs:
eval:
if: startsWith(github.head_ref, 'prompt/')
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: {python-version: "3.11"}
- run: pip install openai
- name: Run eval
env:
KEY: ${{ secrets.LLM_KEY }}
run: python eval/run.py --prompt prompts/summarizer/experiment.md --out results.jsonl
- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: eval-results
path: results.jsonlThe job posts a comment with score deltas. Now every git branching workflow for prompt experiments gets the same rigorous gate as code.
Putting it together
Start with one prompt directory and one eval script. Move your current production prompt to main. Next time you want to try “add a constraint to refuse off-topic”, create prompt/summarizer-refuse-20240522, edit, eval, PR. Within a week you’ll have a searchable record of what works. The git branching workflow for prompt experiments is low tech, but it matches how LLM systems actually fail: quietly, and at the boundary of language.