Storing prompt metadata alongside your git commits

Storing prompt metadata in git commits turns your LLM prompts from ephemeral strings into auditable, diffable assets. This article walks through a concrete workflow that treats prompts as code and captures model parameters, versions, and eval results next to the source.

Step 1: Define a prompt schema

Start by rejecting bare .txt prompt files. Use a structured format that separates static template from mutable config. YAML is readable and diff-friendly.

# prompts/summarize_v1.yaml
id: summarize
version: 1
model: gpt-4o-mini
params:
  temperature: 0.2
  max_tokens: 512
  top_p: 1.0
system: "You are a concise technical editor."
user_template: "Summarize the following {{section}} in two sentences: {{text}}"
metadata:
  owner: platform-team
  created: 2024-11-01

The metadata block holds team-specific fields. Keep it flat to avoid merge conflicts. Versioning inside the filename (_v1) lets you run old and new prompts side by side during migration.

Step 2: Validate prompts in CI

A prompt that fails schema validation should block the merge. Write a small Pydantic model and run it in your pipeline.

from pydantic import BaseModel, Field
from pathlib import Path
import yaml

class PromptSpec(BaseModel):
    id: str
    version: int
    model: str
    params: dict
    system: str
    user_template: str
    metadata: dict

for f in Path("prompts").glob("*.yaml"):
    data = yaml.safe_load(f.read_text())
    PromptSpec(**data)  # raises if malformed

Run this in GitHub Actions or a pre-merge hook. It takes 20 lines and saves hours of debugging missing variables or typo’d model names.

Step 3: Stamp commit metadata with a pre-commit hook

Storing prompt metadata in git commits requires recording which commit introduced or changed a prompt. A pre-commit hook writes a registry file that maps prompt IDs to the current commit hash, author, and timestamp.

Create .git/hooks/pre-commit:

#!/usr/bin/env python3
import subprocess, json, datetime, os

def git(*args):
    return subprocess.check_output(["git", *args]).decode().strip()

try:
    head = git("rev-parse", "HEAD")
except subprocess.CalledProcessError:
    head = "0000000"  # initial commit

author = git("config", "user.name")
ts = datetime.datetime.utcnow().isoformat() + "Z"

registry_path = "prompt_registry.json"
reg = {}
if os.path.exists(registry_path):
    reg = json.load(open(registry_path))

for f in subprocess.check_output(["git", "diff", "--cached", "--name-only"]).decode().split():
    if f.startswith("prompts/") and f.endswith(".yaml"):
        pid = os.path.basename(f).split("_")[0]
        reg[pid] = {"commit": head, "author": author, "updated": ts, "file": f}

json.dump(reg, open(registry_path, "w"), indent=2)
subprocess.call(["git", "add", registry_path])

The hook updates prompt_registry.json on every commit that touches prompts/. It stages the registry automatically. This makes storing prompt metadata in git commits mechanical rather than manual.

Testing the hook

Before relying on it, stage a dummy prompt and run git commit. Inspect the registry:

git commit -m "test hook" && cat prompt_registry.json

If the file updated with your prompt ID, the hook works. If not, ensure the script is executable (chmod +x .git/hooks/pre-commit) and uses Python 3.

Step 4: Capture runtime invocation metadata

Prompt files define intent; runtime defines reality. After a prompt runs, write its actual model, token usage, and provider routing to a sidecar file. If you route through a gateway such as n4n.ai, the response payload includes provider routing and cache hints—serialize those into your metadata file before committing eval results.

Call an OpenAI-compatible endpoint and log:

import openai, json, os

client = openai.OpenAI(base_url="https://api.n4n.ai/v1", api_key=os.environ["KEY"])
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize: ..."}],
    temperature=0.2,
)
usage = resp.usage.model_dump()
meta = {
    "model": resp.model,
    "provider": resp.provider if hasattr(resp, "provider") else "unknown",
    "prompt_tokens": usage["prompt_tokens"],
    "completion_tokens": usage["completion_tokens"],
}
json.dump(meta, open("prompts/summarize_v1.run.json", "w"), indent=2)

Commit the .run.json only after you’ve validated the output. Treat it as a cache, not a source of truth. The sidecar lets you reconstruct cost and latency per prompt version later.

Step 5: Link eval results to the commit

Create an evals/ directory keyed by prompt ID and short commit hash.

mkdir -p evals/summarize/$(git rev-parse --short HEAD)
echo '{"rouge": 0.41, "hallucinations": 0}' > evals/summarize/$(git rev-parse --short HEAD)/score.json
git add evals/ && git commit -m "eval: summarize @ $(git rev-parse --short HEAD)"

Now each prompt version has its template, its runtime metadata, and its eval score anchored to the same commit graph.

Step 6: Verify the workflow end to end

Run a full cycle to confirm the system holds:

Edit prompts/summarize_v1.yaml (bump version).
python validate.py (from Step 2).
git add prompts/ && git commit -m "prompt: tune summarize temp"
The pre-commit hook should print nothing but update prompt_registry.json. Verify:
```
git show HEAD:prompt_registry.json | grep summarize
```
You should see the new commit hash.
Run the prompt via the script in Step 4, then git add prompts/summarize_v1.run.json && git commit -m "run: summarize metadata"
Check history:
```
git log --oneline -- prompts/summarize_v1.yaml
```
You should see both the template change and the run metadata commit.

If prompt_registry.json is missing or stale, the hook failed—check executable bit (chmod +x .git/hooks/pre-commit). This verification proves storing prompt metadata in git commits actually happened.

Operational caveats

Secrets: Never put API keys in prompt metadata. The .run.json should contain only usage stats.
Merge conflicts: The registry file will conflict if two branches edit the same prompt. Use a post-merge hook to rebuild it from git log, or store metadata in Git notes (git notes add) to keep the tree clean.
Large histories: If eval JSON grows, use Git LFS or store only pointers.
Diff noise: Keep user_template on one line per sentence to minimize YAML diff churn.

Storing prompt metadata in git commits is not about git magic; it’s about enforcing a contract that prompts are code, runs are observable, and evaluations are reproducible. The above steps give you a minimal, extensible foundation.

Step 7: Extend with tags and branches

Once the base flow works, tag prompt releases like software:

git tag -a prompt-summarize-1.2 -m "stable summarize v1.2"

Branch per experiment:

git checkout -b prompt/summarize-aggressive-temp

Because metadata travels with the commit, checking out an old tag reconstructs the exact prompt, model, and eval context. That’s the payoff: prompt-as-code with the same forensic power as application source.

Automation tip

Add a CI job that fails if a prompt YAML is modified without a corresponding .run.json commit in the same PR. This keeps the metadata honest as the team scales.

That’s the full loop. Implement it this week; your future debugging self will thank you.