Code review checklists for prompt pull requests

A code review checklist for prompt changes should treat prompts as first-class artifacts in your repo, not scattered strings in application code. If you merge prompt edits without the same scrutiny as logic changes, you ship unpredictable model behavior to production. The following items are what I expect every prompt pull request to clear before it gets a green check.

1. Separate prompt templates from application code

Prompt text does not belong inline in a Python function or a TypeScript handler. Store each template as a standalone file—JSON, YAML, or Markdown with frontmatter—so the diff is readable and the content is reviewable without parsing code.

{
  "id": "summarize_ticket",
  "version": "1.4.0",
  "model": "gpt-4o-mini",
  "messages": [
    {"role": "system", "content": "You summarize support tickets. Output JSON."},
    {"role": "user", "content": "Ticket: {{ticket_text}}"}
  ]
}

A reviewer should see exactly what the model sees. If the prompt is buried in f"..." strings, the PR hides changes behind code formatting noise. Enforce this with a CI lint that fails if prompt literals appear outside the prompts/ directory.

2. Pin model versions and record metadata

Model behavior drifts even when the model name stays the same. Your code review checklist for prompt changes must require an explicit model identifier and a version field. Avoid bare "gpt-4"; use "gpt-4-0613" or the provider’s snapshot tag.

# prompts/summarize_ticket.yaml
model: anthropic/claude-3.5-sonnet-20240620
temperature: 0.2
cache_control: { type: ephemeral }

Record the author and a short rationale in the commit message or PR body. When a prompt breaks later, you need to know which model version it was tuned against. This metadata is also what lets a gateway route correctly and meter per-token usage.

3. Require a deterministic eval harness

Every prompt PR should include or reference an eval set: a fixed set of inputs and expected output properties. The harness runs offline in CI and fails the build on regression.

def test_summarize_ticket():
    prompt = load_prompt("summarize_ticket", version="1.4.0")
    out = client.chat.completions.create(
        model=prompt["model"],
        messages=render(prompt, ticket_text=FIXTURE)
    )
    data = json.loads(out.choices[0].message.content)
    assert "priority" in data and "summary" in data

If the PR only says “improves tone,” it is not reviewable. A code review checklist for prompt changes demands evidence, not vibes. Store eval fixtures in evals/ and version them alongside prompts.

4. Audit variable interpolation and injection

Prompt templates use placeholders. Review how those are filled. Never use raw string formatting with untrusted input.

# Bad
messages = [{"role": "user", "content": f"Ticket: {user_input}"}]

# Good
messages = render_template("summarize_ticket", ticket_text=user_input)

The good path escapes or structurally separates user content, often via distinct message objects. A reviewer must confirm that a malicious ticket containing “Ignore previous instructions” cannot hijack the system role. Note this in your code review checklist for prompt changes as a security line item.

5. Measure token count and cost delta

Prompt edits change token consumption. Compute the delta against the previous version using a tokenizer locally.

python -m tiktoken prompt_old.txt prompt_new.txt --model gpt-4o
# old: 412 tokens, new: 489 tokens (+18.7%)

If the new prompt adds 20% tokens per call, that multiplies across millions of requests. The PR should state the expected throughput impact. A checklist item: “Token budget change documented.” This keeps LLM spend from silently creeping.

6. Verify model compatibility and fallback

Prompts often exploit provider-specific features. If your template uses Anthropic’s prompt caching or OpenAI’s JSON mode, it will not port cleanly. When you route through n4n.ai, the single OpenAI-compatible endpoint exposes 240+ models with automatic fallback when a provider is degraded; your review should confirm the prompt does not rely on a single provider’s extension without a documented fallback.

{
  "model": "openai/gpt-4o-mini",
  "route": {"fallback": ["anthropic/claude-3.5-sonnet", "google/gemini-1.5-pro"]},
  "response_format": {"type": "json_object"}
}

If the prompt hard-codes a system prompt that only works on one vendor, flag it. The code review checklist for prompt changes should require a note on cross-model behavior or a test that runs against at least two providers.

7. Check cache-control and logging hints

Providers support cache-control to reduce cost and latency. If the PR adds cache_control: ephemeral on a large system prompt, verify the gateway forwards it. n4n.ai honors client routing directives and forwards provider cache-control hints, so the PR should include the exact hint block.

{"role": "system", "content": "Long static instructions...", "cache_control": {"type": "ephemeral"}}

Also confirm the prompt does not leak secrets into logs. Review the logging middleware: only the rendered prompt (with user data redacted) should be stored. A missing redaction filter is a blocker.

8. Ensure rollback and feature flag

Merging a prompt is a deployment. The PR must include a rollback path: either a version pin in config or a feature flag that switches between prompt versions.

// config.ts
export const PROMPT_CONFIG = {
  summarize_ticket: flags.isEnabled("prompt_v1_5") ? "1.5.0" : "1.4.0"
};

If the new prompt degrades quality, you flip the flag instead of rushing a revert. The final item on your code review checklist for prompt changes is: “Rollback verified in staging.”

Summary

Treat prompt PRs like any other production change. The above code review checklist for prompt changes keeps your LLM surface area observable, testable, and safe.