Linting prompts before they merge with a pre-commit hook

A prompt linting pre-commit hook treats prompt templates as code and blocks malformed changes at the git boundary. If you already run black and ruff on your Python, your .prompt and .json prompt files deserve the same static gates before they merge.

Step 1: Define the prompt contract and failure modes

Pick a single on-disk format for prompts and stick to it. I use JSON files with a strict schema: system, user_template, max_tokens, and optional cache_control. The linter enforces that contract so a missing variable or unbalanced brace never reaches inference.

Concrete checks I enforce:

• Required keys exist (system, user_template).
• user_template uses {variable} syntax and every referenced variable is declared in a variables array.
• Braces are balanced; no stray { or }.
• max_tokens is an integer between 1 and 32_768.
• File size stays under 16 KiB to avoid silent truncation at the gateway.

If you skip this step, your linter becomes a vague style checker. Define the contract first.

Step 2: Write the linter script

Create tools/lint_prompts.py. It takes a list of files and exits non-zero on the first violation. Keep it dependency-free; the standard library is enough.

#!/usr/bin/env python3
import sys
import json
import glob
import re
from pathlib import Path

REQUIRED_KEYS = {"system", "user_template"}
VARIABLE_RE = re.compile(r"\{(\w+)\}")

def check_file(path: Path) -> list[str]:
    errors = []
    text = path.read_text(encoding="utf-8")
    if len(text.encode("utf-8")) > 16 * 1024:
        errors.append(f"{path}: file exceeds 16 KiB")
    try:
        data = json.loads(text)
    except json.JSONDecodeError as e:
        return [f"{path}: invalid JSON ({e})"]

    missing = REQUIRED_KEYS - data.keys()
    if missing:
        errors.append(f"{path}: missing keys {missing}")

    user_tpl = data.get("user_template", "")
    open_braces = user_tpl.count("{")
    close_braces = user_tpl.count("}")
    if open_braces != close_braces:
        errors.append(f"{path}: unbalanced braces ({open_braces}/{close_braces})")

    declared = set(data.get("variables", []))
    used = set(VARIABLE_RE.findall(user_tpl))
    undeclared = used - declared
    if undeclared:
        errors.append(f"{path}: undeclared variables {undeclared}")

    mt = data.get("max_tokens")
    if not isinstance(mt, int) or not (1 <= mt <= 32_768):
        errors.append(f"{path}: max_tokens must be int in [1, 32768]")

    return errors

def main() -> int:
    files = sys.argv[1:] or glob.glob("prompts/**/*.json", recursive=True)
    all_errors = []
    for f in files:
        all_errors.extend(check_file(Path(f)))
    if all_errors:
        print("\n".join(all_errors))
        return 1
    print(f"OK: {len(files)} prompt files passed")
    return 0

if __name__ == "__main__":
    sys.exit(main())

Run it directly to confirm behavior:

python tools/lint_prompts.py prompts/summarize.json

A clean file prints OK: 1 prompt files passed. A broken one exits 1 with a precise message.

Step 3: Wire it into pre-commit

Use the pre-commit framework. Add .pre-commit-config.yaml at repo root:

repos:
  - repo: local
    hooks:
      - id: prompt-lint
        name: prompt linting pre-commit hook
        entry: python tools/lint_prompts.py
        language: system
        files: ^prompts/.*\.json$
        pass_filenames: true

The files regex scopes the hook to your prompt directory. pass_filenames: true means only staged prompt files get checked, keeping commits fast.

Step 4: Install and test locally

Install the framework once:

pip install pre-commit
pre-commit install

Now create a deliberately broken prompt:

{
  "system": "You are a helper.",
  "user_template": "Summarize {text} from {source",
  "variables": ["text"],
  "max_tokens": 500
}

Stage and commit:

git add prompts/broken.json
git commit -m "add broken prompt"

The prompt linting pre-commit hook fails the commit with unbalanced braces and undeclared variables. Fix the file, git add, and commit again. That tight loop is the entire point.

Step 5: Validate provider-specific hints

If you route prompts through an OpenAI-compatible gateway, provider extensions leak into your files. For example, n4n.ai forwards client cache_control hints verbatim to upstream providers, so a malformed cache_control block wastes a request instead of erroring locally. Extend the linter to reject unknown roles or misplaced keys.

def check_cache_control(data: dict, path: Path) -> list[str]:
    errors = []
    cc = data.get("cache_control")
    if cc is None:
        return errors
    if not isinstance(cc, dict) or "type" not in cc:
        errors.append(f"{path}: cache_control needs a type field")
    if cc.get("type") not in {"ephemeral"}:
        errors.append(f"{path}: unsupported cache_control type")
    return errors

Call it inside check_file. This keeps your local schema aligned with what the gateway will actually forward.

Step 6: Promote the hook to CI

A hook that only runs on a developer’s machine is advisory. Mirror it in GitHub Actions so --no-verify cannot sneak bad prompts into main.

name: prompt-lint
on: [pull_request]
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python tools/lint_prompts.py

For teams already using pre-commit, call pre-commit run --all-files in the workflow instead. Either path gives you a second gate that matches local behavior.

Step 7: Verify success and handle bypasses

Verification is concrete: introduce a regression, watch the pipeline fail, then watch it pass after fix. I keep a prompts/_negative_tests/ directory with intentionally broken files that the linter must reject; a small pytest wrapper asserts exit code 1 on those paths.

import subprocess, pathlib

def test_negative_samples():
    for p in pathlib.Path("prompts/_negative_tests").glob("*.json"):
        r = subprocess.run(["python", "tools/lint_prompts.py", str(p)])
        assert r.returncode == 1, f"{p} should fail lint"

Document that git commit --no-verify exists but CI will block the merge. If someone repeatedly bypasses the prompt linting pre-commit hook, the fix is a repo policy, not a thicker script.

Step 8: Iterate on the contract

Prompt engineering changes weekly. When you add a new field like stop_sequences, add the check in the same commit that introduces the field to the schema. Treat the linter as part of the prompt spec, not an afterthought.

A good prompt linting pre-commit hook is small, fast, and opinionated. It does not rewrite your text; it refuses to ship nonsense. That boundary lets you refactor prompts with the same confidence you refactor code.