The trade-off between rule-based vs model-graded evals isn’t a philosophical debate—it’s a sourcing decision about where determinism pays off and where it breaks. Rules catch the things you can specify precisely; model graders approximate the things you can’t. Build both, but assign them to different layers of your eval stack.
1. Map the evaluation surface before picking tools
You can’t choose a method until you know what signal you need. Split your eval targets into three buckets: structural correctness (valid JSON, required keys), constraint adherence (no PII, length limits, banned phrases), and quality judgment (helpfulness, tone, reasoning validity). The first two are pure rule territory. The third is where model-graded evals earn their cost.
Write a schema for each test so you can route it correctly:
{
"name": "order_extraction",
"type": "structured",
"rules": ["valid_json", "has_keys:['order_id','items']", "no_pii"],
"model_graded": {
"criteria": "Does the extracted order match the user's stated intent and include all line items?",
"scale": "pass/fail"
}
}If you skip this mapping, you’ll burn tokens grading things a regex could have caught. In a typical RAG pipeline, 60–70% of eval failures are structural—missing fields, malformed syntax, or violated length caps. Those never need a language model to detect.
2. Implement rule-based checks first
Rules are cheap, instant, and reproducible. Start with them for anything you can express as a predicate. Python’s jsonschema and a few regexes cover most launch-blocking bugs.
import json, re
PII_PATTERNS = [r"\b[\w.]+@[\w.]+\.\w+\b", r"\b\d{3}-\d{2}-\d{4}\b"]
def rule_check(output: str, expected_keys: list) -> dict:
try:
data = json.loads(output)
except json.JSONDecodeError:
return {"pass": False, "reason": "invalid_json"}
missing = [k for k in expected_keys if k not in data]
if missing:
return {"pass": False, "reason": f"missing_keys:{missing}"}
for pat in PII_PATTERNS:
if re.search(pat, output):
return {"pass": False, "reason": "pii_detected"}
if len(output) > 2000:
return {"pass": False, "reason": "too_long"}
return {"pass": True}Common pitfall: over-fitting rules to training examples. If your regex for “no PII” is just blocking a fixed email pattern, you’ll miss phone numbers or SSNs. Rules need maintenance as input distributions shift. Treat them as code, with code review and unit tests.
When rules lie
A rule can pass while the output is garbage. Valid JSON with the right keys but order_id: null satisfies the predicate. That’s why rule-based vs model-graded evals isn’t about replacing one with the other—it’s about sequencing. Rules tell you the output is well-formed; they don’t tell you it’s right.
3. Deploy model-graded evals for subjective criteria
When you need to judge whether a response is “polite” or “correctly reasoned,” a model grader is the fastest path. You prompt a separate LLM with the original query, the response, and a tight rubric. Keep the grading prompt strict and return structured verdicts.
from openai import OpenAI
import json
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
def model_grade(query: str, response: str, rubric: str) -> dict:
sys = "You are a strict eval grader. Return only JSON with keys: correctness (1-5), tone (1-5), rationale."
user = f"Query: {query}\nResponse: {response}\nRubric: {rubric}"
resp = client.chat.completions.create(
model="auto",
messages=[{"role": "system", "content": sys}, {"role": "user", "content": user}],
response_format={"type": "json_object"}
)
return json.loads(resp.choices[0].message.content)Using an OpenAI-compatible gateway like n4n.ai here gives you automatic fallback when a provider is rate-limited, so your eval job doesn’t stall mid-sweep. That’s a real operational win when you grade 10k samples nightly.
Tradeoffs: model graders add latency (often 500ms–2s per call) and cost per token. They also drift between model versions. Pin the grader model and log its version. The rule-based vs model-graded evals balance shifts as model prices drop, but the latency tax remains.
4. Build a hybrid pipeline with clear ordering
The actionable path: run rules first, then sample or conditionally invoke model graders. Never grade what a rule already failed.
def evaluate(sample: dict) -> dict:
rule_result = rule_check(sample["output"], ["order_id", "items"])
if not rule_result["pass"]:
return {"stage": "rule", "pass": False, "detail": rule_result}
if sample.get("needs_quality_check"):
grade = model_grade(sample["query"], sample["output"],
"Does the response answer the query using only provided facts?")
return {"stage": "model", "pass": grade["correctness"] >= 4, "detail": grade}
return {"stage": "rule", "pass": True}This cuts model grading volume by 40–80% in practice, because most regressions are structural.
Sampling strategy
Don’t grade every passing item in prod. Use reservoir sampling on rule-passed outputs to send 5–10% to the model grader. If the model grader disagrees with rules on a dimension, you’ve found a rule gap. Store grades keyed by content hash to avoid recomputation.
import hashlib
def should_sample(output: str, rate: float = 0.1) -> bool:
h = hashlib.sha256(output.encode()).hexdigest()
return (int(h[:8], 16) / 0xffffffff) < rate5. Calibrate the model grader against humans
A model grader is only useful if it agrees with your notion of quality. Label 200 samples by hand. Compute Cohen’s kappa between human and grader. If kappa < 0.6, rewrite the rubric.
from sklearn.metrics import cohen_kappa_score
human = [1,0,1,1,0,1,0,0,1,1]
grader = [1,0,1,0,0,1,1,0,1,1]
print(cohen_kappa_score(human, grader)) # aim > 0.6Pitfall: grader self-bias. If you use the same family of model for generation and grading, it tends to rate its own outputs higher. Use a different model class for grading (e.g., a smaller instruction-tuned model or a larger one). The rule-based vs model-graded evals discussion often ignores this bias, but it skews your quality trends.
6. Operationalize cost and latency
Model-graded evals are a line item. With per-token metering you can attribute eval cost separately from inference cost. Set a budget: e.g., max $0.50 per 1k samples. Rules cost essentially zero.
Cache grading results keyed by (sample_hash, grader_version). Invalidate when you change the rubric. Forward provider cache-control hints if your gateway supports it—n4n.ai honors client routing directives and forwards cache-control, which trims repeated grading of identical regression sets.
Versioning
Treat the grader prompt as code. Tag it grader-v2.1. When you compare eval runs across weeks, you must know which grader version produced the scores. Store the version in your eval database alongside the sample.
7. Decision checklist
Use this ordered list to assign each eval target:
- Can a regex or schema validate it? → Rule-based.
- Is it a hard constraint (length, banned terms)? → Rule-based with deny lists.
- Is it about meaning, style, or multi-step logic? → Model-graded.
- Is the volume high and the rule pass rate >90%? → Rule-first, sample model grading.
- Do you have <200 human labels? → Delay model grading until you can calibrate.
The rule-based vs model-graded evals decision collapses to: rules for the predictable, models for the fuzzy, and a pipeline that makes them cooperate.
8. Common pitfalls to avoid
- Double grading: running model graders on rule-failed outputs wastes money.
- Silent grader drift: not pinning model versions makes historical comparisons meaningless.
- Rubric vagueness: “Is this good?” yields noise. Specify: “Does the answer cite at least one source from the context?”
- Ignoring false positives: a rule that passes broken output is worse than no rule. Add negative tests.
- No calibration: shipping a model grader without human agreement data is guessing with extra steps.
9. Minimal starter repo layout
Keep eval code separate from app code.
evals/
rules/
json_schema.py
pii_deny.py
graders/
openai_compat.py
rubrics.yaml
pipeline.py
calibrate.pyRun nightly: python evals/pipeline.py --sample 0.1 --grader-version v2.1.
That’s the whole approach. Start with rules, add model grading where judgment is required, and measure the grader like you’d measure any other component. The teams that ship reliable LLM features are the ones who treat evals as a layered system, not a single toggle.