Haystack evaluation tutorial: exact match vs semantic

When you set up a Haystack evaluation pipeline, the first decision is how to score generated answers: exact match or semantic similarity. The haystack evaluation exact match vs semantic choice determines whether you measure string equality or meaning overlap, and it cascades into your test data requirements, compute budget, and how you interpret regression signals. This post breaks down both approaches across the dimensions that matter in production.

What exact match evaluation does

Haystack’s ExactMatchEvaluator compares the predicted answer string against the ground truth using normalized string equality. By default it lowercases both strings and strips whitespace and punctuation before comparing. You can disable normalization if you need strict character-for-character matching.

from haystack.components.evaluators import ExactMatchEvaluator

evaluator = ExactMatchEvaluator(
    ignore_case=True,
    ignore_punctuation=True,
    ignore_numbers=False
)

result = evaluator.run(
    ground_truth_answers=["Paris"],
    predicted_answers=["paris"]
)
# result["individual_scores"] == [1.0]

The component returns a binary score per example (1.0 or 0.0) and an aggregate accuracy metric. It has no model dependencies, runs in microseconds, and produces deterministic results. The trade-off: it treats “Paris, France” and “Paris” as different answers unless you preprocess ground truth to match your generator’s output style.

What semantic evaluation does

SemanticAnswerSimilarityEvaluator embeds both the predicted answer and the ground truth with a sentence transformer, then computes cosine similarity. The default model is sentence-transformers/all-MiniLM-L6-v2 (384 dimensions, ~22M parameters), but you can pass any SentenceTransformersDocumentEmbedder-compatible model.

from haystack.components.evaluators import SemanticAnswerSimilarityEvaluator
from haystack.components.embedders import SentenceTransformersDocumentEmbedder

embedder = SentenceTransformersDocumentEmbedder(
    model="sentence-transformers/all-mpnet-base-v2"
)

evaluator = SemanticAnswerSimilarityEvaluator(
    document_embedder=embedder,
    score_threshold=0.75  # optional: binarize for "correct/incorrect"
)

result = evaluator.run(
    ground_truth_answers=["The capital of France is Paris."],
    predicted_answers=["Paris is France's capital city."]
)
# result["individual_scores"] == [0.92...]

Scores range from 0.0 to 1.0. The evaluator also returns mean_score and std_score across the batch. Because it uses a neural model, it captures paraphrase equivalence — but introduces latency, GPU/CPU dependency, and non-determinism across hardware or model versions.

Comparison at a glance

Capabilities: what each catches and misses

Exact match is precise but brittle. It catches regressions where the generator drops entities, changes numbers, or flips yes/no. It misses legitimate variations: “42” vs “forty-two”, “U.S.” vs “United States”, or a reordered clause that preserves meaning. Normalization helps — Haystack’s defaults handle case, punctuation, and whitespace — but you still need to align ground truth formatting to your generator’s output style. If your RAG pipeline emits “Answer: Paris” but ground truth is “Paris”, exact match scores 0.0 unless you strip the prefix in a pre-processing step.

Semantic similarity handles paraphrase, synonym substitution, and syntactic reordering gracefully. “The Eiffel Tower is in Paris” scores ~0.95 against “Paris hosts the Eiffel Tower.” But it has failure modes: it can reward hallucinated but semantically plausible additions (“Paris, the capital of France, is known for the Eiffel Tower” vs “Paris”), and it may penalize concise correct answers if the ground truth is verbose. The score_threshold parameter lets you binarize for CI gates, but picking 0.75 vs 0.80 vs 0.85 is a calibration exercise — run a labeled validation set first.

Both evaluators support ground_truth_answers as a list of lists for multi-reference evaluation. Exact match returns 1.0 if any reference matches; semantic returns the maximum similarity across references. This matters for QA where multiple correct phrasings exist.

# Multi-reference: exact match passes if ANY reference matches
result = evaluator.run(
    ground_truth_answers=[["Paris", "Paris, France", "the city of Paris"]],
    predicted_answers=["Paris"]
)
# exact match: 1.0, semantic: max similarity across three references

Cost and latency in practice

Exact match is effectively free. A 10,000-example evaluation completes in under 50 ms on a single CPU core. You can run it in every PR check without thinking about infrastructure.

Semantic evaluation cost scales with model size and hardware. The default all-MiniLM-L6-v2 embeds ~1,000 sentences/second on a modern CPU core (batch size 32). On an A10G GPU, throughput jumps to ~8,000–10,000 sentences/second. For a nightly eval of 50k examples: ~50 seconds CPU, ~6 seconds GPU. If you swap to all-mpnet-base-v2 (110M params) for higher quality, expect 3–4x slower.

Cloud inference pricing makes this concrete: embedding 1M sentences with all-MiniLM-L6-v2 on a spot GPU instance costs roughly $0.10–$0.30. On CPU spot, $0.50–$2.00. These are trivial compared to LLM inference costs, but they’re non-zero and require a GPU-enabled CI runner if you want fast feedback.

If you run evaluations through an inference gateway like n4n.ai, you can route embedding calls to the same endpoint you use for generation, simplifying auth and observability — but the latency and cost profile remains tied to the embedding model, not the gateway.

Ergonomics and ecosystem integration

Both evaluators are native Haystack components, so they slot into EvaluationPipeline and the evaluate() helper without adapters. They share the same input/output contract: ground_truth_answers: List[List[str]], predicted_answers: List[str], returning individual_scores: List[float] and aggregate metrics.

from haystack import EvaluationPipeline, Pipeline
from haystack.components.evaluators import ExactMatchEvaluator, SemanticAnswerSimilarityEvaluator

eval_pipeline = EvaluationPipeline()
eval_pipeline.add_component("exact_match", ExactMatchEvaluator())
eval_pipeline.add_component("semantic", SemanticAnswerSimilarityEvaluator())

# Run both in one pass
results = eval_pipeline.run({
    "exact_match": {"ground_truth_answers": gt, "predicted_answers": preds},
    "semantic": {"ground_truth_answers": gt, "predicted_answers": preds}
})

Haystack’s EvaluationResult object and to_pandas() method work identically for both. You can log to MLflow, Weights & Biases, or a local CSV with the same code. The semantic evaluator requires a DocumentEmbedder instance — this is explicit, not magic, so you control model versioning and device placement.

One ergonomic gap: neither evaluator exposes per-token or per-span alignment. If you need to know which part of a long answer failed, you’ll need a custom component or a post-hoc diff. For RAG answer evaluation, this rarely matters; for long-form generation, it’s a limitation.

Limits and edge cases

Exact match breaks down when ground truth contains variable content: timestamps, IDs, generated UUIDs, or model-specific phrasing. You can pre-filter with regex or normalize aggressively, but each normalization rule adds maintenance burden and risks false positives (e.g., stripping all numbers breaks “42” vs “43” detection).

Semantic similarity has subtler limits. The default model was trained on general English web text — it may under-score domain-specific equivalence (medical codes, legal citations, code snippets). Fine-tuning a sentence transformer on your domain helps, but adds a training pipeline. Short answers (< 5 tokens) produce noisy similarities; the evaluator works best with sentence-length inputs. Very long answers (> 256 tokens) get truncated by the embedder’s max length unless you configure a longer-context model.

Both evaluators assume the ground truth is correct. If your annotation process has noise, exact match amplifies it (every typo = failure), while semantic similarity dampens it (typo “Pariis” still scores ~0.85). Neither fixes bad labels — clean your test set first.

Determinism deserves emphasis. Exact match is bit-for-bit reproducible across Python versions, OS, and hardware. Semantic similarity is not: different BLAS libraries, GPU architectures, or PyTorch versions can shift scores by 0.01–0.03. Pin your sentence-transformers, torch, and transformers versions in requirements.txt and run evals on the same runner image if you need stable CI thresholds.

Which to choose

Use exact match when:

• Answers are short, structured, or entity-heavy (IDs, codes, names, numbers)
• You need deterministic, zero-cost CI gates that run in seconds
• Ground truth formatting is controllable and stable
• False negatives from paraphrase are acceptable (you’d rather catch regressions than reward variation)
• Evaluating extractive QA, slot filling, or classification-style outputs

Use semantic similarity when:

• Answers are generative, conversational, or long-form
• Paraphrase and synonym variation are expected and correct
• You can tolerate ~100–500 ms per 1k examples and a GPU/CPU dependency
• You have a validation set to calibrate score_threshold for pass/fail
• You need a continuous quality signal for trend lines, not just binary gates

Use both when:

• You want a regression safety net (exact match) and a quality trend (semantic)
• Your test set mixes structured and generative answers — route by answer type
• You’re A/B testing generators and need both precision and recall perspectives

A practical pattern: run exact match on every PR (fast, strict), run semantic nightly (richer signal, slower), and alert on either regressing. This gives you the developer velocity of binary gates with the semantic awareness that catches “correct but different” improvements.

# CI gate: exact match must not drop
assert results["exact_match"]["accuracy"] >= 0.92

# Nightly dashboard: track semantic mean_score trend
log_metric("semantic_mean", results["semantic"]["mean_score"])

The haystack evaluation exact match vs semantic decision isn’t permanent — you can swap evaluators in the pipeline without changing test data. Start with exact match for speed and determinism, add semantic when you have evidence that paraphrase variation is masking real quality. Most teams end up running both.