DeepEval metrics explained: DeepEval is an open-source Python framework that treats LLM outputs as testable units, shipping metric classes that score responses with either heuristic checks or an LLM-as-judge. The three metrics most teams reach for are G-Eval for custom quality scoring, Faithfulness for retrieval grounding, and Bias for stereotypical association detection.
What DeepEval metrics actually are
DeepEval provides a Metric base class. Each metric consumes an LLMTestCase—a structured record with input, actual_output, expected_output, context, and retrieval_context. The metric returns a score (0–1 or 1–10 depending on the metric) and a reason string generated by the judge model. You invoke them synchronously with .measure() or asynchronously with .a_measure() inside pytest or standalone scripts.
The framework assumes you will iterate on prompts and RAG pipelines like traditional software, using scores as gates in CI. That mindset is the whole point: evaluation is a test suite, not a notebook you run once.
G-Eval: LLM-as-judge with constrained scoring
How G-Eval works
G-Eval is the flexible metric. You supply criteria (a natural-language rubric) and evaluation_params (which fields of the test case the judge may read). Under the hood, DeepEval first asks the judge to generate evaluation steps from your criteria, then performs chain-of-thought reasoning, and finally forces a numeric score on a fixed scale (default 1–10) via constrained decoding or strict parsing. The CoT step reduces position bias and vague grading; the numeric constraint makes runs comparable across time.
The judge model is configurable. By default it calls an OpenAI chat model, but any OpenAI-compatible endpoint works.
Code example
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase
test_case = LLMTestCase(
input="Summarize the refund policy.",
actual_output="You can get a refund within 30 days.",
expected_output="Refunds are issued within 30 days of purchase if item is unused."
)
g_eval = GEval(
name="SummaryAccuracy",
criteria="Does the summary preserve the key constraint about item condition?",
evaluation_params=["actual_output", "expected_output"]
)
g_eval.measure(test_case)
print(g_eval.score, g_eval.reason)If the output dropped the “unused” condition, G-Eval’s CoT should catch the omission and score low. The reason field tells you which evaluation step failed.
Score interpretation
A 1–10 score is not a percentage. Calibrate thresholds against your own golden set. Many teams map 7+ to “acceptable” and treat 4–6 as a warning band. Because the judge is an LLM, run the same case multiple times to estimate variance before trusting a hard cutoff.
Faithfulness: grounding against retrieval context
Mechanism
FaithfulnessMetric checks whether actual_output is entailed by the provided context (or retrieval_context). It uses an LLM to decompose the output into discrete claims, then verifies each claim against the context. The final score is the fraction of supported claims. This is the primary guardrail for RAG systems: hallucinations outside retrieved docs tank the score.
Unlike G-Eval, you do not write criteria. The metric has a fixed entailment protocol and requires context to be non-empty.
Code example
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase
test_case = LLMTestCase(
input="What does the warranty cover?",
actual_output="The warranty covers accidental damage for 2 years.",
context=["The warranty covers manufacturing defects for 2 years."]
)
faith = FaithfulnessMetric()
faith.measure(test_case)
# score near 0 because "accidental damage" is not in context
print(faith.score)In a pipeline, a faithfulness threshold of 0.8 might block merges to main.
Limits of entailment
The metric only sees what you put in context. If your retriever pulls the wrong document, the output can be faithfully entailed by bad context and still be wrong. Faithfulness measures grounding, not correctness against ground truth.
Bias: detecting stereotypical associations
How BiasMetric scores
BiasMetric probes whether the output relies on stereotypes across gender, race, religion, or other attributes. It prompts the judge model to assess the response for biased language and returns a score where higher means more biased (0 = neutral, 1 = strongly biased). It catches overt stereotypical phrasing, not systemic disparate treatment.
The metric works on actual_output alone, though supplying input helps the judge understand context.
Code example
from deepeval.metrics import BiasMetric
from deepeval.test_case import LLMTestCase
test_case = LLMTestCase(
input="Describe a nurse.",
actual_output="The nurse is a caring woman who loves children."
)
bias = BiasMetric()
bias.measure(test_case)
print(bias.score, bias.reason)A score > 0.5 should trigger a review, but treat it as a signal, not a verdict.
Why these metrics matter in production
Shipping LLM features without evaluation is flying blind. G-Eval lets you encode product-specific quality bars (tone, format adherence) as tests. Faithfulness catches the most common RAG failure: the model ignoring retrieved facts. Bias provides a first-pass safety net for user-facing copy.
Together they form a layered gate: faithfulness ensures grounding, G-Eval ensures fitness for purpose, bias ensures baseline neutrality.
A concrete evaluation pipeline example
Assume a RAG chatbot for internal docs. You store golden questions, expected answers, and retrieved contexts in a dataset. A pytest session runs:
import pytest
from deepeval.metrics import FaithfulnessMetric, GEval
from deepeval.test_case import LLMTestCase
@pytest.mark.parametrize("case", load_golden_cases())
def test_rag(case):
faith = FaithfulnessMetric()
faith.measure(case)
assert faith.score >= 0.8
g = GEval(
name="Helpfulness",
criteria="Is the answer helpful and concise for an engineer?",
evaluation_params=["actual_output", "input"]
)
g.measure(case)
assert g.score >= 7Run this in CI on every prompt change. If you use an OpenAI-compatible gateway like n4n.ai, the judge calls automatically fail over when a provider is rate-limited, keeping eval runs green without manual key swapping.
Common misconceptions about DeepEval metrics
Misconception 1: G-Eval is just a prompt
Engineers new to DeepEval metrics explained often assume G-Eval is a single prompt to an LLM. It is not. The metric constructs a multi-step protocol: it generates evaluation steps from your criteria, performs CoT, then constrains the score. That structure is what makes scores reproducible enough to gate releases.
Misconception 2: Faithfulness catches all hallucinations
Faithfulness only checks output against supplied context. If your retriever pulls the wrong document, the output can be faithfully entailed by the bad context and still be wrong. The metric cannot see ground truth unless you put it in context. Pair it with G-Eval against expected_output for end-to-end checks.
Misconception 3: Bias metric is a full safety audit
BiasMetric detects stereotypical phrasing in a single response. It does not measure differential performance across demographic groups over a large sample, nor does it catch subtle exclusion. Use it as a smoke test, not a compliance artifact.
Tuning metrics for your stack
DeepEval metrics explained here are defaults; you can adjust judge models, score scales, and thresholds. For G-Eval, narrow criteria to binary pass/fail if you want stricter gates. For Faithfulness, increase the claim-decomposition temperature if you see false positives on vague outputs.
When the judge model is a cost or latency bottleneck, batch test cases and use smaller models. The framework does not care which endpoint serves the tokens as long as it speaks OpenAI chat completions. Set the base URL and key via environment variables and the same test code runs against any provider.
Configuring the judge model
from deepeval.models import GPTModel
g_eval = GEval(
name="Correctness",
criteria="...",
evaluation_params=["actual_output", "expected_output"],
model=GPTModel(model="gpt-4o-mini")
)Swap GPTModel for any custom wrapper that implements the chat interface. Keep the judge model distinct from the model under test to avoid self-preference bias.