How MT-Bench evaluates multi-turn conversations

MT-Bench is a multi-turn benchmark that evaluates LLM conversation quality using an LLM-as-a-judge approach across 80 questions spanning eight categories. Unlike single-turn benchmarks such as MMLU or HumanEval, MT-Bench specifically tests a model’s ability to maintain context, follow instructions across turns, and produce coherent extended dialogues. The benchmark uses GPT-4 as the default judge to score model outputs on a 1-10 scale, producing results that correlate strongly with human preferences.

How mt-bench works

MT-Bench consists of 80 multi-turn questions organized into eight categories: writing, roleplay, extraction, reasoning, math, coding, STEM knowledge, and humanities. Each question has two turns — an initial prompt and a follow-up that requires the model to reference or build upon its first response. This structure directly tests context retention and instruction following in conversation.

The evaluation pipeline runs as follows:

1. Model inference: The candidate model answers both turns of each question, producing 160 total responses (80 questions × 2 turns).
2. Judge scoring: GPT-4 (or another strong LLM) evaluates each response pair against a reference answer and scoring rubric, assigning a 1-10 score.
3. Aggregation: Scores are averaged per category and overall, yielding a single MT-Bench score.

The judge prompt is critical. It instructs the evaluator to assess helpfulness, relevance, accuracy, and coherence — with explicit guidance to penalize hallucination, instruction violations, and context loss. The official implementation uses a pairwise comparison format where the judge sees both the model output and a reference answer, then scores the model output relative to that reference.

# Simplified MT-Bench judge prompt structure
JUDGE_PROMPT = """You are an impartial judge evaluating an AI assistant's response.
Question: {question}
Reference Answer: {reference_answer}
Model Answer: {model_answer}

Score the model answer from 1-10 on:
- Helpfulness and relevance
- Accuracy and factual correctness
- Coherence and context awareness
- Instruction following

Provide your score and brief reasoning."""

The benchmark runs in two modes: single-answer grading (absolute scoring) and pairwise comparison (model A vs model B). Pairwise mode reduces position bias and calibration drift by having the judge directly compare two outputs side-by-side. The original paper recommends pairwise for model selection decisions, single-answer for leaderboard reporting.

Why multi-turn evaluation matters

Single-turn benchmarks miss failure modes that only appear in conversation. A model can ace MMLU but collapse when asked to “rewrite the previous answer for a non-technical audience” or “debug the code you just wrote given this error message.” MT-Bench catches:

• Context dropping: The model forgets constraints, variables, or facts established in turn one.
• Instruction drift: The model follows the first instruction but ignores modifications in the second turn.
• Hallucination amplification: An initial hallucination compounds when the model builds on it in turn two.
• Style inconsistency: The model shifts tone, format, or persona between turns.

These failures are invisible to static benchmarks but fatal in production chat applications. If you’re building a coding assistant, customer support bot, or any multi-turn workflow, MT-Bench scores predict real-world usability better than any single-turn metric.

Concrete example: a coding question walkthrough

Consider this MT-Bench coding question (simplified from the actual dataset):

Turn 1: “Write a Python function that takes a list of integers and returns the top 3 most frequent elements with their counts. Handle ties by returning the smaller elements first.”

Turn 2: “Now modify the function to accept an optional k parameter for the top-k elements, defaulting to 3. Also add type hints.”

A strong model produces a correct Counter-based solution in turn one, then cleanly refactors it in turn two — preserving the tie-breaking logic, adding k: int = 3 with List[Tuple[int, int]] return type, and updating the docstring. A weak model might:

• Forget the tie-breaking rule when adding the k parameter
• Change the return type to Dict[int, int] (losing order)
• Drop the default=3 and break backward compatibility
• Hallucinate a heapq implementation that doesn’t match the spec

The judge scores the pair of responses. Even if turn one is perfect, a turn two regression drags the score down. This mirrors how users actually experience conversational agents — the whole thread must hold together.

Common misconceptions

“MT-Bench measures factual knowledge”

It doesn’t. The questions are designed to be answerable without specialized knowledge — they test reasoning, instruction following, and coherence. A model that hallucinates a citation in a writing task gets penalized for hallucination, not for “not knowing facts.” For knowledge evaluation, use MMLU, GPQA, or SimpleQA.

“GPT-4 as judge is circular / biased”

GPT-4 judging GPT-4 outputs would be circular. But MT-Bench evaluates other models against GPT-4 as judge. The judge model is fixed; the candidate varies. The original paper validated this by showing GPT-4 judgments correlate with human annotators at ~0.85 Spearman correlation — higher than human-human agreement on the same task. Subsequent work has replicated this with Claude, Gemini, and open judges like Prometheus.

That said, judge bias exists. GPT-4 prefers verbose, structured outputs and may penalize concise but correct answers. If you’re evaluating a model optimized for brevity (e.g., a mobile assistant), calibrate the judge prompt or use pairwise mode with a style-matched reference.

“MT-Bench score = production readiness”

A high MT-Bench score means the model handles these specific conversation patterns well. It doesn’t guarantee:

• Low latency or throughput targets
• Safety alignment for your domain
• Tool use / function calling correctness
• RAG faithfulness with your corpus
• Cost efficiency at scale

Treat MT-Bench as a necessary but insufficient filter. Pair it with domain-specific evals, latency benchmarks, and red-teaming.

“All 80 questions matter equally”

They don’t for your use case. If you’re building a SQL generator, the coding and extraction categories matter; roleplay and writing matter less. Weight categories by your traffic distribution. The per-category breakdown is more actionable than the aggregate score.

Running mt-bench yourself

The official implementation lives in the FastChat repository. You need:

• The 80 questions (JSONL format)
• A reference answer file (provided for GPT-3.5-Turbo, GPT-4, Claude)
• Access to a judge model (GPT-4 recommended, GPT-4-Turbo works, strong open models like Llama-3-70B-Instruct or Prometheus-2 are viable alternatives)

# Install FastChat
pip install fschat

# Run evaluation (single-answer mode)
python -m fastchat.llm_judge.gen_model_answer \
  --model-path your-model-name \
  --model-id your-model-id \
  --bench-name mt_bench \
  --question-file fastchat/llm_judge/data/mt_bench/question.jsonl \
  --answer-file model_answer.jsonl

# Judge with GPT-4
python -m fastchat.llm_judge.gen_judgment \
  --model-list your-model-id \
  --bench-name mt_bench \
  --judge-model gpt-4 \
  --mode single \
  --model-answer-file model_answer.jsonl \
  --judgment-file judgment.jsonl

For pairwise mode, provide two model answer files and use --mode pairwise. The output includes per-question scores, category averages, and an overall mean.

If you’re running evaluations at scale across many models, consider that judge API costs dominate. A full MT-Bench run (160 judge calls) costs roughly $2-4 with GPT-4-Turbo at current pricing. Open judges eliminate this cost but require validation against your quality bar.

Interpreting scores in context

The original leaderboard (June 2023) placed GPT-4 at 8.99, Claude-v1 at 7.94, and Vicuna-13B at 6.57. Scores have compressed upward as models improve — current frontier models cluster in the 8.5-9.5 range. A 0.5 gap is meaningful; a 0.1 gap is noise.

More useful than absolute scores: category profiles. A model scoring 9.0 overall but 6.0 on coding is a poor choice for a code assistant. A model scoring 8.0 overall but 9.0 on extraction and reasoning may excel at RAG workflows. Always request the per-category breakdown.

When to use mt-bench vs alternatives

MT-Bench remains the standard for multi-turn conversation quality because it’s reproducible, correlates with human preference, and stresses the right capabilities. Use it as a gate before deploying any conversational model — but build your own evals for the last mile.