GPQA explained: graduate-level science questions for AI

GPQA (Graduate-Level Google-Proof Q&A) is a 448-question multiple-choice benchmark spanning biology, physics, and chemistry that requires genuine reasoning rather than fact retrieval. The questions are designed so that internet search alone cannot reliably produce the answer — you need to synthesize concepts, apply domain knowledge, and work through multi-step logic. If you’re evaluating whether a model can actually reason through scientific problems instead of pattern-matching training data, GPQA is currently the gold standard.

What GPQA actually tests

The benchmark was introduced in late 2023 by researchers at NYU, Anthropic, and other institutions (Rein et al., “GPQA: A Graduate-Level Google-Proof Q&A Benchmark”). The name tells you the design goals: graduate-level difficulty, resistant to simple web search, and formatted as Q&A.

Each question targets one of three domains:

• Biology (roughly 150 questions): molecular biology, genetics, evolution, systems biology
• Physics (roughly 150 questions): mechanics, electromagnetism, quantum mechanics, statistical physics, relativity
• Chemistry (roughly 150 questions): organic, inorganic, physical, analytical, biochemistry

What makes GPQA different from MMLU’s science subsets or other knowledge benchmarks is the reasoning depth. MMLU often tests whether you’ve seen a fact during training. GPQA tests whether you can derive an answer from first principles when the exact question hasn’t been memorized.

The “Google-proof” claim is operationalized: the authors verified that domain experts (PhDs in the relevant field) could answer correctly given time and resources, while non-experts with internet access scored near random chance (~34%). This gap — expert performance around 65-75% vs. non-expert search performance at chance — is the benchmark’s validity signal.

How the benchmark works

GPQA uses a straightforward multiple-choice format: one question, four options, exactly one correct answer. But the construction process is where the signal lives.

Question creation pipeline

1. Expert generation: PhD-level domain experts write questions in their specialty, targeting concepts that require multi-step reasoning.
2. Adversarial filtering: Questions are tested against a suite of search strategies and baseline models. If a simple Google query or a strong retriever+reader pipeline finds the answer, the question is discarded or rewritten.
3. Expert validation: A second expert in the same domain solves the question independently. Disagreements trigger revision or removal.
4. Non-expert validation: Non-experts with internet access and 30 minutes per question attempt to solve. Questions where non-experts succeed reliably are rejected.

This pipeline is expensive — the paper reports roughly $200-300 per validated question — which is why the dataset is relatively small (448 questions) compared to MMLU’s 16,000+.

The Diamond subset

Within GPQA there’s a 198-question “Diamond” subset where both validators agreed on the answer and rated the question as high-quality. This is the cleaner signal; most published model results report on Diamond. If you’re running evals, use Diamond for your primary metric and full GPQA for breadth.

Scoring

Standard accuracy: correct answers / total questions. No partial credit, no chain-of-thought evaluation in the base benchmark. Some eval frameworks (like Inspect or custom harnesses) add reasoning trace scoring, but the canonical metric is raw accuracy.

# Minimal GPQA evaluation loop
import json
from pathlib import Path

def evaluate_gpqa(model, dataset_path: str, subset: str = "diamond") -> dict:
    with open(dataset_path) as f:
        data = json.load(f)
    
    questions = [q for q in data if q.get("subset", "full") == subset]
    correct = 0
    
    for q in questions:
        prompt = format_gpqa_prompt(q)
        response = model.generate(prompt)
        predicted = extract_answer(response)
        if predicted == q["correct_answer"]:
            correct += 1
    
    return {
        "subset": subset,
        "total": len(questions),
        "correct": correct,
        "accuracy": correct / len(questions)
    }

def format_gpqa_prompt(question: dict) -> str:
    options = "\n".join(f"{k}. {v}" for k, v in question["options"].items())
    return f"""Answer the following multiple choice question. Output only the letter (A, B, C, or D).

Question: {question["question"]}

Options:
{options}

Answer:"""

Why GPQA matters for reasoning evaluation

If you’re building systems that need to reason through novel scientific or technical problems — not just retrieve documentation — GPQA measures something MMLU doesn’t.

The retrieval vs. reasoning distinction

MMLU’s science questions often look like:

“What is the half-life of Carbon-14?” A) 5,730 years B) 12,000 years C) 1,200 years D) 57,300 years

This is a fact lookup. A model with good training coverage gets it right. A RAG system with a physics textbook gets it right.

GPQA questions look like:

“A researcher is studying a protein that forms a homodimer. The monomer has a molecular weight of 45 kDa. When run on SDS-PAGE under reducing conditions, a single band appears at 45 kDa. Under non-reducing conditions, a single band appears at 90 kDa. The researcher then introduces a point mutation at a cysteine residue suspected to form an intermolecular disulfide bond. Under non-reducing conditions, the mutant protein now shows two bands: one at 45 kDa and one at 90 kDa. What is the most likely explanation?”

This requires understanding:

• SDS-PAGE behavior under reducing vs. non-reducing conditions
• Disulfide bond chemistry
• How a point mutation at a cysteine affects dimerization
• Interpreting band patterns as evidence

You can’t Google the exact scenario. You have to reason through the biochemistry.

Correlation with downstream capability

Empirically, GPQA performance correlates strongly with:

• Code generation on novel algorithms (not LeetCode memorization)
• Mathematical problem solving (AIME, Olympiad-style)
• Multi-step agentic tasks requiring tool use and planning

The common factor: all require composing multiple reasoning steps without a clear template from training data. GPQA is a convenient proxy because it’s multiple-choice (easy to grade) but reasoning-heavy (hard to game).

Model performance landscape (as of mid-2024)

The gap between best models and experts is still meaningful. That gap is where reasoning improvement happens.

Concrete example: a GPQA question walkthrough

Here’s a representative physics question (simplified from the dataset style):

Question: A particle of mass m moves in a one-dimensional potential V(x) = αx⁴, where α > 0. The particle’s total energy is E. What is the period T of oscillation as a function of E?

Options: A) T ∝ E^(-1/4) B) T ∝ E^(1/4) C) T ∝ E^(-1/2) D) T ∝ E^(1/2)

Reasoning trace

Step 1: Identify the physics. This is classical mechanics — period of oscillation in a potential well. The particle moves between turning points where V(x) = E.

Step 2: Find turning points. αx⁴ = E → x = ±(E/α)^(1/4). The amplitude scales as E^(1/4).

Step 3: Period integral. For 1D potential, T = √(2m) ∫ dx / √(E - V(x)) over one quarter-cycle, times 4. T = 4√(2m) ∫₀^(E/α)^(1/4) dx / √(E - αx⁴)

Step 4: Change variables. Let x = (E/α)^(1/4) u. Then dx = (E/α)^(1/4) du, and E - αx⁴ = E(1 - u⁴). T = 4√(2m) (E/α)^(1/4) ∫₀¹ du / √(E(1 - u⁴)) = 4√(2m) E^(-1/4) α^(-1/4) ∫₀¹ du / √(1 - u⁴)

Step 5: Extract scaling. The integral is a dimensionless constant. T ∝ E^(-1/4).

Answer: A

Notice what this requires: setting up the correct integral, dimensional analysis or variable substitution, and extracting the scaling relationship. No single step is graduate-level, but the composition is. A model that has memorized “period scales as E^(-1/2) for harmonic oscillator” will pick C and fail. The quartic potential changes the scaling.

This is the GPQA pattern: familiar concepts, unfamiliar composition.

Common misconceptions

“GPQA is just a knowledge benchmark”

False. Knowledge is necessary but not sufficient. The questions are designed so that having taken the relevant graduate courses doesn’t guarantee the answer — you have to work through the specific scenario. The non-expert + search baseline at 34% (vs. 25% random) proves that search retrieval alone doesn’t crack it.

“High GPQA means the model is a scientist”

Also false. GPQA measures reasoning in scientific contexts, not scientific practice. It doesn’t test:

• Experimental design
• Literature synthesis
• Handling noisy/contradictory data
• Long-horizon research planning
• Tool use (simulation, data analysis, lab automation)

A model scoring 60% on GPQA Diamond cannot replace a PhD researcher. It can, however, be a useful reasoning engine within a research workflow.

“GPQA is saturated / solved”

Not close. The expert ceiling is 65-75%. Best models are at ~59% (Diamond). That 10-15 point gap represents real reasoning capability that doesn’t exist yet. Also: the benchmark is static. Models trained on GPQA questions (or similar reasoning traces) will inflate scores without improving generalization. Track held-out or new GPQA-style questions for honest signal.

“Multiple choice means it’s easy to game”

The adversarial filtering specifically targets this. Questions where option elimination works reliably (e.g., two options are dimensionally inconsistent, one is obviously wrong) are filtered out. The remaining questions typically have all four options physically plausible — you have to compute or reason to the right one.

“I should fine-tune on GPQA to improve my model’s reasoning”

Don’t. Fine-tuning on the benchmark teaches the benchmark, not reasoning. The correct approach: train on reasoning traces for similar problems (synthetic or human-generated), then evaluate on GPQA as a held-out test. If you must use GPQA data, use it only for evaluation, never for training.

Where GPQA fits in your eval stack

GPQA is not a general-purpose benchmark. It has a specific role: measuring multi-step reasoning in technical domains. Here’s how to position it:

Tier 1: Core reasoning evals (run on every model candidate)

• GPQA Diamond — primary reasoning signal
• MATH / GSM8K — mathematical reasoning
• HumanEval+ / MBPP+ — code reasoning (not memorization)
• GPQA + MATH correlation — if a model is high on one and low on the other, investigate why

Tier 2: Domain-specific evals (run when relevant)

• GPQA Biology/Physics/Chemistry subsets — if your use case is domain-specific
• SWE-bench — software engineering reasoning
• MedQA / PubMedQA — medical reasoning

Tier 3: Behavioral evals (run on deployed systems)

• Custom GPQA-style questions for your domain — write 20-50 questions mirroring GPQA’s style but targeting your problem space
• Reasoning trace quality — not just answer correctness, but does the chain-of-thought make sense?

Practical eval harness tips

# Example eval config for GPQA in a CI pipeline
evals:
  gpqa_diamond:
    dataset: "gpqa/diamond.jsonl"
    metric: "accuracy"
    threshold: 0.45  # minimum for "reasoning capable" tier
    temperature: 0.0
    max_tokens: 2048
    prompt_template: "gpqa_cot.j2"  # chain-of-thought prompt
    num_fewshot: 0  # zero-shot is standard
    parallel: 8
    timeout_seconds: 300

Run GPQA zero-shot with chain-of-thought prompting. Few-shot contaminates the reasoning signal (the model learns the format of GPQA reasoning, not general reasoning). Temperature 0 for reproducibility. Expect 3-5 minutes per 200 questions on a typical API endpoint.

Interpreting results for model selection

If you’re routing between n4n.ai for model routing, you can direct GPQA-heavy workloads to models in the 50%+ tier and save cost on simpler tasks.

Closing thought

GPQA exists because the field needed a benchmark that couldn’t be solved by retrieval, memorization, or shallow pattern matching. It’s imperfect — static, multiple-choice, limited domains — but it measures something real: the ability to compose known principles into novel derivations. That capability is the prerequisite for every “agent” or “reasoning” application that isn’t just a wrapped prompt template.

Run it. Track it. Don’t optimize for it. Use it to know whether your model can actually think through a problem it hasn’t seen before.