HumanEval explained: how coding benchmarks work

HumanEval is a benchmark of 164 hand-written programming problems that tests a model’s ability to generate functionally correct Python code from docstrings and function signatures. Each problem includes a natural language description, a function signature, and a hidden test suite that verifies correctness. The metric is pass@k: the probability that at least one of k generated samples passes all tests.

How the benchmark works

The dataset lives in a single JSONL file. Each record contains a task ID, a prompt (the signature plus docstring), a canonical solution, and a test string. The test string is not executed during dataset creation — it’s meant to be run against the model’s completion at evaluation time.

{
  "task_id": "HumanEval/0",
  "prompt": "def has_close_elements(numbers: List[float], threshold: float) -> bool:\n    \"\"\" Return True if any two numbers in the list are closer than threshold. \"\"\"",
  "canonical_solution": "    for i, elem in enumerate(numbers):\n        for j in range(i + 1, len(numbers)):\n            if abs(elem - numbers[j]) < threshold:\n                return True\n    return False",
  "test": "def check(candidate):\n    assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.3) == True\n    assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.05) == False\n    assert candidate([1.0, 2.0, 5.9, 4.0, 5.0], 0.95) == True\n    assert candidate([], 0.5) == False\n    assert candidate([1.0], 0.5) == False\n"
}

The evaluation harness executes the model’s completion alongside the test function in a sandboxed Python process. A pass means zero assertion errors, no exceptions, and completion within a timeout (typically 10–30 seconds per problem).

Pass@k calculation

Pass@k accounts for the stochastic nature of sampling. Given n generated samples per problem (typically n ≥ k), the unbiased estimator is:

pass@k = 1 - comb(n - c, k) / comb(n, k)

where c is the number of correct samples among n. This corrects for the fact that you’re drawing without replacement from a finite set of generations. Most papers report pass@1, pass@10, and pass@100 with n=200 or n=800 samples per problem.

from math import comb

def pass_at_k(n: int, c: int, k: int) -> float:
    if n - c < k:
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)

# Example: 15 correct out of 200 samples
print(pass_at_k(200, 15, 1))   # ~0.075
print(pass_at_k(200, 15, 10))  # ~0.54
print(pass_at_k(200, 15, 100)) # ~0.99

What the problems actually test

HumanEval problems fall into several categories. Understanding the distribution helps you interpret scores:

Category	Examples	Share
String manipulation	Palindromes, anagrams, encoding	~25%
List/array algorithms	Search, sort variants, sliding window	~30%
Math/number theory	Primes, GCD, base conversion	~20%
Data structures	Tree traversal, heap ops, graph basics	~15%
Parsing/format	CSV, JSON, custom grammars	~10%

The problems are intentionally self-contained — no external libraries, no I/O, no multi-file context. Each fits in a single function. This isolates code generation from software engineering skills like dependency management, API design, or reading existing codebases.

Why it matters for model selection

If you’re choosing a model for a coding assistant, HumanEval gives you a baseline signal for algorithmic correctness in isolation. A model scoring 85% pass@1 will reliably produce working snippets for well-specified functions. A model at 45% will hallucinate logic errors that look plausible but fail edge cases.

But the benchmark has blind spots. It doesn’t measure:

Multi-file reasoning — imports, cross-module refactoring, circular dependency handling
Framework knowledge — Django ORM, React hooks, Kubernetes manifests
Debugging — reading a failing test, locating the bug, applying a minimal fix
Long-context coherence — maintaining consistency across 500+ lines
Style and maintainability — type hints, docstrings, error handling patterns

For those, you need SWE-bench, RepoBench, or internal evals on your actual codebase.

A concrete evaluation run

Here’s what a minimal evaluation script looks like. This mirrors the official OpenAI evaluation harness but stripped to essentials:

import json
import subprocess
import tempfile
import os
from pathlib import Path
from concurrent.futures import ProcessPoolExecutor, as_completed

DATASET_PATH = "HumanEval.jsonl"
TIMEOUT = 30  # seconds

def load_problems():
    problems = {}
    with open(DATASET_PATH) as f:
        for line in f:
            p = json.loads(line)
            problems[p["task_id"]] = p
    return problems

def evaluate_completion(task_id: str, completion: str, test: str) -> bool:
    """Run completion + test in isolated process. Return True if all assertions pass."""
    # Wrap in a module that defines the candidate function
    code = f"""
{completion}

{test}

if __name__ == "__main__":
    check(candidate)
"""
    with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f:
        f.write(code)
        f.flush()
        try:
            result = subprocess.run(
                ["python", f.name],
                capture_output=True,
                timeout=TIMEOUT,
                text=True
            )
            return result.returncode == 0
        except subprocess.TimeoutExpired:
            return False
        finally:
            os.unlink(f.name)

def run_eval(model_fn, problems, n_samples=200):
    """model_fn(prompt) -> list[str] of length n_samples"""
    results = {}
    for task_id, problem in problems.items():
        prompt = problem["prompt"]
        test = problem["test"]
        samples = model_fn(prompt)
        correct = 0
        for sample in samples:
            if evaluate_completion(task_id, sample, test):
                correct += 1
        results[task_id] = {"correct": correct, "total": len(samples)}
    return results

Run this against your model endpoint, aggregate with the pass@k formula, and you have a reproducible score. The key engineering decisions: sandboxing (subprocess isolation), timeout handling, and sampling strategy (temperature, top-p, number of samples).

Common misconceptions

“Higher HumanEval means better coding agent”

False. HumanEval correlates with single-function correctness. It does not correlate strongly with multi-file editing success rates. A model can score 90% on HumanEval and fail miserably on a PR that touches 12 files across a Django codebase. Treat it as a necessary but insufficient signal.

“Pass@1 is the only number that matters”

Pass@1 measures greedy or low-temperature correctness. Pass@10 and pass@100 measure the model’s capability ceiling — whether the right answer exists in the distribution. For agents that use search, verification, or self-consistency (e.g., generate 20, pick the one that passes tests), pass@100 is the relevant metric. For autocomplete with no verification, pass@1 matters more.

“The canonical solution is the only correct answer”

The test suite defines correctness, not the canonical solution. Many problems admit multiple valid implementations. The evaluation harness runs your completion against the tests — if it passes, it’s correct. This matters when you’re doing semantic equivalence checking or training reward models.

“HumanEval is saturated”

Top models now exceed 90% pass@1. But the distribution of failures is informative. Models still struggle with:

Off-by-one errors in sliding window problems
Unicode handling in string tasks
Large integer arithmetic without overflow (Python handles this, but logic errors persist)
Nested loop invariants

Saturation on the aggregate metric doesn’t mean saturation on the hard subset. Track per-problem pass rates to see where your model actually fails.

“Temperature 0 gives the best pass@1”

Not necessarily. Some models benefit from slight temperature (0.1–0.3) to escape local minima in the token distribution. Always sweep temperature for your specific model and prompt format. The optimal setting varies by model family and quantization.

Prompt format sensitivity

HumanEval scores swing 5–15 points depending on prompt formatting. The original paper uses a specific format: the prompt ends at the function signature with no closing triple quotes, no pass, no indentation hint. Many open-source evaluations accidentally add a newline or indent the completion, which changes the distribution.

# Correct: completion starts at column 4 (inside the function)
prompt = '''def add(a: int, b: int) -> int:
    """Return the sum of a and b."""'''

# Wrong: extra newline causes completion at column 0
prompt = '''def add(a: int, b: int) -> int:
    """Return the sum of a and b."""
'''

# Wrong: includes pass statement
prompt = '''def add(a: int, b: int) -> int:
    """Return the sum of a and b."""
    pass'''

If you’re comparing models, use identical prompt formatting. The canonical format is in the dataset — don’t improvise.

Extending HumanEval for your use case

The benchmark is a starting point, not a destination. Teams that ship coding agents typically build three layers on top:

HumanEval+ — additional edge-case tests per problem (EvoEval, HumanEval+ datasets)
Domain-specific evals — your framework, your patterns, your bug classes
Regression suites — real PRs from your repo, frozen at specific commits

# Example: augmenting a HumanEval problem with property-based tests
from hypothesis import given, strategies as st

@given(st.lists(st.floats(allow_nan=False, allow_infinity=False), min_size=2),
       st.floats(min_value=0.0, max_value=10.0))
def test_has_close_elements_property(numbers, threshold):
    result = candidate(numbers, threshold)
    # Brute force verification
    expected = any(
        abs(numbers[i] - numbers[j]) < threshold
        for i in range(len(numbers))
        for j in range(i + 1, len(numbers))
    )
    assert result == expected

Property-based tests catch whole classes of errors that fixed test suites miss. They’re slower to run but higher signal per problem.

Where HumanEval fits in your eval stack

┌─────────────────────────────────────┐
│  Production traffic (real PRs)      │  ← Highest fidelity, slowest cycle
├─────────────────────────────────────┤
│  SWE-bench / RepoBench              │  ← Multi-file, realistic context
├─────────────────────────────────────┤
│  Domain evals (your framework)      │  ← Your stack, your patterns
├─────────────────────────────────────┤
│  HumanEval+ / MBPP / EvoEval        │  ← Algorithmic correctness, extended
├─────────────────────────────────────┤
│  HumanEval (pass@1, pass@10, pass@100)  ← Baseline, fast, comparable
└─────────────────────────────────────┘

Run HumanEval on every model candidate. It’s cheap (164 problems × 200 samples ≈ 32k completions), fast (minutes on batched inference), and comparable across papers. But don’t stop there. The models that win on HumanEval don’t always win on your codebase.

Practical takeaways

Use pass@1 for autocomplete, pass@10 for agents with verification. Match the metric to your product flow.
Fix your prompt format. Use the exact dataset formatting. Document it in your eval config.
Track per-problem failure rates. Aggregate scores hide systematic weaknesses.
Build your own layer 2 and 3 evals. HumanEval is table stakes. Your competitive advantage is in domain-specific evaluation.
Run it in CI. A 5-minute HumanEval gate catches regressions from quantization, prompt changes, or model swaps before they hit users.

The HumanEval benchmark explained here is a tool, not a verdict. It tells you whether a model can write a correct function from a clean spec. Everything else — context, style, debugging, architecture — is on you to measure.