SWE-bench vs SWE-bench Verified: what's the difference

The decision between SWE-bench vs SWE-bench Verified usually surfaces the first time a coding agent scores 40% on a benchmark but you can’t tell if the failures are model bugs or dataset rot. SWE-bench packages real GitHub pull requests with failing tests into a repo-level coding task; SWE-bench Verified is a human-audited subset of that corpus designed to remove unresolvable or mis-specified instances. Below is a practitioner’s comparison across the axes that matter when you wire these into CI.

What the two datasets actually contain

SWE-bench construction

SWE-bench scrapes merged PRs from 12 popular Python repos (scikit-learn, django, flask, sympy, etc.). For each PR it records the base commit, the issue description, and the test patch that validates the fix. The task is to generate a code patch that, when applied to the base commit, makes the hidden test patch pass without breaking the repo’s existing suite. The full set ships 2,294 instances. The catch is that some instances have flaky tests, require network access, or depend on environment state the harness can’t reproduce. A non-trivial fraction of “failures” are actually harness bugs.

SWE-bench Verified curation

SWE-bench Verified takes 500 instances from the full set and runs them through manual review. Annotators confirm the issue is self-contained, the test patch actually exercises the fix, and the environment builds deterministically. They drop cases where the gold patch is a no-op, where the spec contradicts the repo’s behavior, or where the test only passes due to a side effect. The result is a benchmark where a zero score is far more likely to mean your agent is wrong, not the dataset is broken.

How evaluation actually works

Both datasets use the same harness mechanics. Each instance defines FAIL_TO_PASS tests (must go from red to green) and PASS_TO_PASS tests (must stay green). The evaluator applies your model_patch, runs the test suite in a container, and computes resolution as the conjunction of both conditions.

# pseudocode for the scoring logic inside swebench
def resolve(instance, model_patch):
    env = build_container(instance.base_commit)
    apply(env, model_patch)
    ftp = run(env, instance.fail_to_pass)  # expect all pass
    ptp = run(env, instance.pass_to_pass)  # expect all pass
    return all(ftp) and all(ptp)

The full set’s noise shows up as PASS_TO_PASS regressions that are actually environment drift, not model errors. Verified’s curation explicitly checks those tests are stable.

Head-to-head dimensions

Capabilities

Both measure the same capability: autonomous generation of a repo-level diff that satisfies a hidden test suite. Neither tests multi-file reasoning beyond what the PR required, nor evaluates agent trajectories, tool use, or human-in-the-loop behavior. Verified does not add new task types; it subtracts noise. If you need signal on agentic workflows (retrieval, iterative debugging), you must wrap either dataset in your own loop.

Price/cost model

There is no licensing fee. The cost is compute. Each instance requires a Docker image build (often 1–5 GB) and a test run that can take 30–300 seconds. At 2,294 instances, a single full pass on a mid-size cloud VM cluster burns hours of wall clock and noticeable CPU dollars. Verified’s 500 instances cut that by roughly 78%, making daily regression runs feasible. The dominant cost is your model inference, not the eval harness—a single GPT-class sweep on full SWE-bench can cost orders of magnitude more than the containers.

Latency/throughput

Full SWE-bench is an overnight job on modest hardware. Verified fits in a lunch break on a single 8-core machine if you parallelize the containers. Throughput is gated by Docker startup overhead, not by dataset size per se, but fewer instances means fewer cold starts. For CI gating, Verified is the only one that won’t block your merge queue.

Ergonomics

The swebench Python package accepts a predictions JSON and emits a resolved rate. With the full set, you will spend time triaging “expected failures” that are environment bugs. Verified gives cleaner error bars: if a patch fails, it’s almost always a real regression. The trade-off is coverage—500 instances means wider confidence intervals on small model deltas (a 2% improvement may not be significant).

Ecosystem

Both share the same loader, same Docker specs, same evaluation script. HuggingFace mirrors princeton-nlp/SWE-bench and princeton-nlp/SWE-bench_Verified. Tooling like OpenCompass, AgentBench, and several agent scaffolds consume both via a --dataset flag. You can swap one for the other without changing your agent code. Leaderboards (e.g., SWE-bench leaderboard) report both, so cross-comparison is straightforward.

Limits

SWE-bench full suffers from label noise and non-determinism; Verified sacrifices breadth for precision and excludes many hard instances that happened to be clean. Neither includes non-Python repos (despite community forks). Both are static—they don’t measure how your agent behaves when the issue tracker changes or when tests are updated post-hoc.

Comparison table

Running both locally

The harness is identical. To evaluate a predictions file against Verified:

python -m swebench.harness.run_evaluation \
  --dataset_name princeton-nlp/SWE-bench_Verified \
  --predictions_path ./preds.json \
  --max_workers 8

For the full set, change the dataset name and brace for longer runtime:

python -m swebench.harness.run_evaluation \
  --dataset_name princeton-nlp/SWE-bench \
  --predictions_path ./preds.json \
  --max_workers 8

A predictions file is just a list of {instance_id, model_patch} objects:

[
  {"instance_id": "django__django-12345", "model_patch": "diff --git a/foo.py b/foo.py\n..."}
]

A minimal agent loop that generates patches might look like:

import openai, json, os

def generate_patch(instance):
    resp = openai.ChatCompletion.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a python repair agent."},
            {"role": "user", "content": f"Fix {instance['issue']} in {instance['repo']}"}
        ]
    )
    return extract_diff(resp.choices[0].message.content)

preds = [{"instance_id": i["instance_id"], "model_patch": generate_patch(i)} for i in instances]
json.dump(preds, open("preds.json", "w"))

If you are generating patches via an LLM gateway, route the agent’s completion calls through a single OpenAI-compatible endpoint. When running sweeps across SWE-bench vs SWE-bench Verified at scale, an inference gateway that honors fallback directives—like n4n.ai—keeps throughput up when a provider rate-limits your batch, while per-token metering keeps cost visible per instance.

Which to choose

Use SWE-bench Verified if

• You run eval in CI and need a green/red signal in under an hour.
• You are tuning a coding agent and need to attribute failures to the agent, not the dataset.
• Your team is small and cannot afford a dedicated benchmark-triage role.
• You want stable PASS_TO_PASS baselines that don’t randomly flip between runs.

Use full SWE-bench if

• You are publishing a paper and need the largest comparable numbers to prior work.
• You have the infra to quarantine flaky instances and maintain an internal “clean full” fork.
• You specifically want to measure performance on the harder, noisier tail of real issues where Verified dropped the instance.

Use SWE-bench Lite (300 instances) if

• You need a middle ground: broader than Verified, cheaper than full, and reasonably stable. Many leaderboards report Lite alongside Verified, and it is the default for quick agent smoke tests.

Verdict by use case

For a startup shipping a coding copilot, Verified is the default. For a research lab claiming SOTA, the full set remains the currency. The comparison SWE-bench vs SWE-bench Verified is not about which is “better” but which noise profile matches your tolerance. Pick Verified for velocity, full for coverage, and revisit quarterly as your agent improves. If you report numbers, always state which split you used—mixing them is the fastest way to make your eval meaningless.