SWE-bench is a benchmark that evaluates language models on real software engineering tasks drawn from actual GitHub repositories. Unlike synthetic coding challenges, it presents models with genuine issues — bug reports, feature requests, and refactoring tasks — from popular Python projects like Django, scikit-learn, and matplotlib. The model must read the issue, navigate the codebase, and produce a patch that passes the project’s existing test suite.
How SWE-bench works
The benchmark consists of 2,294 task instances collected from 12 Python repositories. Each instance contains an issue description, the repository state at the time the issue was created, and the test suite that validates a correct fix. The evaluation harness applies the model’s proposed patch to the codebase and runs the tests. A task passes only if all tests pass — no partial credit, no human grading.
# Simplified evaluation flow
1. Load repository at commit before issue creation
2. Present issue text to model (with repo structure/context)
3. Model outputs a unified diff patch
4. Apply patch to working directory
5. Run project test suite (pytest, unittest, etc.)
6. Record pass/failThe original SWE-bench paper (2023) introduced three splits: SWE-bench (full set), SWE-bench Lite (300 instances, faster iteration), and SWE-bench Verified (500 instances with human-validated fixes). Lite and Verified are the de facto standards for leaderboard comparisons today.
Why it matters for LLM evaluation
Most coding benchmarks — HumanEval, MBPP, CodeContests — test algorithmic problem-solving in isolation. They measure whether a model can write a function that passes unit tests for a self-contained specification. SWE-bench measures something different: can the model operate inside a real codebase?
This distinction reveals capabilities that synthetic benchmarks miss:
- Repository navigation: Finding relevant files across directories, understanding import chains, locating configuration
- Context synthesis: Piecing together information from multiple files to understand the bug
- Minimal editing: Making surgical changes rather than rewriting entire modules
- Test awareness: Understanding existing test patterns and ensuring the fix doesn’t break unrelated functionality
- Dependency reasoning: Handling version constraints, deprecated APIs, and cross-module effects
A model that scores 90% on HumanEval might score 15% on SWE-bench. The gap exists because real engineering is about context, not just code generation.
Concrete example: a Django issue
Consider this real SWE-bench instance from Django (simplified for clarity):
Issue: “Admin list_filter with choices shows all choices even when filtered by another field”
Repository state: Django 4.2, django/contrib/admin/views/main.py and related files
Failing test (added by the Django team to reproduce the bug):
def test_list_filter_choices_respect_other_filters(self):
# Create objects with different status and category combinations
Article.objects.create(status="published", category="tech")
Article.objects.create(status="draft", category="tech")
Article.objects.create(status="published", category="life")
# Filter by category=tech, then check status filter choices
changelist = self.get_changelist({"category": "tech"})
status_filter = changelist.get_filters(request)[0] # status filter
choices = list(status_filter.choices(changelist))
# Should only show "published" and "draft" — not "archived"
# because no "archived" articles exist in category=tech
choice_values = [c["display"] for c in choices]
self.assertNotIn("archived", choice_values)The model receives the issue text, the repository file tree, and typically the contents of 5-15 relevant files (selected by retrieval or provided as context). It must produce a patch like:
--- a/django/contrib/admin/views/main.py
+++ b/django/contrib/admin/views/main.py
@@ -142,7 +142,10 @@ class ChangeList:
def get_filters(self, request):
...
for filter_spec in self.list_filter:
- choices = list(filter_spec.choices(self))
+ # Pass current query params so dependent filters can narrow choices
+ choices = list(filter_spec.choices(self, request.GET))
if choices:
filter_specs.append((filter_spec, choices))The fix requires understanding how ChangeList constructs filters, how request.GET carries the current filter state, and how the choices method on RelatedFieldListFilter uses that state to limit displayed options. This is not a reasoning task you can solve by reading the issue alone — it demands codebase awareness.
What the numbers actually tell you
Leaderboards report resolve rate: percentage of instances where the model’s patch makes all tests pass. As of early 2025, top scores on SWE-bench Verified range from 30-55% depending on the agent architecture and compute budget. SWE-bench Lite scores run higher (40-65%) because the subset favors more approachable tasks.
But resolve rate alone is misleading. Consider these failure modes that still count as “0%”:
| Failure mode | What happened | Why it matters |
|---|---|---|
| Wrong file | Model edits views.py but bug is in filters.py |
Retrieval/navigation failure |
| Syntax error | Patch doesn’t apply or produces invalid Python | Basic code generation failure |
| Test regression | Fixes target issue but breaks 3 unrelated tests | Lack of systems thinking |
| Overfix | Rewrites entire class when a one-line change suffices | Poor minimal-editing discipline |
| Hallucinated API | Calls method that doesn’t exist in this Django version | Version/context awareness gap |
A model with 40% resolve rate that mostly fails on “wrong file” has a different problem than one that mostly fails on “test regression.” The former needs better retrieval; the latter needs better verification loops.
Common misconceptions
“SWE-bench measures coding ability”
It measures software engineering ability in a specific context: Python, established open-source projects, issues with existing reproduction scripts. It does not measure greenfield development, architectural design, performance optimization, or cross-language work. A model that excels at SWE-bench may still struggle to design a new service from scratch.
“Higher resolve rate = better model for my use case”
If you build a code assistant for React frontend work, SWE-bench (Python backend) correlates weakly with your actual needs. The benchmark favors models that understand Django ORM internals, pytest fixtures, and Python packaging quirks. Domain mismatch matters more than a 5-point leaderboard gap.
“The test suite is the ground truth”
The test suite validates the original maintainer’s fix, not all possible correct fixes. A model might produce a valid alternative solution that the existing tests don’t cover — or worse, the tests themselves might be incomplete. SWE-bench Verified mitigates this by human review, but the fundamental limitation remains: passing tests is necessary but not sufficient for correctness.
“Agents solve SWE-bench; LLMs don’t”
The benchmark is agnostic to architecture. Early submissions used single-turn LLM calls with large context windows. Modern top scores come from agent systems: retrieval → planning → editing → test execution → repair loops. The model is one component. Attributing the score to “the LLM” obscures the system design that actually matters.
“SWE-bench is saturated”
Resolve rates have climbed from ~1.9% (GPT-4, zero-shot) to ~55% (best agents, 2025). But the ceiling is not 100%. Many instances involve ambiguous issues, missing reproduction steps, or fixes requiring multi-file refactors that exceed current context windows. The benchmark also grows — SWE-bench Multilingual adds JavaScript, Go, Rust, and Java tasks. Saturation claims usually confuse “current leaderboard top score” with “benchmark ceiling.”
What to look for when evaluating agents on SWE-bench
If you’re assessing an agent framework for your own use, run a stratified subset rather than chasing the full leaderboard:
- Sample by repository — Test against the specific frameworks you use (Django, FastAPI, pandas, etc.)
- Sample by difficulty — SWE-bench tags instances by estimated fix lines and file count. Verify performance on both 1-file and 5+ file tasks.
- Measure repair loops — How many test-run → edit cycles does the agent need? A 40% resolve rate in 3 cycles is more deployable than 45% in 15 cycles.
- Track false positives — Patches that pass tests but introduce semantic bugs (security regressions, performance cliffs) are dangerous in production.
- Cost per resolve — Compute cost varies wildly. A 50% resolve rate at $50/task is a different product than 50% at $2/task.
Where SWE-bench fits in your evaluation stack
SWE-bench is a system-level integration test for coding agents. It sits above unit-level benchmarks (HumanEval, MBPP) and below production A/B tests. Use it to:
- Compare agent architectures (retrieval strategy, planning, verification loops)
- Regression-test your agent after prompt/tool changes
- Identify capability gaps (navigation, multi-file editing, test awareness)
Don’t use it to:
- Select a base model for a non-Python codebase
- Estimate production defect rates
- Replace code review
The practical takeaway
SWE-bench changed the conversation from “can the model write a function?” to “can the model fix a bug in a codebase it didn’t write?” That shift forced the field to build retrieval, tooling, and verification loops — the infrastructure that makes coding agents useful in practice.
If you’re building on top of LLMs for software engineering, run SWE-bench Lite (300 instances, ~$50-200 depending on your agent) before you ship. The failures will teach you more about your system’s gaps than any leaderboard.