Not long ago, evaluating an AI model’s ability to code usually meant giving it a self-contained problem: write a function, complete a method, or make a small collection of unit tests pass.

That tells us something useful. It does not tell us whether the model can work inside a real software system.

A real bug rarely arrives as a clean specification. It arrives as a GitHub issue written by someone who already knows the project. The relevant code may be spread across several files. The visible failure may be far from its cause. Dependencies, tests, conventions, and years of design decisions all constrain what a good solution looks like.

SWE-bench was created to evaluate that harder problem.

Instead of asking an AI to produce an isolated snippet, SWE-bench gives it a real repository at a particular point in its history and a real issue that was later resolved by its maintainers. The AI must inspect the codebase, understand the problem, edit the right files, and produce a patch. That patch is then applied and tested inside a reproducible environment.

The result is a deceptively simple metric: what percentage of real software issues did the system resolve?

SWE-bench has become one of the most influential measures of AI coding ability. It has also become a case study in how difficult it is to measure software engineering at all.

From a GitHub issue to a benchmark task

The original SWE-bench paper, released in 2023, collected 2,294 problems from 12 popular Python repositories. Each problem connects four pieces of history:

  1. A GitHub issue describing a bug or requested change.
  2. The repository at the commit before the maintainers’ fix.
  3. The pull request or commit that eventually resolved the issue.
  4. Tests that distinguish the repository before and after the fix.

For an evaluation, the model receives the issue description and access to the repository at the earlier commit. It does not receive the maintainer’s patch. An agent around the model can search files, inspect definitions, edit code, and run commands. Its final output is a diff.

SWE-bench then applies that diff in a Docker container and runs two important groups of tests:

If the required tests pass and the existing behavior remains intact, the task is counted as resolved.

issue + repository at the base commit
                 ↓
      model and agent scaffold
                 ↓
          candidate patch
                 ↓
     reproducible Docker environment
                 ↓
    failing tests pass, old tests remain green
                 ↓
          resolved or unresolved

This is a meaningful step beyond code generation. The model must localize the problem, recover relevant context, reason about an unfamiliar system, make a coordinated change, and validate it through execution.

The first results showed how difficult that was. The strongest model in the original paper resolved only 1.96 percent of the tasks.

How SWE-bench decides whether a patch is correct

SWE-bench does not normally ask another language model to judge the answer, and it does not grade a patch by measuring how closely it resembles the maintainer’s original code. Its primary grader is an execution-based test harness.

For every task, the harness starts from the historical base commit, applies the agent’s candidate diff, and runs the repository’s evaluation commands inside the task’s Docker environment. The task is marked resolved only when the required FAIL_TO_PASS tests now pass and the PASS_TO_PASS tests remain green.

Three artifacts are easy to conflate:

This distinction is one of SWE-bench’s strengths. Two implementations may look completely different while producing the same correct behavior. By executing tests instead of comparing text, the grader can accept an alternative solution.

It is also the source of the benchmark’s deepest limitation.

SWE-bench treats tests as an executable specification. The grading result is therefore only as complete and fair as those tests.

A narrow test suite may accept a patch that handles the visible examples without implementing the intended behavior. An overly specific test may reject a valid design because it assumes details not stated in the issue. Tests may omit performance, security, maintainability, concurrency, or architectural constraints that mattered to the maintainers.

The harness reports more than a binary score. Its per-instance artifacts include the test output, applied patch, execution log, and report. These are important because an unresolved task is not necessarily an incorrect patch: the container might fail to build, a dependency may be unavailable, or a test may time out. Serious evaluations should separate behavioral failures from infrastructure failures rather than collapse both into one number.

SWE-bench evaluates a system, not only a model

It is tempting to read a SWE-bench score as an IQ test for a language model. That is not what it measures.

Between the issue and the final patch sits an agent scaffold: the software that decides how the model interacts with the repository. It determines which tools are available, how files are presented, how commands are executed, when tests are run, how much context is retained, and when the agent stops.

The model matters enormously. So do the scaffold, prompts, inference budget, retry policy, environment, and even the amount of time allowed for each task.

This is why two systems built on the same model can produce different scores, and why numbers from different leaderboards or benchmark variants are not automatically comparable. A result should be read as:

This model, inside this agent, with this tool interface and compute budget, resolved this fraction of this version of the dataset.

That sentence is less convenient than a single number. It is much closer to the truth.

The main versions

“SWE-bench” now refers to a family of related datasets.

There are also independent extensions such as SWE-bench-Live, SWE-rebench, Multi-SWE-bench, and SWE-Bench Pro. They address different limitations of the original dataset: static tasks, Python-heavy coverage, contamination, short task horizons, and limited repository diversity.

When a model announcement says “SWE-bench,” the first question should therefore be: which one?

Exploring the data on Hugging Face

The official datasets are collected under the SWE-bench organization on Hugging Face. The most useful starting points are:

You can inspect records in the browser or load them with the Hugging Face datasets library:

from datasets import load_dataset

dataset = load_dataset(
    "SWE-bench/SWE-bench_Verified",
    split="test",
)

task = dataset[0]
print(task["instance_id"])
print(task["problem_statement"])
print(task["repo"])
print(task["base_commit"])

A record typically contains the repository, base commit, issue text, human patch, test patch, and identifiers used by the evaluation harness. The human and test patches are needed to construct and grade the benchmark; they should not be shown to an agent being evaluated.

Loading the dataset is easy. Reproducing an evaluation is the expensive part.

Running an evaluation

The official implementation lives in the SWE-bench GitHub repository. It uses Docker so that each patch is tested against the appropriate dependencies and historical environment.

After installing Docker, the basic setup is:

git clone https://github.com/SWE-bench/SWE-bench.git
cd SWE-bench
pip install -e .

Before evaluating a model, it is useful to validate the harness on a single known human patch:

swebench eval verified \
  --gold \
  -i sympy__sympy-20590 \
  --run-id validate-gold

Here --gold tells the harness to use the reference patch. If this run fails, the environment should be fixed before interpreting any model result.

A model or agent normally writes its predictions to a JSON or JSONL file. Each prediction identifies a task and contains the generated diff. The exact fields are documented in the official evaluation guide.

The predictions can then be evaluated with:

swebench eval verified \
  --predictions predictions.jsonl \
  --run-id my-agent-v1 \
  --workers 4

The evaluator applies each patch, runs the task’s test command inside its container, and writes per-instance logs plus an aggregate report. The logs make an important distinction between an unresolved task and an infrastructure failure. A patch may fail because it is wrong, but a run may also fail because an image could not be built, a dependency disappeared, a test timed out, or the host architecture behaved differently.

The harness also caches results by run ID and instance ID. Reusing the same run ID with a different patch can silently reuse an earlier result, so each distinct experiment should receive a distinct run ID.

This is not a lightweight benchmark. The maintainers recommend an x86-64 machine with substantial storage, memory, and CPU capacity. The patches and tests may be deterministic, but their execution still depends on architecture-specific Docker images, compiled dependencies, and system libraries. ARM64 machines such as Apple Silicon Macs can build compatible images locally, though that evaluation path remains experimental. For a first experiment, use one Lite or Verified task, one worker, and a reference patch before attempting a full run.

What the resolved rate captures—and what it misses

Execution makes SWE-bench unusually concrete. The patch must compile or import. The tests must run. The required behavior must change. Existing behavior must remain intact.

But a passing test suite is an observation, not a proof.

Tests can be too narrow and allow an incorrect patch. They can also be too specific and reject an alternative solution that correctly satisfies the issue. The issue itself may omit an assumption that was obvious to the original maintainers. Environment reconstruction can introduce failures unrelated to the candidate patch.

Researchers have now measured these effects. One empirical study of accepted SWE-bench patches found patches that passed the benchmark’s tests but failed additional developer-written tests or behaved differently from the human fix. A later mutation-guided analysis strengthened test suites and reduced the reported resolved rates of leading agents.

There is also contamination. SWE-bench is built from public GitHub history. Once an issue, its solution, and the benchmark itself have appeared in training data, it becomes difficult to distinguish problem-solving from prior exposure.

In February 2026, OpenAI reported that SWE-bench Verified no longer provided a reliable frontier signal. Its audit found material specification or test problems in 59.4 percent of a selected, failure-heavy subset of 138 tasks, and evidence of training exposure among frontier models. That number does not describe 59.4 percent of the entire Verified set, but it suggests that many of the remaining failures could not be interpreted cleanly as model limitations.

OpenAI recommended SWE-Bench Pro, which used longer tasks and public, held-out, and commercial repositories. Five months later, OpenAI published another audit, estimating that roughly 30 percent of Pro’s public tasks were broken.

The lesson is not that executable benchmarks are useless. It is that realistic evaluation inherits the ambiguity of real engineering.

A benchmark under pressure

SWE-bench changed how the field talks about coding models. It replaced “Can the model write this function?” with a more consequential question: “Can an agent enter an unfamiliar repository and resolve an issue that a maintainer once had to solve?”

That shift mattered. It pushed models toward repository exploration, tool use, iterative testing, and coordinated edits. It encouraged reproducible environments and inspectable patch trajectories. It exposed the difference between code completion and software engineering.

Its success also concentrated enormous optimization pressure on one public metric. Models trained on repository data may encounter the tasks. Agent developers tune scaffolds around the benchmark’s structure. As scores approach the ceiling, a few ambiguous tests can determine the leaderboard. The benchmark becomes easier to optimize and harder to interpret at the same time.

The next generation of evaluations is moving in several directions: continuously refreshed tasks, private repositories, manually authored specifications, stronger generated tests, multilingual projects, longer-horizon changes, and finer-grained measures of exploration and diagnosis.

None is a complete solution. Fresh public tasks eventually become old public tasks. Private tasks improve contamination resistance but reduce transparency. Human review improves quality but is slow and expensive. Stronger tests still cannot encode every design constraint that matters.

This is the deeper value of SWE-bench. It did more than give the field a leaderboard. It made the difficulty of evaluating software engineering impossible to ignore.

A patch can pass its tests without reflecting the right understanding. A benchmark can produce a precise score without producing a precise claim.

SWE-bench remains one of the best places to learn how AI agents work on real repositories. It is also a reminder to inspect what a benchmark contains, how an agent was run, and what the resulting number can actually support before calling a problem solved.

Further reading