SWE-bench tests real GitHub issues, not toy problems

2,294 real issues from real repositories

The SWE-bench paper, submitted to arXiv on 10 October 2023, built its test set from 2,294 issue-and-pull-request pairs drawn from twelve popular Python repositories. A model is given an issue description and the surrounding codebase and must produce a patch; the patch is scored by running the repository's own test suite, so success requires the code to actually work rather than merely resemble the accepted fix. The paper reports that this was hard for the systems available at the time: it states that Claude 2 solved a mere 1.96 percent of instances, and that both proprietary models and a fine-tuned SWE-Llama model could resolve only the simplest issues, calling for reasoning that goes beyond ordinary code generation.

Why a human-verified subset followed

Pulling issues directly from real repositories brings real-world noise: some issues turn out to be unsolvable from the description given, underspecified, or paired with tests stricter than the issue itself required. The SWE-bench project repository addressed this with SWE-bench Verified, announced 13 August 2024 as a collaboration with OpenAI's Preparedness team: a subset of 500 problems that practising software engineers confirmed were actually solvable from the given issue text. Restricting to a verified subset changes what a resolve rate means; a score on Verified says a model can close issues known to be well-specified and solvable, while a score on the full set is depressed by instances no model could reasonably be expected to solve, whatever its skill.

What else a resolve rate depends on

The current SWE-bench site lists several further variants, including Lite, Multimodal and Multilingual editions and separate leaderboards, alongside a family of related tools such as SWE-agent for driving models through the repair loop. That proliferation matters for reading any single number: a resolve rate depends not only on the underlying model but on the surrounding scaffold that lets it browse the repository, run tests and iterate, so two labs quoting a SWE-bench Verified result can be comparing different agent harnesses as much as different models. The benchmark also measures single-issue, single-repository patches; it does not by itself establish how a model performs on longer-running or multi-repository engineering work.

Questions to carry into your own evaluation

• Is the quoted score from the full instance set or the human-verified subset, and does the source say which?
• What agent scaffold, tool access and number of attempts produced the number, separate from the underlying model?
• Does the task resemble SWE-bench's single-issue, test-checked patch format, or does it need broader engineering judgement the benchmark does not test?

SWE-bench's contribution was tying a code benchmark to an executable test suite instead of text similarity, and its Verified subset is a documented admission that the original issue set needed human review before a low score could be trusted as a measure of model skill rather than task noise.