A benchmark of real GitHub issues from popular Python repositories where an AI system is given an issue and the repo state and asked to produce a code patch. Success is defined by whether the patch passes the same test cases the human-authored fix passed. Published in 2023 by researchers at Princeton and Chicago.

What is the difference between SWE-bench and SWE-bench Verified?

SWE-bench Verified is a 500-task human-audited subset released by OpenAI in mid-2024. Professional software engineers reviewed each task to ensure it was actually solvable, that tests were appropriate, and that the reference solution was reasonable. Verified scores are systematically higher than original SWE-bench scores and are the version most current tools cite.

Does SWE-bench measure general coding ability?

No. It measures the ability to produce a patch that passes tests for a Python GitHub issue. That excludes non-Python languages, cross-repo work, greenfield development, architectural decisions, code style and review-passability, and any multi-turn iteration. Extrapolating from a high SWE-bench score to general coding ability is a common misuse.

Why do different tools get different scores using the same model?

Because the score is for a model plus a specific agent scaffold plus specific prompts plus specific tools. The same underlying model can score meaningfully differently through OpenHands versus a first-party scaffold. When a launch post cites a score, that is a score for the whole stack — not the model alone.

What other benchmarks should I look at?

Aider's Polyglot leaderboard for multi-language edit tasks, Terminal-Bench for multi-step terminal-agent tasks, and LiveCodeBench for contamination-resistant leetcode-style problems refreshed monthly. Also run the tool against your own five hardest recent bugs — that small, non-generalisable, but for-you highly diagnostic dataset often matters more than any public benchmark.

SWE-bench Methodology Audit: What the Score Actually Measures

SWE-bench is the benchmark every serious AI coding-tool team cites. It appears in launch posts, on pricing pages, in board decks. But the number that gets quoted — "42% on SWE-bench Verified", "68%", the score gets higher each quarter — obscures a lot of what the benchmark actually measures and, more importantly, what it does not.

This is a methodology audit. Not a takedown — SWE-bench is a materially better benchmark than what came before it, and the Princeton team that maintains it has been careful and transparent about updates. But the way the number is used in marketing consistently outruns what the number supports. Reading the leaderboard well requires knowing what is behind it.

What SWE-bench actually tests

The original SWE-bench paper was published in 2023 by researchers at Princeton and Chicago. It collected 2,294 real GitHub issues from 12 popular Python repositories (django, matplotlib, requests, and so on), paired each issue with the actual pull request that resolved it in the real repository, and constructed a test set where the AI system is given the issue text and the repository state at the pre-fix commit and asked to produce a code patch. The patch is judged to have "solved" the issue if it passes the same test cases the human-authored fix passed.

That is a much better benchmark than most that came before. It measures end-to-end capability on real bugs from real codebases, not toy problems.

But the details matter. Three things are worth understanding before quoting a number.

The Verified split, and why it exists

The original SWE-bench had problems. OpenAI released SWE-bench Verified in mid-2024 — a human-audited subset of 500 tasks in which each task was reviewed by professional software engineers to ensure it was actually solvable given the information provided, that the tests were appropriate, and that the "correct" solution was reasonable.

Verified is now what almost every serious cite uses. It has better hygiene, higher reliability, and less noise. But it is also a smaller and easier subset — its scores are systematically higher than raw SWE-bench scores, and comparing "our system scored 68% on Verified" to "another system scored 42% on the original SWE-bench" is not a like-for-like comparison. The Princeton leaderboard shows both.

Tasks it does not include

SWE-bench in all its variants is limited by construction to:

Python only. No TypeScript, no Rust, no Go, no C++, no SQL. Systems that specialise in other languages are not represented. The Multi-SWE-bench work extends the idea to more languages but is younger and less cited.
Repository-scoped bugs. The task is always "given this repo and this issue, produce a patch". Cross-repo work, dependency-graph changes, or infrastructure work are outside the scope.
Test-defined success. A task is solved if the pre-existing tests pass. There is no evaluation of code quality, style, review-passability, or side effects that were not caught by the test suite. In real practice, an engineer who writes a change that passes tests but is unmaintainable has not "solved" the issue.
Single-shot patches. The task is one issue, one patch attempt. The multi-turn back-and-forth of real code review is not part of the benchmark.

None of these are flaws of SWE-bench. They are choices the maintainers made to keep the benchmark tractable. But they are the reason SWE-bench Verified scores overstate real-world capability on any work that is not Python bug-fixing.

The most common misuses

Three specific misuses appear over and over in marketing materials.

Conflating agent scaffold with model quality. A "SWE-bench Verified 68%" score is a score for a specific model plus a specific agentic scaffold plus specific prompts plus specific tools. Change any of these and the score changes. The OpenHands scaffold, for example, has consistently gotten higher scores from a given underlying model than many first-party scaffolds. When a launch post says "GPT-X scored 68%", that is not the model's score — that is the score for the scaffold the launcher ran the model through.

Comparing against non-current numbers. The frontier moves quarterly. A 12-month-old SWE-bench score is not comparable to a current one; the underlying models on both sides have moved. Comparisons across cohorts are common and are usually misleading.

Extrapolating to general coding ability. A high SWE-bench score does not mean the tool will do well on TypeScript, on infrastructure changes, on multi-file refactors, on greenfield work, on architectural decisions, or on any of the other 80% of what a working software engineer does. It means the tool can produce a patch that passes tests for a Python GitHub issue — a real skill, but one skill.

What to look at alongside SWE-bench

If you want a fuller picture of a coding agent's capability, no single benchmark suffices. A defensible sampling:

Aider's Polyglot leaderboard — multi-language edit tasks, updated with every Aider release. Better representation of the non-Python work most engineers actually do.
Terminal-Bench — multi-step terminal-agent tasks. Closer to what CLI-based assistants actually do.
LiveCodeBench — contamination-resistant leetcode-style problems, refreshed monthly to reduce training-data leakage.
Your own repository test. Run the tools you are evaluating against your own five hardest recent bugs. That is a small, non-generalisable, but for you highly diagnostic dataset.

The honest reading

SWE-bench Verified is the best-known measure of coding-agent capability in 2026, and it deserves the citation weight it gets. But the number in a launch post is not the full story. Read the full leaderboard, note the agent scaffold and model versions, and cross-reference against at least one non-SWE-bench evaluation before making a purchase or an architecture decision on the strength of the score.

Sources

SWE-bench paper (Jimenez et al., 2023) — arxiv.org/abs/2310.06770
SWE-bench leaderboard — swebench.com
OpenAI Introducing SWE-bench Verified — openai.com/index/introducing-swe-bench-verified
Multi-SWE-bench (2025) — arxiv.org/abs/2504.02605
OpenHands agent scaffold — github.com/All-Hands-AI/OpenHands