I wrote about Devin’s benchmark methodology getting called out a while back, and the more I dug into that story the more I realized I’d never actually explained what SWE-bench is in the first place, even though I’d cited it a dozen times. So here’s the version I wish I’d had earlier.
What it actually tests: the original SWE-bench is 2,294 real GitHub issues, each paired with the actual pull request that fixed it, pulled from 12 popular Python repositories. An agent gets the issue description and the full repo at the commit right before the fix, and has to produce a patch. Success is measured by running the repository’s own test suite against that patch — pass the tests that should pass, and don’t break the ones that already did. It’s not a toy problem set; every task is something a real maintainer actually had to fix.
Why “SWE-bench” alone is an incomplete answer: almost nobody runs the full 2,294-issue set anymore, because most of the numbers you see cited are from a specific curated subset, and different subsets aren’t interchangeable. SWE-bench Lite trims it to 300 self-contained instances, mostly to cut evaluation cost. SWE-bench Verified — the one OpenAI’s own preparedness team put together, and the one most current model and agent scores refer to — is a human-reviewed 500-issue subset built specifically to remove issues with vague descriptions or weak tests that let wrong answers pass. Beyond those two, researchers have spun off smaller derivatives again: a 50-issue “verified-mini,” an 84-issue overlap set between Lite and Verified, language-specific and multimodal variants. A percentage with no subset named next to it is missing the single most important piece of context.
Even “Verified” has real known problems. Independent audits have found solution leakage in a meaningful chunk of instances — cases where the fix, or something close to it, was already sitting in the GitHub issue thread or comments the model gets to read. Other audits found that 12–22% of “passing” patches were actually wrong or incomplete, just not caught because the existing test suite wasn’t strict enough to notice. And several teams have found that models score 5–10 points lower on fresh, non-benchmark issues from those exact same repositories, which points at some degree of memorization rather than pure reasoning. None of this means the benchmark is useless. Frontier models still only clear 54–81% on Verified, so it’s far from saturated, and it’s still closer to real engineering than something like HumanEval’s isolated function-writing tasks. It just means the number needs a lot more context than a single percentage gives you.
The comparison problem, concretely: this is exactly what tripped up the Devin headline number I wrote about separately — one figure measured on a 25% random subsample, compared directly against a baseline measured on the full set. Different denominators, same chart. It’s an easy mistake to make even without any intent to mislead, because so many overlapping subsets exist under names that all start with “SWE-bench.”
So the next time a tool’s landing page leads with a SWE-bench score, the actual useful questions are: which subset, what scaffold and tools were allowed, how many attempts per issue, and is the comparison baseline measured the same way. All twelve underlying repositories are Python, too, so even a clean number tells you almost nothing about how a tool handles your actual stack if that stack isn’t Python. The score is a real signal. It’s just a much narrower one than the marketing usually implies.
