I don’t usually cover Devin here since it’s Cognition’s closed commercial product, not an open-source tool. But the story behind its original benchmark claim is genuinely useful context for anyone reading SWE-bench numbers on any agent’s marketing page, open-source ones very much included, so I’m making an exception.
When Devin launched, Cognition’s own blog post claimed it resolved 13.86% of issues end-to-end on SWE-bench, compared against a prior state-of-the-art baseline of 1.96% — a headline that read like an order-of-magnitude leap. Researchers went and checked the methodology. Devin’s 13.86% had been measured on a random 25% subsample of SWE-bench. The 1.96% baseline it was being compared against was measured on the full 2,294-problem set. Different test sets, same chart. When independent groups reran the comparison on equal footing, Devin’s apples-to-apples score came out closer to 9–10%. Still a genuine improvement over what came before, just not the leap the original framing implied.
I bring this up now specifically because it’s not an isolated methodology slip — it’s a symptom of a benchmark that has real, independently documented quality problems even in its most trusted form. SWE-bench Verified, the 500-issue, human-reviewed subset that most current scores actually refer to, was built by OpenAI’s own preparedness team specifically to filter out issues with vague descriptions or weak tests. Even so, later audits found solution leakage in a meaningful share of instances — cases where the actual fix, or something close to it, was already sitting in the GitHub issue thread the model gets to read while solving it. Separate reviews found that somewhere between 12% and 22% of “passing” patches on earlier scaffolds were actually wrong or incomplete, just not caught because the existing test suite wasn’t strict enough to notice. And several teams have found models scoring meaningfully lower — 5 to 10 points — on fresh, non-benchmark issues pulled from those exact same repositories, which points at some degree of memorization rather than pure generalizable reasoning.
None of that means SWE-bench is worthless. Frontier models still only clear somewhere between 54% and 81% on Verified, so it’s genuinely far from saturated, and it’s a lot closer to real engineering work than something like isolated function-generation benchmarks. It just means a single percentage, with no subset named and no comparison baseline specified, is close to meaningless on its own — which is exactly the gap Devin’s headline number fell into, intentionally or not.
I’m not bringing this up to single out Cognition specifically. It’s an easy mistake to make even without any intent to mislead, given how many overlapping SWE-bench variants exist under names that all start the same way. If you want the fuller breakdown of what SWE-bench actually measures and which subset is which, I wrote a separate explainer here. The short version for reading any agent’s landing page: which subset, what scaffold and tools were allowed, how many attempts per issue, and is the thing it’s being compared against measured the same way.
