I’ve cited a Terminal-Bench ranking in passing without ever explaining what it actually tests, and given how much ground I’ve already covered on what SWE-bench does and doesn’t measure, the comparison is worth making directly.
SWE-bench is deliberately narrow: real GitHub issues, paired with the pull request that actually fixed them, pulled from a fixed set of Python repositories. It measures one specific, well-defined skill — resolving a real-world software issue given the whole codebase as context.
Terminal-Bench, a joint project from Stanford and the Laude Institute, is built to measure everything else you’d plausibly do at a command line. Software engineering is one category among several — the current version’s 89 hand-crafted, human-verified tasks span roughly sixteen categories including system administration, scientific computing, security, data science, and machine learning workflows. A task might mean compiling a project, training a model, configuring a service, or debugging a broken environment — anything reachable through terminal commands and code execution, not just “here’s a bug, fix it.”
The structure per task mirrors SWE-bench’s rigor, just generalized: each one ships a natural-language instruction, a containerized Docker environment providing the actual working context, a verification test suite that checks the result programmatically, and a human-written oracle solution proving the task is genuinely solvable. Nothing is graded on vibes — an agent either satisfies the test suite running against a real terminal session or it doesn’t.
The versioning history is its own honest admission about benchmark saturation — the same problem I flagged with SWE-bench Verified. The original Terminal-Bench saw frontier model performance climb past 50%, so version 2.0 deliberately raised the difficulty to restore a meaningful gap between capable and merely competent agents, while also fixing tasks that turned out to be unsolvable for accidental reasons or had unreliable verification. Version 2.1 was a further refresh — patched Docker environments, corrected mismatches between a task’s instructions and what its test actually checked — specifically so a low score reflects an agent’s real limits rather than a broken test catching it on a technicality. Frontier agents currently clear less than 65% on 2.0’s 89 tasks.
The practical reason to know the difference: if a tool’s landing page cites a Terminal-Bench score instead of SWE-bench, that’s not a stronger or weaker version of the same claim — it’s a claim about a much broader kind of competence, general terminal and systems work rather than specifically fixing software issues in a codebase. Worth knowing which one you’re actually looking at before treating either number as a stand-in for “how good is this agent,” since neither one alone answers that question completely.
