A coding-agent leaderboard I came across this week scores every agent-and-model pairing with one clean number, and buries the actual methodology in a footnote: that number is an average across three separate benchmarks, weighted equally. Worth pulling apart, since it’s a genuinely different thing than any single benchmark I’ve covered so far.
Why anyone bothers combining benchmarks instead of just picking one: I’ve already gone through why SWE-bench and Terminal-Bench measure genuinely different skills — real GitHub issue resolution versus broad terminal and systems competence. A model that’s excellent at one can be mediocre at the other, and reporting either score alone tells you about that one narrow slice, not “how good is this agent” in any general sense. Averaging several benchmarks together is an attempt to approximate a more rounded answer than any single test can give.
What actually determines the final ranking, beyond the models themselves: the specific choice of which benchmarks get included, and how heavily each one counts. Weight software-engineering benchmarks more heavily, and agents optimized specifically for that narrow task climb the list. Weight general terminal competence more heavily instead, and a more broadly capable but less specialized agent can pull ahead. Equal weighting, which is what that particular leaderboard uses, is itself a real methodological choice with real consequences — it’s treating “resolving a GitHub issue,” “general terminal and systems work,” and whatever a third benchmark specifically measures as equally important to what “best overall” means, which is a defensible stance, but a stance, not a neutral default.
Why this matters more than it looks like it should: a single score with no visible methodology invites treating it as an objective fact — “Model X scored 62.2, Model Y scored 59.7, X is better.” A composite score is doing something closer to “X did better across this particular mix of tests, weighted this particular way.” Change the mix, or the weighting, and a different model could easily come out ahead using the exact same three underlying benchmark scores. That’s not a flaw in the specific leaderboard I ran into — it’s just an inherent property of any composite metric, the same way a school’s overall ranking depends heavily on how much weight got put on test scores versus graduation rates versus everything else that went into it.
The practical habit worth building: when a leaderboard reports one clean number, look for what it’s actually built from before treating the ranking as settled. If the benchmarks included match the kind of work you actually do — say, mostly resolving real issues in an existing codebase rather than broad systems administration — a composite score weighted toward the other kind of task might rank a genuinely worse fit for you above a genuinely better one. The single number is a summary, and like any summary, it’s only as useful as knowing what got left out or downweighted to produce it.
