What coding benchmarks actually say under the hood

Most people meet a coding benchmark as a single number. "This agent solves 45% of real GitHub issues." The number gets quoted, a leaderboard shifts, and everyone moves on.

But a whole family of benchmarks has grown up around the original, SWE-bench, and each new one changes just one thing and measures what happens. Read them side by side and they stop being a scoreboard. They start to read like a manual for getting better results at lower cost. The surprising part: changing the model is usually the last and smallest lever, not the first.

Here is the family, and what each one was really testing.

The SWE-bench family, each entry changing one thing. Each name links to its paper.

Pull the threads together and the cost-and-performance story lands on a handful of clear, repeating levers. Here they are, cheapest first.

1. Fix the plumbing before you touch the model

This is the biggest and cheapest win in the whole table. SWE-agent's interface redesign took the same model from 11% to 18% on SWE-bench Lite. Claw-SWE-Bench watched one fixed model jump from 19% to 73% on 350 tasks purely from better packaging, mostly by letting the agent edit files directly instead of hand-typing a patch as a block of text. That pair is the extreme of their sweep; across all of it, choosing the harness moved the score about as much as choosing the model. SWE-Edit split "look at the code" and "change the code" into two helpers with their own clean workspace, and got both a small accuracy bump and a real cost cut at the same time. That last part is rare: almost everywhere else, accuracy and cost trade off against each other.

None of this needs a smarter model or more context. It is wiring. If your setup has any mechanical friction, like fragile patch formatting or one step's mess leaking into the next, that is the first thing to fix.

2. The right context beats more context, and the wrong context costs you

SWE-ContextBench found that well-matched context improves accuracy and cuts cost, especially on hard tasks. But badly-chosen context gave little benefit, and sometimes made things worse. SWE-Skills-Bench found that 80% of the knowledge documents it handed the agent did nothing at all.

3. Not every task deserves the expensive treatment

SWE-Bench 5G showed that heavy domain context helped a lot on tasks that genuinely needed it, and did essentially nothing on plain mechanical fixes. Put that next to lever #2 (the wrong context can hurt) and the move is obvious: add a cheap triage step. Quickly sort whether a task is mechanical or judgment-heavy, and only spend your expensive tools, deep retrieval, extra reasoning turns, on the ones that need it. Applying the full treatment to every task wastes money on the easy ones and buries your real gains on the hard ones.

4. Bigger models stop paying off fast

The cost curves make this concrete. On the Holistic Agent Leaderboard's 50-task SWE-bench Verified Mini, with the same SWE-agent scaffold, Claude Sonnet 4.5 solved 72% for about $464 while Claude Opus 4.1 solved 61% for about $1,351. The priciest model on the shelf cost three times as much and solved less. If you care about cost per solved issue rather than the peak score, the honest choice is usually a cheaper base model with a good scaffold and good context around it, not the priciest model on the shelf. Worth checking before you assume you need a frontier model at all.

5. Put stable knowledge in the model, changing knowledge in retrieval

Every token you put in a prompt gets paid for on every single call, forever. Something you bake into the model itself gets paid for once, at training time. So the split writes itself. Things that rarely change, like known design patterns and architecture choices, are good candidates to train in. Things that change every week, like the current API surface or this sprint's file layout, belong in retrieval where they can update cheaply.

Get the split backwards in either direction and you lose. Either you pay a recurring token tax for something that never changes, or you bake in something that goes stale and quietly misleads the model. A related study on teaching a model to spot a pattern found that training it in beat retrieval alone, because recognizing a pattern in the wild is a different skill from being able to look up its definition.

6. A cheap, honest test loop is its own saving

Slimmed-down benchmarks, such as the 50-task SWE-bench Verified Mini, are built to track the full 500-task set at a tenth of the cost. That is not about your live system. It is about your own development budget. A fast, trustworthy test loop lets you run more experiments for the same spend, so you can actually answer questions like "is this extra layer earning its keep?" If every test cycle is expensive, you run them rarely, and you end up trusting numbers you did not really check.

7. Watch for wins that are not real

SWE-ABS found that about 1 in 5 "solved" patches on a top leaderboard were false alarms once the tests were tightened. Separately, one benchmark had a data leak that inflated every model's score until it was caught and patched. The cost angle: if your scorecard is gameable or leaky, you will spend real engineering effort chasing a number that does not reflect real ability. That is the most expensive kind of waste, because it stays invisible until someone bothers to check.

Updated 14 October 2026: added links to every paper and replaced three figures we could not trace to a source (SWE-agent "tripled", a $1,400 vs $8 head-to-head, and a half-point mini-benchmark) with the sourced numbers above.

Takeaway

Read as a group, the SWE-bench family quietly agrees on an order of operations for "solve more, spend less":

1. Fix the plumbing first. It is the cheapest lever and often the biggest.
2. Feed the right small amount of context, and be willing to feed none.
3. Triage tasks, so the expensive treatment only lands where it pays off.
4. Split stable knowledge (into the model) from changing knowledge (into retrieval).
5. Keep a cheap, honest test loop so you can tell what is actually working.

Notice what is missing from the top of that list: buy a bigger model. It is on the list, but it is near the bottom, and it is usually the smallest and most expensive win. That is the whole point of reading benchmarks under the hood instead of off the leaderboard. The headline number tells you who is ahead. The design underneath tells you how to get there for less.

It is also the thesis we build on. Prefex is a proxy that sits in front of your model and works these levers automatically: routing easy turns to cheaper models, keeping the prompt cache warm, and trimming context that is not pulling its weight. Better plumbing and better context, without changing the model you already trust.