We measured our own tool. It said less than we hoped.

What we were trying to do

The premise is simple enough. A coding agent dropped into an unfamiliar repository spends its first several thousand tokens rediscovering things the repository already knows: which files everything depends on, which change together, which are one person's work and nobody else's, where the tests actually live.

That information is derivable, deterministically, without a model. So derive it once, keep it fresh, and hand it to the agent instead of making it re-infer the same map every session.

The bet is that better context beats more context, and that the cheapest token is the one you never send. Nothing about that is novel — it is roughly what the benchmark literature has been saying for two years. The question was whether it holds on real repositories with a real agent, and by how much.

Why we stopped trusting our own numbers

For a while we answered "by how much" with a model. Token counts from a scan, an estimate of what an agent would otherwise read, a plausible dollar figure. Every input was real; the conclusion was arithmetic.

That is a reasonable way to size an idea. It is not evidence, and we were increasingly presenting it as though it were. The honest move was to stop quoting it and go measure.

So we built the harness we should have built first: clone a repository at a known-broken commit, run the real agent twice — once on a clean checkout, once after ReadyBase has configured it — and grade both by running the project's own tests. Cost comes from the agent's own reported usage, not our estimate. A task counts as solved only when the failing test passes and the previously-passing tests still do.

What it actually said

Fifteen instances, two arms, two trials each. Sixty real agent runs.

The cost result is the one we can defend. It survived every subsequent fix to the harness unchanged, which is its own kind of evidence.

The resolution result is not a null result. It is worse than that, and more interesting.

The corpus could not answer the question

When we broke the fifteen instances down by outcome, this is what we found:

• 6 solved by both arms, every trial
• 8 solved by neither arm, ever
• 1 where the arms differed

Fourteen of fifteen instances were decided before the experiment began. Either trivially solvable regardless of context, or out of reach regardless of context. There was almost nothing in between for an intervention to move.

That is not evidence that ReadyBase does not help agents solve more. It is evidence that we could not have detected it either way. Adding trials would not have fixed it. Adding more instances from the same pool would not have fixed it. The corpus had no measurement range on the outcome we cared about most, and we had been about to spend real money running more of it.

We later worked out what it would take: roughly fifty instances that are genuinely uncertain, which given our observed yield means screening several hundred candidates. And that is only if the effect is large. A modest effect is out of reach at any sample size we could afford. Knowing that a claim is unaffordable to prove is more useful than a cheap claim we could not stand behind.

The bug that would have told us what we wanted to hear

This is the part that changed how we work.

Agent runs have a spend cap. When a run hits it, the harness records the run and moves on. Our screening logic counted those as failures — the task was attempted and not solved.

They are not failures. A run stopped at a budget cap is a measurement that did not finish. And the error is not symmetric: a run gets killed precisely because it was still working, so the arm that explores more accumulates more of them.

We caught it in the direction that made things look worse than they were — an instance filed as "never solved" when all three of its attempts had simply been cut off. Then, checking the other direction, we found the one that matters:

One instance completed all three trials cheaply and reliably under a lower cap. Under a higher cap, two trials were cut off and one succeeded. Counting the cut-off trials as failures reads that as one out of three — a partial success rate, which is exactly the profile our screening used to select instances as informative.

So a saturated instance would have been selected into the corpus on the strength of variance that was not real. Build a benchmark set that way and regression to the mean can produce an effect in either direction out of nothing — including a positive one. That is the result we would have been least equipped to question, because it is the one we wanted.

What we changed our minds about

Cost is the claim we have. Single-digit, measured, significant, reproducible. We would rather publish 7.4% with a confidence interval than a larger number with a footnote. If that is not enough to be interesting, that is useful to know now rather than after a customer runs their own numbers.

Some claims are unaffordable, and should be dropped rather than weakened. "Agents solve more with ReadyBase" is plausible, and we cannot currently price the experiment that would establish it. So we do not say it.

The instrument deserves more scrutiny than the product. A wrong product feature is visible; someone complains. A wrong measurement is silent, and it makes every decision downstream of it slightly wrong forever. We now spend more review effort on the harness than on the thing it measures.

The most valuable part of the work is not measurable by benchmarks at all. Nothing in a SWE-bench-style corpus tells you whether an agent should have been allowed to edit that file, who owns it, whether one person wrote all of it, or whether anything downstream breaks. Those are governance questions, and they are where we have been putting the effort — not because benchmarks were unkind to us, but because that is what the benchmark work made obvious was missing.

Where this leaves us

ReadyBase is a deterministic analysis pass over a repository — dependency graph, co-change history, ownership and authorship provenance, test correlation, structural drift — that produces context an agent can use and constraints an agent must respect. No model in the loop, so the same repository produces the same answer twice.

What we can tell you it does: it makes agent sessions modestly cheaper, and it gives you a mechanical way to say "not this file, not without a human". What we cannot yet tell you: whether it makes agents meaningfully better at fixing things. We tried honestly to find out, our own corpus was not up to the question, and we would rather say so than round it up.

If you are building something similar, the transferable lesson is small and slightly deflating. Build the thing that can prove you wrong before you build the case that you are right, and then read its output as carefully as you would read a competitor's.