Zero is not the same as "we did not look"
Every scoring system has a moment where an input is missing. The disk read fails, the repo has no commits yet, the toolchain is not installed, the run got killed halfway. What the system does in that moment decides whether its numbers mean anything.
The tempting answer is to score it zero and move on. That is almost always wrong, and it is wrong in a way nobody notices, because a zero looks exactly like a finding.
The same bug, nine times
We went looking deliberately, after being burned by it twice. Here is what one audit turned up.
| Where | What it reported | What was true |
|---|---|---|
| Co-change analysis | 0 file pairs change together | The parser split on a blank line git never emits. It returned zero on every repo, always. |
| Test correlation | 0/4 — your load-bearing files have no tests | The table it read had never been written. |
| Bus factor | 0 unique committers → 0/15, the single biggest penalty | The repo had no commits yet. A fact about git, not about the code. Six of twenty repos in one local scan were in that state. |
| Denylist coverage | Full marks — nothing to protect | The denylist was empty because generating it had failed. The score went up when the guard broke. |
| Benchmark runs | Instance never solved, 0 of 3 trials | All three trials hit the spend cap. Stopped, not defeated. |
| Grader | Patch did not fix it | The repo would not clone. The grader never ran. |
| Test oracle | Oracle test failed | The package did not compile, so the test never reported at all. |
| Cost accounting | $2.89 for this run | The token counts implied $0.23. Twelve times off, and nothing flagged it. |
| Coherence score | v2 -1 | A sentinel meaning "insufficient signal", printed raw on a 0-100 scale. |
Read the middle column as a user would. Every line is a confident, actionable claim. Every one of them is a lie of omission, and each suggests exactly the wrong fix: write more tests for a file whose test data was never collected, add contributors to a repo that has no commits, throw out a benchmark task that was never actually attempted.
The costume that fooled us longest
The instance we are least proud of is the one that had already been "fixed".
Someone noticed the bus-factor penalty firing on repos with no git history and corrected it — not by scoring zero, but by awarding a neutral middle value. Eight out of fifteen. Reasonable-looking. It avoided the false penalty, which was the point.
It also awarded eight points for a measurement that never happened, and then reported the total out of a flat 100. So a repo with genuinely mediocre dependencies and a repo with no toolchain installed came out with identical scores. Two situations needing opposite advice, rendered indistinguishable.
Three of them were hiding in the tests
This is the part worth stealing regardless of what you build.
We run a mutation gate: replace a scoring function's body with a stub that returns nothing, and confirm a test dies. A surviving stub means the test does not actually check the result. On this codebase the gate covered dozens of functions and had been green for months.
It had never been pointed at the benchmark package — the one producing every number we quote publicly. We added nine functions. Eight died immediately. One survived: a small helper that resets test files before the grader applies its reference tests. Gutting it changed no test result.
That helper exists because of a specific past incident, where a model edited test files, collided with the reference patch, and produced a spurious total failure — every task looking unsolved for a reason that had nothing to do with the model. The function guarding against a known, previously-encountered failure had no test checking its output, and the suite was green the entire time.
Two more of the same shape surfaced the same week:
- Our agent guard blocks edits to dangerous files and explains why. The explanation went to stdout; the protocol feeds stderr back to the agent. So every blocked agent got an unexplained failure. The test asserted on the stdout buffer and passed throughout.
- A test whose stated purpose was preventing configuration drift — its comment literally says a hand-maintained list would drift — carried a hand-maintained list. It drifted the first time anyone added an entry, and its failure message then blamed the wrong file.
A test pinned to the wrong surface is worse than no test. It does not merely miss the bug; it certifies the thing it is not checking, and it does so most confidently at the moment you need it.
The fix is a third state, not better arithmetic
Every one of these collapses three distinct facts into one number:
- Measured, non-zero. We computed it. Here is the value.
- Measured, zero. We computed it and the answer is genuinely none. This is a finding.
- Not computable. The input was absent, unreadable, or never produced. There is no measurement.
The first two are findings. The third is the absence of one, and it deserves opposite advice. Substituting a neutral value does not fix it — that is just a quieter fabrication.
What works is excluding the unmeasurable from both the earned points and the available points, then reporting what fraction could be measured at all. A repo scoring 34 out of an available 85 is not making the same claim as one scoring 34 out of 100, and reporting the second is how an unmeasured repo gets mistaken for a bad one.
One property made this safe to adopt across an existing scoring system: when everything is measurable, the new path returns the old number unchanged. So it cannot move a score that was fully measured — which means any score that does move was previously reporting points for something it never measured. That is a test, not a hope.
Two things we would do differently from the start
Make the third state awkward to ignore. Our coherence score had a sentinel for "insufficient signal" — a named constant, correct, documented, with a comment telling callers to render it as unavailable. It was consulted by no consumer anywhere, for its entire life. A bare number that every caller must remember to compare against a constant is unenforceable, and nothing fails when they forget. Return a value plus a flag, or a type the compiler makes people unwrap. A sentinel nothing honours is just a magic number.
Point the mutation gate at whatever produces your headline numbers first. Not the code you think is risky — the code whose output you quote. We had it aimed at scoring functions for months while the package generating our benchmark results had zero coverage, which is exactly where a family of these bugs was living.
Why this matters more than it used to
Agents make decisions from these numbers without a human reading them first. A person seeing "0 unique committers" on a repo they know has one commit thinks "huh, that's broken" and moves on. An agent reads it as a signal and acts.
The bill arrives quietly. A benchmark corpus gets filtered on a measurement that never happened. A governance guard scores itself healthy because the evidence it would have used could not be read. A cost comparison averages in runs whose reported spend does not reconcile with their own token counts.
None of that announces itself. It just makes your numbers slowly stop meaning anything, while every dashboard stays green.
The question worth asking of your own system is narrow and answerable: when an input is missing, what number do you print? If the answer is "zero", or "something reasonable", you have this bug. It is probably somewhere you have already looked.