Four of Our Own Instruments Reported Success Today. All Four Were Measuring Nothing.
We spent today auditing our own tooling instead of anyone else's. Four separate instruments were reporting success. All four were reporting success about nothing, and not one of them was lying in a way an alarm could catch.
None of these were found by monitoring. Every one was found by a person opening the bag and counting what was inside.
One: the alt text was present, so the check passed
Every image we publish gets alt text, and every check we had asked the same question — is there alt text? The answer was always yes.
The answer was yes and 348 characters were missing.
A function named cleanAltText was applying filename rules to prose. One of those rules strips a file extension, which it does by deleting everything after the last period in the string. Run that over a sentence and it deletes the end of the sentence. A figure caption on a post from the day before published 143 of its 491 characters, ending mid-version-number at "the single build 1.43.3".
The second rule turns hyphens into spaces, which is correct for my-diagram.png and catastrophic for prose, because every CVE identifier is hyphenated. [CVE-2026-96652](https://analytics.dugganusa.com/api/v1/dredd/kev-gap?cve=CVE-2026-96652) published as "CVE 2026 96652" — an identifier no reader can search for.
Measured across the archive: 245 author-written image captions, 18 that had lost characters, and 128 images whose CVE identifiers had been flattened. All repaired, all confirmed by reading them back.
Two: the checker was watching the one thing that wasn't broken
The obvious question is why a defect that size survived in the archive. We have a script whose entire job is checking post images.
It read the cover image's alt text, for the first ten posts of the first page of results, and printed no totals. It never looked at a body image once.
So it was sampling ten of 1,772 and reporting on a different field than the one that was broken. It was also, incidentally, correct about those ten.
A checker that samples without stating its denominator cannot tell you whether something is broken. It can only tell you that it didn't personally see it.
Three: HTTP 200, success true, and nothing in the box
A daily job on our platform generates infrastructure clustering from our threat feed. It returns HTTP 200. It returns success: true. Its arrays are empty, its persisted count is zero, and buried in the response is a field reading warning: api_key_required.
It calls our own API with the wrong authentication header. Our feed routes want a bearer token; this one sends a different header entirely. The request fails, the empty result propagates upward, and the wrapper reports success because the function it called returned without throwing.
The job has not been failing. It has been succeeding at nothing, on a schedule. Four other parts of the platform read the index it was supposed to fill, including the one that expands network ranges into a customer-facing blocklist — which has therefore been contributing exactly zero addresses.
We know the header is the fault because we reproduced it independently: later the same day, a verification script of ours hit the same route with the same wrong header and got the same 401 we had never seen because nobody looked at the inner response.
That job is now removed rather than repaired. It is not that the idea was bad; it is that nothing had depended on it working for long enough that nothing would notice it going.
Four: the same network, classified two different ways
This is the one that matters most, because it had a customer-facing edge.
We classify addresses by whether they are actor infrastructure — a machine someone rented, where blocking it is correct and costs nobody anything — or borrowed egress, like a consumer VPN exit, where thousands of ordinary people share one address and blocking it hits all of them without inconveniencing the operator for more than a minute.
The classifier works on the network operator's name. We ran it across 59 addresses from a scanning campaign and it returned 49 actor-infrastructure, 4 borrowed-egress, 6 unclassified, with no complaint.
Four of the addresses it caught and three it missed are the same network. That autonomous system answers to two different names depending on which record you get: one contains the string "VPN" and matched our rule; the other is the vendor's company name and matched nothing. Same network. Two verdicts. Total confidence in both.
A name list is an enumeration, and an enumeration needs a new entry for every vendor forever while its gaps stay silent. So the real fix is not a longer list. It is an invariant: one network, one classification — if any name ever seen for that network says borrowed egress, every address in it is borrowed egress. Borrowed egress deliberately wins ties, because blocking a bystander is the more expensive mistake.
And one correction, which is the fifth instrument
Writing this up, we claimed those three misclassified addresses had been "published to subscribers as actor infrastructure." It followed from the confidence score being above the publishing threshold.
Then we pulled the actual published file. They are not in it.
Our own documentation says, in writing, that the published feed and the indicator corpus are different populations with different filters — and we reasoned straight past our own rule because the inference felt obvious. The true finding is narrower and still worth the fix: they were misclassified in the corpus. Whether they ever reached a subscriber is unknown and we are not going to claim it.
That is the fifth green instrument, and it was us.
What actually connects these
Not one of these four was silent. Every one of them was talking, confidently, in the affirmative.
- Alt text existed, so "does alt text exist" returned true.
- The checker examined its sample honestly and found nothing wrong in it.
- The job's function returned without throwing, so the wrapper logged success.
- The classifier produced a confident label for every address it was given.
A false green is not a broken instrument. It is a working instrument answering a question slightly beside the one you care about. That is why it survives: it is right, locally, forever.
The only defense we have found that works is the denominator. Not "we found nothing" but "we found nothing in 2,175 records." Not "the alt text is fine" but "245 captions checked, 18 damaged." The number next to the result is what converts "nothing is wrong" into "nothing is wrong here, and here is how big here was."
We now meter this on ourselves: defects the membrane surfaced and we fixed, over the chances it had to surface them. Today that reads six verified catches across fifty commits reviewed. We publish the caveat alongside it, because the rate rises when we get sloppier — every catch requires a defect to catch — and because a rate of zero means either a clean month or an instrument that stopped looking, and the number cannot tell you which.
A note on the tone of all this
Read in a row, four defects in one day sounds like a confession. It is closer to the opposite.
Every one of these existed yesterday too, and the day before. What changed today is that somebody looked, and that the looking had numbers attached. A shop with no defects found is not a shop with no defects; it is a shop with no instruments pointed inward. We would rather be the kind of operation that finds four and says so than the kind that finds none and believes it.
We can also say so because of where we are. Two people, no investors, nobody to ask. There is no communications review standing between finding a flattened CVE identifier and writing it down in public, and no quarter in which this is an inconvenient story. That will not last forever — it does not, for anyone — so while it does, the honest write-up is the one we publish. If you work somewhere that cannot publish this post, the defects are still there. Ours are just visible.
Call the whole thing 95% confident. Something in here is wrong too, and the only question is which denominator will show it.
Sources
- All figures measured 2026-10-27 from our own repository, API responses and published artifacts. First-party.
- WordPress core checksum manifest (3,349 files, version 6.9.5), api.wordpress.org — used to classify the scanning campaign referenced above.
- Alt-text repair: 245 captions examined, 125 posts repaired and confirmed by API read-back.
- Egress classification: 59 addresses, classified with our own library before and after the fix.
How do AI models see YOUR brand?
AIPM has audited 250+ domains. 15 seconds. Free while still in beta.