Postmortem
A scanner that cries wolf is worse than none — so I built a 20-flaw answer key before trusting it, and kept it on the machine that owns the data.
A security scanner's hardest problem isn't finding issues. It's not burying the one that matters under fifty that don't.
I built a local Node scanner that audits my web apps across 19
categories of flaw — leaked secrets, .env files tracked in
git, missing auth checks, injection risk, vulnerable dependencies,
wide-open CORS, absent security headers. The first version worked, in
the sense that it printed findings. But I didn't trust it, and I was
right not to: a tool that produces a wall of output trains you to ignore
it, which is strictly worse than no tool, because now you have false
confidence plus alert fatigue.
So before I ran it against anything real, I built it something to be graded on. I wrote a throwaway "practice app" — a script-generated project with 20 deliberately planted problems, each one a flaw from a category the scanner claimed to catch. That fixture is the answer key. The scanner's job was to find all 20 and invent zero. It scored 20/20. Only then did I believe that a clean report meant the app was actually clean, rather than the scanner being asleep.
The payoff came fast. Run against my own projects, it flagged a critical framework vulnerability in a Next.js dependency — a real one, with a real advisory — which I patched the same day, along with missing security headers and some outdated packages. All four apps are now at zero open findings, and "zero" means something because I know the floor it's measured against.
The second decision was where the scanner runs. It runs locally, and
that is a security decision, not a convenience one. Think about what
this tool reads: source code, .env files, secrets, git
history. The entire input is the most sensitive material in the project.
A scanner that shipped that to a cloud API to be "analyzed" would itself
become the largest vulnerability in the stack — I'd have built a tool
that exfiltrates exactly what it's supposed to protect. The data never
leaves the boundary because there is no version of this where sending it
out is acceptable. Local-first isn't a limitation here; it's the whole
threat model.
Two things I'd improve. The planted-flaw fixture only tests the categories I thought to plant, so it measures recall against known-knowns, not the unknown-unknowns — a blind spot I'm honest about. And "20/20" is a point estimate from one fixture; a mature version runs many generated fixtures and reports a distribution, not a number.
The general lesson, and the reason this sits next to the Contract Explainer note: trust in a tool is not a feeling, it's a measurement. Build the thing that grades the tool before you rely on the tool, and keep the sensitive input on the machine that owns it.