The premise of the tool is that a lockfile is a document written by strangers and you should not take its word for anything.

I found five bugs in three days. Every one of them was in a place where I had taken my own word for something. Not in the code I had not got to yet, but in the sentences I had already written down as true, in doc comments and design notes and a test name. That is the post. The tool is on GitHub. The interesting part is the five.

1. A lockfile that erases the findings about itself

Package names and versions came out of the audited file and went to stdout with no filtering at all. Plant this as a version string.

1.0.0\x1b[2K\x1b[1A\x1b[2K\x1b[1A\x1b[2K\r

Now the report renders like this.

  package-lock.json        1 packages   (1 direct · 0 transitive)

     not in corpus · d=1 from "lodash" · root-only, no parent

  risk 77/100    20ms    third-party deps used to compute this: 0

The ⚠ HALLUCINATION RISK heading is gone. So is the package name. \x1b[2J clears the screen outright. The exit code still says 1, so a CI gate still fails, but a human reading the terminal sees a finding with no subject.

What makes it a story rather than a bug report is that the JSON writer had been escaping control characters since the first commit. One program, one author, one weekend, two output paths, and only one of them treated the input as hostile. I applied the rule where escaping is obviously required, in machine readable output that something downstream will parse, and forgot it where it is actually dangerous.

There was also a test named piped_output_carries_no_escape_codes, and it had been passing the entire time. It passed because no fixture contained an escape sequence. A test that asserts a property over inputs which cannot violate it is not a test, it is a sentence about what you believe.

The fix replaces control characters with U+FFFD rather than dropping them, so the cell still takes a column and you can see that something was there. Nothing real is lost. npm permits only URL-safe characters in a name, PyPI normalises to [a-z0-9.-], crates.io to [A-Za-z0-9_-], and no version scheme has a control character in it either.

2. A legal reformat that switches the auditor off

This one is worse, because it is evasion rather than defacement.

pnpm writes integrity inline.

lodahs@1.0.0:
  resolution: {integrity: sha512-AAAA}

YAML also permits the flow mapping on the line below its key, and dumpers emit it that way.

lodahs@1.0.0:
  resolution:
    {integrity: sha512-AAAA}

The second form parsed to a mapping whose key was the literal text {integrity. My block() only ever dispatched to mapping or sequence, so a line opening with { fell through to the plain-scalar key scanner, which stopped at the first : and handed back nonsense. has_integrity went false, the package's origin moved from Registry to Elsewhere, and the slopsquat rule skips every Elsewhere package on the entirely correct grounds that a corpus sampled from one registry cannot speak about a package that never came from it.

inline form:    HALLUCINATION RISK  lodahs@1.0.0   risk 77/100
own-line form:  no findings                        risk 0/100

Same packages. Same file. Legal YAML both ways, no error either way. A malicious package evades a supply-chain auditor by pressing enter.

The cause is one missing call. Four scanners in yaml.rs handle unquoted text. Three of them call reject_indicator() and flow_key did not. The module doc comment claimed all four did. I wrote that doc, and I believed it, because the subset had been designed correctly and I had checked three of the four places the design was enforced.

The lesson is specifically about subset parsers. A subset parser is a promise with two halves. Parse what you support, and refuse what you do not. The second half is load-bearing and it is invisible to every test that uses a real fixture, because a real fixture never contains the thing you refuse. Every single bug I found in these three hand-written parsers was in the refusal half.

3. The program crashed freeing what it had successfully built

This is the one that ate an afternoon.

A TOML table header nested 200,000 segments deep, [a.b.b.b…], killed the process at exit 134. With two lockfiles present the scan runs on worker threads with a 2 MiB stack, and about 30,000 segments, or 70 KB of input, was enough. panic = "abort" makes it unrecoverable, so one hostile file takes every sibling lockfile's findings down with it.

The reason it cost an afternoon is that the instrumentation said this.

parse returned in 333ms, ok=true
fatal runtime error: stack overflow

parse returns Ok. The depth guard existed and was correct, and it applied to arrays and inline tables. Table headers walked straight past it. descend is a loop, so building a 200,000-deep Value::Table chain was genuinely fine. The recursive Drop of that chain is what overflowed the stack. The program died freeing what it had already parsed.

And there was a test enshrining the mistake, with a comment explaining it.

// A long header path is iterative, not recursive, so it has to work.
let header = vec!["a"; 500];

500 sits comfortably under the threshold. I had reasoned about the parse and never once about the free, and then I had written the reasoning down where it would reassure me later.

4. An argument that had the inequality backwards

The hallucination rule needs an edit-distance metric. I shipped unrestricted Damerau-Levenshtein and argued for it in DECISIONS.md, in fixtures/README.md and in the book. lodahs is a transposition of lodash, Damerau scores that 1 and plain Levenshtein scores it 2, so, I wrote, the Levenshtein threshold that catches the typo drags in half the registry with it.

Every clause of that is false, and the arithmetic makes it impossible. Levenshtein is pointwise greater than or equal to Damerau, because Damerau permits every edit Levenshtein does and adds transposition. So Damerau-at-k is always the more permissive of the two. It can never be the tighter filter. Measured against the real 140,066-name npm corpus at the shipped threshold of 2.

lodahs: damerau <= 2  ->  3 names ["loadjs", "lodash", "loodash"]
lodahs: leven   <= 2  ->  1 name  ["lodash"]

Plain Levenshtein catches the planted name and is strictly more selective. I had the right intuition about transpositions and attached it to the wrong parameter.

The fix was not to delete the claim. It was to work out what the metric actually buys, which is three things that all live at k = 1 rather than k = 2. The reported distance says d=1 from "lodash" instead of d=2, which is the difference between "somebody fat-fingered lodash" and "this is two edits from something, like a hundred other names are". The nearest-neighbour tie-break sorts the real parent to the front. And if the crates.io corpus were ever as complete as the npm one, k = 1 becomes arguable, and at k = 1 Damerau finds lodash where Levenshtein finds nothing at all. What it does not buy is extra detections at the threshold that ships, where both metrics fire on exactly the same set of names across every fixture. There is a test holding that now, so the claim cannot rot back into the comfortable version.

Argued confidently, in prose, in three separate files, killed by one measurement that took four minutes to run.

5. The false positives I had written up as bad luck

Two packages had been flagged for most of the weekend, ksni in a Cargo.lock and taze in a pnpm-lock.yaml. Both are real. I had documented them in README LIMITS as unlucky near-misses of the popularity cut, real packages that happen to sit below the top of their registry, and moved on.

Then I measured what clause 2 is worth as a function of name length. Take every name in a corpus, pretend it is missing (which is exactly what a real package below the popularity cut looks like), and ask whether the rest of the list offers it a neighbour within the distance threshold.

At four characters, on npm, clause 2 passes unconditionally. It is not a filter at that length, it is a formality, and the rule silently degrades to "not in the corpus AND in-degree zero", which is a guaranteed CRITICAL for any real package that happens to be unpopular and short. ksni and taze are both four characters. They were never unlucky. They were structural, and the write-up excusing them was wrong.

The fix is an edit budget that scales with length, min(2, len / 5). The five is not a taste call. Read the npm column above, where a hit at distance 1 stops being the likelier outcome at five characters and a hit at distance 2 stops being the likelier outcome at ten. Below a coin flip is the bar, because a clause that fires on most inputs is not evidence about any of them.

Every planted name still fires, at the same distance and against the same parent. The last row is there because "just lower the threshold" is the obvious alternative and it is worse at both ends. It loses a true positive and it still keeps four false positives.

The general shape of this one is worth more than the fix. I had an aggregate number that said the rule worked, and the aggregate averaged over a regime where one of its clauses was vacuous.

The idea that survived

Edit distance alone does not work, which is the reason all of the above exists. lodash.assign is two edits from lodash.assignin and both are real. Any threshold loose enough to catch a typo catches legitimate siblings.

The clause that separates them is not about spelling.

A hallucinated package is a root dependency. Nothing depends on it, because nothing real has ever heard of it. A model put it in your manifest. No maintainer ever put it in theirs.

So the rule is a conjunction. Not in a corpus of known-real names, and within a length-scaled edit budget of a name that is, and in-degree zero in the lockfile graph.

Against the full corpus that third clause is worth exactly nothing, 1.000 precision and 1.000 recall with it and without it. That result is at the top of my own table because a measurement that undercuts my idea is the one most worth publishing. It measures nothing because the corpus contains every package in every fixture, so clause 1 alone suffices and no other clause can show a difference. No real corpus is ever that. npm accepts thousands of names a day, and mine is a snapshot from one afternoon, so a package published after the snapshot is indistinguishable from a package that does not exist.

So the experiment deletes part of the corpus and watches which clause holds the rule up.

The 90% row is the realistic one. Ten percent missing is roughly what a few months of registry growth looks like. False positives go from 36 to 1 and recall does not move. Recall falling at 70% is a real cost rather than an artefact, and it is not clause 3 failing. The thinning deletes the real express that expres needed as its neighbour, so clause 2 has nothing left to match against.

Across every fixture, with a full corpus and the length budget in place, one false positive is left, and it is the honest one. tensorflow-gpu is fourteen characters and one edit from tensorflow-cpu. No length policy reaches it, because at fourteen characters a near-miss really is evidence. It is a real, deprecated PyPI package that fell out of a top-15,000 list, it came through the registry the corpus samples so the origin check has nothing to say, and it sits in a requirements.txt, which records no dependency edges, so there is no clause 3 to save it. It stays in the fixtures.

And the ceiling on all of it, stated plainly. I ran the rule over every lockfile on this machine, 1,477 files and 106,673 packages that nobody planted anything in. It produced 28 findings. All 28 are false positives, not by my judgement but by construction, because each one sits beside a registry URL that resolved. That is about one package in 3,800. The true-positive count on those 1,477 files is zero, because there is no hallucinated dependency on this disk to find. Every true positive this tool has ever demonstrated is one of the names I planted myself. A sample of three is a sample of three.

What the standard library actually made hard

I expected to say \uXXXX surrogate pairing, and it is fiddly. The honest answer is different. Nothing in std made the parsers hard. The hard part was knowing what to refuse, and that is a design problem no library helps with.

Two constraints did bite, and both are in the README rather than buried.

• There is no cryptography in the Rust standard library, at all. Every npm entry carries a sha512-… integrity field. stranger reports whether that field is present and never whether it is correct. That is the constraint biting in public, which is the entire subject of the event.
• There is no RNG. The corpus-thinning ablation has to be reproducible, so it uses a seeded xorshift built from SystemTime nanos. Five lines, and it kills rand, all 1.6 billion downloads of it, which is a satisfying trade for five lines.

The package I made unnecessary

serde_json has 1,227,048,507 all-time downloads and 288,758,389 in the last ninety days, measured 2026-08-28. Almost every Rust program that reads a package-lock.json reaches for it.

rand is bigger, at 1,605,926,795, and it is deliberately not the nomination. Five lines of xorshift is not a case for anything. The claim should go to the substitution with the most work behind it, not to the largest number available.

"I wrote a JSON parser" is not a case for anything. Anyone can write one that reads the happy path. The case is two commands. The first is 29 conformance tests, each citing the RFC 8259 section it comes from, walking the grammar production by production. The second generates and mutates two million inputs and puts my parser next to CPython's json, comparing the accept/reject decision and the parsed value.

1,997,016 agreed. The 2,984 that did not fall into four classes, and every one is a place where RFC 8259 permits both answers, a leading byte order mark and three flavours of unpaired surrogate. None was about a value. Every time both accepted, both built the same thing down to the IEEE-754 bits.

The campaign found no defect. That is a result reported rather than a claim made, which after the five bugs above feels like the only way to say anything.

What I cut, and the two cuts that came back

The plan had a hard cut line at hour 30. Whatever is below it and unstarted gets cut, decided by clock rather than by feeling. Two things went below it, yarn.lock support and a stranger diff subcommand, and I wrote both up as named limitations, because a named limitation reads as judgement where a hidden gap reads as an oversight.

Both came back after the verify pass came back clean, and both cuts turned out to have been justified with a false reason.

yarn.lock was cut partly because I had no yarn fixture on disk. I did. Real yarn lockfiles ship inside npm tarballs, and there were three sitting in a bun cache. The fixture paid for itself in the first hour, because yarn resolves dependency edges by specifier rather than by resolved version. Matching on the version, which is the obvious implementation, returns a complete package list with an empty edge set. No error. Just a tree where every package has in-degree 0, which is the exact shape the slopsquat rule fires on.

stranger diff was cut on the reasoning that two shell redirections and diff already do it. They do not, because scan and diff answer different questions. scan --fail-on gates on the state of the tree, so a repository that already has 211 trivial packages fails on every pull request until somebody turns the gate off. diff --fail-on gates on what the change introduced. Reversing the two arguments exits 0 where scan returns 1 both ways, and that asymmetry is the whole reason the subcommand exists.

Eight lockfile formats and three subcommands shipped. The cut line ended up empty, which I am recording as luck rather than planning.

Closing

The tool's premise is that you should not take a document's word for anything. Every defect I found was in a place where I had taken my own word for something. A doc comment, a design note, a test name, a limitation I had already excused in public. The parsers were fine. The prose about the parsers was where the bugs lived, and prose does not fail CI.

The one habit that actually caught them was to make the assumption a parameter and measure it. The corpus is a parameter of the slopsquat rule rather than a global, which is the only reason the ablation table can exist. An assumption you cannot vary is one you cannot measure, and an assumption you cannot measure is a sentence you will eventually believe.

• The repository is at https://github.com/keirsalterego/stranger, MIT licensed, and the submission is the tag v0.1.0-submission
• The cookbook is at https://keir.is-a.dev/stranger/, including the full ablation table

Built for Zero Dependency 2026, run by Hackathon Raptors. Track A, Rust, standard library only. cargo tree is one line and every number above regenerates from the repository with make ablation, make sweep and make bench.