Troubleshooting

Symptom

import completed, exit 0, document count correct.

Cause

a Windows-1251 source decoded as UTF-8; every non-ASCII byte became

U+FFFD. The pipeline never inspects what it stored.

Cost

shipped, and found only when someone read the text back.

Check

after any ingest, read back a sample and count U+FFFD. A success

claim about writing data must be a claim about the data, not about the write.

Symptom

the task runner's notification said exit 0. The build had failed

earlier, with builder status 127.

Cause

a wrapper script spawned the real builder as a child. Run through

npm run or a background runner, two exit codes exist and they disagree.

The meaningless one arrives first and loudest.

Cost

a stale artifact nearly shipped.

Check

never accept a runner's exit code for a wrapped build. Print the

child's own status and verify the artifact — the manifest, by name and

expected item count. A failed build that clears its output directory first

leaves an absent manifest; that absence is the real signal.

Symptom

health check green, heartbeat fresh, three dispatched jobs

producing nothing.

Cause

the liveness counter advanced on events, not wall clock, so it

ticked on its own schedule regardless of whether work moved — while pointing at

a port with nothing behind it.

Cost

70 minutes, three wasted dispatches, and a scheduler routing more

work at a dead endpoint.

Check

derive health from **wall-clock time since the last completed unit

of work, never from an internal counter. An inverted health signal is worse

than none:** green pulls load in, silence routes elsewhere. Before trusting a

health endpoint, confirm it goes red — kill the worker and watch.

Symptom

git push succeeded; the change was not on the branch.

Cause

a failing pre-commit hook aborted the commit, and the push then

succeeded on the unchanged branch, correctly and with exit 0.

Cost

hit twice in one day before it was named.

Check

a push is never proof a commit landed. Confirm with

git log --oneline -1 and compare the SHA to what you meant to push.

Symptom

a monitor written specifically to diagnose a disk-full incident

watched free space fall 92 MB → 25 MB → 0 across nine minutes, and never once

saw the cause: a 26.8 GB pagefile at the root of the same drive.

Cause

the scan enumerated a directory tree; the pagefile is not in one.

Cost

a 2h15m outage of which 2h13m was mis-measurement; the fix took

two minutes.

Check

ask what an instrument is structurally unable to see, and confirm

its total reconciles with the OS's. A scan whose parts do not sum to the whole

is missing the answer.

Symptom

every secret 401'd, live or dead. Reported conclusion: "the vault

holds malformed keys."

Cause

a hardened wrapper had stopped emitting plaintext and printed a

~320-character English notice instead, at returncode 0 with empty stderr.

The caller captured stdout and bound that prose as a bearer token.

Cost

a false "the vault is corrupt" report; four live tokens nearly

condemned; the answer had been written down seven days earlier.

Check

length is the tell. Real credentials are short and have no

spaces (46–64 chars here); assert shape before sending. And **a 401 is evidence

about the reader, not the key** — record which method produced any negative

credential finding, or the next person re-confirms it the same broken way.

Symptom

a pull request advertised 1,306 additions and zero deletions.

Safe by inspection; merge it.

Cause

the diff stat is computed against the merge-base, not the

current tip of the target branch, so on a weeks-old branch it describes a world

that no longer exists. Against main today it was +301 / −1159 — a

near-total revert of a file that had grown elsewhere.

Cost

caught. All four open PRs in that repo were mis-stated the same way;

two would have deleted a 1,200-line file outright.

Check

git diff main..head (two dots) shows what merging would really

do. main...head (three dots) is merge-base-relative and *reproduces the

illusion* — and the GitHub compare API is three-dot only, so it cannot give you

the true answer. Diff two dots locally, or compare file contents by ref.

Symptom

ingestion failing an integrity check; the diagnosis was a missing

SIG_KEY in one component's config file.

Cause

that component verifies an RSA signature against a key file and

never reads SIG_KEY at all; the packaged app passes one shared environment

object to both services, so the config file examined was never consulted. The

real fault was two instances fighting over one port, signed by different keypairs.

Cost

nearly "fixed" a non-bug in a shipping path days before a demo.

Check

before acting on a config-based diagnosis, **grep the code for the

variable you are blaming**. If nothing reads it, the diagnosis is wrong however

plausible it reads.

Symptom

git commit failing repo-wide with exit 2 and output naming an

import rather than a test. Three separate sessions reported themselves blocked

within two hours, and each blamed a different open PR for it.

Cause

a merged PR added two test files importing numpy, uvicorn,

websockets and pytest-asyncio at module scope, without adding them to

the pre-commit hook's uv run --isolated --with … list. Undeclared imports

kill pytest during collection, not during a test — so the gate dies before

a single test runs, on every branch, and the error names the import. CI never

saw it because the CI workflow installs those four separately: **the thing

that reported health was not the thing that was broken.**

Cost

about two hours of fleet-wide commit blockage, and three sessions

each diagnosing it from scratch and each writing a byte-identical thirteen-wheel

fix. Three times the work for one bug — and because only the first could land,

the other two then became merge conflicts that cost a further session to clear.

Check

when a gate and CI disagree, do not average them: **ask which one

executes the thing you are about to change**, and run that one. The probe here

is to run the hook's own command against a clean checkout of the merge target;

a green CI badge is not evidence about the hook, because they install

dependencies from different lists. Concretely: a PR adding a module-scope

import under tests/ must add the wheel to the hook's list in the same commit,

and two dependency lists that must agree and are never diffed will drift.

The tell, and it is visible before any of the fixes land: *when three

sessions write the same fix, the bug is not in the code — it is that nobody

could see it had already been diagnosed.* Duplicated diagnosis is a measurable

defect in the record, not a coincidence. Three branches touching one file for

one reason inside two hours is the signal; the second independent diagnosis of

a bug should trigger writing it down, not a third fix.

Symptom

a reader quoting a number that had been publicly retracted hours

earlier, from a store that held both the original and its correction. Nothing in the

output said the source was superseded; it was simply ranked first.

Cause

a ranking fix. Coverage was made the primary sort key, correctly, because

a tier weight had been multiplying coverage and inverting it. But the demotion applied

to superseded files was also a multiplier on that same score, so making coverage

primary silently disabled it — and a retired document routinely covers MORE terms than

its correction, because a correction is shorter and narrower almost by construction.

The retired file therefore sorted above the file that replaced it. Fixing the

ordering was still not enough: the result set is then CUT by coverage, and the cut

dropped the low-coverage correction while keeping the high-coverage retired file.

Two stages, the same displacement, and the second one survived the fix to the first.

Cost

the whole point of recording a retraction is that the next reader does not

act on the retracted thing. A store that holds the correction and shows the original

first has spent the storage and kept none of the benefit. I introduced this, having

considered the interaction and judged it acceptable, and it was caught in review

rather than by me.

Check

when you change what a ranking sorts BY, enumerate every existing

adjustment that was expressed in the old units. A demotion written as score * 0.3

means nothing once score is no longer what orders the list, and it fails silently

because the constant is still there and still being applied. And check the CUT as well

as the ORDER: sorted() putting the right thing first is not the same as it surviving

[:cap]. Assert on what the caller is finally shown, at the tightest cut you support.

The general shape. A superseded entry outranking its replacement is how a reader ends up working from a refuted number with a citation. That is worse than a miss: a miss invites a second search, and a confident retrieval of the retired version ends the search. Any store that records supersession has to make it a property of ORDERING AND SELECTION, not a weight on a score that some later change can quietly stop consulting.

Symptom

an email indexer classified a message as carrying a one-time code,

applied its embargo, counted it, and printed snippet WITHHELD for 53 message(s). The

code was in the subject, and the subject went into both tables searchable.

Cause

the embargo replaced the snippet only. Every part of the report was true

about the snippet and false about the message.

Cost

worse than having no counter at all. **A wrong "unknown" invites a look; a

wrong "handled" ends one.** The count converted an open question into a false

assurance, and it did so in the one place someone would check.

Check

a counter must be a claim about the thing it names. If it says

message withheld, assert the message — every field of it — not the field the code

happened to touch. Write the assertion against the STORED ROW, not against the function

that prepared it.

Symptom

a Discord voice listener was discarding a 0.7-second "No." as a

hallucination. Fixed at the word guard, is_hallucination(), and a regression test

added asserting a short refusal survives. Test passed. The room stayed deaf.

Cause

the pipeline is screen() -> transcribe() -> is_hallucination(). The test

called is_hallucination() directly, entering below screen(), which rejects

anything under 0.4 seconds before a transcript exists. So a 0.39-second refusal

never reached the guard the test was exercising. The test verified the stage that had

been fixed while stepping around the stage that was still dropping the word.

Cost

worse than no test. A failing case is a task; a covered case is closed.

The next person to touch that filter reads a green regression test naming the exact

symptom and concludes it cannot be the cause. Review caught it, not the suite — and

the suite is what stands guard afterwards.

Evidence — three inputs, all of which a unit test at the guard passes and the real

pipeline dropped:

| input | dropped by |
|---|---|
| 0.39 s, plainly audible | `screen()` — under the duration floor |
| 0.5 s, quiet (across the room) | `is_hallucination()` — under the loudness bar |
| 0.2 s, loud (a clipped refusal) | `screen()` — under the duration floor |

Every instance found so far. This is one failure with several surfaces, and the count is the point: five separate sessions produced one of these in a single day, 2026-09-01, each writing a test that certified the defect it was meant to catch. None was caught by the suite; all five were caught by review or by reading the code.

| what the test asserted | why it passed anyway | the defect it certified |
|---|---|---|
| a short refusal survives `is_hallucination()` | entered the pipeline BELOW `screen()`, which drops it first | the room stayed deaf (1.13) |
| a promoted replacement renders its correction | the fixture's answer happened to sit unusually early, above the frontmatter the renderer stopped at | the correction never appeared for any real file (1.16) |
| the strong variant of a claim holds | the strong variant certified the WEAK one as covered | the weak case was never exercised (Part 8) |
| `nugget_current` resolves supersession | the fixture CREATED that view, so it existed regardless of the live schema | would have certified a query against a view that might not exist |
| `rc == 0` for an empty index published under `--allow-partial` | the assertion was simply wrong, written by the same session that wrote the guard | publishing an EMPTY index and reporting success, which is the exact thing the guard exists to prevent |

The last two are worth separating. One is a fixture built alongside the code, which inherits its assumptions and agrees rather than checks; it happened to be right, and the only way to know was to open the live database. The other is an expectation written down by the author of the thing being tested -- not a fixture problem at all, but a belief about correct behaviour that was itself the bug, frozen into an assertion where it then blocked its own fix twice before anyone read it properly.

The check that covers all five: ask what the test would do if the code were WRONG. If you cannot state the input that makes it fail, it is not testing anything. And if the fixture was written from the code rather than from what real data looks like, it agrees with the code by construction.

Check

**when you fix a bug in a pipeline, the regression test must enter at the

TOP of the pipeline, not at the stage you fixed.** Feed the raw input the user actually

produces — audio, bytes, the HTTP request — and assert on the far end. A test that

constructs the intermediate value has assumed away every stage that builds it, and

those stages are where the bug was. If entering at the top is awkward, that awkwardness

is the finding: it means the stages are not separable and the bug can hide between them.

Corollary, and it is the cheap version of the same rule: **prove the new test fails

against the old code.** A regression test never run red is a claim nobody checked.

Symptom

install.sh pinned a package version, re-ran over an existing

deployment, printed listen OK, exited 0. The old version was still on disk.

Cause

pip install --target does not replace an existing package directory

without --upgrade. The verification step then imported the package and, finding one,

declared success. Importing proves a package exists; it does not prove it is the one

you shipped. Especially dangerous here: the calling code reaches into that package's

private internals, so a silent version drift changes runtime behaviour with no commit

to blame.

Check

assert the resolved version, do not merely import something.

importlib.metadata.version(name) compared against the pin, as an assertion that can

fail. Deployment claims are the purest form of Part 1: the thing reporting success is

the thing that was supposed to do the work.

The general form, and it is the whole of Part 1 in one line: the more confident and specific the success report, the more damage it does when the thing it describes is narrower than the thing it names.

Symptom

an email index computed message age with time.mktime on a Z-suffixed

UTC timestamp. On a UTC+3 host every message read exactly 3 hours older than it was,

so a 23-hour-old message computed as 1.08 days and fell out of a declared one-day

embargo three hours early.

Cause

time.mktime interprets a struct_time as local time. The corpus

stores UTC.

Cost

for a credential the redactor could not see, that embargo was the only

defence, so the skew was itself a path to storing it raw.

**⚠ The part worth keeping: §6.11 of this document already said *"never build a time

filter from the local wall clock"*, and it was written earlier the same session by the

same author who then committed exactly that.** Not as self-criticism — as evidence about

what documentation can and cannot do.

A written rule is retrospective by nature. It fires when someone is reading, and

nobody reads a troubleshooting document while typing time.mktime — at that moment the

line looks obviously correct. The rule explained the failure perfectly, twice, and

prevented it zero times.

Check

when a rule in here is worth having, convert it into something that runs.

The mechanism that actually closes this one is six lines: a test that ages a synthetic

message of known UTC offset and fails on any skew above fifteen minutes. It fires at the

moment the code is written, which is the only moment that matters.

The rule tells you what went wrong afterwards. The test is what stops it happening. Every entry in this file should be read as a request for a test, not as a thing to remember.

Symptom

none. Nothing failed, nothing was slow, and every test passed. It was

found by review, reading what the values could be rather than what they were meant

to be.

Cause

a retrieval tool learned to follow superseded_by: — a frontmatter field

whose value names the document that replaced this one — so that a reader shown a

retired document is also shown its correction. The name was joined to the store path

and the file was opened and rendered. But that name is corpus data: it is a line

inside a markdown file, and anything that can write a file into the store can write

superseded_by: ../../../anything.md. An absolute path worked too. The store's

contents were choosing which files the retriever opened, and it rendered whatever

came back.

Cost

nothing yet, because it was caught before merge. The class is what matters:

a retriever that can be pointed at any readable file is not a retriever, it is a

file-read primitive that happens to take a store-shaped argument.

Check

**a value read out of a store must never become a path the tool resolves

without containment.** Resolve it, and require the result to be inside the root before

opening it; check the extension too. And notice where this crosses: it is not the

obviously dangerous features that do it. It is a field that names another document —

superseded_by, see_also, explained_by, parent, an include directive, a

thumbnail path. Those exist precisely to be followed, so following them looks like the

whole point rather than like a boundary being crossed.

The general shape. Content and paths are different categories, and a field that names another document is where they get conflated by accident. The question to ask of any such field is not "is this value sensible" but "who can write it" — and in a store that ingests, the answer is usually "anything that can write a file".

And the fixture agreed with the bug. The first fix rendered the replacement's

first three non-empty lines, which on any file with ordinary frontmatter are ---,

name: and description: — so the correction never appeared. The test passed anyway,

because the fixture I wrote happened to put its answer unusually early. A fixture built

alongside the code inherits the code's assumptions; it agrees rather than checks. This

is the second instance of that family recorded here in one day — see 1.13, a regression

test that entered the pipeline below the stage that broke. **Build the fixture from

what real data looks like, not from what makes the assertion pass.**

Symptom

none, again, and by construction: a guard that never fires and a guard

that cannot fire look identical from outside. Both were found by tracing what the

values actually are, not by reading the code as intended.

Cause

two of them, in the same file:

• A self-merge refusal compared an author-attribution string against

"fleet-worker". The function producing that string can only ever return google,

openai, anthropic or unknown — never "fleet-worker" — so the branch was

unreachable and the protection it named had never existed. It read correctly; it

just compared against a value nothing could produce.

• An unknown-author fallback routed unattributable pull requests to Codex for

review, and the cross-family check then compared the reviewer's family against the

author's: "openai" != "unknown", so it passed. Codex could review and authorise

its own work, and the path that made that possible was the one meant to be careful.

Cost

none observed, and that is the point: neither could have been observed. The

merge gate had been running on these for weeks with nothing to distinguish "protected"

from "the check silently evaluates false every time".

Check

a guard that has never been observed to fire has not been shown to work.

Assert the refusal, not only the permission: for every gate, write the test that makes

it say no. Where the guard compares against a value from another function, enumerate

that function's actual return values rather than trusting the name of the constant.

And when attribution is uncertain, fail closed — not knowing who wrote something is

not evidence that somebody else did, which is the exact inference the second guard was

making.

Symptom

a voice listener's should_record(user) had to decide what to do when

the speaker was not yet resolved. Cross-model review round nine said returning True

was wrong: with the Server-Members intent unavailable every speaker arrives

unresolved, so each fleet agent would be logged as "unknown" and duplicate turns the

speaking side already wrote. The value was flipped to False. Round thirteen said

False was wrong: a human's first syllables — or an entire short objection — are

discarded as silent data loss.

Both verdicts were correct. That is the whole finding. A reviewer that contradicts

itself is a broken reviewer; a reviewer that produces a real finding in each

direction is telling you something about the code.

Cause

unresolved is a transient state being represented as a **permanent

verdict**. The SSRC-to-speaker mapping arrives moments after the first packets, and an

utterance is not finalised until a silence gap — so there is a real window in which

the answer is not yet known rather than yes or no. A boolean has no way to say

"not yet". No value of it can be right, so the value oscillates.

The signal, stated so it is recognisable:

When a fix oscillates between two states and both directions produce real findings, stop changing the value and change the type.

The fix was not a third boolean. It was buffering the audio under its stream key and

binding the speaker late, when the mapping resolves — which loses neither side,

because it stops answering a question before the answer exists.

Why it is in Part 1 — each flip reported success. Tests passed, the review round

that requested it went quiet on that point, and the session moved on believing the

matter closed. Three rounds of green, describing three different wrong answers to a

malformed question.

Check

before changing a value a review has objected to, ask whether the previous

value was also objected to. Two correct findings pointing opposite ways is a type

error wearing a value error's clothes. Keep a note of what a parameter has been set to

and why; oscillation is only visible across rounds, and each round on its own looks

like progress.

Related — §1.13 is the same lesson about tests, and the two combine badly: a test

written to lock in one side of an oscillation will actively defend the wrong shape

against every future fix. In this same PR, a test asserting that "you" is never

speech codified a defect — "You." is a legitimate answer to "Who should handle it?"

The denylist was not the problem; the test defending it was, because a list can be

edited and a green test is evidence nobody re-examines. **Any denylist built from

artefacts will contain real words, because artefacts are made of real words** — which

is the argument for deciding on a measured signal rather than on membership, and for

being slow to write a test that asserts a word is never speech.

Symptom

a delegated worker described as idle. "It has done nothing all day", while five expensive sessions did everything by hand.

Cause

it had not been idle. It held thirty sessions, thirteen of them stopped in AWAITING_USER_FEEDBACK since the previous evening, several with completed patches attached. Each had asked a sensible question and stopped, exactly as designed. Nothing was listening. Checked the next day: six of the seven pull requests those sessions were working on had already MERGED by other routes, and the seventh was closed. The work was real, the quota was spent, and every answer went stale before anyone read it.

Cost

strictly worse than never dispatching. Never dispatching costs nothing and leaves the task visibly undone. This spent a bounded daily quota, produced correct output, and left the task looking in flight -- so nobody else picked it up either, and the work was redone by hand at a far higher price.

Check

a queue nobody reads is indistinguishable from work never done, and costs more, because the resource was consumed. If you dispatch to anything asynchronous the poll is not a nicety, it is the second half of the dispatch. Pair every dispatch with a poll in the same tool, and treat an unanswered question as an open incident rather than a pending item.

The general shape, and it is not about task queues. AWAITING_USER_FEEDBACK is a question. A question with no listener is the same failure as a system going quiet on an unresolved decision instead of pointing at the person who can settle it -- the silence looks identical to progress from outside, and the longer it lasts the more confidently it is read as "still working". Anywhere a process can stop and ask, two things must exist: something that notices it asked, and a route to whoever can answer. A worker that can ask but cannot be heard fails silently with extra steps.

⚠ Two traps in the reading itself. A session in this state counts as running to no concurrency check, so it consumes no slot and appears in no "active" figure -- it is invisible to exactly the query most likely to be run. And in the activity list, index [0] is the OLDEST entry, not the newest: reading the first element has already produced a false "still waiting" report about a session that had in fact answered.

Symptom

a measurement had made a module's word-lists unnecessary, so ~470 lines came out. The deletions were done by pattern: cut from def is_hallucination( to the next def , and drop every test block mentioning these identifiers. Both overshot, silently:

| pattern | meant to remove | actually removed |
|---|---|---|
| `def X(` → next `

def | one function | that function and the class Transcriber sitting between it and the next def` |

| test blocks naming dead identifiers | 28 obsolete tests | those 28 **and three helpers that happened to sit between them** (`FakeHTTP`, `transcriber_with`, `heard_through_pipeline`) |

It also left @pytest.mark.parametrize decorators orphaned onto the following function, so a surviving test silently acquired an argument list belonging to a deleted one — a test that still runs and no longer tests what its name says.

Cause

a regex or a split boundary encodes a belief about the file's shape. def assumes the next definition is a function; it is not, when a class follows. Splitting on test boundaries assumes helpers live outside blocks; they do not, when a helper sits between two tests.

Cost

none, because the tests were run. That is the point: the only thing between this and a module shipped without a class was the habit of running the suite rather than trusting the edit's report of itself. The script printed "dropped 28 tests" and that was true.

Check

after any bulk edit, diff the set of definitions, not the line count:

diff <(git show HEAD:path | grep -E '^(def |class )')      <(grep -E '^(def |class )' path)

An empty diff is the claim worth making. Line counts, "N blocks removed", and a clean parse all pass while a class is missing. Then run the tests before believing any of it: verify the state changed, and verify nothing else changed with it.

Symptom

three delegated workers reported the same blocker: a rebase they could not start because git status showed 237-306 changed files. I diagnosed it confidently, in writing, with a figure: "7,205 of 8,292 tracked files are stored LF in the index and appear CRLF in the working tree, so a large modified count is expected noise, not your changes", and told them to mask it with core.autocrlf false plus git update-index --assume-unchanged.

Cause

I measured git ls-files --eol on my own Windows checkout, where that figure is true, and stated it as a property of the repository. The workers run in a Linux sandbox, where the same command reports 140 files as i/lf w/crlf, not 7,205. The real cause of their file count was unrelated: those branches are 219-273 commits behind main, so a whole-tree diff legitimately touches 237-306 files and their sandbox's diff-size safeguard was correctly refusing to proceed.

Cost

worse than saying nothing. One worker followed the instruction exactly, was still blocked, and reported back; two more had already acted on it. The real blocker went undiagnosed for another round, and a second session had to measure the sandbox from scratch to find that my number did not describe it. A wrong diagnosis handed to a worker becomes wasted work that then has to be diagnosed again — the error does not stop at being wrong, it manufactures more work in the shape of itself.

Check

before quoting a measurement to someone operating elsewhere, ask which machine was this taken on, and is it the machine they are standing on? Environment- dependent readings are most of what we measure: line endings, path separators, installed tool versions, $PATH, locale, filesystem case sensitivity, CPU count, what git config says. If you cannot run it in their environment, say what yours reported and label it as yours — "on my Windows checkout I see X; check git ls-files --eol on yours" costs one line and cannot mislead.

The general shape. A measurement taken in one environment and quoted as a fact about another is worse than no answer, because the number makes it credible. "I do not know why you are seeing that" invites investigation. "7,205 of 8,292 files" ends it. Precision is read as authority, and the more specific the figure the more completely it forecloses the question — so a precise number from the wrong context does more damage than a vague guess from the right one.

This is the day's theme with a twist: not a check that failed to check, but a check that worked perfectly and was pointed at the wrong subject. See 1.10 (a green CI badge is evidence about CI's dependency list, not the hook's) — same error, and there too the mistake was reading a true measurement as an answer to a question it was not about.

Symptom

a delegated session reported: "I have aborted all my previous partial rebases and reset everything. Following your precise instructions, I've performed the following exact steps..." It read as one of the more competent replies of the day.

Cause

nothing had happened. Its activity stream contained zero patches, the branch head was unchanged, no new ref had been pushed, and the branch was still the same 238 commits behind main it had started at. Three sibling sessions reported COMPLETED with the same emptiness; one of those had three sessionFailed events in its stream while its state field read COMPLETED.

Cost

it would have been recorded as done. The report was specific, used the right vocabulary, and described the correct procedure — everything except having occurred.

Check

verify the artifact, never the report. For a worker that means: did the ref move, is there a patch in the stream, does the diff exist. Not: does the summary sound right. And when a system offers a state field and an event stream, reconcile them rather than reading the field — a state of COMPLETED over a stream ending in sessionFailed is a green lie inside somebody else's API, and reading the field alone records a failure as a success.

Same class, third surface today. An installer that reported success without installing (1.14). A review stamped on a commit it never read. Now a worker summarising steps it did not run. In each case the report was well-formed and the artifact was absent, and in each case the artifact was one command away.

Symptom

a cross-model review was dispatched against a pinned commit. It ran the full 221 seconds, produced 8 blocking findings, exited 0, and printed a JSON line carrying verdict, blocking: 8, reviewed_sha and a reviewer name. Everything about that line says a review happened. GitHub's review state was untouched.

Cause

--post is action="store_true", and the invocation omitted it. The only tell was "posted_state": null in the middle of a one-line JSON blob -- invisible to a caller reading a tail, and invisible to any agent that reads the verdict field, which is what a caller naturally reads.

Cost

nearly a merge decision taken against a verdict that existed only in a terminal. The PR would have shown its previous review state, on a previous commit, and the fresh findings would have been attributed to a review nobody could retrieve.

Check

a run that did nothing must announce that it did nothing, in the loudest channel it has. Not a null field: a sentence, on stderr, and a distinct exit code (this one now exits 3). The absence of an effect is never adequately reported by the absence of a mention. Verify by reading the effect back from the system that owns it -- gh api .../pulls/N/reviews -- never from the tool's own account of itself.

The rule this produces is bigger than "check the flag". A tool that reports on an EXTERNAL action must verify the action, not its own intention to perform it. The review tool knew it had computed a verdict. It did not know whether GitHub had recorded one, and it reported the first as though it were the second. Every wrapper around a remote effect has this shape available to it -- a push, a deploy, a message sent, a secret written -- and the fix is always the same: read the state back from the owner of the state.

WHAT IT HAD ALREADY COST, found by going and looking. 114 review runs were on disk across the session tree; exactly two carried posted_state: null. One was this one. The other was PR #362, a different session, assessment: "regression", one blocking finding, run against commit 358241fa9003 -- and never recorded anywhere.

That PR merged, on 2026-08-29T17:24:06Z, on that exact commit, with reviewDecision still CHANGES_REQUESTED. Two other regression reviews had posted against the same head the previous night and were merged over as well. The lost third review's text is unrecoverable: the run's output file is 372 bytes and holds only the JSON summary line.

The two surviving findings are live on main today. tools/decisions/tokens.py generate(surfaces) builds its verifier table only from the surfaces passed in and then REPLACES the verifier file without reading the existing one, so a subset call silently revokes every omitted surface and returns success. config.json configures ten surfaces; tools/decisions/e2e_deepseek.py:67 calls generate(["dispatch", "jules", "extractor"]), so running that script revokes seven of them -- chloe-telegram, chloe-timer, claude-code, cli, codex, cron, telegram -- while reporting that it worked. The --rotate CLI is safe only because _surfaces_from_config() happens to pass all ten. The rotation is also non-atomic: the loop writes each token file before the verifier, so a failure mid-loop leaves callers holding new tokens that the old verifier rejects.

So the cost of the silent dry run was not one lost review. It was a third opinion on a defect that was already merging past two, on code that mints the credentials for ten callers, and nobody could have known to look -- because the tool that produced it exited 0 and printed a verdict.

THIS IS THE REVIEWER BUILT TO CATCH THIS CLASS, COMMITTING IT. The tool exists to stop other code from reporting work it did not do. It reported work it did not do. Writing the check does not exempt the checker, and the fact that a tool's PURPOSE is to detect a failure mode is not evidence that it is free of it -- if anything the author's confidence in it is what stops anyone looking. The same defect this file has now catalogued five times: An installer that reported success without installing (1.14). A worker that described the exact steps it had taken, having taken none (1.22). Four Jules sessions reporting COMPLETED with zero artifacts. And now the gate's own review tool. Writing the check does not exempt the checker: the reviewer reported work in the vocabulary of work, and the work had not occurred.

Symptom

eight stale branches were queued to be rebuilt onto main. The obvious first question -- has any of this already landed? -- was put to git cherry origin/main origin/<branch>, which compares patch-ids. It marked every branch as carrying new work. Two of them were carrying none: main already had both fixes.

Cause

git cherry asks was this exact diff applied. That is not the question. The question is does main already do this. Where somebody fixed the same defect independently -- with a better docstring, a wider scope, an extra call site handled -- the diffs differ, the patch-ids differ, and a patch-id check reports "still new" about a change that would be a no-op. Patch-id equality is not outcome equality, and the divergence is worst in the case that matters most: the closer the second fix is to correct, the less it resembles the first, and the more confidently the check reports a duplicate as novel.

Cost

would have been eight rebuilds, two of them producing PRs whose entire content main already had, each then consuming a review cycle to discover it.

Check

screen behaviourally, not structurally: check out the branch's own tests against untouched main and run them. A branch whose tests pass unmodified is redundant; one that fails on a missing module is carrying real work; one that mostly passes has a narrow delta worth reading. Three outcomes, all measured, none inferred. Where a branch ships no test, the fallback is to execute its central assertion against main directly -- one of the two redundancies here was settled by calling anchors() on an unmodified checkout and watching it already return the empty set.

Today's shape in yet another costume. The check ran, was correct about what it measured, and was not about the thing being asked. git cherry measured the artefact -- the diff -- while the claim being made was about the outcome. Compare 1.21, where a true measurement of one machine was read as an answer about another; and 1.10, where a green CI badge was read as evidence about a hook it never ran. The failure is never that the instrument lied. It is that nobody asked what it was pointed at.

Symptom

a --debug flag on a Discord voice listener wrote the live voice transport key into a log file. Cross-model review predicted one route: the receive extension logs the key on its CryptoError branch, and the code raises CryptoError routinely (unmapped packets, non-audio RTP, DAVE failures), so it would have fired on nearly every packet.

The first actual leak came from somewhere else. discord.gateway logs the voice websocket op-4 frame at DEBUG, and that frame carries the key as a matter of protocol. Two independent emitters behind one flag, and the one that fired was not the one anyone named.

Cause

--debug set a whole logger tree to DEBUG. Every library under it then decided for itself what a frame worth logging contains.

The fix, whose shape matters more than the incident: a logging.Filter that drops any record whose rendered message names the key field, installed on the handler — not muting the two known emitters.

A denylist of known emitters cannot cover an unknown one; a filter on the data can. Muting the two would have left the third nobody has found, and the whole point of this incident is that the leak arrived by an unpredicted path.

Same argument as replacing a word denylist with a measurement, in the same PR: enumerate the property you care about, not the places it has been seen. Redact on the way OUT of the process, where every path converges, rather than at each call site — where you must first have thought of the call site.

Verified, because a filter that does not filter is this file's whole subject: op-4 frame DROPPED, CryptoError details DROPPED, ordinary debug line kept.

Blast radius, measured rather than assumed — on the host: 76 unit files visible (75 matching a control term, so the probe was live), 0 enabling debug logging; 29 env files visible, 0 setting a log level; 0 journal lines in 24h naming a key field. Nothing else on that host has this shape today. The risk is in the pattern rather than the inventory: any component that turns on a library's DEBUG logging inherits whatever that library considers a loggable frame. The filter belongs somewhere shared, not only in the module that was bitten.

Symptom

a fix was delegated to a different model, deliberately: the model that writes a bug reasons about it with the priors that produced it. The delegate died partway with memory allocation of 532480 bytes failed.

It left behind a correct dependency-free ordering primitive and a correct test for it, and had wired neither into the module they were for. The test passed. The suite was green. The defect was untouched.

Cause

the worker returned; the work did not complete. Those are different events, and only one of them is visible from outside.

Check

a delegated task needs a completion check that looks at the work, not at whether the worker returned. Ask what the change was supposed to make true and test that: is the new function called from anywhere? Does the behaviour it was meant to fix actually differ? A green suite after a partial delegation is evidence of nothing, because the half that landed is the half that was easy to make pass.

And when it dies mid-operation, read what it left before re-running it. Re-running into the same wall costs the same memory, and a second partial write over a first can produce a state neither run intended.

Symptom

two codex_pr_fix dispatches, against different PRs in different worktrees, both ran to exhaustion and returned files_changed: [] with notes reading "Still no tracked source edits" and "No source files have been edited yet". Both then died with rc=4294967295. The obvious reading -- the one taken at first -- is that Codex failed.

Cause

it did exactly what it was told. Both worktrees contain this repository's CLAUDE.md, whose managed GitNexus block says "MUST run impact analysis before editing any symbol" and "MUST run detect_changes() before committing". Codex read that, took it as binding, and spent its entire run invoking npx gitnexus analyze --index-only on a 138,000-symbol repository. The transcripts show it waiting on that build and reporting, accurately, that it had not edited anything yet. The file even carries an override saying those gates are conditional on the GitNexus MCP being loaded -- but the override sits BELOW the generated block, and a worker reading top-down has already accepted the mandate by the time it arrives.

Cost

unknown, and that is the point. Two dispatches are the two that were diagnosed. Delegated fixes have been returning empty across sessions for some time and every one was read as the model failing, so nobody looked at what the model had been asked to do. An unknown number of runs may have burned themselves on an index build.

Check

a delegated worker inherits every instruction file in the directory you hand it, including ones addressed to a different agent in a different harness with a different toolset. Before blaming a worker that produced nothing, read what it was told -- its own transcript will usually say, plainly, what it spent the time on. And when dispatching, state explicitly which in-repo instructions do NOT apply: codex_pr_fix now carries a rule naming CLAUDE.md, naming GitNexus, and forbidding any command that builds, indexes, installs or downloads.

The failure was invisible because the wrong actor was blamed. "Codex returned nothing" is a complete-sounding explanation, and complete-sounding explanations end investigations. The worker was obeying an instruction nobody remembered writing, in a file nobody thought of as part of the prompt -- and a config file IS part of the prompt the moment an agent can read it.

Symptom

a recall benchmark advertised as model-graded. Its own header spends a paragraph explaining that the older regex grader is invalid: the retriever is lexical and so is the grader, so a result that merely shares vocabulary with the question scores as an answer, and four keys did exactly that until they were caught. The gate returned 0. Everything looked green.

Cause

when the model judge could not run -- the CLI absent from PATH, a timeout, a failed call, unparseable output -- judge() returned None and the tool fell back to the regex keys, compared the resulting rate against the floor, and RETURNED 0. So the model-graded gate could PASS having been graded by the method the same file documents as broken. A warning was printed, but the pytest bridge reads the return code and nothing consumes stdout, so the two outcomes were indistinguishable to every automated consumer.

Cost

unquantifiable by construction. Any green run in the window where codex was unavailable was a regex-graded number wearing a model-graded label, and there is no record distinguishing them.

Check

a gate that cannot run its check has not run. An unavailable judge is now a hard failure with its own exit code and no fallback. Degrading silently from the check you promised to one you have documented as broken is worse than not checking at all: an absent gate is noticed, a degraded one reports success.

AND THE FLAG HAD TO CLOSE TOO. --regex still exists, because the upper-bound figure is genuinely useful when reading the file by hand -- but it can no longer return 0. Closing the automatic fallback while leaving the manual flag able to report a pass would have moved the hole to the front door: any caller who added --regex to turn a red gate green would get exactly the false certification the fix removes. When you close a silent degradation, look for the explicit switch that reaches the same state, because a fallback and a flag are the same defect wearing different clothes. WHY THIS ONE IS WORSE THAN THE OTHERS IN THIS FILE. The installer that reported success without installing (1.14) and the reviewer that posted nothing (1.23) were accidents -- nobody wrote them to behave that way. This fallback was DELIBERATE. Some earlier version of this reasoning decided that a gate which cannot run its grader should carry on rather than stop, and every property that made it dangerous follows from that one intention: it kept the gate green. Ask of any fallback you find: what does this do when the thing it falls back FROM is the whole point? If the answer is "reports success anyway", the fallback is not resilience. It is a bypass with a sympathetic name.

Symptom

a PR reached round 9 of cross-model review. Every round had found something real, which is why it kept going. Then the diff grew past the reviewer's input ceiling and the run reported truncated: 16551 -- sixteen thousand characters the reviewer never saw, in a verdict presented as a judgement on the change.

Cause

rounds add commits, commits grow the diff, and past the ceiling each further round scores a SHRINKING FRACTION of the work while its output looks identical to a complete review. The finding rate had also flattened into bookkeeping: the last round returned a disclosure missing from a JSON branch and a number disagreeing with itself -- both real, neither a risk in the code path the PR exists to fix.

Check

two questions, and they are INDEPENDENT: is the reviewer still seeing the whole change, and is the finding rate describing behaviour or paperwork?

⚠ CORRECTED THE DAY IT WAS WRITTEN, and the correction is the useful half. The first version of this entry collapsed those two questions into one and treated truncation as a stopping condition. It is not. Truncation is a reason to DISTRUST the review, not a reason to end it -- it says fix the review (split the PR, raise the ceiling, review the parts), while only the second question says stop. The cost of getting that wrong was nearly paid the same hour. Round 9 of a PR looked like paperwork -- a disclosure missing from a JSON branch, a number disagreeing with itself -- and the recommendation given was to merge on the next clear verdict. Round 10 then produced two BEHAVIOURAL defects, both introduced by round 8's own fix: the shipped default split the coverage tier its contract promised it never split, and a documented recovery command could not do what the documentation said. Merging on the round-9 read would have shipped a tool violating its own stated contract, with the truncation argument as the justification. One round is not a rate. The finding-rate question needs more than one observation before it answers, and "the last round was thin" is the single most available piece of evidence for stopping and one of the weakest.

Symptom

a review tool was changed to review only the increment since the last reviewed commit rather than the whole PR, and to state that scope in the posted review so no reader could mistake a partial verdict for a complete one. The scope line was written, tested, and worked. Then incremental was made the DEFAULT, and the line silently stopped appearing.

Cause

the disclosure was rendered from the FLAG (a.base) rather than from the base actually used. When somebody typed --base the line appeared -- to a person who, by having typed it, already knew the review was partial. On the default path a.base is None, so the line vanished for every reader who had not been told and had no way to find out.

Cost

one run. The very next review compared the pull request's whole-PR description against a partial diff and reported that the feature the PR exists to add was "absent from the provided diff". A confident, wrong, blocking finding, produced because the reviewer was not told what it was looking at.

Check

a disclosure conditioned on the user having already done the thing that implies knowledge is not a disclosure. Render it from the STATE, never from the flag that produced the state. The test is one question: who sees this warning, and do they need it? If the answer is "the people who asked for the behaviour", it is decoration.

The same shape appears three more times in this file, and it is worth seeing them together: a --post flag whose absence made a review vanish silently, so only a caller who passed it learned anything; a stale-index marker attached to returned rows and therefore absent from the empty result that most needed it; a truncation warning printed at build time and never at query time, where the false absence is actually experienced. In each case the warning existed on the path nobody takes. And note how this one surfaced. The limitation -- that an incremental review cannot judge whole-PR claims -- was written down as a theoretical caveat when the feature was built. It arrived in reality within ONE RUN of making it the default. That is the argument for stating limitations out loud while they still sound theoretical: the interval between "worth mentioning" and "blocking finding" is not long.

Symptom

every credentialed gcloud command died with

ERROR: gcloud crashed (DatabaseError): file is not a database. Reinstalled;

declared fixed; broken again the next morning. Two days of work were built

around the breakage — a hand-written --secret= shim, and a standing belief

that "the vault is read-only on this machine".

Cause

two files in the config directory (%APPDATA%\gcloud), not the

install directory: credentials.db zero-filled, access_tokens.db

malformed. An installer touches neither. File timestamps showed

credentials.db had not been successfully written since 1 June — the tool

had been broken for ~3 months and every workaround built since was scar

tissue around one undiagnosed day.

Cost

three months of degraded tooling, two misdiagnoses, and a shim that

silently broke every build reading a secret when the "repair" overwrote it.

Check

when a reinstall doesn't hold, **stop reinstalling and diff the

state directory**. pragma integrity_check on every SQLite file the tool owns,

and read the first 16 bytes: a real SQLite file starts SQLite format 3. **A

zero-filled file is not a corruption mystery — it is a write that died

partway**, and the first thing to check is free disk. Both volumes here were

under 10%.

Check

a capability has as many layers as it has ways to be invoked.

Verify the one the caller actually uses. "Auth works" is not a claim; *"auth

works via the CLI"* is. See also Part 3.

---

## Part 4b — Two correct changes that combined into a defect

Symptom

after the snippet change, the router answered *"what voice does

Anderson use"* from the wrong pool. It stopped at the curated memory store, whose

six 112-char snippets scored density 17.9 — sailing over the 2.0 threshold —

and none of them contained the answer. Two other pools each held **six rows that

did**, and were never consulted. Both changes were individually correct and

individually tested.

Cause

density is matched-terms-per-character, so shrinking the snippet

inflates it. The precision fix drove the denominator down by an order of

magnitude and made the brake more eager, not less. The threshold had been

calibrated against whole documents and nobody re-derived it against snippets. What

density then measured was "returned some text with the query words in it" — which

is exactly what a sufficiency test must never mean. A pool returning a tight

near-miss scored higher than a pool holding the answer.

Cost

a wrong-pool answer that looked correct, on the one question chosen as

the acceptance test, found only because the answer token was searched for by hand

in the returned text rather than trusting that the right pool had answered.

Check

when a threshold is normalised by size, write down what happens as the

denominator goes to zero, and re-derive it whenever anything changes the size of

what it measures. Better: prefer a criterion that is a property of **what a record

is over one that is a property of how it is shaped**. The fix here was to drop

density entirely and stop only on record authority — a principal record is an

answer because of what it is, and no amount of reformatting can fake that.

The general shape: two changes, each correct, each tested, coupled through a

shared quantity neither one names. The coupling lived in the denominator. Ask of

any new threshold: which other change could move this number without touching this

code?

---

Symptom

--audit-embargo printed a clean, well-formed table of ? and returned exit 0.

Cause

two correct changes. The audit was written to show WHAT an embargo had caught, so that over-firing would surface as "these are ordinary emails" rather than as a number nobody can interpret. Then a later fix — also correct, and closing a real credential leak — widened the embargo to overwrite every stored column. The audit read those columns. So it read EMBARGO_NOTE in each one, rendered a table of placeholders, and reported success.

Neither change was wrong. Neither review would have rejected either. The defect exists only in their composition, which is the whole of Part 4b.

Cost

a silent-success path inside the tool built to prevent silent success. The audit exists because a declared exclusion nobody samples is a claim rather than a measurement; going blind, it became the thing it was built to detect, and said nothing.

A diagnostic that reads its subject THROUGH the mechanism it is diagnosing goes blind exactly when that mechanism is at its most aggressive — which is precisely when you most need to look. The more thoroughly the redactor, the embargo, the quarantine or the filter does its job, the less the instrument pointed through it can see.

Check

a diagnostic must hold its own evidence, captured before the mechanism acts, and never re-derive it from the mechanism's output. Here that is bounded per-row audit evidence — sender domain, safe category, the classes that fired — written at classification time rather than read back from the redacted row.

And incomplete evidence must FAIL. An empty audit and a clean audit look identical; the only difference is a value nobody can see. ? in every column is not a result, and a diagnostic that renders one and exits 0 has told you the opposite of the truth. Returning failure on missing evidence is what separates "nothing was caught" from "I cannot see what was caught" — and those call for opposite responses.

Found by review round 7 on 2026-09-01, in the email index. The repaired audit's first run immediately surfaced a genuine over-fire the blind version had been hiding.

Symptom

memory_recall.py "does anderson bind to loopback or all interfaces"

→ no match for … across 352 file(s). Quantified, specific, and read as a fact

about the estate.

Cause

the matcher required every word of the question to appear

literally in one file, does and or and all included. 0.0.0.0 was in

11 live files of that very store.

Cost

on a 22-question ground-truth set the answer was present for 21 and

the tool returned it for 8 (36%). The seven false "missing/broken" calls of

2026-08-28 came out of this.

Check

ask the retrieval tool a question in the words a person would use,

whose answer you have independently located first. If keyword phrasing works and

question phrasing does not, the term extractor is the fault, not the corpus.

Every hand-check anyone had ever run used keywords.

Symptom

results came back. Ten files, plausible names, good scores. The

answer was in none of them.

Cause

score was occurrences × tier weight, with no stopword removal. A

136 KB append-only ledger has unboundedly many occurrences of what, is and

the, so it outranked the 6.9 KB file that answers the question.

_INTENT_LEDGER.md was the #1 hit for 15 of the 20 questions that returned

anything at all.

Cost

12 of the 14 failures were non-empty. A zero result invites a

second search; a confident non-answer ends one. This is why the fault survived

months of daily use: the tool looked like it was working every single time.

Check

run 20 unrelated questions and count how often the same file is

ranked first. More than two or three, and the ranking is measuring file size.

Symptom

even when the correct file was in the list, the printed excerpt

was frontmatter and boilerplate: "DECISIONS entries carry the why…".

Cause

the snippet was the first N matching lines, not the densest. In a

curated store the first match is almost always the header.

Check

assert that the top hit's own printed snippet contains the answer

string. Returning the right file and displaying the wrong part of it scores as a

hit in every metric and reads as a miss to the human.

Symptom

"how many agents are in the fleet" returned nothing, repeatedly.

Cause

fleet, agents and memory were on the stopword list, so the

query extracted zero terms. An empty query matches nothing everywhere, so

there was no pool it could have been routed to. Corrected nine times while

the answer sat in the decision store as a principal record.

Check

every retrieval surface must print the terms it actually used. A

query that silently became empty is indistinguishable from a corpus that is

empty, and only one of those is fixable.

Symptom

the Khalo manuscripts "were not on this machine". Searched as

Carlo, then Kalo, then khalo.

Cause

spelling. kalo does not match khalo; the h breaks the

substring. Each search reached a corpus that held the string in a form the query

could not match.

Check

a zero hit on a proper noun is a **question about spelling, not about

existence**. Keep variant groups and query every form, and make the expansion

symmetric — so carlo reaches khalo because they are one group, not because a

rule maps corruption to canonical.

Symptom

nothing at all. Coverage simply came back one term short,

everywhere.

Cause

the term matcher is built from [a-z] character classes with an

(?<![a-z0-9]) lookbehind, so it must be handed lowercased text. The pool

router's coverage() named its variable low and never lowercased it. Every

capitalised term silently failed to match: Anderson, Zmeyborn, Groq,

OpenRouter — the proper nouns that are the best things to search on, and the

ones prose writes with a capital.

Check

assert the matcher against a capitalised fixture. A matcher bug

produces no error, no warning and no zero — just quietly fewer matches, which

looks exactly like a corpus that holds less than it does.

Symptom

"across 352 files": true, checkable, and misleading. It is a

count of one pool of eight. It was reported onward as "not recorded

anywhere".

Cause

the tool that MEMORY.md points every session at searches the

curated markdown store only. The other seven pools — cavemem (570,691

observations), session_raw (1,388,728 chunks), decisions, session_kb (86,157

nuggets), repoindex, video_kb, newsletter (20,667 items) — were never asked, and

the output never said so.

Check

a miss must name what it searched and what it did not. An absence

is only meaningful with its scope attached. Two of the 22 answers did live only

in another pool, so the scope gap is real — but it was 1 in 11, while the

ranking fault was 12 in 22. Fix the loud thing, not the interesting one.

Symptom

a test needed a question nothing has. Two attempts failed.

Cause

"the airspeed velocity of an unladen swallow" had been in cavemem

since 2026-08-10: a famous absurdity is the first thing anyone reaches for

when they want an obviously-absent string, which is exactly why it is already in

a corpus built from transcripts. So *"which brand of marmalade did the

lighthouse keeper prefer in Aberdeen"* was invented on the spot — and

marmalade entered cavemem at **14:31, two minutes after the probe that first

typed it**, as the text of that very command.

Check

use a per-run nonce, or test the rule as a pure function with no

corpus at all. Any fixed sentence written into an absence test rots into a false

positive: it passes on the day it is written and fails silently afterwards.

Symptom

a traceback at the end of an integration check nobody read.

Cause

tools/test_retrieval_fires.py unpacked two values from a function

that returns three. It died at its DECISIONS section, so its **READER 2 block —

the checks over the retrieval tool itself — had not executed at all.** The guard

that should have caught a 36% recall was failing two sections early and

reporting it as noise. Separately, the repo's commit gate was dying during

collection on four wheels missing from its dependency list, so three test

files had not run on any branch.

Cost

the fault was invisible for exactly as long as its detector was.

Check

a check that errors is not a check that passed, and a section

that never ran is not a section that was green. Assert that verification reached

its last line: print a completion marker and require it.

Symptom

this document.

Cause

D:\output\TROUBLESHOOTING.md is in none of the eight pools.

Asked for its own most distinctive phrase, the router answers out of cavemem and

session_raw — transcripts of the work, never the document itself. It is findable

only by someone who already remembers it exists, which is the level-4 non-fix

its own Part 7 condemns.

Check

for any document meant to change future behaviour, **query for it by

its content**, through the retrieval path that will actually be used. If it does

not come back, it is a file, not a control.

Symptom

a probe for the mail database at a plausible local path returned nothing,

and the sentence forming in my head was "the email store does not exist."

Cause

the path was guessed. The store is real, live and 174 MB, on another

host entirely. Nothing about the probe was broken; it answered the question it was

asked, which was "is there a file here" — not "does this store exist."

Cost

none, this time. It was caught in the same minute, and only because the

session had spent the day on exactly this failure. That is not a control.

Check

**"the file is not there" is one keystroke from "the store does not

exist", and they are different findings.** Before recording a store as missing, say out

loud where you looked and how you chose that place. If the answer is "I guessed the

path", you have measured your guess. The same probe on the correct host was one

command away.

*The reason this belongs in Part 6 rather than as an anecdote: it is the failure this

whole document was extended to describe, arriving inside the fix for it, on the same

day, in the hands of someone actively looking for it.*

Symptom

an email index reported 236 credential-shaped strings removed on

build. It also held **81 one-time codes and 30 password-reset URLs, at rest, in a

searchable store**. Both statements were true at once.

Cause

the redactor detects credentials by shape: sk-ant-…, an AWS key, a

Luhn-valid card. It is very good at that. A six-digit code has no shape — 123456 is

indistinguishable from an invoice number — and a reset URL's sensitivity is in its path,

not its characters. Both are credentials by context: the token carries no signal,

only the surrounding words do. So no extension of the pattern list could have reached

them. **The redactor was solving the problem it was written for, and being trusted for

a different one.**

Cost

credential material sitting in a store a recall query could surface into a

transcript. Nothing was reported wrong at any point: the build was honest, the count was

accurate, and the conclusion drawn from it was not.

Check

for any component whose output is trusted as a safety property, write down

the question it actually answers and compare it to the question people ask of it.

redact() answers "does this contain a token that looks like a credential". It was

being read as "is this safe to store". Then verify at the layer you are claiming:

query the built artefact for the leak rather than trusting the builder's own report.

"Add more patterns" is the wrong lesson. A shape matcher cannot be extended toward something with no shape. When the signal is in the context and not the token, the fix is a second mechanism, not a longer list — otherwise every new provider's phrasing is discovered by the next leak.

Three further traps found only by re-probing the rebuilt index, each of which looked

fixed after the previous round:

• Context is a property of the message, not the field. Redacting a mail one field at

a time left three codes: the words saying "this is a verification code" were in the

snippet while the digits were in the subject, so the subject was judged without them.

• A phrase list is still a list. Three real codes survived because the wording was

"your code is …" rather than "is your code". Adjacency to the word code

generalises; a sentence does not.

• Redaction rewrites the text the rule reads. Replacing one token moved a code word

within range of a number that had been correctly left alone, so the redactor was not

idempotent — running it twice removed more than running it once. Iterating to a fixed

point makes "a second pass finds nothing" true by construction, which is what turns

it into a leak check over a whole index: 0 of 63,927 stored fields.

Check

for anything you are protecting, ask all three: *does it look dangerous, do

its surroundings say it is, and is it still alive?* A system that only ever asks the

first will keep finding the second and third by leak.

Check

fail-closed must not mean fail-noisy. The same pass leaves order

numbers, postal codes, promo codes and area codes intact, and that is tested. A redactor

whose output looks broken is one people work around.

Symptom

a redaction audit and the redactor it audits disagreed about what counted

as a single-use link. The audit's probe matched https://…/auth/<token>; the

redactor's rule did not. The builder stored the whole token and reported success.

Cause

the check was written from the shape of the leak, the implementation from

the shape of the known cases, and nobody compared the two. **An independent check that

tests for something the implementation does not handle is not a safety net — it is a

tripwire nobody wired to a bell.** It fires on the day someone reads it, which may be

never.

Check

for any checker/implementation pair, state the relationship explicitly:

either the implementation covers at least what the check covers, or the check's

extra coverage is written down as known-uncovered. Silent divergence is the failure;

either resolution is fine.

And the converse, which is the more tempting error. When that audit later flagged

two rows that turned out benign, the easy fix was to narrow the probe until it agreed

with the redactor. That would have destroyed the only property that made it worth

having. **Tuning a checker to agree with the implementation turns an audit into a

mirror.** What was done instead: measure the real separation — genuine leaks put the

context word 13–21 characters from the code, the two benign rows 184 and 237 — and put

the threshold in that measured gap, which teaches the probe nothing about the

implementation. An invented test sentence that sat at 29 characters was **deleted

rather than accommodated**, because no real message resembled it and tuning a window to

satisfy a fiction is how a check stops describing the world.

Symptom

a regression test passes, the defect it is named for is live in

production, and nobody is lying.

Cause

the test exercised the clear form of the input. The clear form is the

one already in mind while writing the fix, which is exactly why it already works.

Three independent instances on 2026-09-01, different subsystems, same shape:

• A credential test used the subject "Your verification code is …" — a phrase the

classifier matches outright. The wording that actually leaked was the bare *"your code

is …"*, which it does not. The strong variant certified the weak one as covered.

• A voice regression test drove the pipeline in a way that **bypassed the stage that was

broken**, so the stage could be deleted and the test stayed green.

• An ingest fixture passed only because its answer happened to sit early in the

document, where any ranking would have found it.

Cost

worse than an absent test. An absent test leaves a known hole; a test named

for the defect converts the hole into a claim of coverage, and the claim is what stops

anyone looking.

Check

**write the weakest form of the input that should still be caught, not the

clearest one.** Then prove it: break the code deliberately and watch that specific test

fail. A test that has only ever passed has not been shown able to fail — and when it

covers two defences at once, remove one and confirm it still fails, or it is asserting

their union and neither is load-bearing.

Symptom

a classifier and a redactor with visibly different rules for "is this a

credential", which looks like an inconsistency waiting to be tidied up.

Cause

it is deliberate, and unifying them would make the system worse.

The classifier is broader on purpose. It drives an embargo that withholds a subject

and a snippet from an index — a reversible action, undone by rebuilding once the

credential is dead. Over-classifying costs a few rows of searchability for a few days.

The redactor is narrower on purpose. It rewrites text permanently, so a false

positive chews a hole in prose that no rebuild restores, and a tool whose output looks

mangled is one people route around.

Check

before unifying two checks that look inconsistent, ask what each one does

when it fires. **Unifying them forces the reversible mechanism down to the irreversible

one's threshold**, which loses exactly the cheap safety the split was buying. Different

blast radius, different bar; write the reason next to both so the next reader does not

"fix" it.

Symptom

a credential redactor shipped with a hole, certified by a harness written specifically to find that class of hole, by an author who had already written the rule about it twice in this document.

What happened. Two review findings pointed opposite ways on one control: a live code was escaping (classify more) and ordinary mail was being withheld (classify less). The fix was right — it moved the discriminator from how code-like is this token to is this token in a delivered-value position. To verify it I wrote an acceptance harness before looking at the fix, so the check could not converge on the answer, and ran both sides in one build. Both passed. Variants passed. The cross-field case passed.

The next review round found Your verification code ABCDEF expires in 10 minutes stored raw. My harness had tested Verification code: AbCdEf. I had written the CLEAR form of a delivery — the one with a colon — into the harness built to catch clear-form failures.

The mechanism, which is stronger than "write the weakest input". A test author's intuition about what the input looks like is drawn from the same well as the implementation's. When I picture a verification-code email, I picture the tidy one: label, colon, value. So does the code — that is why the code already handles it. The test therefore agrees with the implementation by construction, and agreement is not evidence. It is the same intuition, consulted twice, reporting itself as confirmation.

Writing the test first does not break this. Writing it before seeing the fix does not break it either — I did both. Only the INPUTS decide, and they came from the same place the code did.

Check

something external has to force the input set away from the author's mental image. What worked here:

• Take the failing input from the corpus, not from imagination. Real provider mail

says code ABCDEF expires, use code ZXCVBN, code is 4 8 3 9 2 7. My picture of it had a colon in it. The corpus does not care what I picture.

• Ban the feature the previous version leaned on. The old harness leaned on the

delimiter, so every live case in the new one is delimiter-free. Removing the crutch is what surfaces whether anything else is holding the case up.

• Choose the negative cases to sit as CLOSE to the positives as possible, not as far.

Please review the code before ACCEPT is merged and Our code of conduct applies to EVERY contributor are one word away from a credential delivery. Distant negatives prove nothing; adjacent ones are what stop a fix from being a widened pattern.

• Let a different model pick the inputs. The reviewer that found this shares no base

weights with the author, which is precisely why its mental image of the input differed. That is not a nicety of process — on this branch it was the only thing that differed.

A test written by the author of the code — or of the fix — is drawn from the same intuition that produced the code. It will agree with the code about what the input looks like, and that agreement reads exactly like coverage. The input set is the only part a careful author cannot check by being careful.

Third instance of the test-agrees-with-the-bug shape on one branch (2026-09-01), and the first where the anti-pattern was written INTO the check built to detect it. The two earlier ones are §6.15 and §6.16.

Symptom

needle_pool_router answers newsletter questions from

D:\output\newsletter_kb\newsletter_kb.db. Every answer is well-formed,

attributed to a newsletter and a date, and carries no indication of its age.

Cause

that file is a local mirror. The live corpus is

/home/opc/email_db/newsletter_kb.db on Oracle, kept current by a 15-minute

timer. Measured side by side the same day:

| `newsletter_kb` | newsletters | issues | items | last written |
|---|---:|---:|---:|---|
| Oracle (live, 15-min timer) | 650 | **7,403** | **21,007** | minutes ago |
| Local mirror (what the registry reads) | 650 | 7,336 | 20,667 | **3 days earlier** |

67 issues and 340 items existed and were unreachable. The counts agree closely

enough that nothing looks wrong, which is exactly why nobody checked.

Cost

every newsletter answer given over those three days was silently

timestamped wrong. Not incorrect — stale, which reads identically. A question

like "what did the AI press report this week" returns a confident, sourced,

well-attributed answer about last week.

Check

for every pool, ask where does this file come from. If the answer

is "something else writes it", the pool is a mirror and must publish **how far

behind it is** alongside every answer. Compare row counts against the upstream

once; if they differ, the pool has been lying about currency for however long

the gap represents.

The general rule. A pool that reads a mirror must declare that it is a mirror and how far behind it is, or every answer it gives is silently timestamped wrong. Freshness is not a property of the data; it is a property of the copy, and only the copy can report it.

Symptom

all seven pools in tools/needle_pool_router/pools.py report

available = True, including the three-day-stale one.

Cause

Pool.available() resolves to Path.exists(). That is a real

check and it is not the one that matters: it distinguishes wired from

unwired, and says nothing about current versus stale. A pool deleted

yesterday and a pool frozen since May are the same answer.

Cost

the probe's green result is the only freshness signal the router

offers, and it is not a freshness signal at all. The registry's own docstring

argues carefully that a pool must be registered-and-unavailable rather than

absent, so "no pool has this" cannot be confused with "the pool was never

wired". The same reasoning has not been extended one step further: a pool can be

registered, available, and three days behind, and there is no state for that.

Check

read what the probe actually calls. If it is an existence test,

the tool cannot report staleness and must not be read as though it does.

Check

when a staleness signal is added, ask which surface got it. A

signal on the human-facing surface and not the machine-facing one leaves the

larger consumer unprotected, and makes the problem look solved.

Symptom

repos.obsidian_note_path exists as a column in

D:\projects\repoindex\catalog.db.

Cause

someone built toward this same Obsidian integration before and

stopped. The column is populated for 0 of 1,126 rows.

Cost

the work was done twice. Nothing in the schema, the catalogue or any

document said the integration had been started and abandoned, so the second

attempt began from the assumption that nothing existed. The column is not wrong,

not broken, and not flagged by anything — it is simply empty, which is

indistinguishable from a column nobody has gotten to yet.

Check

an unused column, an empty table or a config key nothing reads is

evidence of an earlier attempt, not of a clean slate. Before building an

integration, grep the schema for its nouns. If a field named after the thing you

are about to build already exists, someone was here.

*This is the same failure class as the vault itself: a 5,000-note Obsidian vault

and an installed Obsidian both already existed on this machine when the work to

"install Obsidian and create a vault" began. Checking took two commands.*

---

Symptom

tools/codex_pr_review.py returned a confident verdict with three named blocking findings on PR #389, and the findings were correct. It had not read the code. The diff was capped at 180,000 characters; 52,257 were dropped, and the dropped region was run_one.py and score.py — the executable harness. A second pass on the fixed head dropped 58,077 characters, again the harness. Both verdicts were derived entirely from the README.

Cause

a 208 KB data file (noop_raw.json, 880 measured rows) sat in the same diff as the code and consumed the budget before the reviewer reached it. Nothing was broken: the cap did exactly what it says. The defect is that a partial review and a complete one produce the same shape of output — verdict, assessment, findings — so downstream they are indistinguishable.

Cost

nearly merged on a review that had not seen the reviewed code. Caught only because this tool prints ⚠ Partial review. The diff was truncated to 180000 characters; N characters were not seen by the reviewer, with byte counts, in the posted body. A reviewer that had truncated silently would have produced the identical verdict and nothing would have flagged it.

Check

two parts, and the second is the general one:

1. Keep bulk evidence out of the review payload without removing it from the

repo — by storing it gzipped, not by setting a -diff attribute.

🔴 -diff does not work for this, and I shipped it before checking. path/to/data.json -diff in .gitattributes makes local git diff render "Binary files differ", so it looks fixed. But the reviewer reads GitHub's API diff (Accept: application/vnd.github.v3.diff), and that endpoint ignores .gitattributes and emits the text hunks anyway. Measured on PR #389: after the rule was committed and pushed, the next review truncated 60,349 characters — more than the 58,077 before it, because the rule itself enlarged the diff. Fetching the API diff showed the noop_raw.json hunks still present.

What works is making the file genuinely binary: noop_raw.json.gz, 208 KB → 13 KB, which git and GitHub both treat as binary with no attribute at all. The evidence stays committed and versioned, gunzip -k recovers the exact bytes, and the reader (score.py) opens the .gz directly.

The general form: a fix aimed at a rendering layer must be verified at the layer the consumer actually reads. Local git diff and the GitHub API diff are two different renderers of the same commit, and only one of them honours .gitattributes.

1. **Any review path we rely on must report its own coverage, and a reviewer

that cannot state what it read is not a reviewer.** Before acting on a verdict, read the coverage line — not the verdict — first. If a review tool has no coverage line, that is the bug to fix before trusting any verdict it has ever produced. This generalises past reviews: it is the same defect class as Part 3, a procedure reporting success while measuring an unknown fraction of what it claimed to measure.

(Filed under Part 9 as the nearest fit — the failure is a report of COVERAGE it did not have, rather than of currency. 2026-09-01.)

Symptom

a graph of the eleven retrieval pools showed decisions and memory as isolated nodes: no edge to anything, sitting apart from a dense cluster of the other nine.

Cause

bridges between pools were computed by asking each pool's FTS index whether it held a term. decisions is a JSONL file and memory is a folder of markdown. Neither has an FTS index, so neither was ever asked. Their isolation was a property of the probe, not of the estate — both are among the most heavily cross-referenced stores there are.

Cost

nearly shipped. The reading a viewer would have taken is "the two curated stores connect to nothing", which is false, would have looked authoritative, and would have been acted on: those are exactly the stores a retrieval fix would have then wired harder into everything else, solving a problem that did not exist.

Check

for every entity absent from a visualisation, ask whether the probe could have found it at all. Enumerate what was scanned and compare it against what was drawn. A node with zero edges is a claim; make the generator state which of its inputs it could not read, in the artefact itself, next to the picture.

Symptom

with presence-based bridges replaced by concentration, all 300 edges collapsed onto a single pair, cavemem ↔ session_raw. A tidy, confident, completely uninformative picture.

Cause

those two pools hold 1.4M chunks and 570k observations against, say, 1,904 videos. Any term concentrates there by sheer volume. The graph was drawing corpus size and labelling it shared subject matter.

Cost

the second version looked more rigorous than the first (it had a threshold, and a defensible one) and was wrong in a way the first was not, because now the shape was specific enough to believe. Normalising occurrences per row turned 1 bridged pair into 26.

Check

when a derived measure ranks entities of wildly different sizes, check whether it is ranking the size. Normalise, then see whether the shape survives. A metric that reproduces the row counts is the row counts.

Symptom

260 bridge terms, every one spanning all eleven pools. A hairball: maximal connectivity, zero structure.

Cause

the selection sorted by span descending and took the top N, which is selecting for ubiquity. Measuring the distribution explained why the correction was not merely a threshold tweak: 671 of 900 candidate terms appear in all nine searchable pools, and 190 more in eight.

Cost

nothing, once seen; but the finding underneath is load-bearing for the retrieval work, so it is recorded here rather than left in a commit message:

These stores share nearly all vocabulary. They differ in FORM, not in subject. Term presence therefore cannot discriminate between them, and any attempt to route across these pools by shared terms is structurally weak — not because it was tuned badly, but because the signal is not there to find.

Check

before building on a discriminator, plot its distribution across the population. If most items score the same, it is not a discriminator, and no amount of weighting will make it one.

Check

when several components each measure healthy and the system fails, the fault is in the edges. Draw them.

---

Symptom

I was one sentence away from reporting that the packaged build ships a purpose-built ffmpeg it never calls. The chain was clean: the installed app contains resources/ffmpeg/ffmpeg.exe alongside LGPL notices, so it was bundled deliberately; the collector's resolver runs where ffmpeg and validates whatever PATH returns; nothing in the codebase referenced resources/ffmpeg. Therefore the hardened binary is dead weight, the app silently depends on whatever ffmpeg the user happens to have, and on a clean machine transcription fails. Every step of that was true of the files I had read.

Cause

the files I had read were D:\projects\anycloudllm at HEAD, which was sitting on docs/welcome-formats-and-refusals — a docs branch off a 2.6.2 base. The shipped build is build/2.7.1-daylight-complete, nineteen commits further along. On that branch the resolver checks ANYCLOUDLLM_PACKAGED_FFMPEG_PATH first, injected by desktop/main.js when packaged, with PATH only as fallback; electron-builder.config.js fails the pack if the binary is missing or will not execute; verify-release.js fails the release if the installer does not contain it. The dependency had not been missed. It is one of the more carefully engineered parts of 2.7.1.

Cost

none, because it was checked. Had it shipped as a finding it would have cost more than the usual false alarm: it accuses a specific, careful piece of work of being pointless, and the natural response is to go and "fix" a wiring that is already correct — touching a pack-time verifier and a release gate to solve nothing.

Check

name the ref in the same sentence as the claim. Not "the code does X" but "<branch> does X", and if the claim is about a build, re-run the search against the ref that is actually installed before saying it out loud: git grep <pattern> <branch> -- <paths> is one command and it is the whole check. A conclusion about a shipped artefact that cannot name which ref produced it is not a conclusion yet.

Symptom

the finding required believing that the same people who cross-compiled a custom 8.1.2-anycloudllm-decode-only ffmpeg — --disable-network, --disable-everything, a hand-picked demuxer list, LGPL-clean, 1.8 MB instead of ~80, with a build script, a verifier and a test suite — then forgot to call it.

Cause

nothing. That combination does not happen. Meticulous work and a careless omission inside the same component is not a thing people do; it is what reading two different revisions looks like from the outside.

Cost

this is what actually stopped the report. Not a rule about refs, but the finding failing a plausibility check about people.

Check

when a finding requires someone to have been both unusually careful and obviously careless in the same place, suspect your sources before you suspect them. Ask what you would have to be reading for both halves to be true at once. Usually the answer is "two different versions", and usually you are.

---

Symptom

none yet. This was caught while adding COLLECTOR_HOST, before it could happen, by asking what else writes the file it lives in.

Cause

dumpENV() in server/utils/helpers/updateENV.js regenerates server/.env from protectedKeys, a hardcoded allowlist of names. Any environment key not on that list is simply not written back. So a new setting reads correctly, survives restarts, and behaves exactly as designed — until anyone saves any unrelated setting in the UI, at which point the file is rewritten without it and the value silently reverts to its default. No error. No log line. Nothing in the feature's own code is wrong at any point.

Cost

would have been hours, and they would have been spent in the wrong place. The reported symptom is "the collector setting stopped working"; the evidence is a collector, a port and a hostname; the cause is a save handler on a settings page with no relationship to any of it. The two search spaces do not intersect, so searching the obvious one thoroughly gets you nothing, repeatedly, which reads as "the bug is subtle" rather than "the search is in the wrong file".

Check

when adding a setting, the question is not "is it read?" but "what rewrites this file, and does that writer know about my key?" Grep for the file's own path, not for the key. Any allowlist-based config rewriter has this shape, and they are common: a .env regenerated by a settings save or by a CLI's config set, a systemd EnvironmentFile re-templated by a deploy script, registry policy reapplied by GPO. Each is a component whose job is to forget anything it was not told about.

Symptom

a first test asserted that the string "COLLECTOR_HOST" appeared in a 2,000-character slice of the writer's source.

Cause

that is a search for a token, not a check of a behaviour. It passes if the name appears in a comment, in an adjacent list, or in a variable that is never consulted, while persistence stays exactly as broken. The reviewer called it, and was right.

Cost

none here, but the shape is the expensive one: a green test standing guard over a property it does not test is worse than no test, because it converts "nobody checked" into "somebody checked" for every future reader.

Check

run the writer, intercept the write, assert the output. For a function with a hardcoded destination path that is a spy on writeFileSync, not a real file. And pair it with a negative: a key one letter different must be dropped. Without the negative the positive proves nothing about the allowlist — a writer that emitted every key in the environment would pass it just as happily.

---

Symptom

a pytest run invoked by a pre-commit hook stopped emitting output at 41% and stayed there. pytest-timeout was installed and --timeout=60 was on the command line. It never fired. The process was alive, held no lock anyone could see, and produced nothing for 51 minutes before it was killed.

Cause

commit charge stood at 31.6 GB against a 31.9 GB limit: 0.3 GB of headroom. At that pressure a process is not deadlocked, it is unschedulable — paging against a pagefile with nowhere to grow. pytest-timeout's thread method needs to be scheduled in order to fire, so the mechanism that exists to bound a hang is subject to the same condition as the hang.

Cost

the wrong diagnosis was nearly written down. A test file added minutes earlier monkeypatched os.walk, which is exactly the shape of thing that hangs a walk, and an A/B run "confirmed" it by completing without that file. The A/B was worthless: it used a different interpreter from the hook, one lacking pytest-asyncio and pytest-timeout, so two variables moved at once. Run in the gate's own environment the suspect file passed in 15.3 seconds, and the identical commit that hung for 51 minutes went through in about three minutes once the machine was quieter.

Check

before diagnosing a hang, read the commit charge, not free RAM. Free RAM looked survivable at 0.9 GB and would have been; 0.3 GB of commit was the number that mattered. And treat a timeout that did not fire as evidence about the machine rather than about the code: a watchdog that needs the scheduler is not independent of what it is watching.

Symptom

over one session the commit limit fell from 31.9 GB to 28.7 GB while free RAM improved from 0.9 GB to 5.1 GB. Read separately, the machine looked like it was recovering.

Cause

free space on the volume dropped from 27.0 GB to 24.5 GB, and Windows contracted the pagefile to match. Nothing consumed 3.2 GB of memory; the ceiling came down.

Cost

none, because the build that would have spent 8–12 GB of that volume was held. Had it run, the disk it consumed would have lowered the commit limit further, reproducing 13.1 at a larger scale and in the middle of a release.

Check

a moving commit limit is the pagefile resizing between its floor and its cap, not a disk symptom. 🔴 This section originally read "a falling commit limit is a disk symptom" and told you to look at the volume; that was the same wrong mechanism. What to actually do before a large job: read the commit charge and headroom, and remember the ceiling is bounded by MaximumSize — on this box 15.93 GB of RAM plus at most 16 GB of pagefile, whatever the drives are doing.

---

Symptom

the 2.7.1 entry, under a heading reading "Not in — say so rather than imply otherwise", said:

Audio and video are still not ingested. Transcription needs an ffmpeg binary, and none is shipped … The file that would matter is ffmpeg.exe, and there is no copy of it in the package.

Cause

the installed 2.7.1 contains resources/ffmpeg/ffmpeg.exe: a purpose-built 8.1.2-anycloudllm-decode-only cross-compile, LGPL-clean, --disable-network, 1.8 MB against a stock ~80 MB. The collector resolves ANYCLOUDLLM_PACKAGED_FFMPEG_PATH before falling back to PATH; the packer fails the build if the binary is missing or will not execute; the release verifier fails the release if the installer lacks it. The bullet was written before that work landed and was never revisited when it did — because nothing revisits a "Not in" list. The addition was announced in its own commits, which is precisely why nobody went back to the paragraph saying it did not exist.

Cost

nearly a release. A reader deciding whether to attempt audio ingestion would have been told, by the project's own changelog, not to bother — in a build that ships a hardened decoder specifically to make it work. The note inside the bullet is the sharpest part: it correctly warned that the ffmpeg.dll beside the executable is Chromium's media library and not the thing you want. That warning is still right. The conclusion drawn from it — therefore there is no ffmpeg — stopped being right and read exactly as authoritative either way.

Check

treat every "Not in" entry as expiring at the next release. Re-derive it from the artefact rather than carrying it forward: for each claimed absence, look for the thing in the built package. That is one ls per bullet and it is the only check that can catch this, because the absence has no owner and no event. When correcting one, leave the retraction visible instead of quietly deleting the line — the deletion looks like the claim was never made, and the next person cannot learn what shape of statement went bad.

Symptom

the same entry was headed "(in progress, not released)" and said "Status: not shipped. No installer has been built."

Cause

AnyCloudLLM Setup 2.7.1.exe had been built on 2026-08-30, published to the update bucket at 14:21Z at 817,572,860 bytes, and installed on the author's own desktop for three days. The changelog's own preamble states that three things "must agree and must never drift apart", and names the release process as what keeps them together. The release happened; the process step that updates this file did not.

Cost

the version this session was cutting was being derived from a document that denied the previous version existed. Both errors pointed the same way: understating what had shipped.

Check

the installed binary is the only unambiguous record of what a release contained. A branch name is a claim about intent; a changelog is a claim about memory. Opening the installed artefact settled three questions in one minute here — that the ffmpeg shipped, that 2.7.1 shipped at all, and that a branch called release/2.7-daylight was not what shipped, because the thing it lacks is sitting in the package.

---

## Keeping this current

1. Fed by the daily retro, which already names each day's failures: its last

step appends any entry that has a Check field and drops any that doesn't.

1. Ingested into the knowledge base on write, so entries surface on the

active subject of a conversation instead of waiting for someone to open a

file — being retrievable is weaker than being already present.

1. Reviewed by the test it applies: an entry that has caught nothing and

cannot be stated as a mechanical check gets deleted, not softened.

(Recommended, not built.)

---

*Parts 1–5 and 7 compiled from the daily retro of 2026-08-26, the memory stores behind this work,

and the operator's own corrections. Session-transcript search was unavailable

while writing it — itself an instance of Part 3, stated rather than papered over.*

*Part 6 added 2026-09-01 from a measured root-cause pass over the retrieval

estate — a 22-question ground-truth set run against all eight pools. Its numbers

are reproducible with python tools/test_recall_ground_truth.py in hermes_agents.*

Symptom

a review tool was changed to send only the diff since the last reviewed commit instead of the whole PR. That fixed a real problem: the cumulative diff had outgrown the reviewer and was being silently truncated. It also created a worse one, and it took a day to see.

Root cause -- an incremental diff contains only what CHANGED. A finding raised in an earlier round lives in code that has not changed since, so it is not in the increment. The reviewer never sees it, has no reason to mention it, and returns a clean verdict. That verdict then reads as "the earlier findings are fixed" when its actual basis is "I was not shown them".

Compare the two failure modes:

| | what the artifact said | what it meant |
|---|---|---|
| truncation | "%d characters were not seen by the reviewer" | partial, and it says so |
| incremental | "Verdict: APPROVE" | partial, and it says nothing |

Truncation was the honest failure. It declared its own omission in the posted review, so a reader could discount the verdict. The replacement produced a verdict that looked complete and was not -- which is the exact class the tool exists to catch other code doing.

Fix -- not a note in the docs, and not a person remembering. The mechanism:

1. Read the open set back from the reviewer's OWN posted reviews. GitHub is the

authoritative record; a local ledger can be absent, stale, or on another machine, and the one thing that cannot silently vanish is the review the gate itself posted.

1. Widen the diff base to reach the commit the findings were raised on. The reviewer

runs read-only in an empty directory -- it can only clear what it is SHOWN, so asking it about a fix that landed outside the slice is asking an unanswerable question.

1. Carry every open finding into the prompt and require a per-item status:

resolved / still_present / unknown.

1. Refuse the approval while any is not resolved. Silence about a finding is read as

unknown, and unknown blocks.

unknown being a first-class answer is the load-bearing part. It is the honest reply when the fix is not in the slice, and making it available is what stops the model guessing "resolved" to be agreeable.

The general rule -- when you narrow what a checker looks at, ask what used to be in view that no longer is. Narrowing improves the signal on what remains and silently zeroes the signal on what left, and a zero signal is indistinguishable from a clean one unless something makes it speak.

Corollary, found while fixing this -- the guard that widened the base was written as subprocess.run([...], cwd=ROOT, ...) inside try/except Exception, and ROOT is not defined in that module. The NameError was swallowed, reaches stayed False, and the base was widened unconditionally while appearing to have been tested. A bare except around a safety check turns it into a coin with "skip" on both faces. Catch the error the operation can actually raise -- OSError for a subprocess -- and let a bug in your own code crash.

---

Symptom

a branch could not be rebased, checked out over, or reset --hard. Every git operation refused in turn, naming tools/decisions/run_service.ps1 -- a file the branch had never modified and which had nothing to do with its changes.

Root cause -- the file was a mixed-line-terminator blob under text eol=crlf. Git normalised it on checkout, the working copy never matched the index, and it was permanently dirty. The fix had landed on main a few commits earlier. Every branch that had diverged BEFORE that fix still carried the broken blob and was therefore unrebasable -- and the error pointed at the file, not at the missing fix.

Fix -- rebuild the branch on top of current main rather than trying to move main under it:

git checkout -f -q --detach origin/main
git switch -q -c rb/<n>
git cherry-pick <the branch's own commits>

Clean, no conflicts. The cherry-picks apply because the blob is no longer in play.

The general rule -- when a rebase is blocked by a file the branch never touched, look for a fix that landed after the branch diverged. A defect repaired on main can still block every branch older than the repair, and it presents as an unrelated dirty file rather than as a merge conflict. git log origin/main -- <the named file> finds it in one command.

---

Symptom

a fix was dispatched to a remote Codex instance with a prompt that ended ## THE CODE and no code beneath it -- a bug in the dispatcher. Codex returned {"fixes": [], "notes": "No code was included after \"## THE CODE\""}. It refused to guess, said why, and cost one re-dispatch.

The contrast worth understanding -- PR #320, same tool family, reported that it had "performed the following exact steps" and produced nothing at all. One refused a task it could not do; the other narrated a task it had not done.

The difference is not model quality and should not be assumed to be. The dispatch that refused had a structured output schema with a fixes array -- an empty array is a representable, valid answer, so "I have nothing" was sayable in the tool's own vocabulary. The dispatch that confabulated was returning free prose, where the only available shape is a narrative and the cheapest narrative is a successful one.

The general rule -- give a delegated worker a way to say "nothing" that is a first-class value rather than an absence, and it will use it. A schema whose empty case is valid buys more honesty than any amount of instruction not to fabricate. This is the same mechanism as unknown in 1.32: the honest answer has to be sayable before it can be said.

Symptom

a PR was approved, mergeStateStatus said CLEAN, and gh api repos/O/R/commits/<sha>/status -q .state said pending. Waiting for it to go green cost ten minutes. It was never going to.

Root cause -- statuses=0. The legacy commit-status API had no statuses on that commit at all, because this repo reports CI through check-runs, not statuses. The combined state of an empty set is defined as pending, so the endpoint reports "pending" forever on any repo that does not use the legacy API. The actual result was one directory over:

gh api repos/OWNER/REPO/commits/SHA/check-runs

test: completed / success, plus mergeStateStatus: CLEAN, which is GitHub's own statement that every required check is satisfied.

The general rule -- an aggregate over an empty set is not a measurement. Before believing a rolled-up status, ask how many things it rolled up; pending from zero inputs and pending from a running job are the same string and different facts. The same shape appears in "0 of 0 tests failed", an all-green dashboard with no probes registered, and a health check whose target list is empty.

---

Symptom

across four consecutive review rounds, the same finding came back as STILL PRESENT and then UNKNOWN. The fix had been in the branch since round 5. Each time it was tempting to conclude the reviewer was being stubborn, or that the model had degraded.

Root cause -- the review prompt included a supporting "evidence" diff for carried-over findings, built with one git diff <base>..<head> -- pathA pathB pathC and then cut to a 60,000-character budget. git diff emits files in alphabetical order. tools/codex_pr_review.py consumed the entire budget and tools/pr_assembly_start.py -- where the fix lived -- arrived as nothing. The reviewer said so plainly every round: "the relevant implementation was not visible in the supplied evidence." That was accurate. The finding was unresolvable because the evidence was allocated by filename order.

Fix -- give each path its own share of the budget rather than letting a shared pool be drained first-come, and declare every truncation per path. On the next round the reviewer assessed the finding and cleared it in one pass: "the start path is serialized by an atomic exclusive lock held across read/check/spawn/write."

The general rule -- when a checker keeps returning the same answer, audit what you handed it before concluding it is wrong. A repeated verdict is evidence about the input at least as often as about the checker. And a shared budget consumed first-come is a silent truncation with an ordering bias: whatever sorts first is fully represented and whatever sorts last is invisible, which looks nothing like truncation and exactly like absence.

Corollary -- the reviewer had a way to say "I could not see it" (unknown), and used it, and that is the only reason this was diagnosable at all. A checker with no vocabulary for "not shown" would have had to guess, and a guess here reads identically to a finding.

Symptom

five times in one session, across five different files, a check was covered by a test that passed whether or not the check existed. Every one was written by the same person who had just spent the night finding this defect in other people's code.

| what the test did instead | why it passed regardless |
|---|---|
| copied the production regex into the test body | the copy was correct; the original could be deleted |
| asserted a post-condition of a run | the run returned early; nothing had happened to post-condition |
| exercised a helper the call site never used | the helper was correct and unreachable |
| used a fake that ignored the flag under test | the fake raised whether or not the caller asked it to |
| searched `client/src/**/*.css` for a file in `client/src/` | `**/` needs an intervening directory, so a file at the root of the tree is invisible — the search reported ABSENCE and the code was fine |

They are one defect, not four. The test and the thing it guards were in different rooms — a copy, a proxy, an unreachable path, an inattentive double. In each case the test asserted something true about a stand-in for the mechanism.

The tell was identical every time: the sabotage that should have turned it red didn't. That is the whole detector, and it is cheap — revert the guard, run the suite, require failure. A guard whose removal changes no test result is not guarded.

And the detector was luck, twice, which is why four is a lower bound. Two of the four surfaced only because a patch script asserted on a stale anchor, aborted partway, and left later edits unapplied — while the suite stayed green and reported success. Nothing in the process was looking; the discovery was a side effect of an unrelated failure. Any count of this defect found by accident is a floor, never a total.

The procedure that closes it:

1. After every change to a guard, revert exactly that guard and re-run its tests.
2. Check the anchor actually matched. An aborted patch script and a passing suite

are indistinguishable from the outside — both print success.

1. If the suite stays green, the test is not testing the guard. Find what it is

testing before writing another one.

1. When a new check is added upstream of an old one, re-run the whole sabotage set.

The new check silently takes over the old one's job, and the old test keeps passing while testing nothing. This happened three times in one PR.

Related: 1.32 (a clean verdict that means "I did not look") is the same shape one level up — there the reviewer, not the test, was in the other room.

---

Symptom

a review gate reported, correctly and in detail, that it could not confirm whether a finding had been fixed. It said so again the next round, and the next. The fix had been in the branch the whole time. The system was behaving impeccably and achieving nothing.

Root cause — the gate assembles a supporting "evidence" diff for findings carried over from previous rounds, and decided which files to include by testing whether each one exists. It tested existence in the reviewer's own checkout, which is normally main. Every file a pull request ADDS is absent from that tree — and new code is what most findings are about. Both files a finding named were dropped, the reviewer was handed nothing, and it answered honestly that it could not tell.

The omission was disclosed. The prompt said which paths had been left out and why. That disclosure is what made the failure survive: it read as a system working as designed, so nobody looked past it. Disclosing that you looked in the wrong place does not turn it into looking.

Fix — git cat-file -e <reviewed_sha>:<path> asks the tree the finding is actually about. A git that cannot run returns false and the path is disclosed as omitted, because failing to ask is not evidence the file is there.

The general rule — a component that reports its own limitation accurately is honest, not correct, and the two are easy to confuse because honesty is the thing you were checking for. When a truthful "I cannot tell" repeats, the bug is upstream in what it was given, not in the component saying it. See 1.37: a repeated verdict is evidence about the input at least as often as about the checker.

---

Symptom

a pull request appeared carrying the same six files as an existing one, on a much fresher base. It looked strictly better: nine commits behind main where the original was two hundred and eighty. It had zero open review findings. The original had five.

Root cause — not because they were fixed. Because a re-applied patch arrives as a new pull request with an empty review history. The merge gate carries unresolved findings forward within a PR and refuses to approve while any remain open; nothing carries them across one. Merging the replacement would have discharged five open findings without a single one being answered.

Worse, one of the three code differences between them moved away from what a finding asked for — a non-zero exit changed to return 0 on an empty result, where the open finding was precisely that the command always exits 0.

This is a bypass nobody would recognise as one. No rule was broken, no gate was disabled, and the replacement PR is what a helpful contributor produces when asked to rebase something stale. The review obligations simply do not have identity independent of the pull request that raised them.

Fix, when you meet one: close the replacement, and carry any genuine improvement in it onto the original first — otherwise the close is lossy and the next person recreates it. Record the reasoning on both PRs.

The general rule — review obligations attach to code, but review state attaches to pull requests. Any operation that gives the same code a new pull request — a re-apply, a fresh branch, a squash into a new PR, a fork-and-reopen — silently resets the state while the obligations are unchanged. Before merging a PR that duplicates another, diff the two heads and compare their open findings, in that order.

Symptom

a 219-commit-old branch carrying a five-line security fix. Checking out its files onto current main produced a diff that was correct about the fix and wrong about everything else:

| taking the file wholesale | what it silently did |
|---|---|
| `web/library/rag.html`, `repos/index.html` | **deleted** main's favicon, feedback script, navigation module, design CSS and font preloads — all added since the branch was cut |
| `.jules/sentinel.md` | **deleted** main's 2026-08-10 security record |
| `data/movies.db` | **overwrote** the live database with a 219-commit-old snapshot |

None of those three is in the pull request's intent. All three are in its diff.

Root cause — a branch is not a change; it is a photograph of the whole repository at the moment it was cut. git checkout <branch> -- <file> restores the photograph. Every commit main has gained in that file since is reverted, and nothing announces it: the diff looks like the branch's own contribution because that is exactly what it is.

The rule — port the change, not the files. Concretely:

1. Diff the branch's file against main, not against the branch's own base. If the

only difference is the fix, taking it wholesale is safe — verify, do not assume. (shared.js passed this test; the two HTML files failed it.)

1. Where it fails, apply the specific hunks onto main's current version.
2. For generated or binary artefacts — databases, bundles, lockfiles — take main's,

always. A stale branch's copy is a snapshot of a moving thing.

1. For append-only records — changelogs, security logs, decision journals — merge, do

not replace. Take main's content and append the new entry.

1. Afterwards, diff the rebuilt tree against the branch head and read what is missing.

That is what catches the parts you dropped by accident rather than by choice. It found seven comment lines in one rebuild and a whole email_index parameter in another.

The general form — this is 1.43's sibling. There, review state attached to the pull request rather than the code; here, the whole repository state attaches to the branch rather than the change. In both, an operation that looks like it moves one thing moves everything around it too.

---

Symptom

a live javascript: URI XSS sat on a deployed site, reachable, in a repository with 468 pull requests and three cross-model reviews in its history. It was not missed by the gate. It was never going to be seen by it.

Measured across nine repositories (a floor — the last 40 PRs each, some fetches erroring):

| repository | PRs ever | decision-issuing reviews found |
|---|---:|---:|
| `orchestrator-gpt` | 468 | 3 |
| `voice2voice-buddy` | 173 | 1 |
| `soul-server` | 29 | 3 |
| `openrouter-github-dashboard` | 29 | 0 |
| `ivan-workspace` | 10 | 0 |
| `chloe-android` | 2 | 0 |

COMMENTED is not a verdict, so the Gemini comments in the bottom three do not count.

The thing worth knowing, and it is the opposite of the obvious conclusion — the first read was "the app is not installed in those repositories". It is. Asking the App itself, with its own JWT, rather than asking the user token:

GET /app/installations -> repository_selection: "all" GET /installation/repositories -> 1,138 repositories, all nine among them

So the gate is installed everywhere and invoked in one place. That is a much cheaper problem than a missing installation and a much worse one to leave alone: there is nothing to buy, nothing to approve, and no reason it was not already happening except that nobody ran it. tools/codex_pr_review.py takes --repo owner/name and worked unmodified against three other repositories on first use.

The general rule — "not installed" and "installed and never invoked" look identical from the outside and have completely different fixes. One is a decision; the other is a habit. Settle which one you are looking at before proposing either. The user token cannot answer it — it returns 403 on user/installations — and the answer is one JWT-authenticated call away.

Symptom

a regex that looks correct in the editor, in the diff, and in review, and matches nothing at runtime. Zero matches then read as a finding: "no work verbs in these utterances", "no AI content on these channels". Exit 0 throughout.

Three separate breakages of the same pattern in one session, each from a different layer of escaping:

| attempt | what reached the file | what the engine saw |
|---|---|---|
| heredoc → python → file | `\b` collapsed to `\x08` | a literal **backspace byte** |
| the fix for that | `\b` written verbatim | a literal backslash then `b` |
| built with `chr(92) + "b"` | `\b` | a word boundary |

The backspace byte is the dangerous one: it is invisible in an editor, invisible in git diff, and invisible in review. Everyone who looked at it saw \b.

Root cause — escaping was crossing three layers (shell heredoc → Python string literal → file contents) and each layer consumed one level. Counting the levels by eye is the error; it went wrong in a different direction on the second attempt, which is the signal that the method, not the count, was the problem.

The rule — never accept a pattern because the source text looks right. Import the module, reload it, and assert against the compiled object, with one string that must match and one that must not:

import importlib, sys; sys.path.insert(0, ".")
mod = importlib.reload(importlib.import_module("tools.discord_voice.intent"))
assert mod.WORK_VERB.search("can you finish the corpus"), "work verb not seen"
assert not mod.WORK_VERB.search("can you see something"), "perception read as work"

Both ends are required. Asserting only the positive passes on a pattern that matches everything; asserting only the negative passes on one that matches nothing — and matching nothing is exactly the failure being hunted.

⚠ THE RULE ABOVE IS CORRECT AND IT DID NOT WORK. The session that wrote this entry hit the trap four more times the same day, on the same escape, hours after documenting it. That is the finding, and it is bigger than the escape: a rule that must be remembered at the moment of writing is documentation, not a control. Same shape as a branch guard bypassed by passing a parameter, and a poller reading a permanently-empty key — three in one day.

So the control exists now: tools/safe_patch.py.

from tools.safe_patch import BS, NL, WORD_BOUNDARY, patch, check_compiled
patch(path, old, 'x = "line' + NL + '"', "label")     # anchor-gated
check_compiled(path, lambda m: m.PATTERN.search("please run it"))

• NL, BS, WORD_BOUNDARY are built with chr(92), so no heredoc or outer

string can eat them.

• write_text() parses before it writes — a collapsed escape becomes a refusal

with the file untouched, instead of a broken module found by the next command.

• patch() refuses when the anchor matches zero times or more than once: a

no-op patch reports success exactly as loudly as a real one, which is how an unedited file got deployed three times in one session.

• check_compiled() covers what parsing cannot — a  that became 0x08

parses perfectly and matches nothing. Only the engine can say.

tests/test_safe_patch.py pins all of it, including that NL != " " and WORD_BOUNDARY != "" — the two ways the escape has actually failed here.

Generalises past regexes: any artifact built by string manipulation must be verified by the thing that will consume it, not by reading what was written. Same family as 1.30 (the phrase stripper that inverted meaning) and 1.38 (the test that built its own inputs).

Symptom

two modules existed on exactly one disk. git status reported clean, git add -A added nothing, and a session was one command from reporting "everything is committed". No output anywhere would have contradicted it.

Root cause — a correct secrets rule, _token, matched speaking_token.py and tests/test_speaking_token.py, which hold no secret. Ignoring is designed to be silent, so the absence of a warning is not evidence of anything.

Fix — un-ignore by explicit full path. Never weaken the pattern, and never with a wildcard: !_token.py re-opens the door for a real credential file that ends in .py.

The guard, and how its first version was wrong — the obvious test runs git check-ignore over every source file. That question is "does a pattern match this path", not "is this file invisible" — .gitignore has no effect on a file that is already tracked. It named 13 committed files, including tools/secrets_index.py, which CLAUDE.md instructs every session to run. A guard that cries wolf gets switched off, and then the real case walks through.

Ask git one question that already means what you want:

git ls-files --others --ignored --exclude-standard -z -- tools tests

--others --ignored is precisely "on disk, untracked, and ignored". Deriving it by intersecting two narrower answers is a second implementation of ignore-matching, and two layers disagreeing about what exists was the whole defect. Same family as 1.41.

The guard also plants a file and requires itself to notice it. It had already been green for the wrong reason once; a green check that has never been shown to go red is not evidence.

Symptom

the cross-model review gate produced a verdict that raised three blocking findings against scripts/refresh-free-roster.mjs in a pull request that has never touched that file, marked all three of the PR's real findings UNKNOWN, and truncated 447,915 characters. Read as a defect report it was damning and specific, and I spent an hour fixing the gate.

The gate did not have the bug. tools/codex_pr_review.py landed on main and was improved twice more after the working branch was cut, so on that branch the file is untracked — git status prints ?? tools/codex_pr_review.py — and python tools/codex_pr_review.py runs a copy that predates both fixes. The version on main already contained the fix for the exact symptom, with the measurement in a comment.

| base the stale copy used | chars | files | of which the PR's |
|---|---:|---:|---:|
| the pre-force-push head, orphaned by a rebuild | 630,084 | 35 | 4 |
| the correct PR base | 18,851 | 10 | 10 |

Re-run from main's copy: zero truncation, all three real findings cleared, 75 seconds instead of 227.

Two failure modes stacked, and the second one nearly cost more than the first.

1. An untracked file is invisible to every check that reasons about the branch.

git status reports it, but as noise among other untracked files, and nothing distinguishes "a scratch file I made" from "an older copy of a tool that landed on main since".

1. A measurement taken with the wrong build is still a real measurement. The

630,084 characters were real. The misattributed findings were real. Everything I observed was true of the thing I ran, which made the diagnosis feel well-evidenced at every step — and it pointed at the wrong code. I wrote a fix, tests, and a sabotage harness for a bug that had already been fixed, and the only thing that caught it was porting the change onto main and seeing 534 deletions in the diff (see 1.44 — the rule caught its own author).

IT WAS NOT ONE FILE. IT IS A CLASS, AND IT IS FIVE. Asked properly -- which untracked files in this working tree shadow a file that IS tracked on main -- the branch answered:

| untracked shadow | local | on `main` | |
|---|---:|---:|---|
| `tools/codex_pr_review.py` | 48,674 | 74,059 | stale by two merged fixes |
| `tools/pool_graph_vault.py` | 27,037 | 39,385 | stale |
| `tools/pr_assembly_start.py` | 8,705 | 16,157 | stale |
| `tests/test_pool_graph_vault.py` | 3,331 | 16,859 | stale |
| `tools/pr_assembly_line.py` | 45,865 | 43,137 | **LARGER -- unmerged work, not a shadow** |

The last row is why "delete the untracked copies" is the wrong reflex: it defines six functions main does not have. Size does not tell you which is which; the symbol set does. comm -23 over ^(def|class) on both versions separates stale copies from unmerged work in one command.

The rule — before diagnosing a tool from its output, establish which copy ran. git log --oneline -1 -- <tool> on the current branch versus origin/main answers it in one command, and git ls-files --others --exclude-standard, filtered to names that git cat-file -e origin/main:<path> resolves, enumerates the whole class. Untracked tooling makes the question invisible: git status prints these among genuine scratch files, and nothing distinguishes "a file I made" from "an older copy of a tool that landed on main since". For a repo tool with an estate-wide job, run it from a clean worktree on main, not from the feature branch you happen to be standing on:

git worktree add ../.wt-main origin/main python ../.wt-main/tools/<tool>.py ...

The general form — 1.41 is about guards verified by proxy; this is the same error one level up. There, the test exercised something adjacent to the guard. Here, the diagnosis exercised something adjacent to the code. In both, every observation was genuine and the conclusion was about the wrong object.

---

Symptom

a review was posted on a pull request; Jules pushed a commit "with your requested changes" fourteen minutes later. It was not a fix. It was 35 files and 3,621 deletions, and among them:

| | on `main` | at the reviewed commit | after the automated push |
|---|---:|---:|---:|
| `safeUrl()` — the `javascript:` URI XSS fix merged that morning | present | present | **gone** |
| the service-worker fixes the review had asked for | — | present | **reverted** |
| the tests proving them | — | present | **deleted** |
| `web/council/free-roster.json` | present | present | **deleted (287 lines)** |

The requested change was made; everything the branch had not seen in 42 commits was undone alongside it, and none of it appears in the diff as a deletion. It appears as the branch's own content.

Root cause is 1.44 exactly — a branch is a photograph of the whole repository at the moment it was cut — with one difference that matters: nobody chose it and nobody is reading the diff. A human rebuilding a stale branch might notice 3,621 deletions. An agent reacting to a review comment ships it in fourteen minutes, and the pull request still shows a green preview deploy.

Fourteen minutes is the tell, and it is the whole finding in one number. It is faster than anyone looks. A human reviewer posts a comment and moves to the next PR; the revert lands, the preview deploy goes green, and the next person to open the page sees a pull request that says its requested changes were made. There is no window in which a person is between the review and the write, because the write IS the response to the review.

What made it safe was luck of ordering. main was never touched, so the live site kept the fix; the revert only reaches production if that PR is merged, and a human converted it to draft. Had the review round finished ten minutes earlier, merging would have been the obvious next step. The exposure is structural, not incidental — the same loop deletes any fix that landed after the branch was cut, and nothing about a security fix makes it more visible on the way out than anything else.

The rule — on any repository where an automated agent has write access to PR branches:

1. Never merge on a green check after an agent has pushed. Diff the agent's head

against main and read the deletion count. git diff --stat main <head> | tail -1 is the whole check. A number in the thousands on a nine-file PR is the tell.

1. Re-verify anything you merged that day if an agent has since pushed to a stale

branch. The revert is silent and looks like the branch's own content.

1. A review comment is an input to an agent, not just a record. Posting one on a

stale branch can trigger a rebuild-from-photograph. If the branch needs rebuilding, rebuild it before reviewing it, not after.

1. Switching the agent to a mode where it acts only on explicit mention removes the

trigger, but that is an account-level setting and therefore a decision for the owner, not a fix to apply mid-task.

THE MECHANISM, MEASURED — AND IT IS NOT WHAT I FIRST WROTE. My first reading was "a review is an instruction to a system that can act on it." A peer session (hermes-agents-7a) proposed a narrower and worse explanation, and it is the one the evidence supports.

The test is cheap and decisive: compare the trees. If the agent acted on the review, its push contains work. If it merely woke and published, its tree is byte-identical to the branch head from before.

git rev-parse <old-head>^{tree} -> a14336fb64486a56e5a91f0caa4b417a22d7ec67 git rev-parse <agent-push>^{tree} -> a14336fb64486a56e5a91f0caa4b417a22d7ec67

Identical. It made zero edits. It re-pushed its stale snapshot verbatim, under its own original commit message — a message about regenerating a glossary bundle, which had nothing to do with the review it was supposedly responding to.

So the sequence is not read review → decide → edit → push. It is:

1. A message arrives at a session that reads COMPLETED. COMPLETED is not terminal —

the peer measured four sessions returning to IN_PROGRESS on being messaged.

1. The revived session runs to its completion step.
2. The platform publishes whatever the working tree contains — and that tree is a

depth-1 clone with no network in either direction (fetch and push both exit 128), so it cannot compute a merge base or see one commit of what has landed since.

It never chose to revert the security fix. Its snapshot predates it, and the completion step publishes a tree rather than a diff.

Three consequences, all stronger than the reading they replace:

• The blast radius is the whole tree, always. It is not proportional to what the

message asked for. That is why fourteen minutes was enough: no work was required, only publication.

• An empty message would have done the same damage. So would a status nudge, or

anything else that wakes the session. The trigger is contact, not content.

• "Rebuild before reviewing" holds, for a better reason. It is not "avoid giving it

ideas" — it is do not wake a session whose tree is stale.

The rule, restated — before sending ANYTHING to a parked agent session with write access to a branch, establish how old its tree is. If the branch is behind, rebuild it first or do not make contact. And on any repository where such an agent can push:

1. Never merge on a green check after an agent has pushed.

git diff --stat main <head> | tail -1 is the whole check; a deletion count in the thousands on a nine-file PR is the tell.

1. Compare trees to tell a response from a republication. Equal trees mean nothing

was done and everything was overwritten.

1. Re-verify anything merged that day if an agent has since pushed to a stale branch.

The revert is silent and reads as the branch's own content.

1. Switching the agent to act only on explicit mention removes the trigger, but that is

an account-level setting and therefore the owner's decision, not a mid-task fix.

Second instance, which is what makes it a pattern: the same peer measured a stale-branch dispatch that produced a PR deleting ~1,500 lines (closed unmerged). Two incidents, one mechanism, neither caused by anything in the message.

The general form — 1.45 said the cross-model review gate is an estate-wide policy that exists in one repository. This is the same shape with worse consequences: an agent's push permissions are estate-wide, its picture of the repository is one snapshot old, and nothing reconciles the two at publication time. The publication step trusts the tree, and the tree is the one thing nobody re-checked.

⚠ Related trap from the same session: a merged PR keeps accepting pushes to its head branch, and every local signal reads healthy — pushed, clean, up to date — while the commits are unreachable from anything.

Symptom

four consecutive verifications, all accurate, all reported as evidence for something they did not establish. git push succeeded. local == remote. Working tree clean. Full suite green. Every one true. The work was unreachable the whole time.

Root cause — the checks answered did this operation succeed. The claim being made was this work is safe. Those are different propositions, and no amount of rigour on the first says anything about the second. The verification was sound and pointed at the wrong sentence.

This is the same family as the wrong-layer trap (1.41, 1.47) with the error one step earlier: not measuring the wrong layer, but measuring the right layer for a claim nobody had stated. Rigour makes it worse rather than better — four green checks read as four times the confidence.

What would have caught it — a question about the destination, not the operation:

| the question asked | the question that mattered |
|---|---|
| did the push succeed? | where does this branch's work end up? |
| do local and remote match? | is there anything open that would collect it? |
| is the tree clean? | is this reachable from the default branch? |

git merge-base --is-ancestor HEAD origin/main   # 0 = reachable
gh pr list --head "$(git branch --show-current)" --state open

The rule — before reporting a verification, say out loud what it proves, then compare that sentence to the one being claimed. If they differ, the check is decoration. Ask what would have to be true for the claim to be false, and test that.

Same session, same shape, smaller: grep -c '<<<<<<<' file && cat >> file <<'EOF' … — the grep printed 0 and exited 1, so && skipped the append, while the git add && echo "ported" written after the heredoc terminator ran as a separate statement and printed success anyway. The confirmation message was decoupled from the operation it claimed to confirm. Caught only because a convergence check contradicted it.

Symptom

#399 reported MERGED. Its content never reached main, 54 commits sat on its head branch, and every local signal read healthy.

Root cause — the PR merged into a base branch that had already been merged and abandoned:

#381  feat/proactive-stalled-work -> main                     00:50:31Z
#399  feat/voice-dispatch-v2 -> feat/proactive-stalled-work    05:32:47Z   (4h42m later)

The base was a dead end by the time the PR landed in it. GitHub reports such a PR as MERGED — which is accurate and means nothing about reachability.

Then the head branch keeps accepting pushes. A closed PR does not lock its branch, git push succeeds, the refs match, and CI passes. There is no event that fires when a merge target becomes a dead end — same family as the "Not in" list entry: an absence with nothing to announce it.

Detection — and the wrong way to do it. The obvious probe is "commits on the head branch not reachable from the merge commit". That is wrong for squash merges: a squash commit is never an ancestor of the branch's original commits, so every squash-merged branch shows up. It reported 22 branches. Measured by commit time against mergedAt, which squash topology cannot fake, the real answer was 4 — one of them a genuine orphan from another session, unmerged since 2026-08-29.

gh pr list --state merged --limit 40 \
  --json number,headRefName,mergedAt,mergeCommit > merged.json
# then, per row:  git log --since "$mergedAt" origin/$headRefName

Examine the filter, not just the count: an alarming number that comes from the wrong predicate is worse than no number, because it gets reported.

The fix, when it happens — merge main into the branch and resolve in a merge commit. Do not rebase: rebasing rewrites commits that were already reviewed, and a new PR number is not a new review, it is an erased one.

---

Symptom

the same defect was swept for twice, by two sessions, using two reasonable filters. One found 6 files, the other found 15. Neither filter was wrong. Each was blind to the other's population, and the fact that they agreed on the files both could see read as corroboration.

The defect: shell scripts committed mode 100644 — no executable bit — that are invoked by bare path, where the result is Permission denied, exit 126.

| filter | question it asked | found |
|---|---:|---|
| grep for call sites matching `./name` | *which scripts are invoked that way?* | 6 |
| enumerate one directory and ask who calls its contents | *who invokes the things in here, and how?* | 15 |
| the raw count of `100644` shell scripts | — | **147**, and almost all irrelevant |

147 is a count of a PROPERTY, not of a defect. Most of those scripts are invoked as bash script.sh, where the bit is meaningless. Reporting 147 would have been the more alarming and less useful answer.

The move that turns a property into a defect — ask what would have to be true for this to matter, then measure THAT. Here: not "is the bit missing" but "is it missing on something invoked in a way that needs it". The same question as does this monitor have a source, is this test actually collected, did the state change or only the command succeed.

And each filter's blind spot was structural, not careless. A ./name grep cannot see a caller that builds the path — tools/chloe_probe.py does "/opt/fleet-bin/" + w, and no pattern for ./ will ever match it. A directory-first sweep cannot see scripts outside the directory. Both are correct methods with different denominators.

What closed it was neither sweep. A test that enumerates the whole population from git ls-files and asserts the property on every member found a sixteenth file both sweeps missed — deploy/fleet-bin/_real/pi-agent.sh, in a subdirectory outside the glob, and the passthrough target of keyholder.py's REAL_DIR. The sweep finds what the pattern admits; the test finds what the population contains.

⚠ And measure through a layer that can represent the thing. The obvious check — os.access(path, os.X_OK) — is true for everything on Windows, because NTFS has no executable bit and nothing local consults one. A filesystem check would have passed on that machine forever, no matter what was committed, and looked like coverage. The test reads modes from the git index instead. That is the same class of error as the defect itself: a Windows-authored repository cannot record the bit, so nothing on Windows can notice it is missing — which is exactly why it was missing.

The rule, in three parts:

1. Before reporting a count, ask what would make each instance a defect, and

measure that instead of the property.

1. When two sweeps agree, ask what population each one could not see. **Agreement

on an overlap is evidence about the overlap and nothing else.**

1. Replace the sweep with a test over the enumerated population, measured through

a layer that can represent the property. A sweep is a snapshot of one pattern's reach; a test is the only thing that catches the next one.

Related — 1.48 (which copy of the tool ran), 1.41 (guards verified by proxy), and the chloe-android gradlew instance that started this: mode 100644, broken for as long as the repository existed, findable only by a Linux build, and found within minutes of one finally running.

---

Symptom

an exact search for the link returned nothing. The store's newest observation was 2026-09-25 20:54 UTC, minutes old, across 665,161 rows. The natural reading — "memory is fine, the link was never said" — was wrong.

Cause

the store was fed by several independent ingesters, and only the one for this chat application had stopped. Global freshness (MAX(ts) over the whole table) was being kept current by the other sources. Grouped by source, the chat's newest row was 07:21 UTC that morning — thirteen hours behind, while the table as a whole looked live.

Cost

every message in that chat since 07:21 was unrecallable, including both documents; and the global freshness figure actively argued against looking for an ingest fault.

Check

freshness is per source, never per store. SELECT source, MAX(ts) FROM observations JOIN sessions … GROUP BY source, and compare each against the modification time of that source's raw files. A current store with one stale source is a filter; find which source, then find its filter.

Related — the same pass found that one of the two "missing" links had been recalled all along: the operator had pasted the Opus 5 guide (…/prompting-claude-opus-5), and the search was for the 5.5 URL (…-5-5) he later remembered. A negative on the exact string you remember is not a negative on the thing that was said. Search the raw transcript for a loose form first, then the index for what the transcript actually contains.

Symptom

the ingester's source directory, %APPDATA%\Claude\local-agent-mode-sessions, held 784 transcripts when inspected from a terminal. The scheduled job, reading the identical path, logged candidates: 0 on every run.

Cause

the desktop application is MSIX-packaged. A process running inside the package container — the app itself, and every shell or tool it spawns, including the terminal used to investigate — has reads of %APPDATA%\<App> silently redirected to %LOCALAPPDATA%\Packages\<PackageFamily>\LocalCache\Roaming\<App>. A process outside the container, such as a Task Scheduler job, reads the literal path, which on this machine was an empty directory. They are two distinct directories on disk, not a link.

This fooled two earlier investigations of the same family, for three reasons that are worth knowing before you meet it:

• There is nothing to find with link tools. Get-Item, dir /AL and fsutil reparsepoint query report no junction, symlink or reparse point at any level of either path. The redirect is done by a filesystem minifilter driver, below anything those tools inspect.
• Your probes are inside the container too. From an app-spawned shell both spellings open the same file — same inode, same hash, same mtime — so every comparison you run "proves" they are one directory. Ground truth only exists from outside: register a one-shot scheduled task that writes its observations to a file, run it, read the file.
• The container's view is a union. Files that exist only in the real directory are visible through the redirect; files that exist in both are shadowed by the private copy. So loss is partial and silent: some writes land, some reads see stale copies, and the symptom is "some things are in memory and some aren't" — which reads as flakiness, not as a path bug.

Cost

every scheduled run was a no-op for as long as the job was enabled. The only reason the chat reached memory at all was that someone occasionally ran the script by hand from inside the container, where it worked. Two previous "fixes" addressed the same redirect for a different writer and were each declared done.

Check

never compare paths; compare file identity from each process that will actually use the path. From the scheduled context, record for each candidate spelling: file count, and for one known file its size, mtime and a hash of its first megabyte. Different answers from inside and outside means two files. In code, probe every spelling for real content (package spelling first), dedupe by file identity (st_dev, st_ino / NTFS file index), and refuse to report success when every spelling is empty.

Symptom

the ingest log was a column of identical, healthy-looking lines: candidates: 0 to-ingest: 0 then DONE files=0 new_obs=0 errors=0. Task Scheduler recorded result 0.

Cause

zero was treated as "nothing new", which is a legitimate outcome, instead of "found no source at all", which is not. The script counted what survived its discovery step and never asked what the discovery step excluded. Separately, the task had been bulk-disabled a month earlier along with twenty others, so even a correct script would not have run — two independent faults, each sufficient.

Cost

false green for as long as the job ran, then plain silence once it was disabled; neither was visible, because a result code in a scheduler is somewhere nobody looks.

Check

two mechanical rules. (1) Distinguish the empty source from the empty delta: zero files discovered under every candidate root is an error with its own exit code; zero new files is fine. (2) Verify from the consumer's side, independently of the producer: a check that reads the newest message straight from the raw transcript and asks the store whether that exact text is present. Judge the newest message older than the ingest budget, not the newest message — a fresh message on top will otherwise mask an old one that never arrived. Surface the result where a person already looks (session start), not in a scheduler's history; and when it fails, trigger the backfill rather than only reporting.

Symptom

figures derived from the store's kind column ("the user asked about X N times", "share of conversation that is human input") were used in reports and looked plausible.

Cause

in this store kind = 'user_prompt' records which message slot a row arrived in, not who wrote it. In the chat protocols being ingested, tool results are delivered back to the model inside user-role messages, and one of the ingesters stored them as user prompts with a [tool_result: …] prefix. Measured: 51,406 of 114,993 user_prompt rows — 44.7% — are tool output, not anything a person typed. The share varies wildly by source (roughly 60% of one source's rows, 0% of several others), so it does not even cancel out in aggregate.

Cost

every count, ranking or "what did the user ask" query built on kind silently inherited a near-half contamination, and several already had.

Check

before trusting a categorical column, sample it grouped by source and read twenty rows of each class. A field's name is the author's intent, not a measurement. In queries, exclude the transport prefix explicitly (content NOT LIKE '[tool_result:%') until the ingester stops writing it; in ingesters, skip records whose content is entirely tool results rather than relabelling them.

Symptom

any pipeline that reports healthy while a user cannot find something they know they put in.

Cause

the verification was attached to the component that was easiest to instrument, not to the edge the user depends on.

Cost

here, two documents and thirteen hours of one chat; in general, the belief that "memory is fine" persists exactly as long as nobody walks one real item end to end.

Check

pick one concrete item the user asked about and trace it: raw source (does it exist?) → store row (did it arrive, under which key?) → the consumer's own search tool (is it returned?). Stop at the first layer where it disappears; that layer, and only that layer, is the fault.