S542 — how the Platform corpus is minted, why the sweep deadlocked, and what deleted synthetic-company-overview.md

Interim session between S541 and the next planned session. Point-in-time: describes the state of main at dd59d6a, plus the changes this session landed. Every claim below is measured from code, the corpus manifest, uuid5 arithmetic or CI run logs — not from memory and not from the comments in the files concerned.

Why this exists

Two things blocked the cocoindex nightly, and the ledger recorded them as one thing that had been fixed. They are separate mechanisms with separate fixes, and only one of them had been addressed.

• Defect A — the sweep deadlock. Every nightly run since S539 aborted at step 1 with zero deletes. Not a data accident: a structural contradiction that recurs on every run until one of two lists changes.
• Defect B — the corpus-row hijack. content/synthetic-company-overview.md has no source_documents row. The deleter recorded in id-412’s status_note is the sweep, and it was fixed at 34c94fcd3. There is a second, independent deleter in integration teardown that the S539 narrowing does not touch, and it is the one that removed this document.

Both were confirmed on Platform staging by the owner during the session: the two flagged rows exist, the overview row does not.

Defect A — the sweep and its own guard, deadlocked

What the run actually printed

Run 31263414788, sweep step, verbatim:

NM-6 SCOPE-GUARD VIOLATION: 2 candidate row(s) are SHOWCASE/PLATFORM CONTENT …
ABORTING WITH ZERO DELETES (this fails the run by design).
  id=c9176d5e-e314-53ed-8293-3311bfdbfa46 storage_path="verify/synthetic-capability-statement.pdf" logical_path="content/synthetic-capability-statement.pdf"
  id=07df51fc-4405-5328-8fa3-ee57a37c566c storage_path="verify/synthetic-sector-spend.xlsx"        logical_path="verify/synthetic-sector-spend.xlsx"

Both ids are exactly uuid5(fbfaf1ff-1ee4-583c-9757-1674465b2ec1, "sd:verify/<basename>"). That is not an inference from the paths — it is the seed-contract formula recomputed against the observed ids, and it settles provenance: the verify-driver staging minted these rows, not the corpus walk. synthetic-sector-intel.docx is not flagged because the corpus walk won that race and its row sits at content/….

The mechanism

verify_driver.py FIXTURE_SETS['templates'] was repointed at three Platform-corpus content documents at c3286753f. So verify/ held two populations at once:

• the sweep’s STORAGE_PATH_PREFIX_FAMILIES claims the whole verify/ prefix as test-minted and selects it for deletion;
• WALKED_BASELINE_PATHS — what the NM-6 guard protects — included those same three verify_dest paths.

storage_path freezes at mint, so whichever staging the walk reached first won the name. Two corpus documents ended up frozen at verify/…, permanently inside the sweep’s selection scope and permanently protected from it. Selection and protection contradict each other by construction. The guard fires, the run dies, nothing is deleted, and the next run does the same. Bounded set: only the three verify_dest paths can ever collide, because no content/, qa/ or edge/ path starts with any of the eight selection prefixes.

The repoint bought no coverage in the first place

The nightly has walked the whole Platform corpus since id-412 W3. Content-hash-first identity means staging the same bytes at a second path cannot produce a second row — it resolves onto the corpus row. The “verify-lane copy” of a corpus document was the corpus row wearing a different name. The lane’s actual requirement is proving the /stage → walk → row path works across the three converter MIME routes, and that is satisfied by any three distinct-bytes documents.

The guard that should have caught this could not fire

__tests__/guards/corpus-manifest.test.ts already carried an assertion written for precisely this hazard, whose own comment says “the sweep and the S511 guard would contradict each other and the guard would abort every run.” It checks whether a verify/… dest path starts with a baseline directory prefix (content/, qa/, edge/). That is impossible by construction, so it passed vacuously while the real collision ran the other way: baseline documents whose protected paths sat under the sweep’s selection prefix.

This is the more transferable finding. The guard named the right invariant and watched the direction the defect never travelled in. Verified this session by reintroducing the collision and confirming the replacement assertion fails.

Fixed this session

• verify_driver.py stages form templates again — what its name says and what cocoindex-nightly.yml:133-135 has always declared it stages. Three documents, same MIME spread (xlsx/pdf/docx), bytes distinct from every walked-baseline document per DR-133.
• corpus-manifest.json: verify_dest moved off the three corpus documents onto the three form templates.
• lib/corpus/sweep-scope.ts (new) homes SWEEP_STORAGE_PATH_PREFIX_FAMILIES where a guard can read it. The sweep script cannot be imported — top-level await plus process.exit on refusal — so a list inlined there is a list no guard can see. That is the mechanical reason this went unguarded.
• walkedBaselinePathSet() added to lib/corpus/fixture-manifest.ts; the sweep, the new guard and dropFixture now derive the protected set from one place.
• The replaced guard compares the two lists that actually govern the collision: every protected baseline path against every sweep selection prefix.
• [object Set] in the NM-6 error message fixed (a .join lost at dd59d6a).

Defect B — a byte-identical fixture hijacks the corpus row, then teardown deletes it

The chain

Every link verified in code:

1. stageFixture ships fixture bytes verbatim (readFile, no mutation; /stage does no in-byte injection — OQ-62-6).
2. Byte-identical staging hits resolve_or_mint_source_identity, which is content-hash first: it returns the corpus row’s id. No second row exists; the test and the corpus share one row.
3. _upsert_source_document’s ON CONFLICT (id) DO UPDATE sets logical_path and filename = EXCLUDED.filename. The corpus row now answers to the test’s prefix.
4. pollContentItemsFor matches filename ILIKE '<TEST_PREFIX>%' — and returns the corpus row’s id.
5. dropFixture deletes source_documents by that id.

A Platform-corpus document is destroyed from test teardown, and nothing in the run reports it.

Which test, and the corroboration

legacy-alias-preload.integration.test.ts stages CONTENT.companyOverviewMd to inv-10/[53.14-INV10-…].md. That is the missing document.

The prediction was checked against a second, independent case before being believed: extract-contract-honour.integration.test.ts uses prefix [28.14-INV20-…] with namedClientEngagementsMd, which reproduces the S539 forensic observation exactly — logical_path inv-20/…, filename [28.14-. Two files, two matching predictions, neither of which was used to build the model.

Blast radius: 12 integration files stage walked-baseline documents and call dropFixture — 12× sectorSpendXlsx, 1× companyOverviewMd, 1× namedClientEngagementsMd.

Why it stays gone

The identity resolve lives inside ingest_file, which is @coco.fn(memo=True). An unchanged corpus file memo-hits on every later walk in the same run, so the body never runs and the row is never re-minted. Only 2 of the 12 files pass fullReprocess: true. The nightly’s LMDB is a fresh RUNNER_TEMP directory each run, so a later run would re-mint — but since S539 the sweep aborts before the walk, so no walk has run at all.

Compounding it: dropFixture removes DB rows only. The staged file stays on the corpus volume, so the document then exists at two paths with identical bytes and a later mint can freeze storage_path to the test path.

Fixed this session

dropFixture now reads each candidate row before deleting and refuses any whose storage_path or logical_path is in the walked baseline — the same exact-membership test the sweep’s NM-6 guard uses. It still cleans the test’s genuine rows, then throws naming the offending rows and pointing at DR-133.

Consequence to expect, stated plainly: this converts a silent corpus deletion into a loud teardown failure. Until DR-133 is applied to the 12 files, those files will fail in afterAll. That is the intended trade — the corpus survives and the failure names exactly the work — but it means the nightly’s next red will be in the Vitest tier rather than at step 1.

Corrections to the recorded state

• id-412 status_note says “Deleter IDENTIFIED and fixed (34c94fcd3)”. Accurate for the sweep; incomplete as a claim about the document. dropFixture is a second deleter, was never in scope of that fix, and is what removed the overview row.
• The S540 carry’s leading hypothesis — “cocoindex memo-skip on an unchanged file whose row was deleted” — is right about why the row stays absent and does not address what removed it. Both halves are needed.
• S540 records “content_hash is NOT the file’s raw-byte hash — it hashes extracted text post-conversion.” The current walk path disagrees: flow.py computes content_fingerprint = (await file.content_fingerprint()).hex() and passes that same value to both the resolver and the content_hash column. Flagged, not adjudicated — it changes nothing above, because byte-identical files collapse under either definition. Worth one measurement before either statement is relied on.

Remediation — one row still needs a human, and the guard is right to insist

Corrected after running the fix. Run 31271353859 (934ad1c) still failed at the sweep. The prediction that both stranded rows would self-heal was wrong, and the reason is worth recording because it is the guard behaving correctly.

isShowcasePlatformContent tests either stored path. Of the two rows:

row	storage_path	logical_path	after the fix
07df51fc… sector-spend	verify/…	verify/…	neither path protected — sweepable, self-heals
c9176d5e… capability-statement	verify/…	content/synthetic-capability-statement.pdf	still protected via logical_path — sweep aborts

That is deliberate, and the sweep’s own comment states it: “a corpus document whose logical_path still names a baseline file is showcase content whatever its storage_path says, and this guard’s job is to refuse, not to select.” Removing verify_dest retires the storage_path leg only. A corpus document mis-filed at a test storage_path is a data remediation, and an automated sweep is exactly the wrong actor to perform it.

Owed, one-off, by a human with staging access: delete source_documents c9176d5e-e314-53ed-8293-3311bfdbfa46 and its children on Platform staging. The next cold-LMDB walk re-mints it at content/synthetic-capability-statement.pdf (5f609eea-a1b8-5dc3-b38f-0fd00ad2b122). The absent overview row re-mints in the same walk. Until that delete happens the nightly stays red at step 1 — the code fix removes the recurring deadlock but cannot clear the residue the old configuration froze.

Method note on this correction. The flagged-row count could not be read from run 31271353859’s log: get_job_logs clips to a character budget from the end of the job, and every tail window lands past the sweep block, so a larger tail_lines returns the same clipped region. The artefact route is closed too — the egress policy blocks results-receiver.actions.githubusercontent.com.

So it was executed instead of asserted: selectCandidates’ prefix test and isShowcasePlatformContent were replayed over the two row states logged verbatim in run 31263414788, against the manifest as it now stands on main:

c9176d5e…capability:  selected=true showcaseGuard=true  -> ABORTS THE RUN
07df51fc…sectorspend: selected=true showcaseGuard=false -> deleted (self-heals)
NM-6 violations => 1

The one assumption left is that neither row changed since that log, which holds because the sweep aborts before the walk and no walk has run. Anyone re-checking this should re-run the same two predicates rather than trust the number.

Open questions, framed as requirements rather than verdicts

The pre-reframe test population. Most of the cocoindex integration suite was written before the corpus reframe, and 12 of those files stage walked-baseline documents in a way DR-133 now prohibits. It is tempting to read that as “these tests are obsolete”. That conclusion is not available from anything measured here. The question each file needs is “what requirement does this test serve, and is that requirement still live under the current corpus model?” — not “does this test still work?”. What is measured:

• 12 files stage baseline documents and delete by resolved id;
• DR-133 makes id-415 AC-1/2/3 (“repoint to Platform-corpus content docs”) non-executable as written, so those ACs describe work that cannot be done;
• stage-5-row-counter (Inv-11) asserted items.length >= 2 from one fixture staged twice — unsatisfiable under content-hash identity, i.e. at least one test was measuring nothing.

That last point is evidence that some of this population encodes a superseded corpus model, not merely stale paths. It is not evidence about any other file. UNDECIDABLE until each file’s requirement is named and sourced; carrying it forward as a go/no-go on the suite would be exactly the error the framing exists to prevent.

Is there a subtask ready to execute DR-133? No. id-396 designed the distinct-bytes rules but is a design task with no repo code. id-398 (done) was the em:/er: PK re-seed, a different mechanism. id-415 AC-1/2/3 is the nearest home and currently instructs the opposite of DR-133. Applying DR-133 needs those ACs rewritten or a successor minted — an owner mint decision, not a pickup.

Corpus-volume accumulation is unowned. /stage writes into the walk source dir at runtime and nothing removes it. Recorded twice as unowned — the S540 carry (“stageFixture writes corpus bytes nothing removes”) and id-414’s S539 progress note (“They left the baseline; per-test staging still injects one and nothing removes it”) — and in no task’s acceptance criteria. id-396’s accumulation/reset contract covers the staging database, not the corpus volume.

Note this is not caused by docker cp, and the Dockerfile move will not fix it: id-416 explicitly scopes out the corpus-seeding steps. Hop B only ever adds files — there is no prune — so it cannot clear residue either. The one thing that would change is indirect: if the corpus ever lives in an image layer rather than under the /cocoindex-state volume mount, runtime-staged files land in the container’s writable layer and vanish on container replacement, bounding accumulation by container lifetime instead of volume lifetime. That is a mitigation, not a fix, and it is not what id-416 is scoped to do.

What this session did not check

• No queries against Platform staging (no credentials in the session container); the row states quoted are the owner’s checks and the CI log, not mine.
• The full Vitest suite was run, but xlsx could not be installed (egress policy blocks cdn.sheetjs.com), so xlsx-dependent files are not covered here.
• No integration test was executed against a live sidecar. The dropFixture refusal is covered by its own reasoning and unit-level suites, not by an observed nightly.
• Nothing was verified through mempalace: the knowledge-hub MCP server needs an interactive OAuth flow this session could not run.