id-396 TECH — one retention-class-aware corpus model

id-396 — One corpus model across the three fixture surfaces

Status: ALL RULINGS RATIFIED — {396.3} owner sign-off recorded (S511 board). D1, D2, D4 (amended), D5–D8 ratified; D3 overruled to adopt; D9 executed per owner annotation. Board preserved at reports/s511-id396-corpus-rulings-board.html. Empirical ground: ./RESEARCH.md (S511 inventory — read it first; code-intel orientation was satisfied by that agent’s consumer mapping rather than a fresh gitnexus pass, since the subject is fixture bytes and staging behaviour, not symbol-level call graphs).

Context

Three fixture surfaces drift independently: the nightly forms corpus (docs/testing/test-data/, 11 files + __tests__/fixtures/cocoindex-chunking/, 2), the vendored platform synthetic corpus (scripts/cocoindex_pipeline/fixtures/platform-corpus/, 10, shape-guarded), and the e2e/eval seeds (scripts/seed-e2e-users.ts, e2e/fixtures/test-data-fixture.ts, __tests__/fixtures/eval-gold/). Only one of the three has a rot guard; two nightly fixtures are already orphaned; the staging DB accumulates rows from 19/41 specs that never clean up; and byte-identical re-staging run-over-run collides with rel_path-seeded em/er PKs (F4, census #40 — the nightly is hard-down until id-398).

The board (S509, verdict S1) asked this spec to settle: retention-class assignment per fixture family, distinct-bytes rules, the staging-DB accumulation/reset contract, and fixture-authoring conventions — so id-397 can re-derive what the lane proves and id-398 can fix F4 “the charter way”.

Proposed model

1. One corpus manifest + one conformance guard

A checked-in manifest docs/testing/corpus-manifest.json covering ALL fixture trees (nightly test-data, cocoindex-chunking, platform-corpus, eval-gold). Per fixture: id, path, format, sha256, fixture_class (§2), staging_mode (verify-driver | per-test | never-staged | db-seed), consumers (test/spec paths), and notes. platform-corpus-shape.test.ts generalises into __tests__/validation/corpus-manifest.test.ts: exact file-set per tree, SHA match, byte-distinctness across the whole manifest, magic-byte checks carried over, synthetic- prefix rule for synthetic trees, and an orphan rule — a fixture with zero declared consumers fails the guard. The existing platform-corpus invariants (DR-014 no forms/, no __qa__/) move in unchanged.

• Anti-drift: the manifest is the census’s fixture register; the S6 sweep and every census read the same file the guard enforces.
• [D3 RATIFIED — owner overruled the retire proposal] The two orphans (ITT Evaluation Matrix.xls, rfp_onlinetdcops.doc) are adopted: mint legacy .xls/.doc mime-coverage tests that consume them, relocate the files to sit with the other corpus forms (owner note S511 — exact destination settled when the manifest lands), and declare their consumers in the manifest.

2. Fixture retention classes (test-side ↔ product-side)

Product retention classes attach at the binding gate (R1/R4, S440 note §130-175): ingest-once vs keep-and-watch; derived rows of ingest-once sources must outlive engine orphan-cleanup (“what survives the engine”, item 7). The corpus model gives every fixture a test-side class AND makes the corpus exercise both product classes so the lane can prove R4 semantics (id-397’s ground):

fixture_class	Semantics	Examples
staged-ephemeral	Staged per run/test; MUST be swept by the pre-run sweep (§4); never asserted across runs	all stageFixture uses; verify_driver templates
durable-seed	Deliberately persistent; declared in manifest with reason; never swept	[E2E-PUB-REVIEW-FIXTURE] (cross-shard race documented)
vendored-read-only	Never staged; consumed as files	platform-corpus tree, eval-gold, chunking pair

Product-class coverage: the nightly corpus declares at least one ingest-once fixture lineage (walked once, then its derived rows asserted to SURVIVE later walks + orphan cleanup) and one keep-and-watch lineage (re-walked on byte change). Today’s corpus tests neither explicitly — this is new proof surface for id-397 to specify, not new fixtures to invent now.

3. Distinct-bytes rules

• Default: distinct bytes by construction. Any two fixtures that may coexist in the staging DB carry different bytes (manifest SHA-uniqueness enforces this globally — already true, RESEARCH.md surface 1).
• Same-bytes staging is reserved for tests whose SUBJECT is the hash-identity path (memo/idempotency/full-reprocess family — the eight-test population in RESEARCH.md). These tests keep byte-identical re-staging deliberately; their correctness rides id-398’s PK re-seed, not fixture changes.
• No in-byte injection in stageFixture — keep it byte-faithful. Tests needing distinct bytes use distinct fixture files (the entity-variants pattern: paired variant files whose docstrings name their consumers).

4. Staging-DB lifecycle contract — the load-bearing ruling

[D1 RATIFIED S511] Byte-identical re-staging (same bytes, run N and run N+1) is designed upsert-absorb behaviour, not a reset-boundary error: the hash-first resolver (DR-024) correctly maps re-staged bytes to the stored source_documents row, and id-398 re-seeds engine-path em/er PKs on source_document_id so re-declares become idempotent upserts instead of unique-tuple collisions. This is the re-seed shape id-398 waits on. The alternative (full DB reset per nightly) is rejected: the staging DB is shared state (migration-serial lane), a reset destroys the durable-seed class, and it would mask exactly the ingest-once-survival semantics the lane must prove.

With upsert-absorb designed in, the contract is:

• Pre-run sweep (new, in the nightly workflow before /stage): delete fixture-prefixed rows (VERIFY-* dests + per-test dest prefixes) from source_documents and let CASCADE take derived rows — one central step, closing the 19/41 no-dropFixture gap without touching 19 specs. Per-spec dropFixture stays as good hygiene, no longer load-bearing.
• What accumulates by design: durable-seed rows; pipeline_runs/census telemetry (comparability requires history); promoted records (promotion boundary, R3 — walks never touch them); and — owner amendment (S511) — the Platform corpus data itself: the Platform staging surface is not just a test substrate, it must persist data as the showcase (the purpose of Platform) and as the promote-to-client surface. The sweep is therefore scoped strictly to fixture-prefixed TEST rows (VERIFY-* + declared per-test dest prefixes); showcase/platform content is never sweep-eligible.
• Census comparability: with the sweep, every census starts from a known fixture population; run-over-run deltas become signal.
• [D2 RATIFIED S511] entity_pair_resolutions caches verdicts across TIERS (mock verdicts replay into real-LLM runs — S507 §8.4). Tier joins the cache key. Implementation belongs to id-395 ({395.4}); the policy is settled here.

5. Fixture-authoring conventions ({396.1} final clause)

Every new fixture: manifest entry (class + consumers) in the same PR; consuming tests named in the fixture docstring where the format allows (entity-variants pattern); synthetic- basename prefix in synthetic trees; no client IP anywhere (guard-enforced); distinct bytes unless the test’s subject is hash identity (§3); staged fixtures use a declared dest prefix so the pre-run sweep owns them.

6. Adjacent verdicts this model implies

• [D4 RATIFIED S511 (amended)] e2e body rule: every seeded source_documents row carries a position-0 content_chunks body sourced from manifest corpus files, OR is declared body-less in the manifest. E2e keeps programmatic state manufacture (the two lanes keep their two jobs — ingestion proves corpus→rows; e2e proves app behaviour around exact states); full corpus-driven seeding remains the ingestion lane’s job. The e2e/fixtures/test-data-fixture.ts fix rides the S8 e2e retire-and-replace mint.
• id-370 feed-ins consumed by this model: the S494 data posture (synthetic, no backfill, delete freely) licenses the pre-run sweep; the reference_items.source_document_id adjudication and q_a_extractions routing stay id-370’s rulings — this spec takes a dependency on their outcomes but does not decide them. [D5 RATIFIED S511] id-370 is pulled forward now. Owner-supplied context for it (S511): specs/id-127-platform-pipeline/notes/id-127-additional-context-for-item-3.md — two form states (empty = complete-from-corpus / coverage-review; completed = historic q_a-pair seed or platform-completed win-rate data); Q&A sidecar watching is retired in favour of a ONE-TIME extraction of the client’s ready-made bid library with the DB as first-class source for latest verified pairs; conflation sources named (pipeline now also transformation/ETL, bid→procurement pivot, content_items retirement). Related ids: 392, 294 (possibly outdated), 127 ({127.20} s495 prompt unrun — that work lands here instead); id-364 rides option A, id-363 rides option E, both subject to the id-370 review.
• ID-127 {127.4} corpus-as-specced is confirmed retired surface (RESEARCH.md surface 2b); the vendored platform-corpus tree + this manifest replace it. PROPOSAL.md’s surviving DB-seed half rides later phases, unowned here.
• Staleness riders for whichever lane next touches the workflow: :202 corpus comment (8→11 files), verify_driver.py:97 single-set note.

{396.2} — content-echo §6 verdict table

Test	§6 verdict (S507)	Verdict under this model
cross-document-dedup	structurally unsatisfiable (same bytes ×2)	Fixture shape ALREADY ADEQUATE — PR #146 entity-variants pair; per-run satisfiable now. Run-over-run correctness rides id-398 + §4 sweep
stage5-canonical-name-freshness	same	same
cross-workspace-isolation	same	same — W1/W2 stage different variant files
(constraint) unresolved-mention-retains-canonical	must stay single-variant	preserved: stays on the CSP xlsx single-variant surface; manifest notes the constraint

The satisfiability problem has MOVED: from fixture bytes (solved) to the write path under re-staging (id-398) and accumulation (§4). {398.3}‘s “multi-staging tests unblocked at the write path” verifies on the first post-398 dispatched nightly ({398.4} census read).

Testing and validation

• corpus-manifest.test.ts guard (§1) — runs in bun run test; the regression surface for every convention in §3/§5.
• First post-398 nightly (dispatched, mock tier): expect the same-bytes population green at the write path, zero UniqueViolations in the sidecar log, census read against #40 ({398.4}); the three §6 tests green end-to-end.
• id-397 then owns invariant-by-invariant proof re-derivation (incl. the two product-class lineages, §2) — out of scope here by design (board S2).

Follow-ups / handoffs

• id-398: consumes §4’s upsert-absorb ruling as the re-seed shape; migration for existing rel_path-seeded rows under DR-093 (delete, don’t backfill).
• id-397: lane proof targets over this corpus (R4 lineages, class-2 disposition).
• id-395: {395.4} implements D2; mock-residue items re-verified against this model.
• S6 sweep/census: manifest = fixture register; R8-protected columns list lives in the census protocol doc (id-399, {399.4}).

Owner rulings ({396.3})

#	Question	Status (S511 board)
D1	Byte-identical re-staging semantics	RATIFIED: upsert-absorb + pre-run sweep, sweep scoped to test rows only — Platform data persists as showcase + promote-to-client surface
D2	entity_pair_resolutions tier leakage	RATIFIED: tier joins the cache key; id-395 implements
D3	Two orphan fixtures	OVERRULED→ADOPT: keep both, relocate with other corpus forms, mint legacy .xls/.doc mime tests
D4	Body-less e2e bulk seeder	RATIFIED (amended): corpus-derived bodies + programmatic states; fix rides the S8 mint
D5	id-370 scheduling	RATIFIED: pulled forward now, with owner context doc (§6)