Skip to content

ID-138 {138.3} TECH — Corpus durable home + record identity/curation durability

TECH — Corpus durable home + record identity/curation durability

Section titled “TECH — Corpus durable home + record identity/curation durability”

Status: DRAFT — 03/07/2026. Authored by a FRESH Planner (Q-PLANNER-2 — a different instance from any predecessor spec author). Spec tier TECH+PLAN: RESEARCH/PRODUCT are deliberately out of scope — the product basis is the ratified S441 corpus reframe (specs/id-138-corpus-durable-home/notes/s440-corpus-durable-home-decision.md §10) plus the in-force decision-register rulings DR-025 (corpus reframe), DR-026 (promotion boundary), DR-024 clause (i) (admission-minted identity), DR-027 (ontology CV medium), DR-023 (source-binding model, refined by DR-025), DR-020 (two-tier admission gate), DR-015 (ingress posture). This spec settles the FIVE MUST-SETTLE design points and defines the migration plan; {138.4} PLAN decomposes it into implementation Subtasks.

Behaviour basis note: because there is no PRODUCT.md, the “Proposed changes per invariant” mapping runs against the DR rulings + the five settled design points (below), which serve as the numbered acceptance criteria the Checker verifies against. Each ruling R(a)–R(e) is testable.


ID-138 gives the admitted corpus a durable home and makes record identity + curation survive the ingest engine. Under the S441 reframe a “source file” is three things (DR-025): evidence bytes (inputs, per-binding retention class), the permanent provenance register (source_documents rows that survive byte discard — citations degrade to the register, never orphan), and authoritative artefacts (promoted records + the OKF bundle — the canonical layer, durable via DB backups + bundle changelog). The feature surface:

  • A connector-first source-binding register on source_documents: net-new columns origin_type, locator, retention_class (incl. external-referenced — consumed in place, never ingested), cadence, auth, plus an admission_status lifecycle and content_hash-keyed identity.
  • Admission-minted record identity (DR-024 i): identity minted once at admission, STORED, never re-derived from path; rel_path becomes a mutable attribute; rename detected via content_hash → attribute update, same identity. This is the FIRST-BUNDLE-PUBLICATION gate (id-132 BI-20/21 blocks on it).
  • The “what survives the engine” contract per record class (§10.7 / DR-026) — the load-bearing design decision (settled at R(e), §2.5).
  • The promotion boundary (DR-026): pipeline walks never mutate promoted/curated records; engine writes stop at the staging layer; changes arrive as proposals.
  • A GDPR tombstone cascade, an orphaned-sd reaper as register-tombstone semantics, a citations→reference_items CASCADE pre-flight gate, writer fencing, the corpus bucket (demoted per DR-025), pull-sync (shrunk per §10.5), and the write-back re-point from VPS-volume file writes to Storage PUT.

The corpus bucket is DEMOTED (DR-025, s440 §10.6): it is the kept-evidence store + gated-upload landing zone + optional ingest-once archive — not the canonical layer. The VPS is pinned (client-owned at rollout); the invariant is nothing irreplaceable on compute.

1.2 Code-intelligence orientation (verbatim — Checker verifies this step ran)

Section titled “1.2 Code-intelligence orientation (verbatim — Checker verifies this step ran)”

Tool discipline per .gitnexus/CLAUDE.md + .ast-dataflow/CLAUDE.md. ast-dataflow/gitnexus do NOT cover Python/SQL, so those corpora were grep-swept in addition (per .ast-dataflow/CLAUDE.md “does not cover Python or SQL”).

  • gitnexus_query({query:"corpus admission source document retention class pull sync bucket"}) → top execution flows are the write-back POST chain (proc_84/85/86_postwriteBackFileFirst, resolveAbsolutePath, writeFileFirstWithRestore); definitions surfaced flow.py:_upsert_source_document, flow.py:_to_source_relative (1740-1759), and reseed-tenant-instance.ts ensureBrandingBucket (167-196) / createBucket — the existing Storage-provisioning precedent. Confirms the corpus surface is not greenfield: the write-back + upload legs already exist and must be re-pointed, not invented.
  • gitnexus_context({name:"writeBackFileFirst"})incoming callers app/api/items/[id]/route.ts:patchHandler, lib/edit-intent/sweep.ts:runSweep + rollbackSweep; outgoing tryQuery, isOk (lib/supabase/safe.ts), writeFileFirstWithRestore, resolveAbsolutePath. Blast radius of the Storage re-point is these three callers + the sweep path.
  • gitnexus_context({name:"stageAndWalk"}) → outgoing assertCorpusRelativeDestPath, resolveWorkerConfig, FolderDropError; accesses destPath. No incoming edges indexed (the /stage leg is the confirmed-broken transport DR-020 retires).
  • grep (SQL): source_documents base DDL — supabase/migrations/20260617130000_ squash_baseline.sql:7462-7487: PK id uuid DEFAULT gen_random_uuid(), content_hash text NOT NULL (indexed idx_source_documents_content_hash :9274), storage_path text NOT NULL, source_url, status, version, parent_id. id-131 added publication_ status + classification family (20260628191700_id131_sd_classification_cols.sql:33-49). origin_type / locator / retention_class / admission_status / cadence / auth are ABSENT today (grep returned empty) → all net-new for ID-138.
  • grep (Python): identity is minted uuid.uuid5(_KH_PIPELINE_DOC_NS, f"sd:{rel_path}") at flow.py:2047 (content fork), :2222 (content branch), :2674 (qa_sidecar branch); derived-row seeds ci:{rel_path} (:2223), chunk:{rel_path}:{position} (:2367), qa:{rel_path}:{idx} (:2409, :2713) — the whole identity graph keys on rel_path, which is precisely the attribute clients mutate (DR-024 problem). The sd parent is written OFF the engine by _upsert_source_document (flow.py:2871-2974) via a raw-pool INSERT … ON CONFLICT (id) DO UPDATE (columns: id, storage_path, content_hash, filename, mime_type, file_size, op_id, extraction_method, source_url — no lifecycle columns touched). Engine-declared targets (what update_blocking(full_reprocess=True) delete-then-re-exports and orphan-cleanup targets) are mounted at flow.py:3700-3776: ci_target, qa_target (q_a_extractions), sd_target, em_target, er_target, cc_target (content_chunks), ri_target, re_target (record_embeddings — one row per content_chunk, flow.py:1469-1492). q_a_pairs is NOT an engine target — it is populated by the TS promotion path lib/q-a-pairs/promote-corpus.ts (q_a_extractions_promotion_candidates() RPC → q_a_pairs; repromoteCarriedFields re-sync at :300/:585). The __qa__/ qa_sidecar branch (flow.py:2625-2737, frozen prefix per id-45 {45.3}) is the existing one-shot extraction precedent — it mints source_documents + q_a_extractions ONLY, “and NOTHING ELSE.”

gitnexus orientation: existing symbols matched — this is a re-point of live code, not a greenfield surface.

1.3 The current-state facts the design turns on

Section titled “1.3 The current-state facts the design turns on”
  1. SEED-CONTRACT (identity-neutral migration hinge): object key ≡ storage_path ≡ admission-time rel_path; the uuid5 seeds are pure functions of rel_path. Lifting the existing corpus into a corpus bucket at object_key = storage_path mints zero new identities.
  2. Engine lifecycle: engine-declared targets get delete-then-re-export on full_reprocess and orphan-cleanup when a source leaves the walk scope. source_ documents (raw-pool, off-engine) and q_a_pairs (promotion path) already survive it.
  3. Cross-target FK hazards already tamed: _upsert_source_document’s docstring (flow.py:2885-2947) documents the RESTRICT/CASCADE ordering fixes; notably citations.cited_reference_item_id → reference_items is ON DELETE CASCADE (20260628191703_id131_cite_ext_winrate_fix.sql) — a live full_reprocess of the URL ledger FIRES that cascade and deletes citation rows (:2931-2936). This is the citations-CASCADE pre-flight hazard, in code today.
  4. Storage API is already in production: supabase.storage.createBucket / .from(bucket).upload|remove|download at scripts/reseed-tenant-instance.ts:134,186, app/api/procurement/[id]/templates/route.ts:220, app/api/source-documents/[id]/binary-url/route.ts:47 (DOCUMENTS_BUCKET='documents'). The re-point introduces no new external-library symbol (§8, Empirical verification).
  5. Write-back today (write-back.ts:192-336) resolves abs = COCOINDEX_SOURCE_PATH + storage_path and rewrites the VPS volume file; storage_path is consumed verbatim because it is the uuid5 PK seed (INV-1, :184-186). Idle mode (COCOINDEX_SOURCE_PATH unset) already returns fileBacked:false — the graceful no-file path.

2. The five settled design points (the acceptance criteria)

Section titled “2. The five settled design points (the acceptance criteria)”

Brief-point map (the five MUST-SETTLE points → where each is ruled): (a) bucket object-key / SEED-CONTRACT freeze → §2.1 R(a); (b) retention lifecycle defaults → §2.6 R(b); (c) pull-sync change-detection → §2.3 R(c); (d) promotion boundary (supersedes the killed record-override store) → §2.4 R(d); (e) survives-the-engine per-record-class contract → §2.5 R(e). Two further in-scope rulings this Task also settles: R(id) (admission-minted identity, DR-024 i) → §2.2; R(ops) (erasure / reaper / fencing) → §2.6.

2.1 R(a) — Bucket object-key/path scheme freeze × SEED-CONTRACT

Section titled “2.1 R(a) — Bucket object-key/path scheme freeze × SEED-CONTRACT”

RULING. The corpus-bucket object key ≡ source_documents.storage_path (= the admission-time rel_path, e.g. markdown/Phew-Bid-Library-2026-v4_4.md). The uuid5 SEED-CONTRACT (sd: / ci: / chunk: / qa: seeds on rel_path, flow.py:2047/2222/ 2367/2409/2674) is frozen at first bundle publication (BI-20/21) and documented in a SEED-CONTRACT.md alongside the bundle. Migration is identity-neutral: existing rows’ storage_path values become the object keys verbatim → seeds unchanged → zero identity churn (s440 §5.1, DR-023).

Interplay with R(id) — identity (DR-024 i): the frozen object key is the admission-time key, used for the ONE-TIME deterministic mint only. Post-freeze, source_documents.id is authoritative and never re-derived; a client-facing rename updates the mutable path attribute (§2.2) and does not move the bucket object or re-mint identity. The object key therefore records provenance (“admitted as X”), not current display path. This severs the DR-024 identity problem from the bucket-key freeze cleanly.

2.2 R(id) — Admission-minted identity, rename-tolerant (DR-024 clause i STANDS)

Section titled “2.2 R(id) — Admission-minted identity, rename-tolerant (DR-024 clause i STANDS)”

RULING. Stop deriving identity from the live rel_path at walk time. Concretely:

  • Mint once at admission (binding gate / upload leg): id = uuid5(NS, "sd:"+rel_path) on first admission (deterministic → idempotent), then STORE it. content_hash is stored alongside (already NOT NULL + indexed).
  • Walk resolves identity by content_hash, not path. _ingest_file_body / _upsert_source_document change from “derive sd:{rel_path} every walk + ON CONFLICT (id)” to “resolve the stored identity by content_hash first; same bytes at a new path → UPDATE the mutable path attribute, same id; only a genuinely new content_hash mints a new identity.” rel_path becomes a mutable attribute (add source_documents.rel_path or repurpose storage_path as the frozen key + a new mutable logical_path; PLAN picks the column shape).
  • Derived-row seeds re-key onto the stored source_document_id (registry-keyed), NOT rel_path — so a rename does not re-mint the entire derived graph (ci:/chunk:/qa: seeds move to <source_document_id>:…). This converges with R(e). Transition (seeds already minted under the old rel_path formula): engine-owned derived rows absorb the reseed via a one-time full_reprocess (pre-launch tool, DR-023) — these classes are already rebuildable and no bundle has published yet; q_a_pairs is unaffected because its PK is independently minted at promotion, never derived from the qa: seed.

This is the hard mechanical core; first bundle publication BLOCKS on it (DR-024, id-132 BI-20/21).

2.3 R(c) — Pull-sync change-detection preservation

Section titled “2.3 R(c) — Pull-sync change-detection preservation”

RULING. Pull-sync materialises bucket objects onto the VPS volume content-hash-gated, never mtime-naive: write a local file only when the bucket object’s hash differs from the on-disk copy (compare source_documents.content_hash / object metadata to the local file hash); leave unchanged files untouched so cocoindex’s content-hash memoisation short-circuits them (flow.py COCO.10 two-tier memo, :2650-2654). Naive re-download churns mtime and defeats memoisation → full re-extraction bill (§8.4). Scope shrunk (§10.5): pull-sync covers ONLY the keep-and-watch class + the synthetic Platform corpus. Ingest-once sources are extracted once and never re-walked → never pull-synced. The LMDB engine store is non-rebuildable and documented as such (R(ops), §2.6). Direction is outbound HTTPS (VPS→Supabase) — needs zero new ingress, strengthens DR-015.

2.4 R(d) — SUPERSEDED: resolves to the DR-026 promotion boundary, NOT a store

Section titled “2.4 R(d) — SUPERSEDED: resolves to the DR-026 promotion boundary, NOT a store”

RULING (explicit). The former {138.3} design point (d) — “record_overrides / RETAIN store shape (option a/b/c)” — is KILLED by DR-026. There is no record-curation store, no re-apply-after-rebuild contract. (d) resolves entirely to the promotion boundary:

  • Pipeline walks never mutate promoted/curated records. Engine writes stop at the staging layer (q_a_extractions et al.). Grounded: q_a_pairs is already non-engine-declared (§1.2) — the boundary is encoded in code today for the flagship class; this ruling makes it a stated invariant and audits the other classes to it.
  • Changes to promoted records arrive as proposals through the existing machinery (promote-corpus.ts promotion loop + repromoteCarriedFields, id-120 dedup, id-134 confidence). Human review at the knowledge-admission gate is the LAUNCH posture; auto-apply is earned per progressive trust and MUST NOT be encoded as a hard, permanent boundary (DR-026 owner note; product precedent id-71 lane-a W9.2/W9.3 + A22 — propose-with-provenance).

2.5 R(e) — The “what survives the engine” contract per record class (LOAD-BEARING)

Section titled “2.5 R(e) — The “what survives the engine” contract per record class (LOAD-BEARING)”

The problem (§10.7): derived rows (chunks / embeddings / entities / extractions) of ingest-once sources must OUTLIVE engine orphan-cleanup. All four are engine-declared targets (flow.py:3700-3776), so when an ingest-once source leaves the walk scope, the incremental engine’s orphan-cleanup would DELETE them — with no re-walk to rebuild them, they are gone forever.

Candidates weighed (§10.7): (i) one-shot flows outside the walk scope; (ii) export-at-promotion into durable tables (the q_a_pairs shape); (iii) registry-keyed persistence.

RULING — hybrid (i)+(iii); (ii) rejected for derived rows:

  • Ingest-once sources are extracted through a one-shot path (i) that writes derived rows off the engine’s declared targets (raw-pool, keyed to the stored source_document_id) — generalising the existing off-engine precedents _upsert_source_document (flow.py:2871) and the ingest_url / __qa__ qa_sidecar branch (flow.py:2625). Because the source key never enters the memoised walk manifest, orphan-cleanup structurally cannot reach these rows.
  • Their rows are registry-keyed (iii) to the admission-minted source_document_id and stamped with the source’s retention_class, so the register-tombstone reaper (§2.6) distinguishes ingest-once-derived (KEEP by construction) from genuinely-orphaned (tombstone).
  • Candidate (ii) is REJECTED for chunks/embeddings/entities: export-copying them into a parallel durable table doubles the record_embeddings (EMB-STORE) store and fights its single-home invariant. Export-at-promotion (ii) stays reserved for the curated-record boundary — which is exactly what q_a_pairs already is (DR-026, §2.4).

Per-record-class contract table:

Record classWritten by (today)Engine orphan-cleans it?Survives-the-engine mechanism
source_documents (register)_upsert_source_document raw-pool, off-engine (flow.py:2871)NoRegistry-keyed (iii) — permanent (DR-025); tombstone-only, never hard-reaped
q_a_pairs (promoted/curated)TS promotion path (promote-corpus.ts), not an engine targetNoExport-at-promotion (ii) — DR-026 boundary; walks never mutate
reference_items (promoted evidence)ri_target engine, sd taken off-engineDelete-then-re-exportPromotion boundary (ii); CASCADE pre-flight (§2.6)
q_a_extractions (staging)qa_target engineYesKeep-and-watch: rebuildable staging. Ingest-once: one-shot (i) + registry-key (iii)
content_chunkscc_target engineYesKeep-and-watch: rebuildable. Ingest-once: one-shot (i) + registry-key (iii)
record_embeddings (content-tier, EMB-STORE)re_target engine (1/chunk)YesSame as content_chunks
entity_mentions / entity_relationshipsem_target / er_target engineYesSame as content_chunks
citationsCASCADE from reference_items / conceptVia CASCADEDegrade-to-register (DR-025); CASCADE pre-flight gate (§2.6)

2.6 R(b) + R(ops) — Retention lifecycle defaults, erasure, reaper, CASCADE pre-flight, writer fencing (§8.4/§10.9)

Section titled “2.6 R(b) + R(ops) — Retention lifecycle defaults, erasure, reaper, CASCADE pre-flight, writer fencing (§8.4/§10.9)”

R(b) — retention lifecycle defaults (RULING). retention_class ∈ {keep_and_watch, ingest_once, live_connected, external_referenced} is assigned at the binding gate (DR-020 light tier) and defaults per origin: uploaded documents → keep_and_watch; id-45 onboarding Q&A docs / transcripts / completed forms → ingest_once (genuinely living documents may be classed keep_and_watch at classification time, §10.5); a connected external system (e.g. CRM) → external_referenced (never ingested — consumed in place, DR-025). Window default at launch: NO auto-discard timer — ingest-once bytes are RETAINED by default, because the register + the degrade-to-register citation story make the bytes the only re-extraction source, so a discard timer would be a footgun. Discard is a per-client, client-initiated action (the ingest_once byte archive is optional per DR-025 / §10.6), routed through the SAME erasure workflow as the GDPR tombstone (below). external_referenced bindings store only a locator + auth binding and zero bytes.

  • admission_status lifecycle drives admission + GDPR erasure (bucket holds bytes; the column drives lifecycle — §8.3, DR-023). Enum incl. admitted and tombstoned.
  • GDPR tombstone cascade (none exists today — a compliance hole for a sovereignty pitch): admission_status → tombstoned cascades to derived records (chunks/embeddings/entities/extractions/pairs) via an erasure workflow. Bytes discard from the bucket; the register row survives (DR-025) — citations degrade to it, never orphan.
  • Orphaned-sd reaper = register-TOMBSTONE semantics (§10.3): full_reprocess never deletes off-engine sd rows, so orphans accumulate. The reaper tombstones them (not hard-delete — the register is permanent), and uses retention_class + admission_status to distinguish erased/discarded from sync-broken vs intentionally-ingest-once.
  • Citations→reference_items CASCADE pre-flight (unchanged, §10.3): before a live full_reprocess of the URL ledger, verify citations is empty or the cascade is acceptable (flow.py:2931-2936).
  • Writer fencing (§8.4): sync / walk / upload / write-back / operator bulk-load (ID-69) are FIVE bucket-or-volume writers — a mutual-exclusion barrier prevents interleaving; ID-138 owns the barrier, and ID-69’s bulk-load MUST acquire it (wiring ID-69’s participation is ID-69’s own Task-level work). LMDB non-rebuildability is documented: losing the VPS LMDB store silently converts the next walk into a full re-extraction bill.

Grouped by subsystem. Each row cites the governing DR + the design-point ruling. Migration discipline per supabase/CLAUDE.md: DDL via supabase migration new + db push (never MCP execute_sql); every new PL/pgSQL fn SET search_path = public, extensions + explicit REVOKE EXECUTE … FROM anon; types regen --schema public,api.

#Migration (filename stem)ChangeGoverns
M1id138_sd_source_binding_colsALTER source_documents ADD origin_type text, locator text, retention_class text (CHECK keep_and_watch|ingest_once|live_connected|external_referenced), cadence text, auth jsonb, admission_status text (CHECK incl. admitted|tombstoned, DEFAULT admitted), + the mutable path attribute (R(id)). All additive/nullable-or-defaulted (sd is empty at reset — id-131 precedent). Comments cite DR-025/DR-024.R(a),R(b),R(id); DR-023/024/025
M2id138_admission_identity_fnpublic.resolve_or_mint_source_identity(p_content_hash, p_rel_path, …) SECURITY DEFINER, SET search_path, REVOKE … FROM anon — content_hash-first identity resolution the walk + upload leg call (R(id)).R(id); DR-024 i
M3id138_erasure_cascade_fnpublic.tombstone_source_document(p_id) — sets admission_status='tombstoned', cascades to derived rows, records the erasure. SET search_path; anon REVOKE; RLS: editor/admin only via get_user_role().R(ops); DR-025
M4id138_orphan_reaper_fnpublic.reap_orphaned_source_documents() — register-tombstone (not delete) of off-engine orphans, retention/admission-aware. Pre-flight helper citations_cascade_preflight().R(ops)

REVOKE EXECUTE ON FUNCTION public.<fn>(…) FROM anon; is MANDATORY on every function above (CLAUDE.md gotcha). RLS predicates role-based via get_user_role().

#Symbol (file:line)ChangeGoverns
P1_ingest_file_body (flow.py:1962), :2047/2222/2674Replace uuid5("sd:"+rel_path) walk-time derivation with a call to the M2 content_hash-first resolver; treat rel_path as the mutable attribute, not the seed.R(id)
P2_upsert_source_document (flow.py:2871)Add lifecycle columns to the payload (admission_status, retention_class, …); change ON CONFLICT (id) to preserve stored identity + update the path attribute on content_hash match. Do NOT clobber promoted/curated fields (R(d)).R(id),R(b),R(d)
P3derived-row seeds ci:/chunk:/qa: (:2223/2367/2409/2713)Re-key onto the stored source_document_id (registry-keyed), not rel_path.R(id),R(e)
P4one-shot ingest-once path (new; precedents ingest_url flow.py:2740+, _ingest_qa_sidecar_branch :2625)Extract ingest-once sources off the engine’s declared targets → orphan-cleanup cannot reach them (R(e) i). retention_class stamp gates the reaper (R(e) iii).R(e)
P5pull-sync (new; pre-_walk_handler server.py:450, _seed_workspace_manifest :800)Content-hash-gated bucket→volume materialise; keep-and-watch + Platform corpus scope only; writer-fenced. Retires sync-platform-corpus.sh to fixtures-to-bucket seeding.R(c),R(ops)
#Symbol (file:line)ChangeGoverns
T1writeBackFileFirst + resolveAbsolutePath (write-back.ts:192-336)Re-point the file leg from COCOINDEX_SOURCE_PATH-join file write to Storage PUT into the corpus bucket at object_key = storage_path (same key), then nudge a re-walk. Preserve the idle-mode / source-less / dangling guards (:255-320). Callers patchHandler, runSweep, rollbackSweep unchanged in contract.R(a); DR-025
T2stageAndWalk / FolderDropResult (folder-drop.ts:184-309)Retire the /stage transport (DR-020); the {131.24} upload leg becomes gate-pass → Storage PUT + source_documents row (admission-minted identity via M2) in one flow. Closes the walkable-incomplete flag.R(a),R(id); DR-020
T3corpus-bucket provisioning (new; precedent reseed-tenant-instance.ts:167-196 createBucket)Provision a private corpus bucket per client project; service-role writer; key-scoped to project-ref (env-isolation, §2.6 / {127.20}).R(a); DR-023
T4q_a promotion PATCH (app/api/q-a-pairs/[id]/route.ts:202+)No behavioural change required — it is already the propose/promote path (R(d)); the file-leg sidecar re-points to Storage with T1. Preserve auth.success + authFailureResponse(auth) (:202).R(d)

Quality bars carried into every Subtask: sb()/tryQuery() (never raw client), auth.success + authFailureResponse(auth), semantic tokens only, no barrel re-exports, TanStack Query, bun run test (never bun test), behaviour-first tests (reference/test-philosophy.md). New API routes → add to proxy.ts publicRoutes only if intentionally public (they are not; they are login-gated).

  • Writer fence (R(ops)): a mutual-exclusion barrier across sync/walk/upload/write-back/operator-bulk-load — the fifth writer, ID-69’s bucket bulk-load, acquires the ID-138-owned barrier; wiring that participation is ID-69’s work.
  • LMDB non-rebuildability documented in the runbook (loss → full re-extraction bill).
  • Env-isolation / cross-tenant write guard: bucket + key scoping bound to project-ref; backup/restore drill; per-client walk observability — routes {127.20}/GA (§2.6, §8.5).

Each ruling R(a)–R(e) + R(id)/R(ops) maps to a concrete check (the Checker’s acceptance criteria):

  • R(a) identity-neutral migration: a test asserts that for a fixture corpus, lifting to the bucket at object_key = storage_path leaves every source_documents.id and derived uuid5 UNCHANGED. bun run test + python3 -m pytest scripts/tests/.
  • R(id) rename tolerance: pytest — same bytes at a new rel_path → same source_document_id, rel_path attribute updated, derived rows NOT re-minted; a genuine new content_hash mints a new identity. Behaviour-first (no re-extraction on rename).
  • R(b) retention defaults: test — an ingest_once binding retains bytes with no auto-discard timer; a client-initiated discard tombstones via the erasure workflow; an external_referenced binding stores locator + auth only, zero bytes.
  • R(c) pull-sync memoisation: pytest — an unchanged bucket object does NOT rewrite the local file (mtime/hash preserved) and the subsequent walk hits memo (no LLM re-call).
  • R(d) promotion boundary: test — a walk over a corpus whose extraction differs from a promoted q_a_pairs row does NOT mutate the promoted row; the change surfaces as a proposal via the promotion path.
  • R(e) survives-the-engine: pytest — an ingest-once source’s chunks/embeddings/entities SURVIVE a full_reprocess and an incremental walk where the source is absent; a keep-and-watch source’s derived rows ARE orphan-cleaned when its source is removed.
  • R(ops): tombstone cascade test (tombstone → derived rows cascaded, register row survives, citations degrade); orphan reaper tombstones (never deletes); citations-CASCADE pre-flight refuses a live full_reprocess when citations is non-empty; writer-fence concurrency test.

Manual verification gate (Liam’s eyes): a staging dry-run of the corpus→bucket migration on the synthetic Platform corpus, asserting zero identity churn + a clean re-walk.


5. Coordination seams ({138.4} PLAN cross-Task escalation points)

Section titled “5. Coordination seams ({138.4} PLAN cross-Task escalation points)”

Sibling-only Subtask deps apply; these are Task-level dependencies (never bent into cross-Task Subtask deps — escalate split/merge if one is wanted):

  • id-131 (schema family) — source_documents columns; owns the content_items elimination + the source-documents/[id]/route.ts re-point (DR-012). M1 must not collide with id-131 migrations.
  • id-132 (publication gate) — BLOCKS first bundle publication on the R(id) identity contract + the R(d) promotion boundary (DR-024, DR-026; BI-20/21).
  • id-127 / {127.20} (DR-015 ingress) — the pull direction REMOVES the push-ingress need; env-isolation / cross-tenant write guard (§2.6, §8.5). ID-127 must not close until {127.20} lands (DR-015 owner instruction).
  • id-134 (sync scripts → bucket seeding) — sync-platform-corpus.sh retires to fixtures-to-bucket seeding (P5).
  • id-45 (first-client ingest-once onboarding) — id-45 = one-time onboarding ingest (R4/§10.5); the R(e) ingest-once contract is the mechanism it relies on.
  • ID-69 (onboarding bulk-load) — operator bulk-load writes the bucket directly (service role), replacing rsync/docker-cp.
  • id-133 (ontology CV medium, DR-027) — NAMED SEAM, NOT ID-138 WORK: base CVs move docs-site → canonical repo; every bundle carries the materialised effective ontology. ID-138 does not absorb this; it is flagged so PLAN can sequence around id-133’s register-medium change.

  • Identity-migration miss (CRITICAL): if R(id) ships wrong, first bundle publication orphans every citation. Mitigation: identity-neutral migration (R(a)) + the pre-first- publication gate + the staging dry-run; this is “the last cheap moment” (§8.2).
  • LMDB loss → silent full re-extraction bill: documented + writer-fenced (R(ops)); the VPS store is non-rebuildable.
  • Citations CASCADE fires on full_reprocess: pre-flight gate (§2.6); full_reprocess is demoted to a pre-launch tool (DR-023).
  • Env-isolation blast radius: canonical buckets raise the class from “rebuildable records” to “the irreplaceable byte layer” — but the VPS-pinned / nothing-irreplaceable- on-compute invariant (§10.6) keeps the bucket (in the client’s OWN project) the only durable byte store; project-ref scoping + backup drill before client #2 (§8.5).
  • DOCX→md lossy on value-bearing tables (§8.6.4): under the bucket the converted artefact becomes the durable object — conversion QA is an owned roadmap risk, flagged not solved here.

7. Follow-ups (deferred — roadmap, not ID-138)

Section titled “7. Follow-ups (deferred — roadmap, not ID-138)”
  • Accretive-ingest / source-retention lifecycle tiers + connectors (Notion/SharePoint/ transcripts) — the extract-and-discard fork (s440 §6/§8.6; roadmap theme 15).
  • Comms-loop (Slack tag-Claude → propose → human-gate → write-back → re-ingest) — a consumer of the promotion boundary (§10.11; id-71 lane-a convergence).
  • RDF/JSON-LD export via the effective-ontology bundle (DR-027; vault-ld precedent) — explicitly deferred, no commitment.
  • Consumption indexer over the bundle (concept embeddings/graph) — id-132 §F / id-135 (§10.10 two-loop ruling; the bundle is never pipeline ingest input).

8. Empirical verification (external-library API surface)

Section titled “8. Empirical verification (external-library API surface)”

Per shared-discipline §Empirical verification — scope is external-library symbols only. This spec introduces no new external-library symbol. The Storage API surface it relies on (@supabase/supabase-js ^2.105.4, pinned in package.json) is already imported and called in production:

  • 03/07/2026@supabase/supabase-js@^2.105.4storage.createBucket(...)PRESENT (scripts/reseed-tenant-instance.ts:186).
  • 03/07/2026storage.from(bucket).remove([...])PRESENT (app/api/procurement/[id]/templates/route.ts:220).
  • 03/07/2026storage.from(bucket) (download/upload handle) → PRESENT (scripts/reseed-tenant-instance.ts:134; binary-url/route.ts:47).

Result: no ABSENT / SIGNATURE_DRIFT / BEHAVIOUR_DRIFT. The re-point reuses proven call sites; no ratification-blocking verification failure.


Owner ratified this spec + PLAN.md on 2026-07-03 (S443 ratification board). OQ-138-A/B/D/E accepted as authored; OQ-138-C accepted with amendment — the {138.7} erasure mechanism must be coherent with the historic handover guide handover-guides/gdpr-data-export.md.

Seam correction (id-69 ruling): id-69 remains closed. Every reference in this spec and PLAN.md to “ID-69’s bulk-load” / “wiring ID-69’s participation is ID-69’s own work” reads as id-45-owned (reframe R4: id-45 is the onboarding ingest; OQ-138-A: execution stays in id-45). The obligation is materialised as ledger subtask {45.7} — the operator bulk-load acquires the {138.9} barrier; correct dep direction id-45 → ID-138.