ID-138 {138.3} TECH — Corpus durable home + record identity/curation durability
TECH — Corpus durable home + record identity/curation durability
Section titled “TECH — Corpus durable home + record identity/curation durability”Status:
DRAFT — 03/07/2026.Authored by a FRESH Planner (Q-PLANNER-2 — a different instance from any predecessor spec author). Spec tier TECH+PLAN: RESEARCH/PRODUCT are deliberately out of scope — the product basis is the ratified S441 corpus reframe (specs/id-138-corpus-durable-home/notes/s440-corpus-durable-home-decision.md§10) plus the in-force decision-register rulings DR-025 (corpus reframe), DR-026 (promotion boundary), DR-024 clause (i) (admission-minted identity), DR-027 (ontology CV medium), DR-023 (source-binding model, refined by DR-025), DR-020 (two-tier admission gate), DR-015 (ingress posture). This spec settles the FIVE MUST-SETTLE design points and defines the migration plan; {138.4} PLAN decomposes it into implementation Subtasks.Behaviour basis note: because there is no PRODUCT.md, the “Proposed changes per invariant” mapping runs against the DR rulings + the five settled design points (below), which serve as the numbered acceptance criteria the Checker verifies against. Each ruling R(a)–R(e) is testable.
1. Context
Section titled “1. Context”1.1 What is being built
Section titled “1.1 What is being built”ID-138 gives the admitted corpus a durable home and makes record identity + curation
survive the ingest engine. Under the S441 reframe a “source file” is three things
(DR-025): evidence bytes (inputs, per-binding retention class), the permanent provenance
register (source_documents rows that survive byte discard — citations degrade to the
register, never orphan), and authoritative artefacts (promoted records + the OKF bundle —
the canonical layer, durable via DB backups + bundle changelog). The feature surface:
- A connector-first source-binding register on
source_documents: net-new columnsorigin_type,locator,retention_class(incl. external-referenced — consumed in place, never ingested),cadence,auth, plus anadmission_statuslifecycle andcontent_hash-keyed identity. - Admission-minted record identity (DR-024 i): identity minted once at admission,
STORED, never re-derived from path;
rel_pathbecomes a mutable attribute; rename detected viacontent_hash→ attribute update, same identity. This is the FIRST-BUNDLE-PUBLICATION gate (id-132 BI-20/21 blocks on it). - The “what survives the engine” contract per record class (§10.7 / DR-026) — the load-bearing design decision (settled at R(e), §2.5).
- The promotion boundary (DR-026): pipeline walks never mutate promoted/curated records; engine writes stop at the staging layer; changes arrive as proposals.
- A GDPR tombstone cascade, an orphaned-sd reaper as register-tombstone semantics, a citations→reference_items CASCADE pre-flight gate, writer fencing, the corpus bucket (demoted per DR-025), pull-sync (shrunk per §10.5), and the write-back re-point from VPS-volume file writes to Storage PUT.
The corpus bucket is DEMOTED (DR-025, s440 §10.6): it is the kept-evidence store + gated-upload landing zone + optional ingest-once archive — not the canonical layer. The VPS is pinned (client-owned at rollout); the invariant is nothing irreplaceable on compute.
1.2 Code-intelligence orientation (verbatim — Checker verifies this step ran)
Section titled “1.2 Code-intelligence orientation (verbatim — Checker verifies this step ran)”Tool discipline per .gitnexus/CLAUDE.md + .ast-dataflow/CLAUDE.md. ast-dataflow/gitnexus
do NOT cover Python/SQL, so those corpora were grep-swept in addition (per
.ast-dataflow/CLAUDE.md “does not cover Python or SQL”).
gitnexus_query({query:"corpus admission source document retention class pull sync bucket"})→ top execution flows are the write-back POST chain (proc_84/85/86_post→writeBackFileFirst,resolveAbsolutePath,writeFileFirstWithRestore); definitions surfacedflow.py:_upsert_source_document,flow.py:_to_source_relative(1740-1759), andreseed-tenant-instance.tsensureBrandingBucket(167-196) /createBucket— the existing Storage-provisioning precedent. Confirms the corpus surface is not greenfield: the write-back + upload legs already exist and must be re-pointed, not invented.gitnexus_context({name:"writeBackFileFirst"})→ incoming callersapp/api/items/[id]/route.ts:patchHandler,lib/edit-intent/sweep.ts:runSweep+rollbackSweep; outgoingtryQuery,isOk(lib/supabase/safe.ts),writeFileFirstWithRestore,resolveAbsolutePath. Blast radius of the Storage re-point is these three callers + the sweep path.gitnexus_context({name:"stageAndWalk"})→ outgoingassertCorpusRelativeDestPath,resolveWorkerConfig,FolderDropError; accessesdestPath. No incoming edges indexed (the/stageleg is the confirmed-broken transport DR-020 retires).- grep (SQL):
source_documentsbase DDL —supabase/migrations/20260617130000_ squash_baseline.sql:7462-7487: PKid uuid DEFAULT gen_random_uuid(),content_hash text NOT NULL(indexedidx_source_documents_content_hash:9274),storage_path text NOT NULL,source_url,status,version,parent_id. id-131 addedpublication_ status+ classification family (20260628191700_id131_sd_classification_cols.sql:33-49).origin_type/locator/retention_class/admission_status/cadence/authare ABSENT today (grep returned empty) → all net-new for ID-138. - grep (Python): identity is minted
uuid.uuid5(_KH_PIPELINE_DOC_NS, f"sd:{rel_path}")atflow.py:2047(content fork),:2222(content branch),:2674(qa_sidecar branch); derived-row seedsci:{rel_path}(:2223),chunk:{rel_path}:{position}(:2367),qa:{rel_path}:{idx}(:2409, :2713) — the whole identity graph keys onrel_path, which is precisely the attribute clients mutate (DR-024 problem). The sd parent is written OFF the engine by_upsert_source_document(flow.py:2871-2974) via a raw-poolINSERT … ON CONFLICT (id) DO UPDATE(columns: id, storage_path, content_hash, filename, mime_type, file_size, op_id, extraction_method, source_url — no lifecycle columns touched). Engine-declared targets (whatupdate_blocking(full_reprocess=True)delete-then-re-exports and orphan-cleanup targets) are mounted atflow.py:3700-3776:ci_target,qa_target(q_a_extractions),sd_target,em_target,er_target,cc_target(content_chunks),ri_target,re_target(record_embeddings — one row per content_chunk,flow.py:1469-1492).q_a_pairsis NOT an engine target — it is populated by the TS promotion pathlib/q-a-pairs/promote-corpus.ts(q_a_extractions_promotion_candidates()RPC →q_a_pairs;repromoteCarriedFieldsre-sync at :300/:585). The__qa__/qa_sidecarbranch (flow.py:2625-2737, frozen prefix per id-45 {45.3}) is the existing one-shot extraction precedent — it mintssource_documents+q_a_extractionsONLY, “and NOTHING ELSE.”
gitnexus orientation: existing symbols matched — this is a re-point of live code, not a greenfield surface.
1.3 The current-state facts the design turns on
Section titled “1.3 The current-state facts the design turns on”- SEED-CONTRACT (identity-neutral migration hinge): object key ≡
storage_path≡ admission-timerel_path; the uuid5 seeds are pure functions ofrel_path. Lifting the existing corpus into acorpusbucket atobject_key = storage_pathmints zero new identities. - Engine lifecycle: engine-declared targets get delete-then-re-export on
full_reprocessand orphan-cleanup when a source leaves the walk scope.source_ documents(raw-pool, off-engine) andq_a_pairs(promotion path) already survive it. - Cross-target FK hazards already tamed:
_upsert_source_document’s docstring (flow.py:2885-2947) documents the RESTRICT/CASCADE ordering fixes; notablycitations.cited_reference_item_id → reference_itemsis ON DELETE CASCADE (20260628191703_id131_cite_ext_winrate_fix.sql) — a livefull_reprocessof the URL ledger FIRES that cascade and deletes citation rows (:2931-2936). This is the citations-CASCADE pre-flight hazard, in code today. - Storage API is already in production:
supabase.storage.createBucket/.from(bucket).upload|remove|downloadatscripts/reseed-tenant-instance.ts:134,186,app/api/procurement/[id]/templates/route.ts:220,app/api/source-documents/[id]/binary-url/route.ts:47(DOCUMENTS_BUCKET='documents'). The re-point introduces no new external-library symbol (§8, Empirical verification). - Write-back today (
write-back.ts:192-336) resolvesabs = COCOINDEX_SOURCE_PATH + storage_pathand rewrites the VPS volume file;storage_pathis consumed verbatim because it is the uuid5 PK seed (INV-1, :184-186). Idle mode (COCOINDEX_SOURCE_PATHunset) already returnsfileBacked:false— the graceful no-file path.
2. The five settled design points (the acceptance criteria)
Section titled “2. The five settled design points (the acceptance criteria)”Brief-point map (the five MUST-SETTLE points → where each is ruled): (a) bucket object-key / SEED-CONTRACT freeze → §2.1 R(a); (b) retention lifecycle defaults → §2.6 R(b); (c) pull-sync change-detection → §2.3 R(c); (d) promotion boundary (supersedes the killed record-override store) → §2.4 R(d); (e) survives-the-engine per-record-class contract → §2.5 R(e). Two further in-scope rulings this Task also settles: R(id) (admission-minted identity, DR-024 i) → §2.2; R(ops) (erasure / reaper / fencing) → §2.6.
2.1 R(a) — Bucket object-key/path scheme freeze × SEED-CONTRACT
Section titled “2.1 R(a) — Bucket object-key/path scheme freeze × SEED-CONTRACT”RULING. The corpus-bucket object key ≡ source_documents.storage_path (= the
admission-time rel_path, e.g. markdown/Phew-Bid-Library-2026-v4_4.md). The uuid5
SEED-CONTRACT (sd: / ci: / chunk: / qa: seeds on rel_path, flow.py:2047/2222/ 2367/2409/2674) is frozen at first bundle publication (BI-20/21) and documented in a
SEED-CONTRACT.md alongside the bundle. Migration is identity-neutral: existing rows’
storage_path values become the object keys verbatim → seeds unchanged → zero identity
churn (s440 §5.1, DR-023).
Interplay with R(id) — identity (DR-024 i): the frozen object key is the admission-time
key, used for the ONE-TIME deterministic mint only. Post-freeze, source_documents.id is
authoritative and never re-derived; a client-facing rename updates the mutable path
attribute (§2.2) and does not move the bucket object or re-mint identity. The object
key therefore records provenance (“admitted as X”), not current display path. This severs
the DR-024 identity problem from the bucket-key freeze cleanly.
2.2 R(id) — Admission-minted identity, rename-tolerant (DR-024 clause i STANDS)
Section titled “2.2 R(id) — Admission-minted identity, rename-tolerant (DR-024 clause i STANDS)”RULING. Stop deriving identity from the live rel_path at walk time. Concretely:
- Mint once at admission (binding gate / upload leg):
id = uuid5(NS, "sd:"+rel_path)on first admission (deterministic → idempotent), then STORE it.content_hashis stored alongside (already NOT NULL + indexed). - Walk resolves identity by
content_hash, not path._ingest_file_body/_upsert_source_documentchange from “derivesd:{rel_path}every walk +ON CONFLICT (id)” to “resolve the stored identity bycontent_hashfirst; same bytes at a new path → UPDATE the mutable path attribute, sameid; only a genuinely newcontent_hashmints a new identity.”rel_pathbecomes a mutable attribute (addsource_documents.rel_pathor repurposestorage_pathas the frozen key + a new mutablelogical_path; PLAN picks the column shape). - Derived-row seeds re-key onto the stored
source_document_id(registry-keyed), NOTrel_path— so a rename does not re-mint the entire derived graph (ci:/chunk:/qa:seeds move to<source_document_id>:…). This converges with R(e). Transition (seeds already minted under the oldrel_pathformula): engine-owned derived rows absorb the reseed via a one-timefull_reprocess(pre-launch tool, DR-023) — these classes are already rebuildable and no bundle has published yet;q_a_pairsis unaffected because its PK is independently minted at promotion, never derived from theqa:seed.
This is the hard mechanical core; first bundle publication BLOCKS on it (DR-024, id-132 BI-20/21).
2.3 R(c) — Pull-sync change-detection preservation
Section titled “2.3 R(c) — Pull-sync change-detection preservation”RULING. Pull-sync materialises bucket objects onto the VPS volume content-hash-gated,
never mtime-naive: write a local file only when the bucket object’s hash differs from the
on-disk copy (compare source_documents.content_hash / object metadata to the local file
hash); leave unchanged files untouched so cocoindex’s content-hash memoisation
short-circuits them (flow.py COCO.10 two-tier memo, :2650-2654). Naive re-download churns
mtime and defeats memoisation → full re-extraction bill (§8.4). Scope shrunk (§10.5):
pull-sync covers ONLY the keep-and-watch class + the synthetic Platform corpus.
Ingest-once sources are extracted once and never re-walked → never pull-synced. The LMDB
engine store is non-rebuildable and documented as such (R(ops), §2.6). Direction is
outbound HTTPS (VPS→Supabase) — needs zero new ingress, strengthens DR-015.
2.4 R(d) — SUPERSEDED: resolves to the DR-026 promotion boundary, NOT a store
Section titled “2.4 R(d) — SUPERSEDED: resolves to the DR-026 promotion boundary, NOT a store”RULING (explicit). The former {138.3} design point (d) — “record_overrides / RETAIN store shape (option a/b/c)” — is KILLED by DR-026. There is no record-curation store, no re-apply-after-rebuild contract. (d) resolves entirely to the promotion boundary:
- Pipeline walks never mutate promoted/curated records. Engine writes stop at the
staging layer (
q_a_extractionset al.). Grounded:q_a_pairsis already non-engine-declared (§1.2) — the boundary is encoded in code today for the flagship class; this ruling makes it a stated invariant and audits the other classes to it. - Changes to promoted records arrive as proposals through the existing machinery
(
promote-corpus.tspromotion loop +repromoteCarriedFields, id-120 dedup, id-134 confidence). Human review at the knowledge-admission gate is the LAUNCH posture; auto-apply is earned per progressive trust and MUST NOT be encoded as a hard, permanent boundary (DR-026 owner note; product precedent id-71 lane-a W9.2/W9.3 + A22 — propose-with-provenance).
2.5 R(e) — The “what survives the engine” contract per record class (LOAD-BEARING)
Section titled “2.5 R(e) — The “what survives the engine” contract per record class (LOAD-BEARING)”The problem (§10.7): derived rows (chunks / embeddings / entities / extractions) of
ingest-once sources must OUTLIVE engine orphan-cleanup. All four are engine-declared
targets (flow.py:3700-3776), so when an ingest-once source leaves the walk scope, the
incremental engine’s orphan-cleanup would DELETE them — with no re-walk to rebuild them,
they are gone forever.
Candidates weighed (§10.7): (i) one-shot flows outside the walk scope; (ii)
export-at-promotion into durable tables (the q_a_pairs shape); (iii) registry-keyed
persistence.
RULING — hybrid (i)+(iii); (ii) rejected for derived rows:
- Ingest-once sources are extracted through a one-shot path (i) that writes derived
rows off the engine’s declared targets (raw-pool, keyed to the stored
source_document_id) — generalising the existing off-engine precedents_upsert_source_document(flow.py:2871) and theingest_url/__qa__qa_sidecarbranch (flow.py:2625). Because the source key never enters the memoised walk manifest, orphan-cleanup structurally cannot reach these rows. - Their rows are registry-keyed (iii) to the admission-minted
source_document_idand stamped with the source’sretention_class, so the register-tombstone reaper (§2.6) distinguishes ingest-once-derived (KEEP by construction) from genuinely-orphaned (tombstone). - Candidate (ii) is REJECTED for chunks/embeddings/entities: export-copying them into a
parallel durable table doubles the
record_embeddings(EMB-STORE) store and fights its single-home invariant. Export-at-promotion (ii) stays reserved for the curated-record boundary — which is exactly whatq_a_pairsalready is (DR-026, §2.4).
Per-record-class contract table:
| Record class | Written by (today) | Engine orphan-cleans it? | Survives-the-engine mechanism |
|---|---|---|---|
source_documents (register) | _upsert_source_document raw-pool, off-engine (flow.py:2871) | No | Registry-keyed (iii) — permanent (DR-025); tombstone-only, never hard-reaped |
q_a_pairs (promoted/curated) | TS promotion path (promote-corpus.ts), not an engine target | No | Export-at-promotion (ii) — DR-026 boundary; walks never mutate |
reference_items (promoted evidence) | ri_target engine, sd taken off-engine | Delete-then-re-export | Promotion boundary (ii); CASCADE pre-flight (§2.6) |
q_a_extractions (staging) | qa_target engine | Yes | Keep-and-watch: rebuildable staging. Ingest-once: one-shot (i) + registry-key (iii) |
content_chunks | cc_target engine | Yes | Keep-and-watch: rebuildable. Ingest-once: one-shot (i) + registry-key (iii) |
record_embeddings (content-tier, EMB-STORE) | re_target engine (1/chunk) | Yes | Same as content_chunks |
entity_mentions / entity_relationships | em_target / er_target engine | Yes | Same as content_chunks |
citations | CASCADE from reference_items / concept | Via CASCADE | Degrade-to-register (DR-025); CASCADE pre-flight gate (§2.6) |
2.6 R(b) + R(ops) — Retention lifecycle defaults, erasure, reaper, CASCADE pre-flight, writer fencing (§8.4/§10.9)
Section titled “2.6 R(b) + R(ops) — Retention lifecycle defaults, erasure, reaper, CASCADE pre-flight, writer fencing (§8.4/§10.9)”R(b) — retention lifecycle defaults (RULING). retention_class ∈ {keep_and_watch, ingest_once, live_connected, external_referenced} is assigned at the binding gate
(DR-020 light tier) and defaults per origin: uploaded documents → keep_and_watch; id-45
onboarding Q&A docs / transcripts / completed forms → ingest_once (genuinely living
documents may be classed keep_and_watch at classification time, §10.5); a connected
external system (e.g. CRM) → external_referenced (never ingested — consumed in place,
DR-025). Window default at launch: NO auto-discard timer — ingest-once bytes are
RETAINED by default, because the register + the degrade-to-register citation story make the
bytes the only re-extraction source, so a discard timer would be a footgun. Discard is a
per-client, client-initiated action (the ingest_once byte archive is optional per
DR-025 / §10.6), routed through the SAME erasure workflow as the GDPR tombstone (below).
external_referenced bindings store only a locator + auth binding and zero bytes.
admission_statuslifecycle drives admission + GDPR erasure (bucket holds bytes; the column drives lifecycle — §8.3, DR-023). Enum incl.admittedandtombstoned.- GDPR tombstone cascade (none exists today — a compliance hole for a sovereignty
pitch):
admission_status → tombstonedcascades to derived records (chunks/embeddings/entities/extractions/pairs) via an erasure workflow. Bytes discard from the bucket; the register row survives (DR-025) — citations degrade to it, never orphan. - Orphaned-sd reaper = register-TOMBSTONE semantics (§10.3):
full_reprocessnever deletes off-engine sd rows, so orphans accumulate. The reaper tombstones them (not hard-delete — the register is permanent), and usesretention_class+admission_statusto distinguish erased/discarded from sync-broken vs intentionally-ingest-once. - Citations→reference_items CASCADE pre-flight (unchanged, §10.3): before a live
full_reprocessof the URL ledger, verifycitationsis empty or the cascade is acceptable (flow.py:2931-2936). - Writer fencing (§8.4): sync / walk / upload / write-back / operator bulk-load (ID-69) are FIVE bucket-or-volume writers — a mutual-exclusion barrier prevents interleaving; ID-138 owns the barrier, and ID-69’s bulk-load MUST acquire it (wiring ID-69’s participation is ID-69’s own Task-level work). LMDB non-rebuildability is documented: losing the VPS LMDB store silently converts the next walk into a full re-extraction bill.
3. Proposed changes
Section titled “3. Proposed changes”Grouped by subsystem. Each row cites the governing DR + the design-point ruling. Migration
discipline per supabase/CLAUDE.md: DDL via supabase migration new + db push (never MCP
execute_sql); every new PL/pgSQL fn SET search_path = public, extensions + explicit
REVOKE EXECUTE … FROM anon; types regen --schema public,api.
3.1 Migrations (M)
Section titled “3.1 Migrations (M)”| # | Migration (filename stem) | Change | Governs |
|---|---|---|---|
| M1 | id138_sd_source_binding_cols | ALTER source_documents ADD origin_type text, locator text, retention_class text (CHECK keep_and_watch|ingest_once|live_connected|external_referenced), cadence text, auth jsonb, admission_status text (CHECK incl. admitted|tombstoned, DEFAULT admitted), + the mutable path attribute (R(id)). All additive/nullable-or-defaulted (sd is empty at reset — id-131 precedent). Comments cite DR-025/DR-024. | R(a),R(b),R(id); DR-023/024/025 |
| M2 | id138_admission_identity_fn | public.resolve_or_mint_source_identity(p_content_hash, p_rel_path, …) SECURITY DEFINER, SET search_path, REVOKE … FROM anon — content_hash-first identity resolution the walk + upload leg call (R(id)). | R(id); DR-024 i |
| M3 | id138_erasure_cascade_fn | public.tombstone_source_document(p_id) — sets admission_status='tombstoned', cascades to derived rows, records the erasure. SET search_path; anon REVOKE; RLS: editor/admin only via get_user_role(). | R(ops); DR-025 |
| M4 | id138_orphan_reaper_fn | public.reap_orphaned_source_documents() — register-tombstone (not delete) of off-engine orphans, retention/admission-aware. Pre-flight helper citations_cascade_preflight(). | R(ops) |
REVOKE EXECUTE ON FUNCTION public.<fn>(…) FROM anon; is MANDATORY on every function above
(CLAUDE.md gotcha). RLS predicates role-based via get_user_role().
3.2 Python pipeline (P)
Section titled “3.2 Python pipeline (P)”| # | Symbol (file:line) | Change | Governs |
|---|---|---|---|
| P1 | _ingest_file_body (flow.py:1962), :2047/2222/2674 | Replace uuid5("sd:"+rel_path) walk-time derivation with a call to the M2 content_hash-first resolver; treat rel_path as the mutable attribute, not the seed. | R(id) |
| P2 | _upsert_source_document (flow.py:2871) | Add lifecycle columns to the payload (admission_status, retention_class, …); change ON CONFLICT (id) to preserve stored identity + update the path attribute on content_hash match. Do NOT clobber promoted/curated fields (R(d)). | R(id),R(b),R(d) |
| P3 | derived-row seeds ci:/chunk:/qa: (:2223/2367/2409/2713) | Re-key onto the stored source_document_id (registry-keyed), not rel_path. | R(id),R(e) |
| P4 | one-shot ingest-once path (new; precedents ingest_url flow.py:2740+, _ingest_qa_sidecar_branch :2625) | Extract ingest-once sources off the engine’s declared targets → orphan-cleanup cannot reach them (R(e) i). retention_class stamp gates the reaper (R(e) iii). | R(e) |
| P5 | pull-sync (new; pre-_walk_handler server.py:450, _seed_workspace_manifest :800) | Content-hash-gated bucket→volume materialise; keep-and-watch + Platform corpus scope only; writer-fenced. Retires sync-platform-corpus.sh to fixtures-to-bucket seeding. | R(c),R(ops) |
3.3 TypeScript app (T)
Section titled “3.3 TypeScript app (T)”| # | Symbol (file:line) | Change | Governs |
|---|---|---|---|
| T1 | writeBackFileFirst + resolveAbsolutePath (write-back.ts:192-336) | Re-point the file leg from COCOINDEX_SOURCE_PATH-join file write to Storage PUT into the corpus bucket at object_key = storage_path (same key), then nudge a re-walk. Preserve the idle-mode / source-less / dangling guards (:255-320). Callers patchHandler, runSweep, rollbackSweep unchanged in contract. | R(a); DR-025 |
| T2 | stageAndWalk / FolderDropResult (folder-drop.ts:184-309) | Retire the /stage transport (DR-020); the {131.24} upload leg becomes gate-pass → Storage PUT + source_documents row (admission-minted identity via M2) in one flow. Closes the walkable-incomplete flag. | R(a),R(id); DR-020 |
| T3 | corpus-bucket provisioning (new; precedent reseed-tenant-instance.ts:167-196 createBucket) | Provision a private corpus bucket per client project; service-role writer; key-scoped to project-ref (env-isolation, §2.6 / {127.20}). | R(a); DR-023 |
| T4 | q_a promotion PATCH (app/api/q-a-pairs/[id]/route.ts:202+) | No behavioural change required — it is already the propose/promote path (R(d)); the file-leg sidecar re-points to Storage with T1. Preserve auth.success + authFailureResponse(auth) (:202). | R(d) |
Quality bars carried into every Subtask: sb()/tryQuery() (never raw client),
auth.success + authFailureResponse(auth), semantic tokens only, no barrel re-exports,
TanStack Query, bun run test (never bun test), behaviour-first tests
(reference/test-philosophy.md). New API routes → add to proxy.ts publicRoutes only if
intentionally public (they are not; they are login-gated).
3.4 Ops / infra (O)
Section titled “3.4 Ops / infra (O)”- Writer fence (R(ops)): a mutual-exclusion barrier across sync/walk/upload/write-back/operator-bulk-load — the fifth writer, ID-69’s bucket bulk-load, acquires the ID-138-owned barrier; wiring that participation is ID-69’s work.
- LMDB non-rebuildability documented in the runbook (loss → full re-extraction bill).
- Env-isolation / cross-tenant write guard: bucket + key scoping bound to project-ref; backup/restore drill; per-client walk observability — routes {127.20}/GA (§2.6, §8.5).
4. Testing and validation
Section titled “4. Testing and validation”Each ruling R(a)–R(e) + R(id)/R(ops) maps to a concrete check (the Checker’s acceptance criteria):
- R(a) identity-neutral migration: a test asserts that for a fixture corpus, lifting to
the bucket at
object_key = storage_pathleaves everysource_documents.idand derived uuid5 UNCHANGED.bun run test+python3 -m pytest scripts/tests/. - R(id) rename tolerance: pytest — same bytes at a new
rel_path→ samesource_document_id,rel_pathattribute updated, derived rows NOT re-minted; a genuine newcontent_hashmints a new identity. Behaviour-first (no re-extraction on rename). - R(b) retention defaults: test — an
ingest_oncebinding retains bytes with no auto-discard timer; a client-initiated discard tombstones via the erasure workflow; anexternal_referencedbinding stores locator + auth only, zero bytes. - R(c) pull-sync memoisation: pytest — an unchanged bucket object does NOT rewrite the local file (mtime/hash preserved) and the subsequent walk hits memo (no LLM re-call).
- R(d) promotion boundary: test — a walk over a corpus whose extraction differs from a
promoted
q_a_pairsrow does NOT mutate the promoted row; the change surfaces as a proposal via the promotion path. - R(e) survives-the-engine: pytest — an ingest-once source’s chunks/embeddings/entities
SURVIVE a
full_reprocessand an incremental walk where the source is absent; a keep-and-watch source’s derived rows ARE orphan-cleaned when its source is removed. - R(ops): tombstone cascade test (tombstone → derived rows cascaded, register row
survives, citations degrade); orphan reaper tombstones (never deletes); citations-CASCADE
pre-flight refuses a live
full_reprocesswhencitationsis non-empty; writer-fence concurrency test.
Manual verification gate (Liam’s eyes): a staging dry-run of the corpus→bucket migration on the synthetic Platform corpus, asserting zero identity churn + a clean re-walk.
5. Coordination seams ({138.4} PLAN cross-Task escalation points)
Section titled “5. Coordination seams ({138.4} PLAN cross-Task escalation points)”Sibling-only Subtask deps apply; these are Task-level dependencies (never bent into cross-Task Subtask deps — escalate split/merge if one is wanted):
- id-131 (schema family) —
source_documentscolumns; owns thecontent_itemselimination + thesource-documents/[id]/route.tsre-point (DR-012). M1 must not collide with id-131 migrations. - id-132 (publication gate) — BLOCKS first bundle publication on the R(id) identity contract + the R(d) promotion boundary (DR-024, DR-026; BI-20/21).
- id-127 / {127.20} (DR-015 ingress) — the pull direction REMOVES the push-ingress need; env-isolation / cross-tenant write guard (§2.6, §8.5). ID-127 must not close until {127.20} lands (DR-015 owner instruction).
- id-134 (sync scripts → bucket seeding) —
sync-platform-corpus.shretires to fixtures-to-bucket seeding (P5). - id-45 (first-client ingest-once onboarding) — id-45 = one-time onboarding ingest (R4/§10.5); the R(e) ingest-once contract is the mechanism it relies on.
- ID-69 (onboarding bulk-load) — operator bulk-load writes the bucket directly (service role), replacing rsync/docker-cp.
- id-133 (ontology CV medium, DR-027) — NAMED SEAM, NOT ID-138 WORK: base CVs move docs-site → canonical repo; every bundle carries the materialised effective ontology. ID-138 does not absorb this; it is flagged so PLAN can sequence around id-133’s register-medium change.
6. Risks and mitigations
Section titled “6. Risks and mitigations”- Identity-migration miss (CRITICAL): if R(id) ships wrong, first bundle publication orphans every citation. Mitigation: identity-neutral migration (R(a)) + the pre-first- publication gate + the staging dry-run; this is “the last cheap moment” (§8.2).
- LMDB loss → silent full re-extraction bill: documented + writer-fenced (R(ops)); the VPS store is non-rebuildable.
- Citations CASCADE fires on
full_reprocess: pre-flight gate (§2.6);full_reprocessis demoted to a pre-launch tool (DR-023). - Env-isolation blast radius: canonical buckets raise the class from “rebuildable records” to “the irreplaceable byte layer” — but the VPS-pinned / nothing-irreplaceable- on-compute invariant (§10.6) keeps the bucket (in the client’s OWN project) the only durable byte store; project-ref scoping + backup drill before client #2 (§8.5).
- DOCX→md lossy on value-bearing tables (§8.6.4): under the bucket the converted artefact becomes the durable object — conversion QA is an owned roadmap risk, flagged not solved here.
7. Follow-ups (deferred — roadmap, not ID-138)
Section titled “7. Follow-ups (deferred — roadmap, not ID-138)”- Accretive-ingest / source-retention lifecycle tiers + connectors (Notion/SharePoint/ transcripts) — the extract-and-discard fork (s440 §6/§8.6; roadmap theme 15).
- Comms-loop (Slack tag-Claude → propose → human-gate → write-back → re-ingest) — a consumer of the promotion boundary (§10.11; id-71 lane-a convergence).
- RDF/JSON-LD export via the effective-ontology bundle (DR-027; vault-ld precedent) — explicitly deferred, no commitment.
- Consumption indexer over the bundle (concept embeddings/graph) — id-132 §F / id-135 (§10.10 two-loop ruling; the bundle is never pipeline ingest input).
8. Empirical verification (external-library API surface)
Section titled “8. Empirical verification (external-library API surface)”Per shared-discipline §Empirical verification — scope is external-library symbols only.
This spec introduces no new external-library symbol. The Storage API surface it relies
on (@supabase/supabase-js ^2.105.4, pinned in package.json) is already imported and
called in production:
- 03/07/2026 —
@supabase/supabase-js@^2.105.4—storage.createBucket(...)→ PRESENT (scripts/reseed-tenant-instance.ts:186). - 03/07/2026 —
storage.from(bucket).remove([...])→ PRESENT (app/api/procurement/[id]/templates/route.ts:220). - 03/07/2026 —
storage.from(bucket)(download/upload handle) → PRESENT (scripts/reseed-tenant-instance.ts:134;binary-url/route.ts:47).
Result: no ABSENT / SIGNATURE_DRIFT / BEHAVIOUR_DRIFT. The re-point reuses proven
call sites; no ratification-blocking verification failure.
9. Ratification record (S443)
Section titled “9. Ratification record (S443)”Owner ratified this spec + PLAN.md on 2026-07-03 (S443 ratification board). OQ-138-A/B/D/E
accepted as authored; OQ-138-C accepted with amendment — the {138.7} erasure mechanism
must be coherent with the historic handover guide handover-guides/gdpr-data-export.md.
Seam correction (id-69 ruling): id-69 remains closed. Every reference in this spec and PLAN.md to “ID-69’s bulk-load” / “wiring ID-69’s participation is ID-69’s own work” reads as id-45-owned (reframe R4: id-45 is the onboarding ingest; OQ-138-A: execution stays in id-45). The obligation is materialised as ledger subtask {45.7} — the operator bulk-load acquires the {138.9} barrier; correct dep direction id-45 → ID-138.