Skip to content

ID-134 — Platform promotion-confidence E2E corpus + repeatable release gate (TECH)

ID-134 — Platform promotion-confidence E2E corpus + repeatable release gate

Section titled “ID-134 — Platform promotion-confidence E2E corpus + repeatable release gate”

Re-authored S437 (T3, 02/07/2026) against reference/decision-register.md (DR-014 / DR-018 / DR-020 / DR-015) + reference/deployment-architecture.md (the S436 probe-verified topology). This body supersedes the pre-OKF-rearchitecture design (forms folder + forms manifest + a G3 forms gate + a content_items/content_chunks G5) — the prior body’s {134.6}/{134.7} journals are preserved on the ledger. Product intent is settled (reports/okf-record-model-lineage/okf-record-model-v3.md §9); this is a TECH-only re-spec — RESEARCH/PRODUCT/PLAN are not re-run.

Subtask: {134.1} TECH. Tier: TECH+PLAN (product intent settled in the v3 model; {134.2} PLAN decomposed it into {134.3}{134.8}, all but {134.6}/{134.7} done).

Product intent (the de-facto PRODUCT for this Task): reports/okf-record-model-lineage/okf-record-model-v3.md §9 (the {127.4} reframe) + §11 “Task C” + risk #10. Settled; not re-litigated here — this spec turns it into the current-reality implementation plan.

One-line framing. {127.4} is not “shrink to a tiny fixture”. It is the Platform promotion-confidence E2E corpus + a repeatable release gate that proves the same immutable vX.Y.Z image + migration set runs end-to-end on the Platform rig BEFORE it is pinned to a client (runbooks/platform-to-client-promote.md §2–5) — not id-45 content-quality validation. Scale = complete seam coverage, not “tiny”.

Coupling to ID-131 (corrected). The superseded body called this Task “independent of ID-131”. That is no longer true. ID-131 landed the L-records refactor (content_items eliminated; content_chunks/q_a_extractions/entity_* re-parented onto source_documents; the record_embeddings store). The gate is now written against the post-ID-131 schema, and — critically — it can only be validated after the id-131-lockstep cocoindex image is deployed and a fresh walk re-populates both Platform DBs (see §3.1). The corpus files remain shape-stable inputs; the coupling is in the gate’s asserted grains and in the deploy/re-walk precondition.


1. Context — the gap, and how the current system works

Section titled “1. Context — the gap, and how the current system works”

1.1 The live gap (the reason this gate exists)

Section titled “1.1 The live gap (the reason this gate exists)”

The Platform pipeline had, at the {127.4} reframe, 395 pipeline_runs rows with status='completed' yet 0 content_chunks on prod (zjqbrdctesqvouboziae). status='completed' is therefore itself a false-confidence signal — a “completed” walk on Platform prod had never produced the retrieval grain (binary→markdown→chunk→embedding→Postgres). Any release gate asserting on pipeline_runs.status alone would have been green for all 395 of those runs. This is why the gate asserts on actual per-route row deltas keyed on the run’s op_id, not on run status (v3 §9 half-3; risk #10).

Two live-verified facts sharpen the run-select (both carried forward from the {134.5}/{134.6} journals):

  • Governance crons emit empty completed heartbeats into pipeline_runs. A bare status='completed' selector picks those up. The gate must run-select by pipeline_name = 'kh_canonical_pipeline' AND op_id IS NOT NULL (the corpus walk’s canonical name — KH_CANONICAL_PIPELINE_NAME, flow.py:752).
  • {134.5} PROVED the runtime (S429): on both Platform DBs a supervised loopback /walk produced content_chunks=6 with all 6 embedding IS NOT NULL, source_documents=6, q_a_extractions=5 — the first-ever full retrieval-grain write on Platform. That proof was then cleared by the ID-131 W2 debris-wipe (S430), so the gate is currently validation-blocked pending a re-walk on the id-131-lockstep image (§3.1).

1.2 How a walk reads and writes (current head — file:line evidence)

Section titled “1.2 How a walk reads and writes (current head — file:line evidence)”

The deployed entrypoint is app_main in scripts/cocoindex_pipeline/flow.py. The seams a promote changes “only by config” all live in this one walk:

  • Mandatory root manifest (walk guard). flow.py:3373-region loads <source>/.kh-workspace-map.json once at flow start and aborts the whole run (manifest_missing / manifest_invalid, terminal status='failed') if it is absent or schema-invalid. A file literally named .kh-workspace-map.json is skipped by the walk itself. See §2.3 for the two distinct manifest senses — this is the ingest route-tag manifest, not the form_template_requirements J-model.
  • Stage counters. _empty_stage_counts() (flow.py:754–771) fixes the canonical seven stage keys: source_walk, binary_conversion, llm_extraction, embedding, entity_resolution, chunking, postgres_upsert. The webhook enforces all seven via Zod.
  • Content branch (_ingest_content_branch): convert_binary_to_markdown is the binary extractor (md passes through; PDF/DOCX go through the per-MIME adapter — the highest image-bump-risk seam). It declares rows on source_documents (sd_target.declare_row, flow.py:2158), content_items (ci_target.declare_row, flow.py:2193 — legacy, being eliminated by ID-131), then the chunking stage: cc_target.declare_row (flow.py:2258) writes one content_chunks row per chunk, each carrying embedding (flow.py:2272), source_document_id (flow.py:2267, the M2 re-parent FK) and op_id (flow.py:2273), PK uuid5(_KH_PIPELINE_DOC_NS, f"chunk:{rel_path}:{position}") (flow.py:2262) — deterministic, so a re-ingest UPSERTs in place. Q&A-shaped content also yields q_a_extractions (qa_target.declare_row, flow.py:2290, stamping source_document_id at flow.py:2295).
  • __qa__/ reserved prefix. The qa_sidecar route (sd_target/qa_target only, flow.py:2567/:2594) warns qa_sidecar_route_missing if a __qa__/-prefixed path resolves to content. The corpus avoids __qa__/ (RATIFY-2) so the hazard is structurally impossible.
  • Feed / URL slice (the #1 Platform↔client config delta, BI-4): scripts/cocoindex_pipeline/url_source.py FeedUrlSource enumerates feed_articles WHERE passed = true → one item per normalised URL → trafilatura /extract. The URL branch declares source_documents (sd_target.declare_row, flow.py:2974) + reference_items (ri_target.declare_row, flow.py:2991); reference_items.source_document_id is NOT NULL (REFERENCE_ITEMS_SCHEMA, flow.py:1302).
  • Terminal webhook. flow.py emits the terminal pipeline_runs row via POST /api/internal/pipeline-runs/recordrecordPipelineRun (lib/pipeline/record-run.ts:165), carrying status, result.stage_counts, and the run op_id.

The op_id-keyed grains (schema-verified — the gate’s substrate). Every content-side target the gate asserts on carries op_id, so filtering by the run’s op_id isolates this walk’s delta from all historical debris:

targetop_id?key columns for the gateschema const
pipeline_runspipeline_name, op_id, status, result.stage_counts(record-run.ts)
source_documents✓ (flow.py:1256)op_id, source_url (nullable — URL vs localfs); no content_item_idflow.py:1240
content_chunks✓ (flow.py:1423)op_id, source_document_id NOT NULL (:1409), embedding vector(1024) nullable (:1418)flow.py:1404
q_a_extractions✓ (flow.py:1235)op_id, source_document_id nullable (:1219 — sidecar writes None; content branch writes non-null)flow.py:1212
reference_items✓ (flow.py:1305)op_id, source_document_id NOT NULL (:1302)flow.py:1279
entity_mentions✓ (flow.py:1331)op_id, source_document_idflow.py:1319
entity_relationships✗ (no op_id — RULING 2, flow.py:1343)join via source_document_id ∈ this-run SDsflow.py:1348
form_templates✗ (no op_id column — proven S429)not asserted (forms route retired — DR-014)
  • DB seed — BUILT. scripts/seed-synthetic-corpus.ts mints the 6 synthetic workspaces (SYNTHETIC_WORKSPACES, :136) + ~30 form_questions (SYNTHETIC_QUESTIONS, :271), and emits the root manifest via buildManifest (:331) + --emit-manifest --manifest-out. Exported building blocks (not a single seedSyntheticCorpus symbol; the harness is main() at :717 + these exports): parseCorpusArgs (:389), requireApplicationTypeId (:442), resolveFormsWorkspaceId (:506), seedSyntheticWorkspaces (:558), seedSyntheticQuestions (:613), cleanSyntheticCorpus (:680). Feed slice: scripts/seed-platform-feed.ts.
  • The vendored corpus tree — DONE ({134.3}). scripts/cocoindex_pipeline/fixtures/platform-corpus/ (7 walk files + .kh-workspace-map.json.example, cherry-picked to main as ea23a31b). The forms seam (forms/procurement/*) still ships there today; its removal is BL-392 (§2.1).
  • The vendored→on-prem sync — DONE ({134.4}). deploy/onprem/sync-platform-corpus.sh (ebdb55c3).
  • The {134.8} structural guard — DONE. __tests__/integration/cocoindex/platform-corpus-shape.test.ts (d4b246d8) — corpus-rot guard, not the old re-point (§5).
  • The repeatable release gate — MISSING ({134.6}). scripts/verify-platform-promotion-gate.ts is greenfield (confirmed: ls ABSENT + grep -rln verify-platform-promotion-gate scripts/ returns no references anywhere). runbooks/platform-to-client-promote.md §2 is still prose and names only content_items/source_documents, never content_chunks ({134.7}).

1.4 Code-intelligence orientation (cited verbatim, not paraphrased)

Section titled “1.4 Code-intelligence orientation (cited verbatim, not paraphrased)”

gitnexus_query({query:'platform promotion gate content_chunks embeddings verification pipeline runs', repo:'canonical'}) returned no process for a promotion gate (it is greenfield) and surfaced these definitions (file:line verbatim from the tool output):

  • Function:scripts/quality-gate.ts:embedding_coverage (lines 586–637)
  • Function:scripts/quality-gate.ts:chunk_coverage (lines 640–698)
  • Function:scripts/cocoindex_pipeline/flow.py:_empty_stage_counts (lines 754–771)
  • Function:lib/q-a-pairs/promote-corpus.ts:promoteCorpusExtractions (lines 228–508) — the promoter tier (raw q_a_extractionsq_a_pairs); confirms the gate must assert the raw q_a_extractions tier, not q_a_pairs (promoter-tier, no op_id).

gitnexus_context({name:'tryQuery', file_path:'lib/supabase/safe.ts', repo:'canonical'})Function:lib/supabase/safe.ts:tryQuery (lines 159–180), outgoing call to Class:lib/supabase/safe.ts:SupabaseError, ~30 incoming callers across app/api/** (e.g. app/api/admin/provenance/pipeline-runs/route.ts:GET). Companion: sb (lib/supabase/safe.ts:91), SupabaseError (:13). The gate imports these DIRECTLY (no barrel).

Greenfield disclaimer. scripts/verify-platform-promotion-gate.ts does not exist on main (gitnexus_query surfaced no such symbol; ls + grep -rln fallback confirm). The gate is net-new; its harness mirrors the proven seed-synthetic-corpus.ts shape (--target resolution + project-ref guard + sb()/tryQuery()), it does not re-use one.

Python write-sites (grep — gitnexus is TS-only). flow.py content branch: sd_target.declare_row :2158, ci_target :2193, cc_target :2258 (embedding :2272, op_id :2273, source_document_id :2267), qa_target :2290 (sd_id :2295), em_target :2399, er_target :2488; url branch sd_target :2974, ri_target :2991; sidecar sd_target :2567, qa_target :2594. Constant KH_CANONICAL_PIPELINE_NAME = "kh_canonical_pipeline" flow.py:752.

This is not a greenfield walk — the walk, manifest gate, stage counters and webhook are all built and tested. ID-134 adds (a) the corpus input (done: {134.3}) and (b) the assertion harness over its output ({134.6}, this spec).


Design rule (v3 §9): one file per route × format × notable-branch, no more. The corpus touches every seam a config delta could break: the binary extractors (highest image-bump risk), a content route producing content_chunks WITH embeddings, the Q&A raw-tier write, the feed/URL slice, and the graceful-degradation fork.

2.1 The tree (vendored source of truth → mounted at /cocoindex-state/corpus)

Section titled “2.1 The tree (vendored source of truth → mounted at /cocoindex-state/corpus)”
platform-corpus/ # vendored in-repo source of truth (RATIFY-1)
.kh-workspace-map.json.example # ingest route-tag manifest TEMPLATE — stamped per-target at sync (RATIFY-1b)
content/
synthetic-methodology.md # content route · md → content_chunks w/ embedding
synthetic-capability-statement.pdf # content route · PDF binary
synthetic-sector-intel.docx # content route · DOCX binary
qa/
synthetic-qa-pairs.md # content route · Q&A-shaped md → q_a_extractions (NOT __qa__/ — RATIFY-2)
edge/
synthetic-sparse-edge.md # content route · graceful-degradation edge
[forms/procurement/*.pdf,*.docx] # RETIRED under DR-014 / BL-392 — see below

Forms retirement (DR-014, tracked as BL-392). Forms enter the system app-side via manual upload (§J Path-A/B/C → form_template_requirements), not the gated corpus walk. The production forms corpus-route (workspace_resolver RouteKind "forms", the flow.py forms fork, the BI-7(c) manifest-gate test, and the vendored forms/procurement/ seam) is being retired so okf-v3 §8.2 supersedes the live-code behaviour. That retirement is a cross-cutting task (BL-392) — it touches live routing code + the {134.8} shape-guard toEqual set + a diverged track branch, so it is NOT a {134} corpus-tidy edit (the {134.6} removal executor surfaced exactly this coupling on main). The {134.6} gate asserts the POST-retirement no-forms state (no form_templates assertion, no G3). Until BL-392 lands, the 2 vendored forms binaries may still be walked and still write form_templates; that is orthogonal to the gate (form_templates has no op_id, so the gate cannot and does not assert on it).

Plus, seeded into the DB (not files): N_feed feed_articles passed=true via seed-platform-feed.ts (the URL/feed slice), and the 6 synthetic workspaces + ~30 form_questions via seed-synthetic-corpus.ts.

2.2 File → route → expected op_id-keyed row-delta (the gate’s contract)

Section titled “2.2 File → route → expected op_id-keyed row-delta (the gate’s contract)”

All content is invented UK-procurement prose (no client IP — BI-3 / ID-68); filenames use synthetic- tokens only.

rel_pathroutewalk branchexpected op_id-keyed row-delta
.kh-workspace-map.json— (manifest)loaded at flow start, skipped by walkload succeeds, no manifest_missing abort
content/synthetic-methodology.mdcontent (UnmappedPath)_ingest_content_branch+1 source_documents, +K content_chunks each w/ embedding, +entity rows
content/synthetic-capability-statement.pdfcontentcontent branch (convert_binary_to_markdown)+1 source_documents, +K content_chunks w/ embedding
content/synthetic-sector-intel.docxcontentcontent branch (binary→md)+1 source_documents, +K content_chunks w/ embedding
qa/synthetic-qa-pairs.mdcontentcontent branch+1 source_documents, +K content_chunks, +M q_a_extractions (each source_document_id NOT NULL)
edge/synthetic-sparse-edge.mdcontent (UnmappedPath soft-warn)content branchgraceful degradation: soft-warn logged, COALESCE/_field(...,default) forks fire without flipping flow_status; still +1 source_documents (no content_item — S429 proved edge yields no content-item)
(feed) feed_articles WHERE passed=true ×N_feedURL/feedURL branch (flow.py:2974/:2991)+1 source_documents + +1 reference_items (source_document_id NOT NULL) per URL

Content-side source_documents count = 5 (methodology, capability, sector-intel, qa-pairs, sparse-edge — the edge file mints an SD even without a content_item). Feed adds N_feed SDs. This is unaffected by whether the forms seam is still present (forms write form_templates, not source_documents).

2.3 The two manifest senses (do NOT conflate)

Section titled “2.3 The two manifest senses (do NOT conflate)”

The superseded body treated these as one thing; they are structurally distinct:

  1. Ingest route-tag manifest — the root .kh-workspace-map.json loaded at flow start (flow.py:3373-region). It is a route-tagger + mandatory walk guard: absent/invalid → the run aborts (manifest_missing/manifest_invalid). Post-DR-014 its forms-workspace mappings may be empty (no forms route in the gated corpus), but the file itself still stands — it gates the walk and tags content/feed routing. This is what .kh-workspace-map.json.example templates and what §3 stamps per-target (RATIFY-1b).
  2. The J-model requirement definitionform_template_requirements (the Path-C catalogue; ~96 rows live, DERIVED). This is an app-side structure (DR-014), never a corpus artefact, never loaded by the walk. Forms match against it; the corpus never carries it.

The gate + runbook ({134.6}/{134.7}) reference only sense 1, and only as the walk guard + content/feed router — never as a forms manifest.


3.1 Validation precondition (BLOCKING — id-131 image coupling)

Section titled “3.1 Validation precondition (BLOCKING — id-131 image coupling)”

The {134.6} gate is spec-complete but VALIDATION-BLOCKED. {134.5} already ran the seed + supervised /walk on both Platform DBs (S429) and proved the runtime, but the ID-131 W2 debris-wipe cleared that proof (S430: content_chunks/source_documents/ q_a_extractions/reference_items/entity_mentions = 0 on both DBs). Re-validation requires, in order:

  1. Deploy the id-131-lockstep cocoindex image. The deployed Platform image MUST carry the current flow.py (the M2 write-sites that stamp source_document_id + op_id on content_chunks/q_a_extractions — verified on main at flow.py:2267/:2273/:2295). An older image mis-keys the walk. Same image-coupling caveat as the schema apply.
  2. Re-seed + one supervised /walk per Platform DB (re-run the §3.2 procedure) to re-populate the retrieval grain under fresh op_ids, and record those op_ids + observed per-table counts for the gate’s expected-count baseline.
  3. Then run the {134.6} gate against each DB to confirm G1–G7.

Mark this precondition explicitly in the {134.6} details and testStrategy: “VALIDATION BLOCKED pending id-131-lockstep image deploy + fresh re-walk (both Platform DBs); the S429 row counts are the reference baseline but MUST be re-baselined against the re-walk.”

3.2 The seed + walk procedure (each step idempotent; owner-approved R-2)

Section titled “3.2 The seed + walk procedure (each step idempotent; owner-approved R-2)”

Ordered steps against a Platform DB (prod zjqbrdctesqvouboziae, then staging rbwqewalexrzgxtvcqrh):

  1. Verify baselineapplication_types present (requireApplicationTypeId, seed-synthetic-corpus.ts:442, fails loud if absent).
  2. Workspacesbun run scripts/seed-platform-workspaces.ts --target=prod --apply.
  3. Synthetic corpusbun run scripts/seed-synthetic-corpus.ts --target=prod --apply → 6 Synthetic — … workspaces + ~30 form_questions.
  4. Feed slicebun run scripts/seed-platform-feed.ts --target=prod --apply → N_feed feed_articles (passed=true).
  5. Stamp the manifest per-targetbun run scripts/seed-synthetic-corpus.ts --target=prod --emit-manifest --manifest-out=<corpus>/.kh-workspace-map.json (RATIFY-1b: the one per-target value).
  6. Sync + walk{134.4} bridge copies the vendored tree → /cocoindex-state/corpus; trigger one supervised loopback POST /walk (Bearer $CRON_SECRET, runbooks/onprem-b1-deploy.md step 7b), watch docker logs, Coolify Stop as kill-switch.

Synthetic rows are Synthetic — -prefixed + --clean-removable (reversible prod pollution, R-2). Safety: dry-run default; --apply required; one DB per run (project-ref guard). Repeat steps 2–6 with --target=staging. This is a live operator run (staff-supervised), not a fire-and-forget worktree executor.


4. The repeatable expected-row-delta release gate ({134.6})

Section titled “4. The repeatable expected-row-delta release gate ({134.6})”

A new re-runnable TS assertion scriptscripts/verify-platform-promotion-gate.ts (greenfield) — mirroring the seed-synthetic-corpus.ts harness: same --target resolution

  • project-ref guard, sb()/tryQuery() from @/lib/supabase/safe (DIRECT import, no barrel), exit non-zero on any failed assertion. RATIFY-GATE-SURFACE: a TS operator script (not a Python pytest) — a release-time gate over a live deployed DB, consistent with the sibling seed scripts and reusable by id-128 Lane B. It reads, never writes.

4.2 The re-derived assertion set (keyed on the run op_id)

Section titled “4.2 The re-derived assertion set (keyed on the run op_id)”

Run-select (the foundation). Read the latest terminal pipeline_runs row WHERE pipeline_name = 'kh_canonical_pipeline' AND op_id IS NOT NULL (NOT bare status='completed' — governance crons emit empty completed heartbeats). Extract its op_id; assert every content-side delta filtered by that op_id.

#assertioncloses
G1pipeline_runs: the run-selected row is terminal status='completed' (NOT failed / completed_with_errors); exactly one terminal row for the runfalse-fail / partial-run
G2result.stage_counts: source_walk ≥ N_walk (lower bound — counts all walked files incl. any still-present forms binaries); chunking > 0; embedding == chunking (every declared chunk embeds synchronously — flow.py:2257)empty / forms-only / partial-embed walk
G3source_documents WHERE op_id=run = 5 (content-side) + N_feed — the PRIMARY grain (content_items is eliminated; SD carries op_id)binary + URL extractor coverage
G4 (HEADLINE)content_chunks WHERE op_id=run > 0 AND every row embedding IS NOT NULL (post-M2 FK source_document_id)THE gap — 0 content_chunks despite 395 “completed” runs
G5q_a_extractions WHERE op_id=run ≥ 1, each source_document_id IS NOT NULL (RAW tier — NOT q_a_pairs, which has no op_id and is promoter-tier)Q&A raw-tier write + re-parent integrity
G6reference_items WHERE op_id=run = N_feed, each source_document_id IS NOT NULLfeed/URL slice (BI-4)
G7(isolation) the gate connects ONLY to the Platform DSN (zjqbrdctesqvouboziae prod / rbwqewalexrzgxtvcqrh staging); it never opens a client DSN — rows land only in the Platform DBpromote-runbook §2 “rows in the Platform DB”
E (informational — REPORT, do not fail)entity_mentions WHERE op_id=run (has op_id) + entity_relationships joined via source_document_id ∈ this-run SDs (no op_id column — RULING 2, flow.py:1343)id-133 register not yet populated → 0 expected; report the counts, never gate on them

Dropped from the superseded gate (do not re-add): the old G3 forms gate (form_templates — forms are manual-upload, no op_id column, DR-014) and the old G5 content_items assertion (content_items eliminated by ID-131). The old G1–G9 numbering is replaced by this G1–G7 + E set.

Expected counts. Express N_walk (walked-file lower bound), N_content=5 (content-side SDs) and N_feed (seeded feed URLs) as gate parameters. Reference baseline (S429, pre-wipe): content_chunks=6 all embedded, source_documents=6, q_a_extractions=5, chunking=6, embedding=6 — but these MUST be re-baselined against the §3.1 re-walk (exact chunk counts depend on the chunker over the deployed corpus).

M5 follow-up (mark with a code comment in the gate). When ID-131 {131.11} lands (record_embeddings populated + the inline vector cols dropped), G4’s embedding check re-homes from content_chunks.embedding to EXISTS (SELECT 1 FROM record_embeddings re WHERE re.owner_kind='content_chunk' AND re.owner_id = cc.id). record_embeddings has no op_id ((owner_kind, owner_id, model) idiom, migration 20260628190001) → join via this run’s chunk ids (content_chunks WHERE op_id=run). This is a forward dependency on the cross-Task id-131 {131.11}, NOT a {134} subtask (a {134}{131.11} sibling dep would be a cross-Task dep — forbidden); it is a code-comment-marked follow-up the Orchestrator routes to id-131 / backlog.

The gate discharges runbooks/platform-to-client-promote.md §2 bullet 1, upgrading its prose (“confirm a real ingest lands content”, naming only content_items/source_documents) into an assertable check with the content_chunks dimension it omits. Run:

  • Per release, at the §2 “Prove on Platform” step, before §3 cuts the vX.Y.Z tag — so only a config that lands the full content write becomes a promote candidate.
  • Re-runnably: byte-stable corpus (deterministic uuid5 seeds) + idempotent seed + in-place UPSERT walk ⇒ the only variable between two promote runs is the image/migrations/app under test.

The runbook edit lands in the docs-site repo (its own .claude/ lane).


5. Composition with id-128 (three non-conflatable E2E lanes)

Section titled “5. Composition with id-128 (three non-conflatable E2E lanes)”

Per id-128 S423 the convergence is on schema shape, not execution — three lanes:

  • id-128 Lane A — Playwright APP tests; SQL fixtures + precomputed embeddings; ephemeral Supabase branch; no worker.
  • id-128 Lane B — the cocoindex TS/Python integration specs run a real /walk over the synthetic-corpus structure. Lane B self-contains its fixtures (TS stageFixture from docs/testing/test-data/templates/; Python tmp_path) — the vendored tree has zero test consumers.
  • {127}/{134} — the deployed Platform rig (real Coolify app+worker+Traefik+webhook+ image vX.Y.Z) — a release/deploy-time gate, not a CI test.

{134.8} is a structural guard, NOT a re-point (mechanism corrected — OQ oq-c0d96c319a2ff88a). The original “re-point Lane B at the vendored corpus” premise was false (Lane B self-contains; a re-point would change behaviour). {134.8} instead ships a read-only corpus-shape guard (__tests__/integration/cocoindex/platform-corpus-shape.test.ts) asserting the §2.1/§2.2 file-tree shape — a corpus-rot guard (the vendored tree is a real deploy artefact via {134.4}). BL-392 (forms retirement) must update this guard’s toEqual set when the forms binaries are removed — that update is owned by BL-392, not a new {134} subtask.


6. ID-131 coupling — golden-snapshot deferral (re-homed)

Section titled “6. ID-131 coupling — golden-snapshot deferral (re-homed)”

S423 §6 proposed a golden snapshot: run /walk once, freeze the retrieval grain to a versioned seed, replay it deterministically in Lane A (real output, no worker, no token burn). DEFER the FREEZE until after ID-131 {131.11} (M5). ID-131 already re-parented content_chunks onto source_documents (M2) and introduced record_embeddings (M1b); M5 ({131.11}) will drop the inline vector cols and make record_embeddings the sole embedding store. A snapshot frozen on today’s inline-content_chunks.embedding shape is invalidated the moment M5 lands. The corpus FILES are shape-stable now (inputs); only the frozen OUTPUT + the gate’s G4 embedding-home (§4.2 M5 note) wait. Disposition: build the gate now against content_chunks.embedding; the M5 re-home is a code-comment-marked forward follow-up, not a {134} subtask, not a blocker.


The gate (§4) IS the primary validation surface:

invariant / intentverified by
v3 §9 half-1 — seam-coverage file tree exists§2 tree ({134.3} done); G2–G6 exercise every route×format
v3 §9 half-2 — seed runs on both Platform DBs on the id-131 image§3 (re-walk precondition §3.1); G3/G4/G5 non-zero after the re-walk
v3 §9 half-3 — repeatable row-delta gate closes the 0-content_chunks gapG4 (content_chunks > 0 with embeddings)
BI-3 — synthetic only, never client datareviewer confirms no client file (all synthetic-) + {134.8} shape guard
BI-7 — mandatory manifest, walk does not abortG1 + manifest load succeeds (no manifest_missing)
BI-4 — URL via /extract, no PullMDG6 (reference_items via feed /extract)
DR-014 — forms NOT corpus-walkedgate asserts no form_templates; §2.3 two-manifest distinction
graceful degradationedge file: soft-warn logged, flow_status not flipped

Behaviour-first (reference/test-philosophy.md): the gate asserts observable row deltas, not pipeline internals. It is an operator script against a live DB (not a bun run test unit), while the file-tree shape is guarded by the {134.8} structural test (which runs in bun run test).

Idempotency check: run the gate, re-walk, re-run — G1–G7 hold with a new op_id and identical counts (deterministic uuid5 UPSERT).

Empirical-verification scope note. This TECH introduces no new external-library API surface — the gate calls only internal Canonical symbols (sb/tryQuery from @/lib/supabase/safe; the @supabase/supabase-js client is reached through those wrappers, already in use repo-wide). No pinned third-party symbol requires an import-and-call check per shared-discipline.md §Empirical verification.


8. RATIFY decisions (preserved / corrected)

Section titled “8. RATIFY decisions (preserved / corrected)”
  • RATIFY-1 — Corpus ownership: VENDOR committed in-repo. STILL VALID. Vendored at scripts/cocoindex_pipeline/fixtures/platform-corpus/; local-fs-platform/corpus is a derived working copy (BI-3: synthetic → no IP block on committing). Done ({134.3}).
  • RATIFY-1b — Manifest per-target stamping. STILL VALID, corrected: the stamped file is the ingest route-tag manifest (§2.3 sense 1); its forms-workspace mapping may be empty post-DR-014, but the file + per-target stamp stand (walk guard + content/feed routing).
  • RATIFY-2 — __qa__/: avoid it; route Q&A via the content branch. STILL VALID. G5 covers the q_a_extractions raw-tier via the content branch.
  • RATIFY-3 — Golden-snapshot freeze: defer. STILL VALID, re-homed: the deferral now targets ID-131 M5 {131.11} (record_embeddings + inline-col drop), not just the M2 re-parent (§6).
  • RATIFY-4 — Run the seed on Platform PROD. DONE (S429, both DBs). Outstanding action is the id-131-image re-walk (§3.1), not the initial seed.
  • RATIFY-GATE-SURFACE — op_id-keyed TS gate script. STILL VALID (§4.1).
  • CORRECTED by the OKF re-arch: forms route + G3 forms gate DROPPED (DR-014 / BL-392); content_items G5 DROPPED (ID-131); “independent of ID-131” framing removed; {134.8} re-point → structural guard.

  1. /extract depends on a live public feed URL. Mitigation: a stable canonical gov.uk procurement URL; if it 404s, G6 fails loud (better than silent skip). (OQ-3 feed-URL determinism remains open — see PLAN note.)
  2. Binary fixtures drift from real shapes. The synthetic PDF/DOCX must be real binaries (file(1) reports PDF/Office, not text-renamed). Mitigation: {134.3} used cupsfilter / textutil; the {134.8} guard asserts magic bytes + OOXML parts. G4 (chunks from those files) proves the extractor ran.
  3. Token burn on every release walk. Mitigation: small corpus (5 content-side files); operator-triggered at release time only. The golden snapshot (§6, deferred) is the eventual token-free CI replay.
  4. Stale image at re-walk. The single biggest correctness trap (§3.1): a Platform image predating the id-131-lockstep flow.py mis-keys the grain. Mitigation: verify the deployed image carries the M2 write-sites before the re-walk; G4/G5 fail loud if source_document_id is unstamped.
  5. Prod pollution. Mitigation: Synthetic — prefix + --clean; owner-approved (R-2).
  6. BL-392 timing. Until forms retirement lands, forms binaries still walk and write form_templates. Mitigation: the gate asserts nothing on form_templates; G2 uses a source_walk ≥ N_walk lower bound; G3 counts source_documents (forms-independent).

10. {134.6}/{134.7} implementation scope (forward note)

Section titled “10. {134.6}/{134.7} implementation scope (forward note)”

{134.2} PLAN already decomposed this Task; the outstanding slices are:

  • {134.6} — build scripts/verify-platform-promotion-gate.ts with the §4.2 G1–G7 + E set (headline G4 content_chunks>0 with embeddings), op_id-keyed, run-selected by pipeline_name='kh_canonical_pipeline' AND op_id IS NOT NULL; VALIDATION-BLOCKED on §3.1. Deps: {134.3}, {134.5}.
  • {134.7} — wire the gate into promote-runbook §2 (docs-site): drop forms language, headline content_chunks>0-with-embeddings, state the run-select filter, distinguish the two manifest senses (§2.3). Deps: {134.6}.

Open questions carried forward (from {134.6} journals; not TECH-blocking): OQ-2 (entity tier as a hard gate vs informational — this spec rules informational/E, revisit when id-133 populates) and OQ-3 (feed-URL determinism — risk #1). The M5 gate re-home and the golden-snapshot freeze are post-ID-131 forward follow-ups, not {134} subtasks.