Skip to content
Canonical

ID-93 {93.1} RESEARCH — Pre-re-ingest intent-gap register (coupling-classified)

ID-93 {93.1} RESEARCH — Pre-re-ingest intent-gap register

Section titled “ID-93 {93.1} RESEARCH — Pre-re-ingest intent-gap register”

Status: RESEARCH complete — decision queue PENDING Liam (§6). Authored S-untracked (parallel to S324-adjacent sessions), 07/06/2026. Purpose: before ID-45 (T7) full-corpus re-ingest, classify ALL yet-to-be-built / yet-to-be-specced functionality by re-ingest coupling, so nothing that is cheap today becomes expensive after the run. Routes into the ID-64 §7 gate checklist BEFORE {64.7}/{64.8} land. This doc is also a re-litigation killer: §5 records verified-safe items with evidence so they are not re-investigated.

  • Task: ID-93 (in_progress) — RESEARCH-only; no implementation here.
  • Sibling surfaces: ID-64 RESEARCH (schema/FK/retention gate-set), v1-completion-sequence.md (ordering board), deferral-register.md (V1.1/V2 deferrals).
  • Prod row counts (07/06/2026, read-only): content_items 631 · content_chunks 926 · entity_mentions 1,397 · content_history 1,225 · feed_articles 410 · workspaces 4 · q_a_extractions 3 · bid_questions 0 · bid_responses 0 · content_citations 0 · q_a_pairs 0 · form_templates 0 · source_documents 0 · reference_items 0 · bid_questions rows with matched_content_ids 0.

1. Method

Section titled “1. Method”

1.1 Coupling classes

Section titled “1.1 Coupling classes”
ClassMeaningCost if deferred past re-ingest
C1 ingest-capturePipeline must write/compute it DURING the runSecond LLM run or LLM backfill — expensive
C2 zero-row windowRename/constraint cheap while tables empty (origin_kind precedent, ID-64 §3)Becomes a data migration
C3 id-coupledKeyed to content_items/source_documents ids (renumbered at re-ingest)Orphans; re-link work
C4 shape-onlyNew tables/columns; data arrives post-ingestNear-zero (ALTER later); only the handover-completeness policy question
C5 verified-decoupledNothingZero — recorded with evidence to kill re-litigation

1.2 Adversarial verification discipline

Section titled “1.2 Adversarial verification discipline”

First-pass doc sweeps over the (historic) intended-architecture corpus produced false HARD classifications — e.g. question_matches claimed ingest-coupled; V11-4 sidecar claimed to force a second ingest. Both refuted against the live write surface. Rule: no C1/C2/C3 claim enters this register without a citation against scripts/cocoindex_pipeline/flow.py, the migration corpus, or a live row count. Truth set used: flow.py mount_table_target write surface (flow.py:2987–3045), migration SQL, prod row counts above, v1-completion-sequence.md Lane B verification.

Pipeline write targets (verified): content_items, source_documents, q_a_extractions, entity_mentions, content_chunks, form_templates, form_template_fields, reference_items. Never q_a_pairs, question_matches, citations, content_history (flow.py:2987–3045).

2. C1 — ingest-capture (must be right BEFORE the run)

Section titled “2. C1 — ingest-capture (must be right BEFORE the run)”
ItemFindingAction
bl-92 Provider-Disambiguation holder-rule fix (S203 WP-C1)Classification-quality fix spanning DB CHECK + TS union + Python frozenset + skill table + inlined bundle. If the cocoindex extraction prompts (scripts/cocoindex_pipeline/extraction.py + stage prompts) do NOT carry the fix, the entire fresh corpus is classified with the known-defective rule.Pre-run porting check (new gate): verify fix propagates to the cocoindex prompt surface; port if not. Small.
bl-93 Table-heavy entity recall + Pass-1 truncation 5000→32000 (S203 WP-C2)Same class — entity recall at ingest. S203 framing targets the OLD pipeline surfaces; cocoindex path needs the equivalent check.Same pre-run porting check.
Q&A alternate phrasings at extraction (ID-71 candidate)NOT captured today (q_a_extractions.extracted_question_text singular). Post-hoc derivable by batch LLM over stored extraction text — so not strictly C1 — but capturing during the run is the cheapest moment if wanted.Liam call (§6.7). Recommend DEFER (post-hoc batch).

Everything else evaluated for C1 was refuted — see §5. Notably: document summaries are post-ingest BY DESIGN (generate_summary tool; ingest-time generation would be lost on content update); chunk-heading hierarchy is post-hoc derivable (non-LLM markdown re-parse; re-embed cost only); cross-document links derive from stored entity_mentions co-occurrence (SQL, no LLM).

3. C2 — zero-row window (cheap NOW, data migration after)

Section titled “3. C2 — zero-row window (cheap NOW, data migration after)”

3.1 The headline: the ENTIRE bid→form terminology surface is at zero rows TODAY

Section titled “3.1 The headline: the ENTIRE bid→form terminology surface is at zero rows TODAY”

Prod: bid_questions 0 · bid_responses 0 · content_citations 0 · q_a_pairs 0. OQ-64-3 was parked as “defer out of pre-re-ingest scope” — that deferral default should be re-decided deliberately, because the window closes twice over:

  1. Re-ingest + UC5 promotion fills q_a_pairs (ID-45 promotes extractions) — the origin_kind-adjacent surfaces gain rows.
  2. Bid-testing UAT (bid-testing-strategy-spec.md) creates bid_questions / bid_responses / content_citations rows the moment it starts — possibly before the re-ingest run.

Rename surface if ratified: bid_responses + bid_response_history tables, bid_questions table, search_for_bid_response RPC, content_citations → citations (coordinates with ID-58/T11 polymorphic redesign — T11 may REPLACE rather than rename it, which is an argument for deciding jointly), citing_entity enum value, ~18+ TS consumer files (.from('bid_responses') etc.). All code+DDL only at current row counts. → §6.1.

3.2 Other C2 items

Section titled “3.2 Other C2 items”
ItemFindingAction
{64.2} origin_kind value renameAlready gated (ID-64 §7 A2)None — stands
bl-74 q_a_pair_history snapshot misses superseded_by + source_workspace_idq_a_pairs/history at 0 rows. ID-45 itself fills them (promotion step). Landing the trigger/column extension AFTER promotion = history rows missing lineage fields forever (history is append-only). Currently parked with a Lane-B “consume at ID-58 spec” note — too late.Re-home: land BEFORE the ID-45 promotion step. → §6.5
bl-42 DB-layer empty-string coercion (BEFORE INSERT trigger, classification cols)Trigger guards the FRESH corpus writes — pointless to add after 631+ rows land un-guardedFold into Lane A pre-run window
bl-43 content_item_workspaces PK constraint renameCosmetic; cheapest pre-runFold into Lane A (batch with other DDL)
bl-106 Zod ingestion_source enum 4→9 (Python writes 5 values Zod rejects)Run-record correctness DURING the run — read-path validation of pipeline_runs/rows written by the re-ingest itselfFold into Lane A pre-run window

4. C3 — id-coupled (renumbering exposure)

Section titled “4. C3 — id-coupled (renumbering exposure)”
ItemFindingAction
content_history (1,225 rows)Known — OQ-64-9 openStands ({64.7})
feed_articles (410) content_item_id re-linkKnown — {64.7}Stands
content_citations.content_item_id + bid_questions.matched_content_ids uuid[]0 rows today — but UAT before the run would create rows citing ids that renumber. matched_content_ids is an array — no FK protects it; orphaning is silent.NEW GATE for {64.8}: re-verify zero rows on bid_questions/bid_responses/content_citations immediately pre-run; if UAT preceded the run, extend the {64.7} re-link plan to cover them. → §6.3
edit_intent storage (ID-59/T9 sub-9: content_history column vs sidecar .md metadata vs cocoindex flow arg)No column exists today; ID-59 TECH explicitly defers storage shape. Two of three options couple to re-ingest surfaces: a content_history column couples to OQ-64-9 disposition; a sidecar file couples to the cocoindex source-file layout being frozen at re-ingest.Decide storage option JOINTLY with OQ-64-9, before {64.7} finalises. No implementation needed pre-run. → §6.8

5. C4 + C5 — verified-safe register (re-litigation killers)

Section titled “5. C4 + C5 — verified-safe register (re-litigation killers)”

5.1 C4 shape-only — one policy decision covers all

Section titled “5.1 C4 shape-only — one policy decision covers all”

VERIFIED: the pipeline never writes q_a_pairs, question_matches, citations, or any entity-spine table (flow.py:2987–3045). ID-57/58/60 are fully post-ingest. The ONLY remaining question is policy (OQ-64-5 generalised): should the handover DB be structurally complete (land empty T10/T11/T12 tables pre-cutover) or minimal (add post-cutover when each spec ratifies)? Recommend minimal — specs not ratified; speculative DDL would be churned by spec outcomes; ALTER/CREATE post-cutover is cheap. One ratification closes A6 for good. → §6.2

Shape-only items that stand as deferred: V11-11/12 form-supersession pointer columns (pipeline never writes them — confirmed form_templates has no policy/supersession columns); V11-1 private_to_workspace_id; V11-7 state_machine_config; V2-1 form_type policy JSONBs; scope_tags CV table (absent; q_a_extractions.scope_tags is free text[] — UC9 governance post-cutover); reserved satellite workspace tables (exist, minimal shape — per-app columns at feature-spec time).

5.2 C5 verified-decoupled

Section titled “5.2 C5 verified-decoupled”
Claim killedEvidence
V11-4 markdown sidecar is ingest-coupled — REFUTEDq_a_extractions retains full extracted_question_text + extracted_answer_text + the 4 form-metadata fields (flow.py:1162–1180, 2100–2122). Sidecar is post-hoc derivable, zero LLM re-run. Reclass C4.
question_matches/citations/entities needed at ingest — REFUTEDNot in write targets (flow.py:2987–3045); Lane B verification corroborated.
entity_relationships table exists / Stage-5 writes a graph — REFUTEDNo such table; Stage-5 writes entity_mentions UPDATEs only (stage_5.py). ID-60 KG is post-ingest; cross-doc links derivable from stored mentions (SQL co-occurrence).
reference_items renumber at re-ingest — REFUTEDuuid5(ns, "ri:"+url) deterministic ids (flow.py:2504); same URL → same id. Stable across runs.
Forms Path-B couples form re-ingest to content corpus — REFUTEDForms land ZERO content rows (flow.py:2622–2628); form_templates/form_template_fields only.
Document summaries need ingest capturePost-ingest generate_summary tool is the ratified pattern; raw markdown retained in content_items.content.
Template-driven completeness at riskMVP built; template_requirements + embeddings decoupled from content ids.
ID-59 arbitration rule itselfPure write-site function post-CRDT-merge; no ingest coupling (storage option is the only C3 — §4).

5.3 ID-71 ingest-time data demands — verdict

Section titled “5.3 ID-71 ingest-time data demands — verdict”

Within proxy sources (06-mcp-tooling.md, p0-23 spec, current lib/mcp/tools surface — Liam’s 54-tool review notes still pending handover, §7): no NEW ingest-time capture is required for the v1 AI surface. Classification/embeddings/chunk-position/entity mentions/Q&A extraction all captured; summaries + headings + cross-links post-hoc derivable from retained raw material.

One contract defect found: content_chunks.heading_text/heading_level/heading_path columns EXIST (pre_squash:3454–3456) and the search_content_chunks RPC RETURNS them (pre_squash:3029) — but the cocoindex RecursiveSplitter never populates them (budget-driven, non-LLM). Post-re-ingest every chunk row has NULL/empty headings flowing through the MCP chunk-search contract. {71.1} input: either populate post-hoc (markdown re-parse + re-chunk + re-embed — embedding cost only, no LLM) or drop the fields from the tool contract. Not a re-ingest gate.

6. Liam decision queue

Section titled “6. Liam decision queue”

RATIFICATIONS 08/06/2026 (Liam): §6.1 = rename pre-run → new {64.14} (naming target form_* vs procurement_* confirmed at dispatch, STEP 0). §6.2 = structurally complete, based on ratified specs → T10/T11/T12 schema DDL lands pre-cutover ONLY per ratified ID-57/58/60 specs ({64.8} gate G6). §6.3–§6.6 = approved → {64.8} gates G1/G2/G3/G5; bl-74 flipped ready; bl-42/43/106 Lane-A folds noted. §6.7 = capture at ingest → new Task ID-94 ({64.8} gate G4). §6.8/§6.9/§6.10 remain open — carried in {64.7} (joint edit_intent/OQ-64-9 decision), {71.1} (notes handover), OQ-64-7.

CLOSE-OUT 11/06/2026 (Liam + session review): the queue is now fully dispositioned. §6.1 EXECUTED — {64.14}+{64.16} bid→form rename DONE (staging S331, prod S336 with the 7-mig cutover). §6.3 was subsequently DROPPED as a gate (08/06, {64.8} journal: platform not live, surface cannot gain rows pre-run; S332 caveat — revisit only if bid-testing UAT starts before the ID-45 run). §6.4 G1 SATISFIED (bl-93 recall guidance ported df4463cd; bl-92 N/A on the cocoindex path — relationship/holder extraction is owned by ID-101, which supersedes it as the C1 gate; both backlog items retired). §6.5 EXECUTED — {64.15} DONE. §6.6 EXECUTED — bl-42/43 landed via {64.17}; bl-106 ESCALATED (no canonical ingestion_source list ratified; 8 observed values — still pending Liam). §6.7 EXECUTED — ID-94 DONE (+ prod push S329). §6.8 RESOLVED — storage shape decided in the ID-59 TECH chain and built: edit_intent/arbitration_inputs columns ({59.5}, staging S331 → prod S336); no longer coupled to OQ-64-9 (resolved separately, below). §6.9 RESOLVED — 54-tool notes handed over; ID-71 in-flight (lane-a workshop outcomes ratified; see platform-direction.md). §6.10 OQ-64-7 RESOLVED — the workspace concept/purpose (MAT Auditing · NHS Digital Cyber Alerts · Education Sector Monitor) carries to the new DB, not the rows themselves; GNEWS-DEDUP deleted. OQ-64-9 RESOLVED — clean-slate content_history; pre-export sweep first identifies any client actions (keyed by client UUIDs — expected minimal). Residual opens routed to ID-64: bl-106 ratification ({64.8} G5 tail) + ID-57 schema-slice spec-ratification ({64.8} G6 tail).

  1. OQ-64-3 RE-DECISION (bid→form rename) — the window is NOW. Entire surface at 0 rows (§3.1). Options: (a) rename pre-run/pre-UAT (code+DDL only, coordinate content_citations with ID-58 T11 which may replace it outright); (b) ratify permanent bid_* naming; (c) accept later data migration. Default-defer is the one wrong answer.
  2. OQ-64-5 as POLICY: handover DB structurally complete vs minimal for T10/T11/T12 tables. Recommend minimal (§5.1).
  3. Pre-run zero-row re-check gate on bid_questions/bid_responses/ content_citations (+ explicit UAT-before-run sequencing call): ratify as a new {64.8} checklist line (§4).
  4. bl-92/93 cocoindex porting check as a new pre-run gate (§2). Small, high corpus-quality leverage.
  5. bl-74 re-home: q_a_pair_history snapshot columns land BEFORE the ID-45 promotion step (currently parked; §3.2).
  6. bl-42 + bl-43 + bl-106 fold into Lane A pre-run window (§3.2).
  7. Q&A alternate phrasings: extend ingest extraction now vs post-hoc batch later. Recommend defer (post-hoc derivable; keeps run risk down).
  8. edit_intent storage option (ID-59/T9): decide jointly with OQ-64-9 content_history disposition before {64.7} finalises (§4). Decision only — no build.
  9. ID-71 confirmation: accept the “no new ingest-time capture for v1” verdict (§5.3)
    • unblock the 54-tool review-notes handover so {71.1} can re-test it against the real outcome list; disposition the content_chunks headings contract defect inside {71.1}.
  10. OQ-64-7 (workspace ontology / workspaces retention) — pre-existing, still pending, unchanged by this evaluation; restated for completeness.

7. Input gaps

Section titled “7. Input gaps”
  • Liam’s 54-tool review notes (ID-71 {71.1} seed) not yet handed over — §5.3 verdict is proxy-source-based and must be re-tested when the notes land. This is the single open input that could add a C1 item.
  • Staging row counts not separately swept (prod is the retention-authoritative source; re-ingest target is a fresh preview branch per OQ-64-8).

8. Routing (proposals — Orchestrator/Liam write the ledger, not this doc)

Section titled “8. Routing (proposals — Orchestrator/Liam write the ledger, not this doc)”
  • ID-64: new Lane-A/{64.x} items per §6.3/§6.4/§6.6 (+ §6.1 rename migration if ratified); {64.8} gains the zero-row re-check line; {64.7} gains the §6.8 joint decision dependency.
  • ID-45 spec chain: promotion-step ordering note (bl-74 before promotion); C1 porting-check gate as a pre-flight item.
  • ID-57/58/60: explicitly post-cutover under the §6.2 minimal policy (if ratified); ID-58 absorbs the content_citations disposition.
  • ID-71 {71.1}: §5.3 verdict + headings contract defect as seeded inputs.