Skip to content

ID-64 {64.1} RESEARCH — Pre-re-ingest readiness (schema/FK migration + correctness gate-set)

ID-64 {64.1} RESEARCH — Pre-re-ingest readiness (schema/FK migration + correctness gate-set)

Section titled “ID-64 {64.1} RESEARCH — Pre-re-ingest readiness (schema/FK migration + correctness gate-set)”

Status: RESEARCH (findings + gate checklist + decomposition proposals + Open Questions). This artefact does NOT run migrations and does NOT ratify the final plan. DDL is authored and applied via the Supabase CLI only (supabase migration new + supabase db push) — never via MCP execute_sql / apply_migration (CLAUDE.md Supabase gotcha). All {64.2+} subtask proposals below are returned to the Orchestrator for writing into docs/reference/task-list.json; this Planner does not edit the ledger.

  • Task: ID-64 — Pre-re-ingest readiness umbrella (status: spec_needed, priority: high, deps: [56, 62, 63], effort: L).
  • Worktree base: main @ fffe0aeb (env-reported 11cd62e5 is the worktree HEAD pre-fetch; the Orchestrator integrates against current main).
  • Authoring session: S279 (continuation), DD/MM/YYYY = 29/05/2026.
  • Decomposition principle: ID-64 is an umbrella. Its correctness/capability gates (chunking, fixture-staging, ontology/CV) live in Tasks 56 / 62 / 63 and are wired as Task-level dependencies (ID-64.dependencies = [56, 62, 63]) — NOT as {64.x} subtasks (sibling-only constraint, §3.3). The {64.2+} subtasks proposed here own ONLY the schema-migration + FK + data-retention work that has no other home, plus the single readiness-surface subtask that aggregates the gate checklist.

1. Code-intelligence orientation (run before authoring; cited verbatim)

Section titled “1. Code-intelligence orientation (run before authoring; cited verbatim)”

Per .gitnexus/CLAUDE.md “Always Do”. SQL migrations and Python pipeline are OUTSIDE the TS index (ts-morph corpus only — .ast-dataflow/CLAUDE.md), so the gitnexus queries are supplemented with grep sweeps over supabase/migrations/*.sql and scripts/cocoindex_pipeline/*.py. Repo disambiguation required: repo: /Users/liamj/Documents/development/knowledge-hub.

1.1 gitnexus_query({query: 'origin_kind derived_from_bid_response'})

Section titled “1.1 gitnexus_query({query: 'origin_kind derived_from_bid_response'})”

Returned no TS execution flow that reads/writes origin_kind as a symbol (it is a DB CHECK-constraint string value, not a TS symbol). Top definitions surfaced were the QA-form extraction tests (scripts/tests/test_cocoindex_extraction.py:TestExpectedResponseKindParity, test_only_two_values) and lib/ast-dataflow/queries/flow-trace.ts:resolveOrigin (an unrelated ast-dataflow internal). Conclusion: origin_kind lives entirely in SQL + Python; the rename is a SQL CHECK-constraint + Python-string-literal change, with NO TS consumer to refactor. Confirmed by grep (§3.2).

1.2 gitnexus_query({query: 'row level security workspace scoping form_templates'})

Section titled “1.2 gitnexus_query({query: 'row level security workspace scoping form_templates'})”

Surfaced app/api/procurement/[id]/templates/route.ts:POST (proc_46) and .../[templateId]/fill/route.ts:POST (proc_110) as the app-side template flows, plus scripts/cocoindex_pipeline/flow.py:ingest_file / _trim_stale_form_fields and lib/templates/template-coverage.ts:listAvailableTemplates / fetchTemplateRequirements. Conclusion: form-template reads happen on BOTH the app side (TanStack-fed routes) and the pipeline side (cocoindex mount_table_target). Any RLS change is exercised by the live integration test, not by these TS flows — the durable acceptance artefact is the integration test (§2.1), not a unit assertion.

1.3 gitnexus_query({query: 'foreign key content_items source_documents canonical pipeline'})

Section titled “1.3 gitnexus_query({query: 'foreign key content_items source_documents canonical pipeline'})”

Surfaced scripts/cocoindex_pipeline/flow.py:ingest_file / app_main, scripts/kb_pipeline/store.py:insert_content_item / insert_content_history_entry / check_url_exists, and lib/intelligence/pipeline.ts:storeAsContentItem / processFeedSource / ensureWorkspaceLink. Conclusion: there are TWO writer families into content_items: the cocoindex flow (flow.py, the new canonical pipeline) and the intelligence/RSS path (lib/intelligence/pipeline.tsstoreAsContentItem, plus feed_articles). This bears directly on the content_items-vs-feed_articles disposition (OQ-64-2).

1.4 grep sweeps (SQL + Python — outside TS index)

Section titled “1.4 grep sweeps (SQL + Python — outside TS index)”
  • origin_kind CHECK constraints: only q_a_pairs + q_a_pair_history carry it (§3.2).
  • templates_select … USING (true): confirmed in supabase/migrations/20260416102457_pre_squash_reconciliation.sql:6346 (§2.1).
  • cocoindex write targets: mount_table_target for content_items, q_a_extractions, source_documents, entity_mentions, form_templates, form_template_fields (scripts/cocoindex_pipeline/flow.py lines 1472–1507) (§4).

gitnexus_context on row-type symbols was not productive: DB row shapes are generated types in supabase/types/database.types.ts (Tables<'x'>), not call-graph symbols, so the canonical source for column/FK shape is the generated types file (read directly in §3–§4), which is the project-mandated source of truth (CLAUDE.md “TypeScript conventions”). No symbol-edit is proposed by this RESEARCH, so gitnexus_impact is N/A here; each {64.x} implementation subtask brief below carries the impact-analysis-before-edit / detect-changes-before-commit discipline for its Executor (Inv 2 → Inv 3).


2. Inv-25 RLS — role-based single-tenant USING(true) is intended (CLOSED S292; not a migration)

Section titled “2. Inv-25 RLS — role-based single-tenant USING(true) is intended (CLOSED S292; not a migration)”

form_templates was renamed from templates (supabase/migrations/20260520120828_t2_combined_pr_intel_shape_b_form_type_split.sql SUB-TASK 2, lines 204–239) and inherited the original four templates_* policies (supabase/migrations/20260416102457_pre_squash_reconciliation.sql):

PolicyCommandPredicate
templates_selectSELECTTO authenticated USING (true) — role-gated single-tenant (intended)
templates_insertINSERTWITH CHECK (get_user_role() IN ('admin','editor'))
templates_updateUPDATEUSING (get_user_role() IN ('admin','editor'))
templates_deleteDELETEUSING (get_user_role() = 'admin')

USING (true) is role-gated (any authenticated user) but NOT workspace-scoped. Empirically, a workspace-A viewer can SELECT workspace-B form_templates rows — confirmed S278 against live staging with a viewer-role JWT (test.user3, get_user_role()='viewer'); the cross-workspace row WAS returned.

This is the INTENDED model, not a defect (ratified S292 — OQ-64-1 CLOSED). Liam ratified that role-based single-tenant USING(true) is the intended RLS model for the v1 Phew pilot: workspace membership is NOT required, and cross-workspace form_templates readability is acceptable under a single-tenant deployment. The earlier S279 framing of this USING(true) as a defect-to-fix — and the “Model A workspace_members junction” prerequisite — are REVERSED. ID-52 PRODUCT Inv-25’s per-workspace scoping (“A user’s view of ingested form instances is scoped to workspaces they may access … the catalogue, having no workspace scope, is a shared platform resource.”) describes a per-workspace multi-tenant posture the platform has deliberately NOT adopted (see §2.4 and ID-69/RESEARCH.md §4) — not a pre-re-ingest gate, and not a planned requirement. form_templates is the single-workspace Path-B activity output (workspace_id NOT NULL), correctly single-tenant; the canonical corpus layer (content_items) is, by ratified ID-69 design, NOT workspace-organised at all (§2.4). Evidence for the single-tenant disposition: procurement-workspaces/PRODUCT.md:382 (v1 single-tenant Phew pilot), reserved-workspace-seats/TECH.md:160 (role-based delegation), the bl-207 fix caveat (no membership model = a larger separable task), ID-69/RESEARCH.md:155 (correctly single-tenant), and the consolidated disposition in docs/themes/canonical-pipeline/reference/pipeline-writepath-schema-gap-s295.md §S296 (“OQ-64-1 — CLOSED S292”).

form_templates.workspace_id is NOT NULL with FK form_templates_workspace_id_fkey → workspaces(id) (supabase/types/database.types.ts form_templates block) — so workspace-scoping the SELECT predicate is structurally feasible without a schema change to the table itself.

2.2 Sibling tables to audit (Inv-25 scope)

Section titled “2.2 Sibling tables to audit (Inv-25 scope)”
  • form_template_fields — current template_fields_select … USING (true) (pre_squash:6308). Has NO workspace_id of its own; it links via form_template_fields.template_id → form_templates(id). Scoping must therefore be transitive (a field is visible iff its parent form_templates row is visible).
  • form_template_requirements (catalogue) — template_requirements_select … USING (true) (pre_squash:6327). This USING(true) is CORRECT per Inv-25 (the catalogue is a global platform resource, template_type → form_types only, no workspace_id). The acceptance test asserts the catalogue remains readable regardless of workspace (form-extraction-rls.integration.test.ts second it). Do NOT scope this one.

2.3 The single-tenant RLS model — USING(true) is intended, no membership junction by design

Section titled “2.3 The single-tenant RLS model — USING(true) is intended, no membership junction by design”

Ratified S292 (OQ-64-1 CLOSED): the absence of a workspace-membership model is BY DESIGN for the v1 single-tenant Phew pilot. Role-based USING(true) is the intended SELECT model; there is NO pre-re-ingest RLS migration required. This REVERSES the earlier S279 framing (below restated as historical context) that treated the missing junction as a blocking gap.

The empirical state of the schema (still accurate):

  • The workspaces table has no membership junction — a user associates with a workspace only through workspaces.created_by / updated_by. There is no workspace_members (or equivalent many-to-many) table.
  • Every workspace-bearing table SELECT-scopes with USING (true), not by workspace: confirmed for source_documents ("Authenticated users can view source documents" … USING (true), pre_squash:5835), bid_questions (bid_questions_select … USING (true), pre_squash:5924), form_templates, form_template_fields, form_template_requirements. The get_item_workspaces() / get_workspace_counts() functions are SECURITY INVOKER RPCs, not RLS predicates.

Under the ratified single-tenant model this uniform USING(true) is correct: a single deployment serving one tenant (Phew) does not need per-workspace SELECT isolation. The evidence base for the ratification: procurement-workspaces/PRODUCT.md:382 (v1 single-tenant Phew pilot), reserved-workspace-seats/TECH.md:160 (role-based delegation), the bl-207 fix caveat (no membership model = a larger separable task), ID-69/RESEARCH.md:155 (correctly single-tenant), and the consolidated disposition in docs/themes/canonical-pipeline/reference/pipeline-writepath-schema-gap-s295.md §S296 (“OQ-64-1 — CLOSED S292”).

2.4 The corpus is canonical and shared, NOT workspace-organised (ID-69, ratified)

Section titled “2.4 The corpus is canonical and shared, NOT workspace-organised (ID-69, ratified)”

The deeper reason per-workspace isolation is not a looming requirement: the ratified ID-69 vision makes the corpus a Wikipedia-style canonical record layer, NOT a workspace-organised one. One record is authored once and reused across workspaces/activities. The schema already supports this and the access model is intentionally shared:

  • content_items (the canonical record) has no workspace_id — by design. Workspace reuse is expressed by the many-to-many content_item_workspaces junction (composite PK (content_item_id, workspace_id), FK ON DELETE CASCADE to both parents, ciw_select USING(true)) — “the Wikipedia one-record-many-workspaces model” (ID-69/RESEARCH.md §4.1).
  • Path-A canonical ingest (scripts/cocoindex_pipeline/flow.py:ingest_file) never writes content_item_workspaces today (ID-69/RESEARCH.md §4.2/§4.3): the single resolved workspace_id feeds only the Path-B form_templates write. Canonical content is workspace-agnostic at ingest by design, not by omission.
  • form_templates.workspace_id is the only NOT NULL single-workspace column — the Path-B activity output, correctly single-tenant (ID-69/RESEARCH.md §4.1).
  • The curated-clean-corpus posture (content enters only when AI-ready) supersedes the older “point cocoindex at the whole client doc-dump” framing — a further reason the corpus is a shared canonical asset, not per-tenant siloed data.

Consequently USING(true) on the corpus/content tables is correct by architecture, and per-workspace SELECT isolation is not on the roadmap as a requirement. Recorded here only to forestall re-litigation: a workspace_members + is_workspace_member() scheme would be the mechanism only if the platform ever adopted a genuine multi-tenant SaaS posture for activity surfaces — which the ratified architecture (canonical shared corpus + single-tenant form-activity) does not require. Cross-ref ID-69/RESEARCH.md §4.

The S278 acceptance artefact __tests__/integration/form-extraction-rls.integration.test.ts (ID-52.13) was authored under the earlier multi-tenant framing; under the S292 single-tenant model it is not a pre-re-ingest gate (its cross-workspace-denial assertion describes a per-workspace multi-tenant model the platform has NOT adopted — §2.4). The catalogue-stays-global assertion remains valid regardless. That test would matter only if a genuine multi-tenant SaaS posture were ever adopted — which the ratified canonical-corpus + single-tenant-form-activity architecture does not require. Not ID-64, not on the roadmap.


3. KNOWN schema-migration item 2 — origin_kind value rename

Section titled “3. KNOWN schema-migration item 2 — origin_kind value rename”

3.1 What origin_kind actually is (correcting the brief’s “enum” framing)

Section titled “3.1 What origin_kind actually is (correcting the brief’s “enum” framing)”

origin_kind is not a Postgres ENUM type — it is a text column governed by a CHECK constraint, and it exists on exactly two tables: q_a_pairs and its history mirror q_a_pair_history (confirmed via type-block scan of supabase/types/database.types.ts and the migration grep). It does not exist on content_items, source_documents, entity_mentions, or any other pipeline table. So the rename is narrow and isolated to the QA subsystem, not a corpus-wide enum.

supabase/migrations/20260520225456_t6_q_a_pairs_full_schema.sql (SUB-TASK 1c, lines 60–76) is the latest reconciliation:

ALTER TABLE public.q_a_pairs DROP CONSTRAINT IF EXISTS q_a_pairs_origin_kind_check;
ALTER TABLE public.q_a_pairs ADD CONSTRAINT q_a_pairs_origin_kind_check
CHECK (origin_kind IN (
'extracted_from_corpus',
'curated_explicit',
'derived_from_bid_response', -- ← rename target
'imported_legacy'
));
ALTER TABLE public.q_a_pairs ALTER COLUMN origin_kind SET DEFAULT 'curated_explicit';

q_a_pairs and q_a_pair_history are both 0 rows today (T6 just created the shape), so a DROP+ADD CHECK swap is safe and no data backfill/UPDATE is required. This is the exact reason it MUST land before re-ingest — once the corpus writes any derived_from_* row, the rename becomes a data migration.

Rename the CHECK value derived_from_bid_response → derived_from_form_response on BOTH q_a_pairs and q_a_pair_history (the history mirror must carry the same constraint or the history trigger can write a value the mirror rejects). Migrate the matching Python string literal(s): grep shows the value referenced in scripts/tests/test_cocoindex_extraction.py (TestExpectedResponseKindParity, test_only_two_values) and the QA-flow extractor pathway — these expect the parity set, so they update in lockstep. No TS consumer references the literal (§1.1).

3.4 The bid_responses table-name question (DO NOT silently rename — OQ-64-3)

Section titled “3.4 The bid_responses table-name question (DO NOT silently rename — OQ-64-3)”

The brief says “verify bid_responses naming”. Findings:

  • A real bid_responses table EXISTS (created in pre_squash:3360) with bid_response_history, the RPC search_for_bid_response, and FKs bid_responses_question_id_fkey → bid_questions, …_approved_by/_drafted_by/ _last_edited_by → user_profiles. It is the procurement bid-answer record, distinct from the QA corpus.
  • The terminology is mid-migration: ID-31 PLAN §11 records “Pre-S234 bid_response_id NOT NULL framing fully retired (RATIFIED-RETIRE)”, and the citations design (ID-58 / T11, PLAN §P-22) still defines a citing_entity polymorphic enum whose first value is bid_response. So “bid” → “form” is a broader, unresolved terminology migration, not a single value swap.
  • Recommendation: scope {64.x} to the origin_kind value rename only (derived_from_bid_response → derived_from_form_response), which is internally consistent and 0-row-safe. Whether the bid_responses table (and the citing_entity enum value) should also rename to form_* is OQ-64-3 — a separate decision the Orchestrator routes to Liam, because it touches procurement bid surfaces (live UI), the search_for_bid_response RPC, and the T11 citations design that is not yet built. Do not couple it into the pre-re-ingest readiness migration.

4. KNOWN schema-migration item 3 — systematic missing-FK audit

Section titled “4. KNOWN schema-migration item 3 — systematic missing-FK audit”

Audit method: for each canonical-pipeline table, enumerate *_id / *_by columns from the generated Row shape and diff against the declared Relationships (FK) block in supabase/types/database.types.ts. A column with no declared FK is a candidate; candidates are then triaged into REAL-MISSING (should be enforced) vs INTENTIONAL (correlation id / polymorphic / cross-store).

TableFK-enforced (sample)Unenforced *_id/*_by candidatesTriage
content_itemscreated_by,updated_by→user_profiles; source_bid→workspaces; superseded_by→content_itemssource_document_id, parent_id, content_owner_id, archived_by, verified_by, governance_reviewer_id, op_idREAL-MISSING: source_document_id→source_documents (canonical pipeline link!), parent_id→content_items (self-ref). user_profiles refs (content_owner_id,archived_by,verified_by,governance_reviewer_id) → enforce. op_id = INTENTIONAL (cocoindex correlation text, no FK target — §4.2).
source_documentsarchived_by,uploaded_by→user_profiles; parent_id→source_documents; pipeline_run_id→pipeline_runs; workspace_id→workspacesop_id, pullmd_share_idop_id INTENTIONAL; pullmd_share_id INTENTIONAL (external pullmd share token, not a local table) — confirm in §4.2.
q_a_pairssource_workspace_id→workspaces; superseded_by→q_a_pairs(none beyond enforced)OK.
q_a_pair_historyq_a_pair_id→q_a_pairschanged_byREAL-MISSING: changed_by→user_profiles.
q_a_extractionspromoted_to_pair_id→q_a_pairs; source_content_item_id→content_itemsop_idop_id INTENTIONAL.
entity_mentionscontent_item_id→content_itemsop_idop_id INTENTIONAL.
content_chunkscontent_item_id→content_items; parent_chunk_id→content_chunks(none)OK. (op_id add is owned by {56.6}, not here.)
feed_articlescontent_item_id→content_items; feed_source_id→feed_sources; prompt_version_id→feed_prompts; workspace_id→workspacesexternal_idexternal_id INTENTIONAL (RSS GUID string, not a local table).
feed_sources/feed_promptscreated_by→user_profiles; workspace_id→workspaces(none)OK.
bid_responsesquestion_id→bid_questions; approved_by/drafted_by/last_edited_by→user_profiles(none)OK.
bid_questionsassigned_to/created_by→user_profiles; template_requirement_id→form_template_requirements; workspace_id→workspaces(none)OK.
form_templatescreated_by→user_profiles; form_type→form_types; workspace_id→workspaces(none)OK (FK-wise; RLS is the §2 problem).
form_template_fieldsquestion_id→bid_questions; template_id→form_templates(none)OK.
form_template_requirementstemplate_type→form_types(none)OK.
entity_aliases{} (none)(whole table)RETAIN (OQ-64-4 resolved S296). Zero declared FKs is CORRECT: no canonical entities table exists, so there is no FK target. entity_aliases is admin-curated read-only input to Stage-5 entity resolution (ID-53). Do NOT add an FK, do NOT truncate. The real entity-spine FK is T12/ID-60 future work.
citations, question_matches, entitiesNOT YET IN SCHEMAFuture tables (T10/T11/T12 = ID-57/58/60). Gate question: must they land BEFORE re-ingest? OQ-64-5.

4.2 op_id is deliberately NOT a FK (do not “fix” it)

Section titled “4.2 op_id is deliberately NOT a FK (do not “fix” it)”

op_id is the cocoindex per-flow operation id (text), stamped on content_items, q_a_extractions, source_documents, entity_mentions, pipeline_runs by T8 (supabase/migrations/20260521203414_t8_op_id_propagation.sql; COMMENT ON COLUMN … 'Cocoindex per-flow op_id; T8'). There is no operations table — the only op/run-shaped table is pipeline_runs, and op_id is a correlation stamp, not a parent key. The pipeline test test_missing_op_id_fails enforces presence, not referential integrity. The FK audit must explicitly exclude op_id from the add-missing-FK set, or it will introduce a constraint the pipeline cannot satisfy.

4.3 The canonical-pipeline FK that genuinely matters

Section titled “4.3 The canonical-pipeline FK that genuinely matters”

content_items.source_document_id → source_documents(id) is unenforced today yet it is the structural link the new canonical pipeline relies on (cocoindex writes content_items AND source_documents in the same flow — §1.3, flow.py 1472–1507). Re-ingesting with this FK unenforced risks orphan content_items if a source_documents write fails mid-flow. This is the highest-value FK to add before re-ingest. Add with ON DELETE SET NULL (matches the q_a_extractions.source_content_item_id … ON DELETE SET NULL precedent in the T6 migration) so a source-document purge does not cascade-delete content. Open nullability question folded into the subtask brief: source_document_id is nullable (manual/UI items have no source document), so the FK must permit NULL.


Section titled “5. KNOWN input 5 — gate-links NOT owned by ID-64 (checklist references only)”

These are Task-level dependencies of ID-64 (deps: [56, 62, 63]) and the origin-of-truth for each gate lives in that Task. ID-64 only references their GREEN status in the checklist (§7); ID-64 does NOT author or own their work.

  • ID-56 {56.8} chunking (Variant B = RecursiveSplitter, ratified S278/{56.5}). The cocoindex chunking stage must be landed + integration-green ({56.9}) before re-ingest, or the corpus is chunked by the about-to-be-retired app-side chunker. {56.6} (content_chunks.op_id) and {56.7} (search RPC nullable headings) are sibling prerequisites inside Task 56 — not ID-64 subtasks.
  • ID-62 fixture-staging live tierHARD gate. GREEN before re-ingest. Until the /stage server + in-VPC reachability harness exists ({62.1} RESEARCH in progress), the 39-test cocoindex integration tier cannot run live — it skips clean. Re-ingesting without this is re-ingesting unverified. (The §2 Inv-25 RLS acceptance test is NOT part of this gate — role-based USING(true) is the intended single-tenant model and the corpus is canonical/shared per ID-69 §2.4; per-workspace scoping is not part of the ratified architecture, not a pre-re-ingest gate; S292.)
  • ID-63 ontology / CV integration — the controlled vocabularies (form_types, taxonomy, requirement_type) feed Path-A classification during ingest. A parity-drifted CV load means systematically wrong classification across the whole corpus. Must be verified end-to-end ({63.1} RESEARCH pending) before re-ingest.

6. Liam’s data-retention list — per-table disposition

Section titled “6. Liam’s data-retention list — per-table disposition”

Source: docs/research/db-retain-investigation.md (read in full). For each table: RETAIN (carry data through unchanged), RE-SEED (truncate + reload from a seed/fixture), MIGRATE (transform existing data into the new shape), RE-INGEST (content re-run through the new pipeline; classification/embedding discarded), or DROP (no longer relevant under the new architecture). Dispositions below are proposed; the ones marked (OQ) need Liam ratification.

TableRecordsProposed disposition
company_profiles1RETAIN — single real record; client/org profile, not pipeline output. Verify it survives any workspace-ontology change.
feed_flags2RE-SEED — feed moderation flags; tied to feed_articles which re-ingest. Re-seed from current 2 if still relevant, else DROP.
form_types(CV)RETAIN / RE-SEED via ID-63 — controlled vocabulary. Disposition OWNED by ID-63 CV-load (the source of truth), not ID-64.
guidesRETAIN — authored guidance content, not pipeline output. Confirm not workspace-ontology-coupled (OQ if it is).
intelligence_workspacesMIGRATE/RETAIN (OQ) — depends on the workspace-ontology decision (§ workspaces below). Likely retained as workspace rows.
procurement_vehicle_instancesRETAIN — reference data (frameworks/lots). Not re-ingested.
procurement_vehiclesRETAIN — reference data. Not re-ingested.
TableProposed disposition
user_rolesRETAIN (mandatory) — drives get_user_role() which ALL RLS depends on. Must survive re-ingest untouched.
user_profilesRETAIN (mandatory) — FK target for nearly every *_by/owner column across the schema (§4). Untouched.

6.3 Content items (the core re-ingest question)

Section titled “6.3 Content items (the core re-ingest question)”
TableProposed disposition
content_itemsRE-INGEST (content only). Retain ONLY rows that are (a) Phew-employee added/edited OR (b) successfully ingested articles/URLs — keyed by ingest_source ∈ {'rss feed','manual','url_import'} (column is nullable text; verify exact stored values, OQ-64-6). For retained rows, the source content is re-run through the new cocoindex pipeline — old classification_*, embedding, quality_score, classification_cache_* are discarded and regenerated. Implementation shape = export-retained-content → drop derived columns → re-ingest, NOT an in-place UPDATE.
content_historyRE-SEED/DROP (OQ). Audit trail of edits. If re-ingest rewrites content_items ids, history rows orphan. Either migrate history forward keyed to the new ids, or accept a clean-slate history at handover. Liam decision.

6.4 “Needs investigation — new-architecture impact”

Section titled “6.4 “Needs investigation — new-architecture impact””
TableProposed disposition
feed_articlesRETAIN structure; RE-INGEST linked content. feed_articles ALREADY EXISTS as the RSS-ingestion record (workspace-scoped; FKs to content_items/feed_sources/feed_prompts). Its content_item_id is a NULLABLE link to the promoted content_item. RSS/URL items therefore already have a home distinct from content_items — this is the core of OQ-64-2. Retain feed_articles rows whose linked content is retained; re-link content_item_id after re-ingest.
feed_promptsRETAIN — RSS relevance-prompt versions; referenced by feed_articles.prompt_version_id. Keep for lineage.
feed_sourcesRETAIN — RSS source configs (URLs, schedules). Re-ingest re-reads from these; do not lose.
si_processing_queueRE-SEED/DROP (OQ) — sector-intelligence processing queue (transient work items). Almost certainly truncate-and-rebuild on re-ingest; confirm nothing durable is parked there.
taxonomy_domainsRETAIN / OWNED by ID-63 — taxonomy CV. Disposition is ID-63’s call (CV-load source of truth). ID-64 references, does not own.
taxonomy_subtopicsRETAIN / OWNED by ID-63 — as above.
entity_aliasesRETAIN (OQ-64-4 resolved S296) — admin-curated read-only Stage-5 input; no entities table → no FK target; do NOT truncate, do NOT add FK. (The entity_pair_resolutions determinism cache is also RETAINED — truncating breaks idempotency.) Cross-ref docs/themes/canonical-pipeline/reference/pipeline-writepath-schema-gap-s295.md §S296.
workspacesMIGRATE/RETAIN (OQ-64-7) — the workspace-ontology change (cross-ref ID-63, “based on changes to workspace ontology”) may invalidate some current rows. Retain workspaces that map to the new ontology. (Decoupled from RLS: per S292 the single-tenant USING(true) model needs no membership junction — §2.3 — so this is purely a retention question.) The single biggest cross-cutting retention question.

6.5 Tables NOT in Liam’s list but written during re-ingest (must be clean-slate-able)

Section titled “6.5 Tables NOT in Liam’s list but written during re-ingest (must be clean-slate-able)”

These are cocoindex/QA write targets (§4) with ~0 rows today; they are truncate-and-let- the-pipeline-repopulate on re-ingest, but the readiness subtask must confirm each is empty (or safely truncatable) before cutover: source_documents, q_a_pairs, q_a_pair_history, q_a_extractions, entity_mentions, content_chunks, pipeline_runs.


The single readiness surface. Re-ingest to the chosen target (staging OR new preview branch — OQ-64-8) proceeds only when every HARD gate is GREEN. Verify on the target, then cut over to prod. DDL via Supabase CLI only.

A. Schema / FK migrations (owned by ID-64 {64.x}):

  • A1 (NOT A GATE — resolved S292) Inv-25 RLS workspace-scoping is NOT a pre-re-ingest gate. Role-based USING(true) on form_templates is the intended single-tenant model (S292, OQ-64-1 CLOSED — §2.3); no workspace-scoping migration is required pre-re-ingest. The bl-207 form_templates SELECT-parity fix is optional and not undertaken here — do NOT undertake an RLS migration. form_template_requirements (the catalogue) stays globally readable regardless. Per-workspace SELECT isolation is not part of the ratified architecture (the corpus is canonical/shared — no content_items.workspace_id, M2M reuse via content_item_workspaces per ID-69; form_templates is single-tenant) — not a roadmap requirement, not this checklist (§2.4).
  • A2 (HARD) origin_kind value rename derived_from_bid_response → derived_from_form_response on q_a_pairs + q_a_pair_history CHECK constraints, plus matching Python literal(s). Must precede any derived_from_* row write.
  • A3 (HARD) Add content_items.source_document_id → source_documents(id) ON DELETE SET NULL (nullable). The canonical-pipeline link (§4.3).
  • A4 (MEDIUM) Add remaining REAL-MISSING FKs: content_items.parent_id, content_items.{content_owner_id,archived_by,verified_by,governance_reviewer_id}, q_a_pair_history.changed_by. Explicitly exclude op_id (§4.2).
  • A5 (GREEN — resolved S296) entity_aliases = RETAIN: no entities table → no FK target; admin-curated read-only Stage-5 input; do NOT truncate/FK; the entity_pair_resolutions determinism cache is also retained. See §S296.
  • A6 (gate question) citations/question_matches/entities (T10/T11/T12) — decide land-before-re-ingest vs after (OQ-64-5).
  • A7 Regenerate supabase/types/database.types.ts after all DDL (supabase gen types …); supabase-types-parity CI green.

B. Correctness / capability gates (owned by Tasks 56/62/63 — references only):

  • B1 (HARD) ID-62 fixture-staging live tier GREEN (/stage server + in-VPC reachability). Unblocks the 39-test integration tier. (Not coupled to A1 — A1 is resolved S292 and is no longer a gate.)
  • B2 (HARD) ID-56 {56.8}/{56.9} cocoindex chunking (Variant B) landed + integration-green; app-side chunker retired ({56.11}).
  • B3 (HARD) ID-63 ontology/CV integration verified end-to-end (Path-A classify + form-cataloguing + T10 matching).

C. Data retention (owned by ID-64 readiness subtask + Liam ratification):

  • C1 Retention dispositions (§6) ratified by Liam (all (OQ) rows resolved).
  • C2 Retained-content export verified (content_items keep-set by ingest_source; feed_articles re-link plan; content_history decision).
  • C3 Mandatory-RETAIN tables (user_roles, user_profiles, reference data) confirmed untouched by the re-ingest procedure.
  • C4 Clean-slate write-target tables (§6.5) confirmed empty/truncatable.

D. Cutover:

  • D1 Re-ingest executed on the chosen non-prod target (OQ-64-8); corpus spot-verified (classification parity, embedding presence, FK integrity, role-based RLS — USING(true) single-tenant; no cross-workspace denial expected, the corpus is canonical/shared per §2.4).
  • D2 cat supabase/.temp/project-ref verified before any prod db push; relink to prod only at the deliberate cutover step (CLAUDE.md project-ref drift gotcha).
  • D3 Prod cutover; post-cutover verification = the client-handover DB state.

8. {64.2+} SUBTASK PROPOSALS (TM-shape — for the Orchestrator to write)

Section titled “8. {64.2+} SUBTASK PROPOSALS (TM-shape — for the Orchestrator to write)”

Sibling-only deps (integers, local to Task 64). No cross-Task subtask deps — the chunking/fixture-staging/ontology gates are Task-level (ID-64.dependencies=[56,62,63]), correctly NOT expressed here. Decomposition = 7 subtasks (well within the 25 soft ceiling). Each details is load-bearing (an Executor needs no other doc); each carries the code-intelligence tool-discipline instruction (Inv 2 → Inv 3) and the Supabase-CLI-only + project-ref + SET search_path gotchas.

[
{
"id": 2,
"title": "Migration — origin_kind value rename derived_from_bid_response -> derived_from_form_response",
"description": "Rename the origin_kind CHECK value on q_a_pairs + q_a_pair_history and the matching Python literal(s); 0-row-safe DROP+ADD; must land before any derived_from_* row is written.",
"details": "RESEARCH §3. origin_kind is a TEXT CHECK constraint (NOT a pg ENUM), present ONLY on public.q_a_pairs and public.q_a_pair_history. Current authoritative def: supabase/migrations/20260520225456_t6_q_a_pairs_full_schema.sql lines 60-76 (CHECK IN ('extracted_from_corpus','curated_explicit','derived_from_bid_response','imported_legacy'), DEFAULT 'curated_explicit'). NEW migration via `supabase migration new` (DDL via CLI ONLY, never MCP execute_sql): DROP CONSTRAINT q_a_pairs_origin_kind_check; ADD with value renamed to 'derived_from_form_response'; repeat the same DROP+ADD on q_a_pair_history (its CHECK must match or the history trigger can write a rejected value). Both tables are 0 rows so no UPDATE/backfill. Then migrate the Python string literal: grep 'derived_from_bid_response' in scripts/ (test_cocoindex_extraction.py TestExpectedResponseKindParity/test_only_two_values and the QA-flow parity set) and update in lockstep; run `grep -rn derived_from_bid_response supabase scripts lib types` and confirm ZERO residual after. No TS consumer references the literal (gitnexus_query confirmed). DO NOT rename the bid_responses TABLE or the citing_entity 'bid_response' enum value — that is OQ-64-3, out of scope. Pre-push: cat supabase/.temp/project-ref; worker first action `supabase link --project-ref turayklvaunphgbgscat` (staging). Run gitnexus_detect_changes before commit.",
"status": "pending",
"dependencies": [],
"testStrategy": "python3 -m pytest scripts/tests/ for the QA parity tests passes with the renamed value; grep -rn 'derived_from_bid_response' over supabase/scripts/lib/types returns ZERO; q_a_pair_history CHECK accepts 'derived_from_form_response' and rejects 'derived_from_bid_response'."
},
{
"id": 3,
"title": "Migration — add content_items.source_document_id FK to source_documents (nullable, ON DELETE SET NULL)",
"description": "Enforce the canonical-pipeline link content_items.source_document_id -> source_documents(id); nullable (manual/UI items have no source document); ON DELETE SET NULL so a source purge does not cascade-delete content.",
"details": "RESEARCH §4.3. content_items.source_document_id is currently UNENFORCED (confirmed: content_items Relationships block in supabase/types/database.types.ts declares FKs only for created_by/updated_by->user_profiles, source_bid->workspaces, superseded_by->content_items). The cocoindex flow (scripts/cocoindex_pipeline/flow.py mount_table_target lines 1472-1507) writes content_items AND source_documents in the same flow, so an unenforced link risks orphan content_items on a mid-flow source_documents failure. NEW migration via CLI: validate no orphan rows first (SELECT count(*) FROM content_items ci WHERE ci.source_document_id IS NOT NULL AND NOT EXISTS (SELECT 1 FROM source_documents sd WHERE sd.id=ci.source_document_id)); if >0, surface to Orchestrator before adding the constraint. ADD CONSTRAINT content_items_source_document_id_fkey FOREIGN KEY (source_document_id) REFERENCES public.source_documents(id) ON DELETE SET NULL (NOT NULL-able: column is nullable). Precedent: q_a_extractions.source_content_item_id ... ON DELETE SET NULL (T6 migration line 107). Run gitnexus_impact on any TS reader of source_document_id before any code touch (none expected — this is DDL-only). Pre-push project-ref check + staging link. Regen types after (covered by {64.8}). gitnexus_detect_changes before commit.",
"status": "pending",
"dependencies": [],
"testStrategy": "Inserting a content_items row with a non-existent source_document_id is rejected by the FK; deleting a source_documents row sets dependent content_items.source_document_id to NULL (no cascade delete); orphan-precheck query returns 0 before the constraint is added."
},
{
"id": 4,
"title": "Migration — add remaining REAL-MISSING FKs across canonical-pipeline tables (exclude op_id)",
"description": "Enforce the user_profiles + self-ref FKs surfaced by the §4 audit; explicitly exclude op_id (cocoindex correlation text, no FK target).",
"details": "RESEARCH §4.1/§4.2. Add FKs (all via one CLI migration, ON DELETE SET NULL for user refs, self-ref for parent_id): content_items.parent_id->content_items(id); content_items.content_owner_id/archived_by/verified_by/governance_reviewer_id->user_profiles(id); q_a_pair_history.changed_by->user_profiles(id). For EACH: run the orphan-precheck (NOT EXISTS) query first; if any orphan rows, surface to Orchestrator (do not silently add). EXCLUDE op_id on ALL tables (content_items/source_documents/q_a_extractions/entity_mentions/pipeline_runs) — it is the cocoindex per-flow op_id text stamp (COMMENT in supabase/migrations/20260521203414_t8_op_id_propagation.sql; there is NO operations table; pipeline test test_missing_op_id_fails enforces presence not referential integrity). EXCLUDE source_documents.pullmd_share_id (external pullmd token) and feed_articles.external_id (RSS GUID) — not local-table refs. DO NOT touch content_chunks.op_id (owned by {56.6}). Pre-push project-ref + staging link; SET search_path not needed (DDL only, no new functions). gitnexus_detect_changes before commit.",
"status": "pending",
"dependencies": [3],
"testStrategy": "Each added FK rejects an orphan insert and SET-NULLs on parent delete; a verification query confirms op_id columns have NO FK constraint on any table; orphan-precheck returns 0 for every column before its constraint is added."
},
{
"id": 5,
"title": "Migration — Inv-25 workspace-scoped SELECT policy on form_templates (+ transitive form_template_fields)",
"description": "[SUPERSEDED S292 — see details/§2.3/§9: OQ-64-1 CLOSED, USING(true) single-tenant is intended, no workspace-scoping migration; {64.5} re-scoped to RESEARCH-framing correction.] (HISTORICAL S279) Replace form_templates templates_select USING(true) with a workspace-scoped predicate; scope form_template_fields transitively via its parent; leave form_template_requirements global. GATED on the membership-model decision (OQ-64-1).",
"details": "[SUPERSEDED S292: {64.5} re-scoped to RESEARCH-framing correction; OQ-64-1 CLOSED, role-based single-tenant USING(true) is intended, NO workspace-scoping migration — see §2.3 / §9. The proposal text below is retained as the S279 point-in-time decomposition; the workspace-scoping migration it describes is NOT undertaken.] RESEARCH §2. (HISTORICAL S279 framing) BLOCKED until OQ-64-1 (workspace-membership model) is ratified — there is currently NO membership junction and NO is_workspace_member() predicate; EVERY workspace table uses USING(true). Once the predicate exists (Variant A: workspace_members table + SECURITY DEFINER is_workspace_member(uuid) WITH `SET search_path = public, extensions`; or Variant B: narrower creator/JWT-claim scope): NEW CLI migration — DROP POLICY templates_select ON public.form_templates; CREATE POLICY form_templates_select ON public.form_templates FOR SELECT TO authenticated USING (<membership predicate on workspace_id>). For form_template_fields: DROP template_fields_select; CREATE form_template_fields_select FOR SELECT TO authenticated USING (EXISTS (SELECT 1 FROM public.form_templates t WHERE t.id = form_template_fields.template_id AND <same membership predicate on t.workspace_id>)). DO NOT touch template_requirements_select (catalogue is global by Inv-25 — the acceptance test asserts it stays readable). Leave insert/update/delete policies (role-gated) unchanged. Existing live policies: supabase/migrations/20260416102457_pre_squash_reconciliation.sql lines 6308/6346/6327. Acceptance = __tests__/integration/form-extraction-rls.integration.test.ts (skips clean until ID-62 fixture-staging GREEN). If the migration cannot be authored because OQ-64-1 is unresolved, STOP and escalate (do not invent a membership model). Pre-push project-ref + staging link. gitnexus_detect_changes before commit.",
"status": "pending",
"dependencies": [],
"testStrategy": "[SUPERSEDED S292 — describes a per-workspace multi-tenant model the platform has NOT adopted (§2.3/§2.4, ID-69); not a pre-re-ingest gate.] __tests__/integration/form-extraction-rls.integration.test.ts: a workspace-A viewer SELECT of a workspace-B form_templates row returns 0 rows, AND form_template_requirements remains readable regardless of workspace (both assertions pass once the policy lands and ID-62 fixture-staging is wired)."
},
{
"id": 6,
"title": "entity_aliases FK/retention disposition — investigate + apply (OQ-64-4)",
"description": "[SUPERSEDED S296: {64.6} = doc-only RETAIN disposition — no entities table → no FK, no truncate; entity-spine FK is T12/ID-60 future. See §S296] Resolve the zero-FK entity_aliases table: either add the proper FK(s) to the canonical entity id, or ratify truncate-and-rebuild during re-ingest entity-resolution.",
"details": "[SUPERSEDED S296: {64.6} = doc-only RETAIN disposition — no entities table → no FK, no truncate; entity-spine FK is T12/ID-60 future. See §S296] RESEARCH §4.1/§6.4. entity_aliases has ZERO declared FKs (supabase/types/database.types.ts) and is flagged by Liam as possibly holding now-irrelevant data under the new workspace ontology. Its target shape depends on ID-60 (Mempalace-KG entity_mentions). Step 1: read the entity_aliases CREATE in supabase/migrations/* (grep 'entity_aliases') and the ID-60/ID-53 entity-resolution design to determine the intended parent (likely a canonical entities/entity id). Step 2 (decision recorded as OQ-64-4 resolution): if rows are durable + a parent exists, add the FK via CLI migration with orphan-precheck; if rows are transient resolution artefacts, ratify truncate-and-let-Stage-5-rebuild (no FK migration, just a clean-slate note in the readiness surface). Use documentation-and-adrs to record whichever decision. Pre-push project-ref + staging link if a migration results. gitnexus_detect_changes before commit.",
"status": "pending",
"dependencies": [],
"testStrategy": "entity_aliases disposition is recorded (FK added with passing orphan-precheck, OR a ratified truncate-and-rebuild note in the readiness surface) and the readiness checklist item A5 is resolvable to GREEN."
},
{
"id": 7,
"title": "Data-retention export + re-link plan (content_items keep-set, feed_articles re-link, content_history decision)",
"description": "Produce the executable retention procedure: which content_items rows to keep (by ingest_source), how feed_articles re-link to re-ingested content, and the ratified content_history disposition.",
"details": "RESEARCH §6. content_items: build the keep-set query — retain rows where ingest_source IN (the real stored values for 'rss feed'/'manual'/'url_import'; VERIFY exact strings first via SELECT DISTINCT ingest_source FROM content_items — OQ-64-6, the column is nullable text). For retained rows, export the SOURCE content only (content/title/source_url/source_document linkage) — DISCARD classification_*, embedding, embedding_model, quality_score, classification_cache_* (regenerated by the new pipeline). This is export-then-re-ingest, NOT in-place UPDATE. feed_articles: ALREADY EXISTS as the RSS record (workspace-scoped; content_item_id nullable link); plan to re-link content_item_id after re-ingest assigns new content_items ids (retain feed_articles whose linked content is retained). content_history: record Liam's ratified decision (migrate-forward-keyed-to-new-ids vs clean-slate-at-handover — OQ resolution). Confirm mandatory-RETAIN (user_roles, user_profiles, company_profiles, guides, procurement_vehicle*, feed_sources, feed_prompts) untouched. Confirm clean-slate write-targets (source_documents, q_a_pairs, q_a_pair_history, q_a_extractions, entity_mentions, content_chunks, pipeline_runs) empty/truncatable. Output = a runbook-style procedure doc (use documentation-and-adrs) referenced by {64.8}. No DDL here unless a staging/transform table is needed (CLI if so). DO NOT edit task-list.json.",
"status": "pending",
"dependencies": [],
"testStrategy": "The content_items keep-set query returns the expected Phew-authored + successfully-ingested rows (validated against SELECT DISTINCT ingest_source); the export/re-link procedure is reviewable and names every table in §6 with a disposition; no mandatory-RETAIN table appears in any truncate/drop step."
},
{
"id": 8,
"title": "Re-ingest readiness surface — aggregate gate checklist + types regen + cutover runbook",
"description": "Single readiness surface that aggregates §7 gates, gates the non-prod re-ingest, regenerates types after all DDL, and documents the staging/preview -> verify -> prod cutover with the project-ref safeguard.",
"details": "RESEARCH §7. Aggregates ALL gates into one checked surface (a doc under docs/operations/ or docs/runbooks/, NOT the ledger): A-gates (this Task's migrations {64.2}-{64.6}), B-gates (Task-level deps 56/62/63 GREEN — reference only, do not own), C-gates (retention {64.7} ratified), D-cutover. After every DDL subtask is merged, regenerate supabase/types/database.types.ts via the CLAUDE.md command (`/opt/homebrew/bin/supabase gen types typescript --project-id <ref> --schema public > supabase/types/database.types.ts`) and confirm supabase-types-parity CI green (A7). Document the cutover: re-ingest to the chosen non-prod target (staging OR new preview branch — OQ-64-8), spot-verify (classification parity, embedding presence, FK integrity via the new constraints, role-based RLS — USING(true) single-tenant; no cross-workspace denial, the corpus is canonical/shared per ID-69 §2.4), THEN `cat supabase/.temp/project-ref` and relink to prod ONLY at the deliberate cutover step (D2 — CLAUDE.md project-ref drift gotcha: a stale prod link makes db push land on prod silently). Post-cutover verification = the client-handover DB state. Use documentation-and-adrs for the runbook. This subtask is the LAST to complete (it gates cutover).",
"status": "pending",
"dependencies": [2, 3, 4, 5, 6, 7],
"testStrategy": "The readiness surface enumerates every §7 A/B/C/D gate with an owner and GREEN/RED state; supabase-types-parity CI is green after the post-DDL types regen; the cutover runbook contains the explicit `cat supabase/.temp/project-ref` + relink step before any prod db push."
}
]

Decomposition notes for the Orchestrator:

  • {64.5}[SUPERSEDED S292] the original Inv-25 RLS workspace-scoping proposal is withdrawn: OQ-64-1 is CLOSED, role-based single-tenant USING(true) is the intended model, and there is no workspace-scoping migration. {64.5} is re-scoped to this RESEARCH-framing correction (see §2.3 / §9). It is NOT blocked on any membership model and is NOT coupled to ID-62. (The S279 note that it was “BLOCKED on OQ-64-1” no longer applies.)
  • {64.4} depends on {64.3} only to serialise the two content_items FK migrations (avoid two concurrent ALTERs on the same table); they could merge if the Orchestrator prefers a single content_items-FK migration.
  • {64.8} depends on all of {64.2}{64.7} (it is the aggregation/cutover surface).
  • If OQ-64-5 resolves to “land citations/question_matches/entities before re-ingest”, that is NOT a {64.x} subtask — it belongs to Tasks 57/58/60 and would be added as a new Task-level dependency on ID-64. Surfaced, not folded in (sibling-only constraint).
  • Subtask count = 7 ({64.2}{64.8}), comfortably under the 25 soft ceiling.

9. Open Questions (OQ-64-N) — for the Orchestrator to route to Liam

Section titled “9. Open Questions (OQ-64-N) — for the Orchestrator to route to Liam”
  1. OQ-64-1 — Workspace-membership model — CLOSED (S292). Resolution: role-based single-tenant USING(true) is the INTENDED RLS model for the v1 Phew pilot; workspace membership is NOT required, and no workspace_members junction / is_workspace_member() predicate is needed pre-re-ingest. There is no workspace-scoping migration under {64.5}; the bl-207 SELECT-parity fix is optional and not undertaken. Per-workspace isolation is not part of the ratified architecture — the corpus (content_items) is canonical and shared with no workspace_id (ID-69 §4; reuse via the content_item_workspaces M2M), and form_templates is single-tenant — so it is NOT a planned requirement and NOT a pre-re-ingest gate (§2.4). Evidence: procurement-workspaces/PRODUCT.md:382 (v1 single-tenant Phew pilot), reserved-workspace-seats/TECH.md:160 (role-based delegation), the bl-207 fix caveat (no membership model = a larger separable task), ID-69/RESEARCH.md:155 (correctly single-tenant), and the consolidated disposition docs/themes/canonical-pipeline/reference/pipeline-writepath-schema-gap-s295.md §S296 (“OQ-64-1 — CLOSED S292”). (OQ-64-7 workspace-ontology retention is now decoupled from this — it is a pure retention question.)
  2. OQ-64-2 — RSS/URL items: content_items vs feed_articles — CLOSED (verified 11/06/2026; answered by the ID-75 O4 ratification, S316). Resolution: neither of the two framed options — external RSS/URL material lands the REFERENCE LAYER. ID-75 PRODUCT DP-1: content_items = client-adopted knowledge. External material lands a reference layer”; BI-1: a gate-passing feed article lands exactly one source_documents row + one reference_items row — no content_items row, not by the pipeline, not by any fallback. Adoption into content_items is explicit and workflow-driven (ID-58 citations / ID-71), never automatic. feed_articles remains the per-workspace RSS ledger (TS poller inserts; cocoindex enumerates + backlinks reference_item_id — flow.py WP-C step 7, live since the ID-75 M1/M2 migrations); legacy promoted rows are NOT retro-converted (BI-10) — the retained-content keep-set + feed_articles re-link stays in {64.7}. The legacy storeAsContentItem promotion path is an ID-75 BI-11 / ID-42 / ID-46 retirement target.
  3. OQ-64-3 — bid_responses table rename — CLOSED (08/06/2026, ID-93 §6.1). Resolution: rename pre-run. The deferral default was re-decided when ID-93 found the ENTIRE bid_* surface at 0 rows on prod. EXECUTED: {64.14} (tables + search_for_bid_response RPC + consumers) + {64.16} (get_bid_*get_form_* RPCs) — staging S331, prod S336 (7-mig cutover). content_citations disposition was owned by ID-58 (polymorphic citations REPLACE + DROP — also done, prod S336). (Historical text below §3.4 retained as the S279 point-in-time framing.)
  4. OQ-64-4 — entity_aliases disposition — CLOSED (S296). Resolution: RETAIN. No canonical entities table exists → there is no FK target; entity_aliases is an admin-curated, read-only input to Stage-5 entity resolution (ID-53). Do NOT truncate, do NOT add an FK. The entity_pair_resolutions determinism cache is also RETAINED (truncating it breaks idempotency). The real entity-spine FK (to a canonical entities row) is T12/ID-60 future work. {64.6} is therefore doc-only. Evidence: docs/themes/canonical-pipeline/reference/pipeline-writepath-schema-gap-s295.md §S296 (“OQ-64-4 — RETAIN”). (Drives {64.6}.)
  5. OQ-64-5 — T10/T11/T12 tables land order — CLOSED (08/06/2026, ID-93 §6.2; reframed {64.8} journal). Resolution: structurally complete, based on ratified specs — the handover DB carries the T10/T11/T12 schema slices pre-cutover ONLY per ratified ID-57/58/60 specs ({64.8} gate G6); full feature build stays post-cutover. Reframe (08/06): more-than-minimal means STARTING the ID-58/ID-59 spec chains, not speculative empty DDL; ID-60 = NO pre-re-ingest action; ID-57 stays C5-decoupled but its schema slice must spec-ratify pre-cutover (the outstanding G6 tail). Execution since: ID-58 citations REPLACE + ID-59 edit_intent storage BUILT — staging S331, prod S336.
  6. OQ-64-6 — exact content_items.ingest_source stored values. Liam’s keep-set keys on ingest_source ∈ {'rss feed','manual','url_import'} but the column is nullable text. Confirm the exact stored strings (a SELECT DISTINCT ingest_source on the live data, {64.7}) so the keep-set query is correct and no retained row is missed. (Progress 11/06/2026: the {64.17}/bl-106 escalation observed 8 distinct valuesmanual, upload, url_import, upload_autosplit, bid_outcome_integration, … — the canonical list ratification (bl-106) and the {64.7} keep-set verification remain open.)
  7. OQ-64-7 — workspace ontology + workspaces/intelligence_workspaces retention — CLOSED (11/06/2026, Liam). Resolution: the CONCEPT/PURPOSE carries, not the rows. The three live intelligence workspaces (MAT Auditing · NHS Digital Cyber Alerts · Education Sector Monitor) carry forward to the new DB as re-created workspaces with the same purpose/configuration (company-profile binding, guide, relevance threshold — cf. specs/intelligence-workspaces/ S-1/S-2 typed-column promotion); the GNEWS-DEDUP workspace is DELETED (not carried). {64.7} implements: export the three workspace configs, re-create on the fresh target, drop the fourth.
  8. OQ-64-8 — re-ingest target — CLOSED (S296, Liam). Resolution: FRESH Supabase preview branch for the full re-ingest (isolated dry-run, cleaner rollback, no staging-data entanglement); Task 82 proved the git-integrated preview-branch migration replay. Drives the {64.8} cutover runbook + project-ref relink steps (v1 sequence Spine S2).
  9. OQ-64-9 — content_history disposition — CLOSED (11/06/2026, Liam). Resolution: CLEAN SLATE — edit history does not migrate forward at client handover. Pre-condition: the {64.7} export first identifies any client actions taken (rows keyed by client user UUIDs — expected minimal) so nothing client-authored is silently dropped; anything found is dispositioned explicitly before the truncate.

10. Verification (OQ-3 pre-ratification empirical check)

Section titled “10. Verification (OQ-3 pre-ratification empirical check)”
  • External-library API citations: NONE introduced. This RESEARCH cites only KH-internal artefacts (SQL migrations, generated database.types.ts, flow.py). mount_table_target is referenced descriptively as an already-in-use cocoindex symbol read from scripts/cocoindex_pipeline/flow.py (lines 1472–1507) — not a NEW citation requiring an import-and-call check. The OQ-3 import-and-call check therefore does not apply to this artefact. (Where {64.x} Executors touch cocoindex 1.0.3 write APIs, the existing pinned usage in flow.py is the empirical reference; cocoindex’s ExtractByLlm/version drift precedent — docs/research/cocoindex-1.0.3-extractbyllm-spec-reality-investigation.md — is the standing reminder to re-verify against the pin if any new symbol is introduced.)
  • Empirical schema facts in §2–§4 were confirmed against the live-generated supabase/types/database.types.ts and the migration corpus in this worktree (not assumed from prose). The templates_select … USING(true) finding is additionally corroborated by the S278 live-JWT probe recorded in the acceptance test header.
  • Date: 29/05/2026. Worktree: id-64 off main.