ID-64 {64.1} RESEARCH — Pre-re-ingest readiness (schema/FK migration + correctness gate-set)
ID-64 {64.1} RESEARCH — Pre-re-ingest readiness (schema/FK migration + correctness gate-set)
Section titled “ID-64 {64.1} RESEARCH — Pre-re-ingest readiness (schema/FK migration + correctness gate-set)”Status: RESEARCH (findings + gate checklist + decomposition proposals + Open Questions). This artefact does NOT run migrations and does NOT ratify the final plan. DDL is authored and applied via the Supabase CLI only (
supabase migration new+supabase db push) — never via MCPexecute_sql/apply_migration(CLAUDE.md Supabase gotcha). All{64.2+}subtask proposals below are returned to the Orchestrator for writing intodocs/reference/task-list.json; this Planner does not edit the ledger.
- Task: ID-64 — Pre-re-ingest readiness umbrella (
status: spec_needed,priority: high,deps: [56, 62, 63],effort: L). - Worktree base:
main@fffe0aeb(env-reported11cd62e5is the worktree HEAD pre-fetch; the Orchestrator integrates against currentmain). - Authoring session: S279 (continuation), DD/MM/YYYY = 29/05/2026.
- Decomposition principle: ID-64 is an umbrella. Its correctness/capability gates
(chunking, fixture-staging, ontology/CV) live in Tasks 56 / 62 / 63 and are wired as
Task-level dependencies (
ID-64.dependencies = [56, 62, 63]) — NOT as{64.x}subtasks (sibling-only constraint, §3.3). The{64.2+}subtasks proposed here own ONLY the schema-migration + FK + data-retention work that has no other home, plus the single readiness-surface subtask that aggregates the gate checklist.
1. Code-intelligence orientation (run before authoring; cited verbatim)
Section titled “1. Code-intelligence orientation (run before authoring; cited verbatim)”Per .gitnexus/CLAUDE.md “Always Do”. SQL migrations and Python pipeline are OUTSIDE the
TS index (ts-morph corpus only — .ast-dataflow/CLAUDE.md), so the gitnexus queries are
supplemented with grep sweeps over supabase/migrations/*.sql and
scripts/cocoindex_pipeline/*.py. Repo disambiguation required:
repo: /Users/liamj/Documents/development/knowledge-hub.
1.1 gitnexus_query({query: 'origin_kind derived_from_bid_response'})
Section titled “1.1 gitnexus_query({query: 'origin_kind derived_from_bid_response'})”Returned no TS execution flow that reads/writes origin_kind as a symbol (it is a DB
CHECK-constraint string value, not a TS symbol). Top definitions surfaced were the QA-form
extraction tests (scripts/tests/test_cocoindex_extraction.py:TestExpectedResponseKindParity,
test_only_two_values) and lib/ast-dataflow/queries/flow-trace.ts:resolveOrigin (an
unrelated ast-dataflow internal). Conclusion: origin_kind lives entirely in SQL +
Python; the rename is a SQL CHECK-constraint + Python-string-literal change, with NO TS
consumer to refactor. Confirmed by grep (§3.2).
1.2 gitnexus_query({query: 'row level security workspace scoping form_templates'})
Section titled “1.2 gitnexus_query({query: 'row level security workspace scoping form_templates'})”Surfaced app/api/procurement/[id]/templates/route.ts:POST (proc_46) and
.../[templateId]/fill/route.ts:POST (proc_110) as the app-side template flows, plus
scripts/cocoindex_pipeline/flow.py:ingest_file / _trim_stale_form_fields and
lib/templates/template-coverage.ts:listAvailableTemplates /
fetchTemplateRequirements. Conclusion: form-template reads happen on BOTH the app
side (TanStack-fed routes) and the pipeline side (cocoindex mount_table_target). Any RLS
change is exercised by the live integration test, not by these TS flows — the durable
acceptance artefact is the integration test (§2.1), not a unit assertion.
1.3 gitnexus_query({query: 'foreign key content_items source_documents canonical pipeline'})
Section titled “1.3 gitnexus_query({query: 'foreign key content_items source_documents canonical pipeline'})”Surfaced scripts/cocoindex_pipeline/flow.py:ingest_file / app_main,
scripts/kb_pipeline/store.py:insert_content_item / insert_content_history_entry /
check_url_exists, and lib/intelligence/pipeline.ts:storeAsContentItem /
processFeedSource / ensureWorkspaceLink. Conclusion: there are TWO writer
families into content_items: the cocoindex flow (flow.py, the new canonical
pipeline) and the intelligence/RSS path (lib/intelligence/pipeline.ts →
storeAsContentItem, plus feed_articles). This bears directly on the
content_items-vs-feed_articles disposition (OQ-64-2).
1.4 grep sweeps (SQL + Python — outside TS index)
Section titled “1.4 grep sweeps (SQL + Python — outside TS index)”origin_kindCHECK constraints: onlyq_a_pairs+q_a_pair_historycarry it (§3.2).templates_select … USING (true): confirmed insupabase/migrations/20260416102457_pre_squash_reconciliation.sql:6346(§2.1).- cocoindex write targets:
mount_table_targetforcontent_items,q_a_extractions,source_documents,entity_mentions,form_templates,form_template_fields(scripts/cocoindex_pipeline/flow.pylines 1472–1507) (§4).
1.5 gitnexus_context note
Section titled “1.5 gitnexus_context note”gitnexus_context on row-type symbols was not productive: DB row shapes are generated
types in supabase/types/database.types.ts (Tables<'x'>), not call-graph symbols, so
the canonical source for column/FK shape is the generated types file (read directly in
§3–§4), which is the project-mandated source of truth (CLAUDE.md “TypeScript
conventions”). No symbol-edit is proposed by this RESEARCH, so gitnexus_impact is N/A
here; each {64.x} implementation subtask brief below carries the
impact-analysis-before-edit / detect-changes-before-commit discipline for its Executor
(Inv 2 → Inv 3).
2. Inv-25 RLS — role-based single-tenant USING(true) is intended (CLOSED S292; not a migration)
Section titled “2. Inv-25 RLS — role-based single-tenant USING(true) is intended (CLOSED S292; not a migration)”2.1 Empirical current state (confirmed)
Section titled “2.1 Empirical current state (confirmed)”form_templates was renamed from templates
(supabase/migrations/20260520120828_t2_combined_pr_intel_shape_b_form_type_split.sql
SUB-TASK 2, lines 204–239) and inherited the original four templates_* policies
(supabase/migrations/20260416102457_pre_squash_reconciliation.sql):
| Policy | Command | Predicate |
|---|---|---|
templates_select | SELECT | TO authenticated USING (true) — role-gated single-tenant (intended) |
templates_insert | INSERT | WITH CHECK (get_user_role() IN ('admin','editor')) |
templates_update | UPDATE | USING (get_user_role() IN ('admin','editor')) |
templates_delete | DELETE | USING (get_user_role() = 'admin') |
USING (true) is role-gated (any authenticated user) but NOT workspace-scoped.
Empirically, a workspace-A viewer can SELECT workspace-B form_templates rows — confirmed
S278 against live staging with a viewer-role JWT (test.user3,
get_user_role()='viewer'); the cross-workspace row WAS returned.
This is the INTENDED model, not a defect (ratified S292 — OQ-64-1 CLOSED). Liam ratified
that role-based single-tenant USING(true) is the intended RLS model for the v1 Phew pilot:
workspace membership is NOT required, and cross-workspace form_templates readability is
acceptable under a single-tenant deployment. The earlier S279 framing of this USING(true)
as a defect-to-fix — and the “Model A workspace_members junction” prerequisite — are
REVERSED.
ID-52 PRODUCT Inv-25’s per-workspace scoping (“A user’s view of ingested form instances
is scoped to workspaces they may access … the catalogue, having no workspace scope, is a
shared platform resource.”) describes a per-workspace multi-tenant posture the platform has
deliberately NOT adopted (see §2.4 and ID-69/RESEARCH.md §4) — not a pre-re-ingest gate,
and not a planned requirement. form_templates is the single-workspace Path-B activity
output (workspace_id NOT NULL), correctly single-tenant; the canonical corpus layer
(content_items) is, by ratified ID-69 design, NOT workspace-organised at all (§2.4). Evidence for the single-tenant disposition: procurement-workspaces/PRODUCT.md:382 (v1
single-tenant Phew pilot), reserved-workspace-seats/TECH.md:160 (role-based delegation),
the bl-207 fix caveat (no membership model = a larger separable task), ID-69/RESEARCH.md:155
(correctly single-tenant), and the consolidated disposition in
docs/themes/canonical-pipeline/reference/pipeline-writepath-schema-gap-s295.md §S296
(“OQ-64-1 — CLOSED S292”).
form_templates.workspace_id is NOT NULL with FK
form_templates_workspace_id_fkey → workspaces(id)
(supabase/types/database.types.ts form_templates block) — so workspace-scoping the
SELECT predicate is structurally feasible without a schema change to the table itself.
2.2 Sibling tables to audit (Inv-25 scope)
Section titled “2.2 Sibling tables to audit (Inv-25 scope)”form_template_fields— currenttemplate_fields_select … USING (true)(pre_squash:6308). Has NOworkspace_idof its own; it links viaform_template_fields.template_id → form_templates(id). Scoping must therefore be transitive (a field is visible iff its parentform_templatesrow is visible).form_template_requirements(catalogue) —template_requirements_select … USING (true)(pre_squash:6327). ThisUSING(true)is CORRECT per Inv-25 (the catalogue is a global platform resource,template_type → form_typesonly, noworkspace_id). The acceptance test asserts the catalogue remains readable regardless of workspace (form-extraction-rls.integration.test.tssecondit). Do NOT scope this one.
2.3 The single-tenant RLS model — USING(true) is intended, no membership junction by design
Section titled “2.3 The single-tenant RLS model — USING(true) is intended, no membership junction by design”Ratified S292 (OQ-64-1 CLOSED): the absence of a workspace-membership model is BY DESIGN
for the v1 single-tenant Phew pilot. Role-based USING(true) is the intended SELECT model;
there is NO pre-re-ingest RLS migration required. This REVERSES the earlier S279 framing
(below restated as historical context) that treated the missing junction as a blocking gap.
The empirical state of the schema (still accurate):
- The
workspacestable has no membership junction — a user associates with a workspace only throughworkspaces.created_by/updated_by. There is noworkspace_members(or equivalent many-to-many) table. - Every workspace-bearing table SELECT-scopes with
USING (true), not by workspace: confirmed forsource_documents("Authenticated users can view source documents" … USING (true), pre_squash:5835),bid_questions(bid_questions_select … USING (true), pre_squash:5924),form_templates,form_template_fields,form_template_requirements. Theget_item_workspaces()/get_workspace_counts()functions are SECURITY INVOKER RPCs, not RLS predicates.
Under the ratified single-tenant model this uniform USING(true) is correct: a single
deployment serving one tenant (Phew) does not need per-workspace SELECT isolation. The
evidence base for the ratification: procurement-workspaces/PRODUCT.md:382 (v1 single-tenant
Phew pilot), reserved-workspace-seats/TECH.md:160 (role-based delegation), the bl-207 fix
caveat (no membership model = a larger separable task), ID-69/RESEARCH.md:155 (correctly
single-tenant), and the consolidated disposition in
docs/themes/canonical-pipeline/reference/pipeline-writepath-schema-gap-s295.md §S296
(“OQ-64-1 — CLOSED S292”).
2.4 The corpus is canonical and shared, NOT workspace-organised (ID-69, ratified)
Section titled “2.4 The corpus is canonical and shared, NOT workspace-organised (ID-69, ratified)”The deeper reason per-workspace isolation is not a looming requirement: the ratified ID-69 vision makes the corpus a Wikipedia-style canonical record layer, NOT a workspace-organised one. One record is authored once and reused across workspaces/activities. The schema already supports this and the access model is intentionally shared:
content_items(the canonical record) has noworkspace_id— by design. Workspace reuse is expressed by the many-to-manycontent_item_workspacesjunction (composite PK(content_item_id, workspace_id), FKON DELETE CASCADEto both parents,ciw_select USING(true)) — “the Wikipedia one-record-many-workspaces model” (ID-69/RESEARCH.md§4.1).- Path-A canonical ingest (
scripts/cocoindex_pipeline/flow.py:ingest_file) never writescontent_item_workspacestoday (ID-69/RESEARCH.md§4.2/§4.3): the single resolvedworkspace_idfeeds only the Path-Bform_templateswrite. Canonical content is workspace-agnostic at ingest by design, not by omission. form_templates.workspace_idis the onlyNOT NULLsingle-workspace column — the Path-B activity output, correctly single-tenant (ID-69/RESEARCH.md§4.1).- The curated-clean-corpus posture (content enters only when AI-ready) supersedes the older “point cocoindex at the whole client doc-dump” framing — a further reason the corpus is a shared canonical asset, not per-tenant siloed data.
Consequently USING(true) on the corpus/content tables is correct by architecture, and
per-workspace SELECT isolation is not on the roadmap as a requirement. Recorded here only
to forestall re-litigation: a workspace_members + is_workspace_member() scheme would be the
mechanism only if the platform ever adopted a genuine multi-tenant SaaS posture for activity
surfaces — which the ratified architecture (canonical shared corpus + single-tenant
form-activity) does not require. Cross-ref ID-69/RESEARCH.md §4.
The S278 acceptance artefact __tests__/integration/form-extraction-rls.integration.test.ts
(ID-52.13) was authored under the earlier multi-tenant framing; under the S292 single-tenant
model it is not a pre-re-ingest gate (its cross-workspace-denial assertion describes a
per-workspace multi-tenant model the platform has NOT adopted — §2.4). The
catalogue-stays-global assertion remains valid regardless. That test would matter only if a
genuine multi-tenant SaaS posture were ever adopted — which the ratified canonical-corpus +
single-tenant-form-activity architecture does not require. Not ID-64, not on the roadmap.
3. KNOWN schema-migration item 2 — origin_kind value rename
Section titled “3. KNOWN schema-migration item 2 — origin_kind value rename”3.1 What origin_kind actually is (correcting the brief’s “enum” framing)
Section titled “3.1 What origin_kind actually is (correcting the brief’s “enum” framing)”origin_kind is not a Postgres ENUM type — it is a text column governed by a
CHECK constraint, and it exists on exactly two tables: q_a_pairs and its history
mirror q_a_pair_history (confirmed via type-block scan of
supabase/types/database.types.ts and the migration grep). It does not exist on
content_items, source_documents, entity_mentions, or any other pipeline table. So
the rename is narrow and isolated to the QA subsystem, not a corpus-wide enum.
3.2 Current authoritative definition
Section titled “3.2 Current authoritative definition”supabase/migrations/20260520225456_t6_q_a_pairs_full_schema.sql (SUB-TASK 1c, lines
60–76) is the latest reconciliation:
ALTER TABLE public.q_a_pairs DROP CONSTRAINT IF EXISTS q_a_pairs_origin_kind_check;ALTER TABLE public.q_a_pairs ADD CONSTRAINT q_a_pairs_origin_kind_check CHECK (origin_kind IN ( 'extracted_from_corpus', 'curated_explicit', 'derived_from_bid_response', -- ← rename target 'imported_legacy' ));ALTER TABLE public.q_a_pairs ALTER COLUMN origin_kind SET DEFAULT 'curated_explicit';q_a_pairs and q_a_pair_history are both 0 rows today (T6 just created the shape),
so a DROP+ADD CHECK swap is safe and no data backfill/UPDATE is required. This is the
exact reason it MUST land before re-ingest — once the corpus writes any
derived_from_* row, the rename becomes a data migration.
3.3 Required change
Section titled “3.3 Required change”Rename the CHECK value derived_from_bid_response → derived_from_form_response on BOTH
q_a_pairs and q_a_pair_history (the history mirror must carry the same constraint or
the history trigger can write a value the mirror rejects). Migrate the matching Python
string literal(s): grep shows the value referenced in
scripts/tests/test_cocoindex_extraction.py (TestExpectedResponseKindParity,
test_only_two_values) and the QA-flow extractor pathway — these expect the parity set, so
they update in lockstep. No TS consumer references the literal (§1.1).
3.4 The bid_responses table-name question (DO NOT silently rename — OQ-64-3)
Section titled “3.4 The bid_responses table-name question (DO NOT silently rename — OQ-64-3)”The brief says “verify bid_responses naming”. Findings:
- A real
bid_responsestable EXISTS (created in pre_squash:3360) withbid_response_history, the RPCsearch_for_bid_response, and FKsbid_responses_question_id_fkey → bid_questions,…_approved_by/_drafted_by/ _last_edited_by → user_profiles. It is the procurement bid-answer record, distinct from the QA corpus. - The terminology is mid-migration: ID-31 PLAN §11 records “Pre-S234
bid_response_id NOT NULLframing fully retired (RATIFIED-RETIRE)”, and the citations design (ID-58 / T11, PLAN §P-22) still defines aciting_entitypolymorphic enum whose first value isbid_response. So “bid” → “form” is a broader, unresolved terminology migration, not a single value swap. - Recommendation: scope
{64.x}to theorigin_kindvalue rename only (derived_from_bid_response → derived_from_form_response), which is internally consistent and 0-row-safe. Whether thebid_responsestable (and theciting_entityenum value) should also rename toform_*is OQ-64-3 — a separate decision the Orchestrator routes to Liam, because it touches procurement bid surfaces (live UI), thesearch_for_bid_responseRPC, and the T11 citations design that is not yet built. Do not couple it into the pre-re-ingest readiness migration.
4. KNOWN schema-migration item 3 — systematic missing-FK audit
Section titled “4. KNOWN schema-migration item 3 — systematic missing-FK audit”Audit method: for each canonical-pipeline table, enumerate *_id / *_by columns from
the generated Row shape and diff against the declared Relationships (FK) block in
supabase/types/database.types.ts. A column with no declared FK is a candidate;
candidates are then triaged into REAL-MISSING (should be enforced) vs INTENTIONAL
(correlation id / polymorphic / cross-store).
4.1 Findings table
Section titled “4.1 Findings table”| Table | FK-enforced (sample) | Unenforced *_id/*_by candidates | Triage |
|---|---|---|---|
| content_items | created_by,updated_by→user_profiles; source_bid→workspaces; superseded_by→content_items | source_document_id, parent_id, content_owner_id, archived_by, verified_by, governance_reviewer_id, op_id | REAL-MISSING: source_document_id→source_documents (canonical pipeline link!), parent_id→content_items (self-ref). user_profiles refs (content_owner_id,archived_by,verified_by,governance_reviewer_id) → enforce. op_id = INTENTIONAL (cocoindex correlation text, no FK target — §4.2). |
| source_documents | archived_by,uploaded_by→user_profiles; parent_id→source_documents; pipeline_run_id→pipeline_runs; workspace_id→workspaces | op_id, pullmd_share_id | op_id INTENTIONAL; pullmd_share_id INTENTIONAL (external pullmd share token, not a local table) — confirm in §4.2. |
| q_a_pairs | source_workspace_id→workspaces; superseded_by→q_a_pairs | (none beyond enforced) | OK. |
| q_a_pair_history | q_a_pair_id→q_a_pairs | changed_by | REAL-MISSING: changed_by→user_profiles. |
| q_a_extractions | promoted_to_pair_id→q_a_pairs; source_content_item_id→content_items | op_id | op_id INTENTIONAL. |
| entity_mentions | content_item_id→content_items | op_id | op_id INTENTIONAL. |
| content_chunks | content_item_id→content_items; parent_chunk_id→content_chunks | (none) | OK. (op_id add is owned by {56.6}, not here.) |
| feed_articles | content_item_id→content_items; feed_source_id→feed_sources; prompt_version_id→feed_prompts; workspace_id→workspaces | external_id | external_id INTENTIONAL (RSS GUID string, not a local table). |
| feed_sources/feed_prompts | created_by→user_profiles; workspace_id→workspaces | (none) | OK. |
| bid_responses | question_id→bid_questions; approved_by/drafted_by/last_edited_by→user_profiles | (none) | OK. |
| bid_questions | assigned_to/created_by→user_profiles; template_requirement_id→form_template_requirements; workspace_id→workspaces | (none) | OK. |
| form_templates | created_by→user_profiles; form_type→form_types; workspace_id→workspaces | (none) | OK (FK-wise; RLS is the §2 problem). |
| form_template_fields | question_id→bid_questions; template_id→form_templates | (none) | OK. |
| form_template_requirements | template_type→form_types | (none) | OK. |
| entity_aliases | {} (none) | (whole table) | RETAIN (OQ-64-4 resolved S296). Zero declared FKs is CORRECT: no canonical entities table exists, so there is no FK target. entity_aliases is admin-curated read-only input to Stage-5 entity resolution (ID-53). Do NOT add an FK, do NOT truncate. The real entity-spine FK is T12/ID-60 future work. |
| citations, question_matches, entities | NOT YET IN SCHEMA | — | Future tables (T10/T11/T12 = ID-57/58/60). Gate question: must they land BEFORE re-ingest? OQ-64-5. |
4.2 op_id is deliberately NOT a FK (do not “fix” it)
Section titled “4.2 op_id is deliberately NOT a FK (do not “fix” it)”op_id is the cocoindex per-flow operation id (text), stamped on content_items,
q_a_extractions, source_documents, entity_mentions, pipeline_runs by T8
(supabase/migrations/20260521203414_t8_op_id_propagation.sql;
COMMENT ON COLUMN … 'Cocoindex per-flow op_id; T8'). There is no operations table —
the only op/run-shaped table is pipeline_runs, and op_id is a correlation stamp, not a
parent key. The pipeline test test_missing_op_id_fails enforces presence, not
referential integrity. The FK audit must explicitly exclude op_id from the
add-missing-FK set, or it will introduce a constraint the pipeline cannot satisfy.
4.3 The canonical-pipeline FK that genuinely matters
Section titled “4.3 The canonical-pipeline FK that genuinely matters”content_items.source_document_id → source_documents(id) is unenforced today yet it is
the structural link the new canonical pipeline relies on (cocoindex writes content_items
AND source_documents in the same flow — §1.3, flow.py 1472–1507). Re-ingesting with this
FK unenforced risks orphan content_items if a source_documents write fails mid-flow.
This is the highest-value FK to add before re-ingest. Add with ON DELETE SET NULL
(matches the q_a_extractions.source_content_item_id … ON DELETE SET NULL precedent in
the T6 migration) so a source-document purge does not cascade-delete content. Open
nullability question folded into the subtask brief: source_document_id is nullable
(manual/UI items have no source document), so the FK must permit NULL.
5. KNOWN input 5 — gate-links NOT owned by ID-64 (checklist references only)
Section titled “5. KNOWN input 5 — gate-links NOT owned by ID-64 (checklist references only)”These are Task-level dependencies of ID-64 (deps: [56, 62, 63]) and the
origin-of-truth for each gate lives in that Task. ID-64 only references their GREEN
status in the checklist (§7); ID-64 does NOT author or own their work.
- ID-56 {56.8} chunking (Variant B = RecursiveSplitter, ratified S278/{56.5}). The
cocoindex chunking stage must be landed + integration-green ({56.9}) before re-ingest, or
the corpus is chunked by the about-to-be-retired app-side chunker.
{56.6}(content_chunks.op_id) and{56.7}(search RPC nullable headings) are sibling prerequisites inside Task 56 — not ID-64 subtasks. - ID-62 fixture-staging live tier — HARD gate. GREEN before re-ingest. Until the
/stageserver + in-VPC reachability harness exists ({62.1} RESEARCH in progress), the 39-test cocoindex integration tier cannot run live — it skips clean. Re-ingesting without this is re-ingesting unverified. (The §2 Inv-25 RLS acceptance test is NOT part of this gate — role-basedUSING(true)is the intended single-tenant model and the corpus is canonical/shared per ID-69 §2.4; per-workspace scoping is not part of the ratified architecture, not a pre-re-ingest gate; S292.) - ID-63 ontology / CV integration — the controlled vocabularies (
form_types, taxonomy,requirement_type) feed Path-A classification during ingest. A parity-drifted CV load means systematically wrong classification across the whole corpus. Must be verified end-to-end ({63.1} RESEARCH pending) before re-ingest.
6. Liam’s data-retention list — per-table disposition
Section titled “6. Liam’s data-retention list — per-table disposition”Source: docs/research/db-retain-investigation.md (read in full). For each table: RETAIN
(carry data through unchanged), RE-SEED (truncate + reload from a seed/fixture),
MIGRATE (transform existing data into the new shape), RE-INGEST (content re-run
through the new pipeline; classification/embedding discarded), or DROP (no longer
relevant under the new architecture). Dispositions below are proposed; the ones marked
(OQ) need Liam ratification.
6.1 “Trace source — may be seeded”
Section titled “6.1 “Trace source — may be seeded””| Table | Records | Proposed disposition |
|---|---|---|
company_profiles | 1 | RETAIN — single real record; client/org profile, not pipeline output. Verify it survives any workspace-ontology change. |
feed_flags | 2 | RE-SEED — feed moderation flags; tied to feed_articles which re-ingest. Re-seed from current 2 if still relevant, else DROP. |
form_types | (CV) | RETAIN / RE-SEED via ID-63 — controlled vocabulary. Disposition OWNED by ID-63 CV-load (the source of truth), not ID-64. |
guides | — | RETAIN — authored guidance content, not pipeline output. Confirm not workspace-ontology-coupled (OQ if it is). |
intelligence_workspaces | — | MIGRATE/RETAIN (OQ) — depends on the workspace-ontology decision (§ workspaces below). Likely retained as workspace rows. |
procurement_vehicle_instances | — | RETAIN — reference data (frameworks/lots). Not re-ingested. |
procurement_vehicles | — | RETAIN — reference data. Not re-ingested. |
6.2 “User access-related — retain”
Section titled “6.2 “User access-related — retain””| Table | Proposed disposition |
|---|---|
user_roles | RETAIN (mandatory) — drives get_user_role() which ALL RLS depends on. Must survive re-ingest untouched. |
user_profiles | RETAIN (mandatory) — FK target for nearly every *_by/owner column across the schema (§4). Untouched. |
6.3 Content items (the core re-ingest question)
Section titled “6.3 Content items (the core re-ingest question)”| Table | Proposed disposition |
|---|---|
content_items | RE-INGEST (content only). Retain ONLY rows that are (a) Phew-employee added/edited OR (b) successfully ingested articles/URLs — keyed by ingest_source ∈ {'rss feed','manual','url_import'} (column is nullable text; verify exact stored values, OQ-64-6). For retained rows, the source content is re-run through the new cocoindex pipeline — old classification_*, embedding, quality_score, classification_cache_* are discarded and regenerated. Implementation shape = export-retained-content → drop derived columns → re-ingest, NOT an in-place UPDATE. |
content_history | RE-SEED/DROP (OQ). Audit trail of edits. If re-ingest rewrites content_items ids, history rows orphan. Either migrate history forward keyed to the new ids, or accept a clean-slate history at handover. Liam decision. |
6.4 “Needs investigation — new-architecture impact”
Section titled “6.4 “Needs investigation — new-architecture impact””| Table | Proposed disposition |
|---|---|
feed_articles | RETAIN structure; RE-INGEST linked content. feed_articles ALREADY EXISTS as the RSS-ingestion record (workspace-scoped; FKs to content_items/feed_sources/feed_prompts). Its content_item_id is a NULLABLE link to the promoted content_item. RSS/URL items therefore already have a home distinct from content_items — this is the core of OQ-64-2. Retain feed_articles rows whose linked content is retained; re-link content_item_id after re-ingest. |
feed_prompts | RETAIN — RSS relevance-prompt versions; referenced by feed_articles.prompt_version_id. Keep for lineage. |
feed_sources | RETAIN — RSS source configs (URLs, schedules). Re-ingest re-reads from these; do not lose. |
si_processing_queue | RE-SEED/DROP (OQ) — sector-intelligence processing queue (transient work items). Almost certainly truncate-and-rebuild on re-ingest; confirm nothing durable is parked there. |
taxonomy_domains | RETAIN / OWNED by ID-63 — taxonomy CV. Disposition is ID-63’s call (CV-load source of truth). ID-64 references, does not own. |
taxonomy_subtopics | RETAIN / OWNED by ID-63 — as above. |
entity_aliases | RETAIN (OQ-64-4 resolved S296) — admin-curated read-only Stage-5 input; no entities table → no FK target; do NOT truncate, do NOT add FK. (The entity_pair_resolutions determinism cache is also RETAINED — truncating breaks idempotency.) Cross-ref docs/themes/canonical-pipeline/reference/pipeline-writepath-schema-gap-s295.md §S296. |
workspaces | MIGRATE/RETAIN (OQ-64-7) — the workspace-ontology change (cross-ref ID-63, “based on changes to workspace ontology”) may invalidate some current rows. Retain workspaces that map to the new ontology. (Decoupled from RLS: per S292 the single-tenant USING(true) model needs no membership junction — §2.3 — so this is purely a retention question.) The single biggest cross-cutting retention question. |
6.5 Tables NOT in Liam’s list but written during re-ingest (must be clean-slate-able)
Section titled “6.5 Tables NOT in Liam’s list but written during re-ingest (must be clean-slate-able)”These are cocoindex/QA write targets (§4) with ~0 rows today; they are truncate-and-let-
the-pipeline-repopulate on re-ingest, but the readiness subtask must confirm each is
empty (or safely truncatable) before cutover: source_documents, q_a_pairs,
q_a_pair_history, q_a_extractions, entity_mentions, content_chunks, pipeline_runs.
7. RE-INGEST GATE CHECKLIST
Section titled “7. RE-INGEST GATE CHECKLIST”The single readiness surface. Re-ingest to the chosen target (staging OR new preview branch — OQ-64-8) proceeds only when every HARD gate is GREEN. Verify on the target, then cut over to prod. DDL via Supabase CLI only.
A. Schema / FK migrations (owned by ID-64 {64.x}):
- A1 (NOT A GATE — resolved S292) Inv-25 RLS workspace-scoping is NOT a
pre-re-ingest gate. Role-based
USING(true)onform_templatesis the intended single-tenant model (S292, OQ-64-1 CLOSED — §2.3); no workspace-scoping migration is required pre-re-ingest. The bl-207form_templatesSELECT-parity fix is optional and not undertaken here — do NOT undertake an RLS migration.form_template_requirements(the catalogue) stays globally readable regardless. Per-workspace SELECT isolation is not part of the ratified architecture (the corpus is canonical/shared — nocontent_items.workspace_id, M2M reuse viacontent_item_workspacesper ID-69;form_templatesis single-tenant) — not a roadmap requirement, not this checklist (§2.4). - A2 (HARD)
origin_kindvalue renamederived_from_bid_response → derived_from_form_responseonq_a_pairs+q_a_pair_historyCHECK constraints, plus matching Python literal(s). Must precede anyderived_from_*row write. - A3 (HARD) Add
content_items.source_document_id → source_documents(id) ON DELETE SET NULL(nullable). The canonical-pipeline link (§4.3). - A4 (MEDIUM) Add remaining REAL-MISSING FKs:
content_items.parent_id,content_items.{content_owner_id,archived_by,verified_by,governance_reviewer_id},q_a_pair_history.changed_by. Explicitly excludeop_id(§4.2). - A5 (GREEN — resolved S296)
entity_aliases= RETAIN: noentitiestable → no FK target; admin-curated read-only Stage-5 input; do NOT truncate/FK; theentity_pair_resolutionsdeterminism cache is also retained. See §S296. - A6 (gate question)
citations/question_matches/entities(T10/T11/T12) — decide land-before-re-ingest vs after (OQ-64-5). - A7 Regenerate
supabase/types/database.types.tsafter all DDL (supabase gen types …);supabase-types-parityCI green.
B. Correctness / capability gates (owned by Tasks 56/62/63 — references only):
- B1 (HARD) ID-62 fixture-staging live tier GREEN (
/stageserver + in-VPC reachability). Unblocks the 39-test integration tier. (Not coupled to A1 — A1 is resolved S292 and is no longer a gate.) - B2 (HARD) ID-56 {56.8}/{56.9} cocoindex chunking (Variant B) landed + integration-green; app-side chunker retired ({56.11}).
- B3 (HARD) ID-63 ontology/CV integration verified end-to-end (Path-A classify + form-cataloguing + T10 matching).
C. Data retention (owned by ID-64 readiness subtask + Liam ratification):
- C1 Retention dispositions (§6) ratified by Liam (all
(OQ)rows resolved). - C2 Retained-content export verified (content_items keep-set by
ingest_source; feed_articles re-link plan; content_history decision). - C3 Mandatory-RETAIN tables (
user_roles,user_profiles, reference data) confirmed untouched by the re-ingest procedure. - C4 Clean-slate write-target tables (§6.5) confirmed empty/truncatable.
D. Cutover:
- D1 Re-ingest executed on the chosen non-prod target (OQ-64-8); corpus
spot-verified (classification parity, embedding presence, FK integrity, role-based RLS —
USING(true)single-tenant; no cross-workspace denial expected, the corpus is canonical/shared per §2.4). - D2
cat supabase/.temp/project-refverified before any proddb push; relink to prod only at the deliberate cutover step (CLAUDE.md project-ref drift gotcha). - D3 Prod cutover; post-cutover verification = the client-handover DB state.
8. {64.2+} SUBTASK PROPOSALS (TM-shape — for the Orchestrator to write)
Section titled “8. {64.2+} SUBTASK PROPOSALS (TM-shape — for the Orchestrator to write)”Sibling-only deps (integers, local to Task 64). No cross-Task subtask deps — the
chunking/fixture-staging/ontology gates are Task-level (ID-64.dependencies=[56,62,63]),
correctly NOT expressed here. Decomposition = 7 subtasks (well within the 25 soft
ceiling). Each details is load-bearing (an Executor needs no other doc); each carries the
code-intelligence tool-discipline instruction (Inv 2 → Inv 3) and the Supabase-CLI-only +
project-ref + SET search_path gotchas.
[ { "id": 2, "title": "Migration — origin_kind value rename derived_from_bid_response -> derived_from_form_response", "description": "Rename the origin_kind CHECK value on q_a_pairs + q_a_pair_history and the matching Python literal(s); 0-row-safe DROP+ADD; must land before any derived_from_* row is written.", "details": "RESEARCH §3. origin_kind is a TEXT CHECK constraint (NOT a pg ENUM), present ONLY on public.q_a_pairs and public.q_a_pair_history. Current authoritative def: supabase/migrations/20260520225456_t6_q_a_pairs_full_schema.sql lines 60-76 (CHECK IN ('extracted_from_corpus','curated_explicit','derived_from_bid_response','imported_legacy'), DEFAULT 'curated_explicit'). NEW migration via `supabase migration new` (DDL via CLI ONLY, never MCP execute_sql): DROP CONSTRAINT q_a_pairs_origin_kind_check; ADD with value renamed to 'derived_from_form_response'; repeat the same DROP+ADD on q_a_pair_history (its CHECK must match or the history trigger can write a rejected value). Both tables are 0 rows so no UPDATE/backfill. Then migrate the Python string literal: grep 'derived_from_bid_response' in scripts/ (test_cocoindex_extraction.py TestExpectedResponseKindParity/test_only_two_values and the QA-flow parity set) and update in lockstep; run `grep -rn derived_from_bid_response supabase scripts lib types` and confirm ZERO residual after. No TS consumer references the literal (gitnexus_query confirmed). DO NOT rename the bid_responses TABLE or the citing_entity 'bid_response' enum value — that is OQ-64-3, out of scope. Pre-push: cat supabase/.temp/project-ref; worker first action `supabase link --project-ref turayklvaunphgbgscat` (staging). Run gitnexus_detect_changes before commit.", "status": "pending", "dependencies": [], "testStrategy": "python3 -m pytest scripts/tests/ for the QA parity tests passes with the renamed value; grep -rn 'derived_from_bid_response' over supabase/scripts/lib/types returns ZERO; q_a_pair_history CHECK accepts 'derived_from_form_response' and rejects 'derived_from_bid_response'." }, { "id": 3, "title": "Migration — add content_items.source_document_id FK to source_documents (nullable, ON DELETE SET NULL)", "description": "Enforce the canonical-pipeline link content_items.source_document_id -> source_documents(id); nullable (manual/UI items have no source document); ON DELETE SET NULL so a source purge does not cascade-delete content.", "details": "RESEARCH §4.3. content_items.source_document_id is currently UNENFORCED (confirmed: content_items Relationships block in supabase/types/database.types.ts declares FKs only for created_by/updated_by->user_profiles, source_bid->workspaces, superseded_by->content_items). The cocoindex flow (scripts/cocoindex_pipeline/flow.py mount_table_target lines 1472-1507) writes content_items AND source_documents in the same flow, so an unenforced link risks orphan content_items on a mid-flow source_documents failure. NEW migration via CLI: validate no orphan rows first (SELECT count(*) FROM content_items ci WHERE ci.source_document_id IS NOT NULL AND NOT EXISTS (SELECT 1 FROM source_documents sd WHERE sd.id=ci.source_document_id)); if >0, surface to Orchestrator before adding the constraint. ADD CONSTRAINT content_items_source_document_id_fkey FOREIGN KEY (source_document_id) REFERENCES public.source_documents(id) ON DELETE SET NULL (NOT NULL-able: column is nullable). Precedent: q_a_extractions.source_content_item_id ... ON DELETE SET NULL (T6 migration line 107). Run gitnexus_impact on any TS reader of source_document_id before any code touch (none expected — this is DDL-only). Pre-push project-ref check + staging link. Regen types after (covered by {64.8}). gitnexus_detect_changes before commit.", "status": "pending", "dependencies": [], "testStrategy": "Inserting a content_items row with a non-existent source_document_id is rejected by the FK; deleting a source_documents row sets dependent content_items.source_document_id to NULL (no cascade delete); orphan-precheck query returns 0 before the constraint is added." }, { "id": 4, "title": "Migration — add remaining REAL-MISSING FKs across canonical-pipeline tables (exclude op_id)", "description": "Enforce the user_profiles + self-ref FKs surfaced by the §4 audit; explicitly exclude op_id (cocoindex correlation text, no FK target).", "details": "RESEARCH §4.1/§4.2. Add FKs (all via one CLI migration, ON DELETE SET NULL for user refs, self-ref for parent_id): content_items.parent_id->content_items(id); content_items.content_owner_id/archived_by/verified_by/governance_reviewer_id->user_profiles(id); q_a_pair_history.changed_by->user_profiles(id). For EACH: run the orphan-precheck (NOT EXISTS) query first; if any orphan rows, surface to Orchestrator (do not silently add). EXCLUDE op_id on ALL tables (content_items/source_documents/q_a_extractions/entity_mentions/pipeline_runs) — it is the cocoindex per-flow op_id text stamp (COMMENT in supabase/migrations/20260521203414_t8_op_id_propagation.sql; there is NO operations table; pipeline test test_missing_op_id_fails enforces presence not referential integrity). EXCLUDE source_documents.pullmd_share_id (external pullmd token) and feed_articles.external_id (RSS GUID) — not local-table refs. DO NOT touch content_chunks.op_id (owned by {56.6}). Pre-push project-ref + staging link; SET search_path not needed (DDL only, no new functions). gitnexus_detect_changes before commit.", "status": "pending", "dependencies": [3], "testStrategy": "Each added FK rejects an orphan insert and SET-NULLs on parent delete; a verification query confirms op_id columns have NO FK constraint on any table; orphan-precheck returns 0 for every column before its constraint is added." }, { "id": 5, "title": "Migration — Inv-25 workspace-scoped SELECT policy on form_templates (+ transitive form_template_fields)", "description": "[SUPERSEDED S292 — see details/§2.3/§9: OQ-64-1 CLOSED, USING(true) single-tenant is intended, no workspace-scoping migration; {64.5} re-scoped to RESEARCH-framing correction.] (HISTORICAL S279) Replace form_templates templates_select USING(true) with a workspace-scoped predicate; scope form_template_fields transitively via its parent; leave form_template_requirements global. GATED on the membership-model decision (OQ-64-1).", "details": "[SUPERSEDED S292: {64.5} re-scoped to RESEARCH-framing correction; OQ-64-1 CLOSED, role-based single-tenant USING(true) is intended, NO workspace-scoping migration — see §2.3 / §9. The proposal text below is retained as the S279 point-in-time decomposition; the workspace-scoping migration it describes is NOT undertaken.] RESEARCH §2. (HISTORICAL S279 framing) BLOCKED until OQ-64-1 (workspace-membership model) is ratified — there is currently NO membership junction and NO is_workspace_member() predicate; EVERY workspace table uses USING(true). Once the predicate exists (Variant A: workspace_members table + SECURITY DEFINER is_workspace_member(uuid) WITH `SET search_path = public, extensions`; or Variant B: narrower creator/JWT-claim scope): NEW CLI migration — DROP POLICY templates_select ON public.form_templates; CREATE POLICY form_templates_select ON public.form_templates FOR SELECT TO authenticated USING (<membership predicate on workspace_id>). For form_template_fields: DROP template_fields_select; CREATE form_template_fields_select FOR SELECT TO authenticated USING (EXISTS (SELECT 1 FROM public.form_templates t WHERE t.id = form_template_fields.template_id AND <same membership predicate on t.workspace_id>)). DO NOT touch template_requirements_select (catalogue is global by Inv-25 — the acceptance test asserts it stays readable). Leave insert/update/delete policies (role-gated) unchanged. Existing live policies: supabase/migrations/20260416102457_pre_squash_reconciliation.sql lines 6308/6346/6327. Acceptance = __tests__/integration/form-extraction-rls.integration.test.ts (skips clean until ID-62 fixture-staging GREEN). If the migration cannot be authored because OQ-64-1 is unresolved, STOP and escalate (do not invent a membership model). Pre-push project-ref + staging link. gitnexus_detect_changes before commit.", "status": "pending", "dependencies": [], "testStrategy": "[SUPERSEDED S292 — describes a per-workspace multi-tenant model the platform has NOT adopted (§2.3/§2.4, ID-69); not a pre-re-ingest gate.] __tests__/integration/form-extraction-rls.integration.test.ts: a workspace-A viewer SELECT of a workspace-B form_templates row returns 0 rows, AND form_template_requirements remains readable regardless of workspace (both assertions pass once the policy lands and ID-62 fixture-staging is wired)." }, { "id": 6, "title": "entity_aliases FK/retention disposition — investigate + apply (OQ-64-4)", "description": "[SUPERSEDED S296: {64.6} = doc-only RETAIN disposition — no entities table → no FK, no truncate; entity-spine FK is T12/ID-60 future. See §S296] Resolve the zero-FK entity_aliases table: either add the proper FK(s) to the canonical entity id, or ratify truncate-and-rebuild during re-ingest entity-resolution.", "details": "[SUPERSEDED S296: {64.6} = doc-only RETAIN disposition — no entities table → no FK, no truncate; entity-spine FK is T12/ID-60 future. See §S296] RESEARCH §4.1/§6.4. entity_aliases has ZERO declared FKs (supabase/types/database.types.ts) and is flagged by Liam as possibly holding now-irrelevant data under the new workspace ontology. Its target shape depends on ID-60 (Mempalace-KG entity_mentions). Step 1: read the entity_aliases CREATE in supabase/migrations/* (grep 'entity_aliases') and the ID-60/ID-53 entity-resolution design to determine the intended parent (likely a canonical entities/entity id). Step 2 (decision recorded as OQ-64-4 resolution): if rows are durable + a parent exists, add the FK via CLI migration with orphan-precheck; if rows are transient resolution artefacts, ratify truncate-and-let-Stage-5-rebuild (no FK migration, just a clean-slate note in the readiness surface). Use documentation-and-adrs to record whichever decision. Pre-push project-ref + staging link if a migration results. gitnexus_detect_changes before commit.", "status": "pending", "dependencies": [], "testStrategy": "entity_aliases disposition is recorded (FK added with passing orphan-precheck, OR a ratified truncate-and-rebuild note in the readiness surface) and the readiness checklist item A5 is resolvable to GREEN." }, { "id": 7, "title": "Data-retention export + re-link plan (content_items keep-set, feed_articles re-link, content_history decision)", "description": "Produce the executable retention procedure: which content_items rows to keep (by ingest_source), how feed_articles re-link to re-ingested content, and the ratified content_history disposition.", "details": "RESEARCH §6. content_items: build the keep-set query — retain rows where ingest_source IN (the real stored values for 'rss feed'/'manual'/'url_import'; VERIFY exact strings first via SELECT DISTINCT ingest_source FROM content_items — OQ-64-6, the column is nullable text). For retained rows, export the SOURCE content only (content/title/source_url/source_document linkage) — DISCARD classification_*, embedding, embedding_model, quality_score, classification_cache_* (regenerated by the new pipeline). This is export-then-re-ingest, NOT in-place UPDATE. feed_articles: ALREADY EXISTS as the RSS record (workspace-scoped; content_item_id nullable link); plan to re-link content_item_id after re-ingest assigns new content_items ids (retain feed_articles whose linked content is retained). content_history: record Liam's ratified decision (migrate-forward-keyed-to-new-ids vs clean-slate-at-handover — OQ resolution). Confirm mandatory-RETAIN (user_roles, user_profiles, company_profiles, guides, procurement_vehicle*, feed_sources, feed_prompts) untouched. Confirm clean-slate write-targets (source_documents, q_a_pairs, q_a_pair_history, q_a_extractions, entity_mentions, content_chunks, pipeline_runs) empty/truncatable. Output = a runbook-style procedure doc (use documentation-and-adrs) referenced by {64.8}. No DDL here unless a staging/transform table is needed (CLI if so). DO NOT edit task-list.json.", "status": "pending", "dependencies": [], "testStrategy": "The content_items keep-set query returns the expected Phew-authored + successfully-ingested rows (validated against SELECT DISTINCT ingest_source); the export/re-link procedure is reviewable and names every table in §6 with a disposition; no mandatory-RETAIN table appears in any truncate/drop step." }, { "id": 8, "title": "Re-ingest readiness surface — aggregate gate checklist + types regen + cutover runbook", "description": "Single readiness surface that aggregates §7 gates, gates the non-prod re-ingest, regenerates types after all DDL, and documents the staging/preview -> verify -> prod cutover with the project-ref safeguard.", "details": "RESEARCH §7. Aggregates ALL gates into one checked surface (a doc under docs/operations/ or docs/runbooks/, NOT the ledger): A-gates (this Task's migrations {64.2}-{64.6}), B-gates (Task-level deps 56/62/63 GREEN — reference only, do not own), C-gates (retention {64.7} ratified), D-cutover. After every DDL subtask is merged, regenerate supabase/types/database.types.ts via the CLAUDE.md command (`/opt/homebrew/bin/supabase gen types typescript --project-id <ref> --schema public > supabase/types/database.types.ts`) and confirm supabase-types-parity CI green (A7). Document the cutover: re-ingest to the chosen non-prod target (staging OR new preview branch — OQ-64-8), spot-verify (classification parity, embedding presence, FK integrity via the new constraints, role-based RLS — USING(true) single-tenant; no cross-workspace denial, the corpus is canonical/shared per ID-69 §2.4), THEN `cat supabase/.temp/project-ref` and relink to prod ONLY at the deliberate cutover step (D2 — CLAUDE.md project-ref drift gotcha: a stale prod link makes db push land on prod silently). Post-cutover verification = the client-handover DB state. Use documentation-and-adrs for the runbook. This subtask is the LAST to complete (it gates cutover).", "status": "pending", "dependencies": [2, 3, 4, 5, 6, 7], "testStrategy": "The readiness surface enumerates every §7 A/B/C/D gate with an owner and GREEN/RED state; supabase-types-parity CI is green after the post-DDL types regen; the cutover runbook contains the explicit `cat supabase/.temp/project-ref` + relink step before any prod db push." }]Decomposition notes for the Orchestrator:
{64.5}— [SUPERSEDED S292] the original Inv-25 RLS workspace-scoping proposal is withdrawn: OQ-64-1 is CLOSED, role-based single-tenantUSING(true)is the intended model, and there is no workspace-scoping migration.{64.5}is re-scoped to this RESEARCH-framing correction (see §2.3 / §9). It is NOT blocked on any membership model and is NOT coupled to ID-62. (The S279 note that it was “BLOCKED on OQ-64-1” no longer applies.){64.4}depends on{64.3}only to serialise the twocontent_itemsFK migrations (avoid two concurrent ALTERs on the same table); they could merge if the Orchestrator prefers a single content_items-FK migration.{64.8}depends on all of{64.2}–{64.7}(it is the aggregation/cutover surface).- If OQ-64-5 resolves to “land citations/question_matches/entities before re-ingest”, that
is NOT a
{64.x}subtask — it belongs to Tasks 57/58/60 and would be added as a new Task-level dependency on ID-64. Surfaced, not folded in (sibling-only constraint). - Subtask count = 7 (
{64.2}–{64.8}), comfortably under the 25 soft ceiling.
9. Open Questions (OQ-64-N) — for the Orchestrator to route to Liam
Section titled “9. Open Questions (OQ-64-N) — for the Orchestrator to route to Liam”- OQ-64-1 — Workspace-membership model — CLOSED (S292). Resolution: role-based
single-tenant
USING(true)is the INTENDED RLS model for the v1 Phew pilot; workspace membership is NOT required, and noworkspace_membersjunction /is_workspace_member()predicate is needed pre-re-ingest. There is no workspace-scoping migration under {64.5}; the bl-207 SELECT-parity fix is optional and not undertaken. Per-workspace isolation is not part of the ratified architecture — the corpus (content_items) is canonical and shared with noworkspace_id(ID-69 §4; reuse via thecontent_item_workspacesM2M), andform_templatesis single-tenant — so it is NOT a planned requirement and NOT a pre-re-ingest gate (§2.4). Evidence:procurement-workspaces/PRODUCT.md:382(v1 single-tenant Phew pilot),reserved-workspace-seats/TECH.md:160(role-based delegation), the bl-207 fix caveat (no membership model = a larger separable task),ID-69/RESEARCH.md:155(correctly single-tenant), and the consolidated dispositiondocs/themes/canonical-pipeline/reference/pipeline-writepath-schema-gap-s295.md§S296 (“OQ-64-1 — CLOSED S292”). (OQ-64-7 workspace-ontology retention is now decoupled from this — it is a pure retention question.) - OQ-64-2 — RSS/URL items:
content_itemsvsfeed_articles— CLOSED (verified 11/06/2026; answered by the ID-75 O4 ratification, S316). Resolution: neither of the two framed options — external RSS/URL material lands the REFERENCE LAYER. ID-75 PRODUCT DP-1: “content_items= client-adopted knowledge. External material lands a reference layer”; BI-1: a gate-passing feed article lands exactly onesource_documentsrow + onereference_itemsrow — nocontent_itemsrow, not by the pipeline, not by any fallback. Adoption intocontent_itemsis explicit and workflow-driven (ID-58 citations / ID-71), never automatic.feed_articlesremains the per-workspace RSS ledger (TS poller inserts; cocoindex enumerates + backlinksreference_item_id— flow.py WP-C step 7, live since the ID-75 M1/M2 migrations); legacy promoted rows are NOT retro-converted (BI-10) — the retained-content keep-set +feed_articlesre-link stays in{64.7}. The legacystoreAsContentItempromotion path is an ID-75 BI-11 / ID-42 / ID-46 retirement target. - OQ-64-3 —
bid_responsestable rename — CLOSED (08/06/2026, ID-93 §6.1). Resolution: rename pre-run. The deferral default was re-decided when ID-93 found the ENTIRE bid_* surface at 0 rows on prod. EXECUTED: {64.14} (tables +search_for_bid_responseRPC + consumers) + {64.16} (get_bid_*→get_form_*RPCs) — staging S331, prod S336 (7-mig cutover).content_citationsdisposition was owned by ID-58 (polymorphiccitationsREPLACE + DROP — also done, prod S336). (Historical text below §3.4 retained as the S279 point-in-time framing.) - OQ-64-4 —
entity_aliasesdisposition — CLOSED (S296). Resolution: RETAIN. No canonicalentitiestable exists → there is no FK target;entity_aliasesis an admin-curated, read-only input to Stage-5 entity resolution (ID-53). Do NOT truncate, do NOT add an FK. Theentity_pair_resolutionsdeterminism cache is also RETAINED (truncating it breaks idempotency). The real entity-spine FK (to a canonicalentitiesrow) is T12/ID-60 future work.{64.6}is therefore doc-only. Evidence:docs/themes/canonical-pipeline/reference/pipeline-writepath-schema-gap-s295.md§S296 (“OQ-64-4 — RETAIN”). (Drives {64.6}.) - OQ-64-5 — T10/T11/T12 tables land order — CLOSED (08/06/2026, ID-93 §6.2; reframed {64.8} journal). Resolution: structurally complete, based on ratified specs — the handover DB carries the T10/T11/T12 schema slices pre-cutover ONLY per ratified ID-57/58/60 specs ({64.8} gate G6); full feature build stays post-cutover. Reframe (08/06): more-than-minimal means STARTING the ID-58/ID-59 spec chains, not speculative empty DDL; ID-60 = NO pre-re-ingest action; ID-57 stays C5-decoupled but its schema slice must spec-ratify pre-cutover (the outstanding G6 tail). Execution since: ID-58 citations REPLACE + ID-59 edit_intent storage BUILT — staging S331, prod S336.
- OQ-64-6 — exact
content_items.ingest_sourcestored values. Liam’s keep-set keys oningest_source ∈ {'rss feed','manual','url_import'}but the column is nullable text. Confirm the exact stored strings (aSELECT DISTINCT ingest_sourceon the live data, {64.7}) so the keep-set query is correct and no retained row is missed. (Progress 11/06/2026: the {64.17}/bl-106 escalation observed 8 distinct values —manual,upload,url_import,upload_autosplit,bid_outcome_integration, … — the canonical list ratification (bl-106) and the {64.7} keep-set verification remain open.) - OQ-64-7 — workspace ontology +
workspaces/intelligence_workspacesretention — CLOSED (11/06/2026, Liam). Resolution: the CONCEPT/PURPOSE carries, not the rows. The three live intelligence workspaces (MAT Auditing · NHS Digital Cyber Alerts · Education Sector Monitor) carry forward to the new DB as re-created workspaces with the same purpose/configuration (company-profile binding, guide, relevance threshold — cf.specs/intelligence-workspaces/S-1/S-2 typed-column promotion); the GNEWS-DEDUP workspace is DELETED (not carried). {64.7} implements: export the three workspace configs, re-create on the fresh target, drop the fourth. - OQ-64-8 — re-ingest target — CLOSED (S296, Liam). Resolution: FRESH Supabase preview branch for the full re-ingest (isolated dry-run, cleaner rollback, no staging-data entanglement); Task 82 proved the git-integrated preview-branch migration replay. Drives the {64.8} cutover runbook + project-ref relink steps (v1 sequence Spine S2).
- OQ-64-9 —
content_historydisposition — CLOSED (11/06/2026, Liam). Resolution: CLEAN SLATE — edit history does not migrate forward at client handover. Pre-condition: the {64.7} export first identifies any client actions taken (rows keyed by client user UUIDs — expected minimal) so nothing client-authored is silently dropped; anything found is dispositioned explicitly before the truncate.
10. Verification (OQ-3 pre-ratification empirical check)
Section titled “10. Verification (OQ-3 pre-ratification empirical check)”- External-library API citations: NONE introduced. This RESEARCH cites only KH-internal
artefacts (SQL migrations, generated
database.types.ts,flow.py).mount_table_targetis referenced descriptively as an already-in-use cocoindex symbol read fromscripts/cocoindex_pipeline/flow.py(lines 1472–1507) — not a NEW citation requiring an import-and-call check. The OQ-3 import-and-call check therefore does not apply to this artefact. (Where{64.x}Executors touch cocoindex 1.0.3 write APIs, the existing pinned usage inflow.pyis the empirical reference; cocoindex’sExtractByLlm/version drift precedent —docs/research/cocoindex-1.0.3-extractbyllm-spec-reality-investigation.md— is the standing reminder to re-verify against the pin if any new symbol is introduced.) - Empirical schema facts in §2–§4 were confirmed against the live-generated
supabase/types/database.types.tsand the migration corpus in this worktree (not assumed from prose). Thetemplates_select … USING(true)finding is additionally corroborated by the S278 live-JWT probe recorded in the acceptance test header. - Date: 29/05/2026. Worktree:
id-64offmain.