ID-131 {131.1} RESEARCH — OKF L-records refactor (content_items elimination)

RESEARCH — OKF L-records refactor (content_items elimination)

Status: DRAFT — 28/06/2026. Consolidation of the RATIFIED record-model (three deep-grounding rounds + an adversarial critic + the product owner’s round-3 direction, all ratified). This RESEARCH does not re-derive the model — it consolidates it into the durable spec record, re-verifies every load-bearing file:line / DB claim against current head, and structures the open sub-decisions for {131.2} PRODUCT.

Owning Task: ID-131 — the FOUNDATION Task of a 4-Task pre-launch re-architecture. ID-131 owns the irreversible-flavoured decisions (record-identity seed contract, content_items elimination, the embeddings store, the governance facet). The three sibling Tasks (L-concepts + producer; ontology pass; promotion-confidence E2E corpus) depend on ID-131’s outputs and are out of ID-131’s scope.

Spec chain: {131.1} RESEARCH (this doc) → {131.2} PRODUCT → {131.3} TECH → {131.4} PLAN.

Ratified inputs consolidated here (private working notes, not committed): the v3 synthesis, the adversarial critic notes (verdict SOUND_WITH_CAVEATS; the one blocker RESOLVED by the owner 2026-06-28 — “anchors only”), and the owner’s round-3 feedback.

---

0. Code-intelligence orientation — verification block (Checker: confirm this ran)

Every load-bearing claim below was re-verified against current head + the live platform DB (project zjqbrdctesqvouboziae) on 28/06/2026, not trusted from the working notes. Tools used: gitnexus_query / gitnexus_context (consumer-surface orientation), supabase execute_sql (schema/constraints/row-counts), and grep over Python (scripts/cocoindex_pipeline/*.py) + SQL migrations (ast-dataflow is TS-only, so the pipeline + migration claims used grep).

Verification verdict: every ratified claim holds at current head. NO contradiction was found. Six deltas (all line-drift or off-by-one, none changing a decision) are folded in and flagged [VERIFIED-Δ] where they appear:

1. hybrid_search RETURNS 21 columns, not 22 (the “8 mapped” figure is exact). The working-notes headline said 22; the critic corrected to 21; independently confirmed 21 here.
2. The Q&A sidecar block in promote-corpus.ts has drifted to ~535–640 (notes cited 540–566); the DB-assigned-id read-back is at promote-corpus.ts:406 (notes cited 391–406). Substance intact.
3. Forms-matching lives at lib/domains/procurement/form-templating/template-coverage.ts (notes once wrote lib/procurement/...). Confirmed.
4. .from('content_items') footprint = 92 files / 191 sites in app/+lib/+components/+hooks/ alone (the critic’s broader pattern counted 184 sites / 142 files including types/tests/scripts). The footprint is confirmed large; “cheaper than it appears” refers to the value-path output contract (8 mapped columns), not the whole Task.
5. NEW: the pipeline mints two further deterministic seeds beyond the citeable trio — ci:{rel_path} (flow.py:1943, :2112, dies with content_items) and chunk:{rel_path}:{position} (flow.py:2243, internal retrieval grain). Neither enters the bundle-citation SEED-CONTRACT.
6. NEW: citations = 0 rows on prod — strengthens the CITE-EXT mitigation (“no prod citations to orphan”).

---

1. Problem & context

1.1 What content_items is, and why it must go

content_items is a 67-column god-table inherited from an earlier personal-IMS (information management system) fork. It conflates roles that the platform has since split into purpose-built typed records: reference_items and q_a_pairs were extracted out of content_items, yet the generic bucket persists, carrying a long tail of IMS-vestige columns (starred, user_tags, priority, brief, detail, reference, thumbnail_url, author_name, platform, source_domain, …) that no pipeline stage writes and only a deprecated browse/item UI reads.

The platform’s core value is high-quality, structured data accessible by AI. A generic god-table is the opposite of structured: it forces every consumer (search RPCs, citations, governance crons, the MCP find tool, forms matching) to reason over a polymorphic blob instead of a typed record. The OKF (Open Knowledge Format) re-architecture replaces the blob with an explicit three-layer model.

1.2 The OKF three-layer model (RATIFIED)

L-raw       — the client's raw sources (local-fs / Notion / Gmail / .docx / .pdf / live website).
              Owner: THE CLIENT (stays put). Authoritative provenance + audit + citation anchor.
              GATED: we define the source-document structure (at least for the first client) and the
              Q&A format; the pipeline monitors the gated corpus for change.
                      │  cocoindex pipeline derives ↓ (delta-only; per-file content-hash memoised)
L-records   — CANONICAL DB records. Owner: CANONICAL (operated).
              source_documents (provenance + version chain + RECORD-IDENTITY SEED);
              q_a_extractions (raw tier) → q_a_pairs (master canonical answers);
              reference_items (external evidence); content_chunks (retrieval grain);
              entity_mentions + entity_relationships (closed-ontology KG, STAYS in DB);
              citations (polymorphic); record_lifecycle facet; the embeddings store.
              Disposable: a full-replace re-ingest rebuilds it deterministically.
                      │  2nd-pass enrichment agent distills ↓
L-concepts  — the OKF concept bundle. Owner: THE CLIENT (private git repo).
              ~30–50 .md concept files (topics, products, certs, company, case studies, metrics,
              playbooks) + index.md + log.md. Concept-identity = the file path. Concepts CITE
              records; they never STORE them. A Q&A pair is a record, not a concept — it never
              appears here.

Two of the three layers are client-owned (L-raw and L-concepts); only L-records is Canonical-operated. This is what makes “we don’t extract your value / you keep your canonical files” literally true. ID-131 is the L-records layer: it eliminates content_items and rebuilds the typed record model that L-concepts will later cite.

The only legitimate cross-layer pointers are:

• L-records → L-raw (provenance): reference_items.source_url, source_documents.storage_path + content_hash, and the per-Q&A “Source of Information”.
• L-concepts → L-records (citation): a concept’s resource: field + # Citations body carry canonical://<table>/<uuid> pointer values — the only place a Canonical record id touches the client-owned bundle, and always a pointer value, never the concept’s identity key.

1.3 Why pre-launch is the cheapest possible moment

This is a pre-launch code/schema refactor with zero production data lock-in. Verified row counts on the live platform DB (zjqbrdctesqvouboziae, 28/06/2026): content_items = 33 (E2E debris, 0 chunks), content_chunks = 0, q_a_pairs = 0, q_a_extractions = 0, governance_config = 0, content_templates = 0, content_item_workspaces = 0, citations = 0. reference_items = 3, form_template_requirements = 96, pipeline_runs = 496, entity_mentions = 27, entity_relationships = 17.

Because re-ingest is full-replace on a new, empty preview branch and record identity is a deterministic uuid5, the L-records DB is disposable and reproducible: re-ingesting the same gated corpus with the same recipe reproduces the same uuids, so it can be re-seeded any number of times pre-launch. There is no data migration anywhere in ID-131 — every “re-parent” is a schema change plus a pipeline-writer change, not a backfill of live rows.

De-identification note: the real first-client corpus is referred to throughout as “the first client” / “the real first-client corpus”. The client’s name is deliberately omitted from this committed spec.

---

2. Record-identity v3 (the seed model ID-131 ratifies-as-built)

The corrected record-identity model is mostly ratify-as-built, not design-in-this-Task. The earlier v2 recipe (switch source_documents to a content_hash seed; mint a qa:question_hash for Q&A) is fully retracted.

2.1 source_documents — KEEPS rel_path by design

• Seed = uuid5(_KH_PIPELINE_DOC_NS, "sd:" + rel_path) — verified at flow.py:2111 (content branch) and flow.py:2529 (sidecar branch); the URL branch is uuid5(… "sd:" + item.url) at flow.py:2940. This is idempotency-in-place by design: re-ingesting the same file mints the same PK → the row UPSERTs in place.
• Verified: source_documents has only a PRIMARY KEY on id — NO UNIQUE(storage_path), NO UNIQUE(content_hash) (pg_constraint on public.source_documents returns source_documents_pkey only). The seed string IS the idempotency mechanism.
• content_hash stays a stored audit/diff column (plain file hash; feeds id-117 diff + byte-idempotency checks). It is not the PK seed.
• Verified: the table is a lean 21-column provenance table (information_schema.columns: positions 1–19, 21, 22; position 20 dropped). It carries id, filename, original_filename, mime_type, file_size, content_hash, version, parent_id, storage_path, status, extracted_text, extraction_metadata, workspace_id, pipeline_run_id, uploaded_by, created_at, archived_at, archived_by, op_id, extraction_method, source_url. It has NO classification, NO freshness/lifecycle, NO governance, NO title (filename only), NO created_by/updated_by/updated_at, NO verified_*. version/parent_id exist but the pipeline does not write them today (no version-threading logic — an edited same-path file overwrites in place).

2.2 Q&A — the master PK is already DB-only opaque (nothing to switch)

This is the single biggest correction to v2.

• qa:{rel_path}:{idx} is the PK of the RAW tier q_a_extractions, not of q_a_pairs — verified at flow.py:2270 (content) and flow.py:2568 (sidecar). It is the exact analogue of sd:{rel_path} (idempotency-in-place over the gated corpus). KEEP it.
• q_a_pairs (the master/canonical tier) is ALREADY DB-only opaque — verified q_a_pairs.id DEFAULT gen_random_uuid(); the promoter inserts then reads the DB-assigned id back (promote-corpus.ts:406). No path/hash derivation exists for the master PK; nothing external pins it. There is nothing to switch. Opacity is a feature: full-replace re-mints it; nothing external depends on a stable master uuid (and per §4 the bundle never cites a q_a_pair).
• The many-raw→one-master merge anchor is built, not hypothetical: q_a_extractions.promoted_to_pair_id (FK → q_a_pairs.id) is the link; the joint (promoted_to_pair_id, source_document_id) keys the pair; re-merge uses q_a_pairs.superseded_by; cross-file dedup is q_a_pair_dedup_proposals (curator-approved writes only). This is what lets a master survive moving across library files and versions — its identity is opaque and DB-internal, so re-grouping only re-points extractions.

The genuine “Q&A as DB-only” decision (QA-DBONLY): it is not a PK change — it is to retire the {59.x} Q&A sidecar file round-trip (promote-corpus.ts ~535–640 emits promoted pairs back to the corpus as __qa__/*.md and sets q_a_pairs.source_document_id = sdUuid5(rel_path)). A Q&A pair is a record, never a concept, so writing it as a file in the client-owned bundle contradicts the concept≠record split. Retire the sidecar; keep q_a_pairs.source_document_id as pure-DB provenance.

2.3 reference_items — KEEPS ri:{source_url} (settled, constraint-enforced)

• Seed = uuid5(_KH_PIPELINE_DOC_NS, "ri:" + item.url) — verified at flow.py:2941, with a live UNIQUE(source_url) constraint (reference_items_source_url_key). id has no DB default (the seed is mandatory); RI is one-shot / never re-extracted. source_url MUST remain on reference_items (settled — no open decision).

2.4 The reframed irreversibility (the seat moved)

	earlier claim	v3 corrected (RATIFIED)
Hard freeze point	the first DB write of re-ingest	first publication of the client-owned OKF bundle (a sibling Task, not ID-131)
Why	re-ingest hardens the grain	full-replace + deterministic uuid5 ⇒ L-records is disposable & reproducible; re-seed any number of times pre-launch
Owner of the irreversible decision	the DB-write Task	the bundle-producer Task (the resource:/# Citations pointers pin record uuids into a durable client-owned artefact)

What ID-131 must ratify as a stable CONTRACT (not as an irreversible DB fact): the SEED-CONTRACT (§4) — so the later producer Task can rely on it. The one genuine reversible schema item ID-131 owns here is re-parenting q_a_extractions to source_document_id (its current source_content_item_id link dies with content_items; q_a_pairs already carries source_document_id).

---

3. content_items decomposition (CI-DECOMP) — the core of ID-131

Direction RATIFIED (“100%, but cheaper than it appears”). Every figure below is current-head-verified.

3.1 The FK inventory — 10 enforced + 2 unenforced (NOT 11+2)

source_document_diffs was already dropped by id-117, so the earlier 11+2 figure is stale. Verified via pg_constraint WHERE confrelid = 'public.content_items' = exactly 10 enforced FKs (all convalidated = true) + 2 unenforced bare-uuid columns:

FK / column referencing content_items.id	Enforcement (verified)	ID-131 action	Target
content_chunks.content_item_id	unenforced (no FK)	RE-PARENT — ADD source_document_id col (CC has none today)	source_document_id
entity_mentions.content_item_id	unenforced (no FK)	RE-PARENT — ADD source_document_id col	source_document_id
entity_relationships.source_item_id	SET NULL	RE-PARENT	source_document_id
classification_disputes.content_item_id	CASCADE	RE-PARENT	source_document_id
ingestion_quality_log.content_item_id	CASCADE	RE-PARENT	facet owner / SD
feed_articles.content_item_id	SET NULL	RE-PARENT	reference_item_id (verify writers — open sub-decision §5c)
verification_history.content_item_id	CASCADE	MOVE with governance	record_lifecycle facet owner
content_history.content_item_id	SET NULL	DROP or move	SD parent_id chain / OKF history
read_marks.content_item_id	CASCADE	DROP	IMS reading-tracker vestige
content_item_workspaces.content_item_id	CASCADE	DROP (ratified)	query-time scope_tag overlap
content_items.superseded_by (self)	SET NULL	DROP	full-replace removes in-DB supersession
citations.cited_content_item_id	CASCADE	HARD — extend the contract	§3.4 (CITE-EXT)

Plus a 3rd bare-uuid ref the critic surfaced: q_a_extractions.source_content_item_id (handled by the QA-REPARENT line above).

Sequencing note (binds TECH/PLAN): the chunk/entity re-parent needs the NEW source_document_id column AND the pipeline writer (flow.py) writing that parent from day one of the full-replace re-ingest, because content_items will not exist. The write-sites are flow.py:2243 (chunk seed), the entity-mention stage (flow.py:2378) and the relationship stage (flow.py:2450).

3.2 Column homes — the honest reality (“mostly already on SD” is FALSE)

Because source_documents is the lean 21-col table verified in §2.1, re-homing onto it means ADDING ~25–30 net-new classification columns, not “columns already mostly there”. Summary of the ratified calls:

• Pipeline core → SD extracted_text (already there) + content_chunks retrieval grain.
• Classification family (primary_domain, primary_subtopic, ai_keywords, summary, classified_at, …) → net-new on SD (SD has none today).
• embedding / embedding_model → the embeddings store (§ EMB-STORE).
• Q&A duplicate cols (answer_standard, answer_advanced) → DROP (q_a_pairs already has them).
• created_by → SD uploaded_by (SD has no created_by); updated_by/updated_at → net-new.
• source_url → DROP from CI (moot once table gone); KEEP on SD (URL-ingest provenance — SD has it, verified) + RI (NOT NULL).
• content_text_hash (GENERATED md5 of normalised text) → NO SD column (low value over content_hash + rel_path). Migrate normalised-text dedup to q_a_pairs (§3.5).
• IMS-vestige cols (author_name, platform, source_domain, thumbnail_url, starred, user_tags, priority, brief, detail, reference, file_path, source_file) → DROP. thumbnail_url is DROP-or-net-new-on-RI only if a reference-card UI is retained.
• freshness/lifecycle/governance/ownership family → the record_lifecycle facet (§ GOV-FACET), NOT inline on SD. citation_count → DERIVE (rollup); quality_score → DERIVE (cron-computed).

Key correction: of the four columns the owner asked about, none is a “move onto an existing column” — author_name/platform/source_domain are DROP; thumbnail_url is DROP-or-net-new. There is no “~58 cols” requirement.

3.3 The 4 search-RPC bodies + their TS callers (the real consumer cost)

The output contract for the AI/MCP value path is small; the cost is in the RPC bodies.

1. hybrid_search — [VERIFIED-Δ] RETURNS 21 columns (verified pg_get_function_result), of which the MCP find tool maps only 8 (search.ts:191–202: id, title, suggested_title, content_type, primary_domain, primary_subtopic, summary, similarity) — all with clean homes. The work is the body: a single-table scan becomes a polymorphic UNION across source_documents/content_chunks/q_a_pairs/reference_items, and the win_stats scoring CTE keyed on cited_kind = 'content_item' must re-anchor. /api/search/route.ts passes RPC rows wholesale, so its TanStack clients (use-search.ts, use-browse-data.ts) see whatever the rewrite emits.
2. get_content_win_rate — NOT dead, NOT migrated (owner belief corrected). Verified 2 live callers: app/api/items/[id]/effectiveness/route.ts:81 and lib/mcp/tools/procurement.ts:544. RETURNS an aggregate (total_citations, winning_citations, losing_citations, pending_citations, win_rate). Re-anchor to q_a_pair. WINRATE-FIX first (§3.3a).
3. find_related_items — NOT dead (caller app/item/[id]/page.tsx). Verified RETURNS carries the IMS-vestige cols (platform, author_name, source_domain, thumbnail_url, user_tags, …). Re-anchor or split per-type; drops user_tags.
4. filter_by_keywords — verified two variants exist: the live (search_terms text[]) RETURNS SETOF uuid (caller use-browse-data.ts) — low effort to re-point; and a RETURNS SETOF content_items no-op stub to delete.

S438 {131.25} reconcile: Items 3–4 above (the find_related_items re-anchor / user_tags-drop and the filter_by_keywords re-point) are superseded by {131.20} §9: both RPCs are DROPPED ENTIRELY (§7.4 related-records → an ontology-grounded backlog item; §7.5 keyword pre-filter → a future hybrid_search facet param), as shipped on the {131.11} search branch. Separately, the EMB-STORE codename used throughout this doc is the shipped record_embeddings polymorphic store.

Plus the dedup family (find_duplicate_pairs, find_exact_duplicates + content.ts/supersession.ts) re-targeted to q_a_pairs (§3.5), and the quality_issues_pending VIEW over content_items (verified present in squash_baseline; 0 rows) — an uncounted consumer to rewrite/drop with the table.

3.3a WINRATE-FIX — resolve the dual outcome-source bug FIRST

Verified: two different outcome sources feed one concept. get_content_win_rate derives outcome from form_outcome_types.counts_toward_win_rate (id130 migrations), while hybrid_search.win_stats derives it from workspaces.domain_metadata->>'outcome' (id61 / squash_baseline migrations). Pick the canonical outcome source before re-anchoring either to q_a_pair, or the bug propagates. This is an open PRODUCT call (§5).

3.4 The one hard re-home: citations (CITE-EXT) — partly pre-built

Verified: citations carries dormant cited_q_a_pair_id and a CHECK citations_cited_one_of_chk enforcing exactly-one-of {content_item, q_a_pair} over the cited_target_kind enum. But owner belief is incorrect: cite_content does not yet source from a dedicated table — it still writes cited_kind: 'content_item' (verified procurement.ts:463; the draft-stream auto-citer draft-stream/route.ts:330 likewise).

CITE-EXT = extend cited_target_kind to add {reference_item, source_document, concept}; add the matching cited_*_id columns (or a generic cited_target_id); rewrite the CHECK; re-point all writers (procurement.ts:463, draft-stream:330) + readers (items/route.ts, effectiveness/route.ts) + both win-rate consumers off cited_kind='content_item'. DB-record targets use the §2 record uuid; concept targets use the concept path. Mitigation amplifier: citations = 0 rows on prod, so no live citation can orphan during the cutover — but CITE-EXT + WINRATE-FIX must land together (see §6 risks).

3.5 Where dedup is actually valuable

• VALUABLE for q_a_pairs — the corpus’s own merge (“masters from many Q&As”) IS normalised-question dedup; migrate a normalised question hash there (matching the existing two-tier merge model).
• Largely VESTIGIAL for source_documents — SD already gets file-level idempotency from content_hash + rel_path; single-tenant + a gated corpus we structure make byte-identical dupes a curation-time concern. Do NOT add content_text_hash to SD.
• The IMS dedup tooling (find_exact_duplicates, find_duplicate_pairs, dedup_status stamping in content.ts/supersession.ts) is personal-KB heritage — port to q_a_pairs, do not blindly re-home onto documents.

3.6 The legacy IMS browse/item UI = ELIMINATION, not rewrite

components/content/content-card.tsx, content-row.tsx, item-detail/metadata-sidebar.tsx, editor-view.tsx, browse/filter-panel.tsx, app/item/[id]/item-detail-client.tsx bind the dropped IMS columns. This is removal of a deprecated surface, not contract preservation — scope these as deletions, not migrations.

3.7 Can content_items be fully eliminated? YES

Pipeline core → SD + CC; IMS cols → DROP; Q&A cols → DROP (dupes); URL → RI; governance/lifecycle → facet; distilled content_types → L-concepts (a sibling Task). No role remains that the typed records + facet + concepts cannot absorb.

---

4. The SEED-CONTRACT (ID-131 declares it; the producer Task freezes it)

Re-ingest reproduces citable uuids only if both the seed strings and the namespace constant are unchanged. ID-131 must declare this as a stable contract and gate it with an asserting test, so that when a later Task publishes the first client-owned bundle the canonical://<table>/<uuid> pointers remain valid across every subsequent full-replace re-ingest.

Frozen artefacts (all verified at current head):

• Namespace constant: _KH_PIPELINE_DOC_NS = uuid.UUID("fbfaf1ff-1ee4-583c-9757-1674465b2ec1") (flow.py:1640).
• Citeable seed strings: sd:{rel_path} (flow.py:2111/:2529; URL branch sd:{item.url} :2940), ri:{source_url} (flow.py:2941), and qa:{rel_path}:{idx} (flow.py:2270/:2568, the raw-tier idempotency anchor).
• [VERIFIED-Δ] Two non-citeable internal seeds exist and are explicitly OUT of the contract: ci:{rel_path} (flow.py:1943/:2112) dies with content_items; chunk:{rel_path}:{position} (flow.py:2243) is internal retrieval grain, never bundle-cited.

The anchors-only bundle-citation clause (owner-ratified 2026-06-28 — resolves the critic’s blocker):

Bundle citation target set = {source_document, reference_item, concept} ONLY. q_a_pair citation is DB-INTERNAL only.

Rationale: provenance flows source_document → concept → q_a_pair; a concept citing a downstream Q&A answer is backwards. source_document (rel_path seed) and reference_item (source_url seed) are deterministic across full-replace; q_a_pairs is opaque gen_random_uuid() and re-mints on every rebuild — so a bundle pointer to a q_a_pair would orphan on the next re-ingest, even pre-launch. Keeping q_a_pair citation DB-internal (draft reuse, forms completion, win-rate — all rebuilt on full-replace) preserves the opaque-DB-native master PK and closes the hole. A metric/dataset concept that needs Q&A data points at q_a_pairs as a table/query via resource:, never per-row.

The asserting test (ID-131 deliverable): assert the namespace value + the three citeable seed-string formats; gate it in CI before any bundle-producer Task publishes. If anyone “tidies” the seed strings or namespace after a bundle ships, every canonical://<table>/<uuid> citation silently orphans.

Empirical-verification note (OQ-3): ID-131 introduces no net-new external-library API call. The EMB-STORE index uses pgvector via a Supabase migration (the vector type is confirmed live in the DB — five vector columns exist today), and ID-131 deliberately AVOIDS cocoindex declare_vector_index (flow.py:3668-3673). The cocoindex producer APIs (declare_file, the Source protocol) are a sibling Task’s surface — their import-and-call verification belongs to that Task’s spec, not ID-131.

---

5. The EMB-STORE and GOV-FACET (net-new schema ID-131 owns)

5.1 EMB-STORE — dedicated polymorphic embeddings store (net-new)

Verified: embeddings are stored inline across 6 scattered columns — 5 vector columns (content_items.embedding, content_chunks.embedding, q_a_pairs.question_embedding, reference_items.embedding, form_template_requirements.requirement_embedding) + company_profiles. company_embedding stored as TEXT (an existing inconsistency to absorb). content_items.embedding dies with the table.

ID-131 builds a dedicated polymorphic store (owner_kind ∈ {source_document, content_chunk, q_a_pair, reference_item, concept}, owner_id, model, vector) so ONE store serves both L-records hybrid_search AND later L-concept bundle traversal. The pgvector index is declared via Supabase migration, not declare_vector_index. Sequencing dependency: the store must be ratified in ID-131 before the producer Task relies on it for concept-vector indexing.

5.2 GOV-FACET — shared polymorphic record_lifecycle facet (net-new)

Promote freshness + review-cadence + governance-review + ownership into its own shared, polymorphic facet (e.g. record_lifecycle), keyed (owner_kind, owner_id) with owner_kind ∈ {source_document, q_a_pair} and a denormalised domain for the policy join. reference_item is EXCLUDED by design (BI-16, enforced by the owner_kind CHECK — absence of a facet row = governance-free). governance_config stays exactly as-is (per-domain policy, UNIQUE(domain); verified 0 rows).

Why a facet beats SD-only / per-record columns: the expiry sweep / recalculate_all_freshness() becomes ONE set-based UPDATE; the review queue / cadence / governance crons / whats_in_my_queue unify across documents AND canonical answers. q_a_pairs already carries a divergent lifecycle subset, so SD-only would freeze Q&A into a parallel governance model (drift) or regress Q&A lifecycle (publication_status is NOT NULL and load-bearing in q_a_search/dedup).

The hot/cold split (RATIFIED):

• Cold (cron/queue-driven, cross-cutting) → the facet: freshness, freshness_checked_at, previous_freshness, lifecycle_type, expiry_date, next_review_date, review_cadence_days, governance_review_status, governance_review_due, governance_reviewer_id, verified_at, verified_by, content_owner_id, domain.
• Hot (read-path filters) → KEEP INLINE on the typed records: publication_status, superseded_by, valid_from, valid_to (q_a_search filters publication_status on every query). Do not push these behind a facet join.
• DERIVE, do not store: quality_score (cron-computed), citation_count (rollup).

Consumer-rewrite scope (~66 governance TS files, critic-verified): 3 crons (quality-score, freshness-transitions, review-cadence), app/api/governance/review, app/api/review/{queue,cadence, stats}, app/api/source-documents/[id]/send-to-review, MCP governance.ts + review.ts, and 3 DB fns (recalculate_all_freshness, get_freshness_breakdown, get_review_breakdown_stats). Convert the freshness-recalc fn and the review queue first. Re-parent the CASCADE sidecars (verification_history, ingestion_quality_log) to the facet owner; review_assignments is already filter-based — extend filter_content_types to span owner_kind. A write-time sync keeps facet.domain == owner.primary_domain after reclassification.

5.3 Forms-matching re-point (the one required forms change in ID-131)

Verified path: lib/domains/procurement/form-templating/template-coverage.ts. Its fetchContentForMatching SELECTs FROM content_items (incl. content_type='q_a_pair'). Eliminating content_items REQUIRES re-pointing this matching engine to q_a_pairs (primary) + reference_items + source_documents, and form_responses.source_content_ids / form_questions.matched_content_ids (uuid[]) become q_a_pair/reference_item ids. This is the forms-side of the CITE-EXT consumer rewrite; it simplifies forms (match against canonical Q&A answers, not a generic bucket). Preserve the blank-vs-answered fork (id-80): answered forms → Path-A knowledge (q_a_pairs); blank instruments → Path-B. Calibration risk: the matching thresholds (0.55/0.35) and cosineSimilarity were tuned against content_items embeddings — re-calibrate against the EMB-STORE vector space or matching silently degrades.

5.4 Out of ID-131 scope (sibling Tasks — referenced for sequencing only)

The L-concepts producer (self-updating-wiki declare_file + the reference_agent two-pass, the ADK+Gemini → Anthropic agent-loop port, the resource: URI scheme, the git knowledge-sync writer), the ontology pass (the 9-item three-layer re-alignment + the allowed_types/allowed_relations register that gates extraction AND concept writes), and the {127.4} promotion-confidence E2E corpus are separate Tasks. ID-131 re-parents the 3 extraction write-sites (entities stay in the DB — owner §F) but does NOT touch the producer, the ontology register, or the synthetic corpus.

---

6. Open sub-decisions for {131.2} PRODUCT (enumerated)

These are the genuine product calls the PRODUCT spec must resolve as numbered, testable invariants. None re-litigates a ratified decision; each is a sub-decision the ratified model deliberately left open.

• (a) Q&A freshness clock — own or derived? Does a Q&A answer get its OWN freshness clock, or derive it from its source_document’s freshness? If derived, the facet’s owner_kind set may differ per axis (freshness = {source_document}; review/publication = {source_document, q_a_pair}) — i.e. possibly two facets, not one. (Also decide whether entity_mentions, which carries a JSON expiry path metadata->>'expiry_date' swept by freshness-transitions, joins the facet or stays on the metadata path.)
• (b) Facet FK idiom. Per-kind nullable FKs + an exactly-one CHECK (mirroring citations_cited_one_of_chk) vs (owner_kind, owner_id) + a validating trigger. Trade referential-integrity strength against polymorphism ergonomics.
• (c) feed_articles re-parent target. reference_item vs source_document — verify the feed writers (flow.py URL branch mints both an SD and an RI for a feed URL at :2940/:2941) before fixing the target.
• (d) content_text_hash / dedup home. Confirm normalised-text dedup belongs on q_a_pairs and is NOT needed on source_documents (the §3.5 recommendation — make it a testable invariant).
• (e) Manual / TS authoring seed. If a non-file create path survives (a user with Claude + the Canonical MCP, or a headless agent creating a record a user approves), define how each of the three types is seeded when there is no rel_path/source_url — e.g. a deterministic seed for an authored source_document, or an opaque PK with a recorded provenance origin. Removing the content_item abstraction should make this clearer, not require re-templating (content_templates is dropped).
• (f) WINRATE canonical outcome source. Pick form_outcome_types.counts_toward_win_rate vs workspaces.domain_metadata->>'outcome' as the single source before re-anchoring win-rate to q_a_pair (§3.3a).

---

7. Blast-radius & sequencing notes for {131.3} TECH / {131.4} PLAN

7.1 Migration ordering (the hard edges)

1. Net-new schema first: the record_lifecycle facet + EMB-STORE land before any re-parent or drop, so the cron/queue and embedding consumers have a destination.
2. Re-parents next: add source_document_id to content_chunks/entity_mentions (+ the enforced-FK re-parents) AND update the flow.py write-sites (:2243/:2378/:2450) and the extraction stamp in lockstep — the pipeline must write source_document_id from day one of the full-replace re-ingest because content_items will not exist.
3. CITE-EXT + WINRATE-FIX land together (one atomic slice): extend cited_target_kind, rewrite the CHECK, re-point writers + readers + both win-rate consumers, after resolving the canonical outcome source. citations = 0 rows makes this safe, but a split landing breaks every citation path.
4. Search-RPC bodies rewrite after the typed-record columns exist (they UNION over them).
5. Drops last: content_item_workspaces, content_templates, the content_items table itself, the quality_issues_pending view, and the legacy IMS browse/item UI — once no consumer references them.
6. Declare the SEED-CONTRACT test independently (no schema dependency) — it can land early and gates later Tasks.

7.2 Parallel / serial groupings (by file-ownership, for the Executor wave)

• Schema-net-new (facet DDL + EMB-STORE DDL) — parallelisable; distinct migration files.
• Pipeline re-parent (flow.py + extraction.py stamp) — serial within scripts/cocoindex_pipeline/ (single file-ownership boundary); run gitnexus impact on _ExtractionStamp / stamp_extraction_base / flow_context first (5 flow_context tests reference content_items_id).
• Citations + win-rate (procurement.ts, draft-stream/route.ts, effectiveness/route.ts, items/route.ts + the 2 RPC bodies) — one ownership group (CITE-EXT/WINRATE land together).
• Governance facet consumers (~66 TS files + 3 crons + 3 DB fns) — the largest serial group; the freshness-recalc fn and review queue convert first.
• Forms-matching (template-coverage.ts + the uuid[] arrays) — folds into the CITE-EXT consumer rewrite; re-calibrate thresholds against EMB-STORE.
• Search RPCs (hybrid_search, find_related_items, filter_by_keywords bodies + use-search.ts / use-browse-data.ts / app/item/[id]/page.tsx callers) — after typed columns exist.
• IMS-UI deletion (components/content/*, app/item/[id]/*) — independent; land last.

7.3 Mandatory tool-discipline for the implementation wave (propagate into briefs)

Before modifying any symbol the Executor MUST run gitnexus impact({direction: 'upstream'}) (warn on HIGH/CRITICAL) and gitnexus detect_changes() before committing; renames go through gitnexus rename, never find-and-replace. ast-dataflow column-reads/column-writes pins every content_items consumer site (TS); grep covers the Python pipeline + SQL migrations (ast-dataflow is TS-only). The named high-blast-radius symbols are hybrid_search, get_content_win_rate, _ExtractionStamp, and the governance freshness fns.

7.4 Top risks carried into TECH/PLAN

1. Citation contract break (HIGH) — cite_content still writes content_item; CITE-EXT + the record-uuid re-anchor + WINRATE-FIX must land together (mitigated by citations = 0 rows pre-launch).
2. Governance facet consumer-rewrite under-scope (HIGH) — ~66 TS files + 3 crons + 3 DB fns; convert the freshness-recalc fn and review queue first; gate with gitnexus impact + ast-dataflow.
3. EMB-STORE sequencing (MEDIUM) — must be ratified in ID-131 before the producer Task relies on it; re-tunes the forms-matching thresholds.
4. Forms-matching blast radius (MEDIUM) — fetchContentForMatching + the two uuid[] arrays; form completion breaks if not re-pointed.
5. Seed-contract drift (MEDIUM, irreversible-flavoured) — re-ingest reproduces uuids only if the seed strings AND _KH_PIPELINE_DOC_NS are unchanged; the asserting test is the guard.

---

8. Summary of ID-131’s ratified scope (the register {131.2}+ builds on)

Decision	What ID-131 does	Reversibility
CI-DECOMP	Eliminate the 67-col content_items; re-home 10 enforced + 2 unenforced FKs; ~25–30 net-new SD classification cols; DROP IMS cols; delete the legacy browse/item UI	EXPENSIVE as code; ZERO data migration
D5-v2 / SEED-CONTRACT	KEEP sd:{rel_path} / qa:{rel_path}:{idx} / ri:{source_url}; q_a_pairs master PK opaque (ratify-as-built); freeze the three seed strings + namespace + an asserting test; anchors-only bundle clause	CONTRACT-FREEZE at bundle publication (a sibling Task), not the DB write
EMB-STORE	Dedicated polymorphic embeddings store; absorb the 6 scattered vector cols; pgvector via Supabase migration	CHEAP–MEDIUM (additive)
GOV-FACET	Shared record_lifecycle facet (owner_kind ∈ {source_document, q_a_pair}; reference_item excluded = BI-16 CHECK); hot/cold split; denormalised domain; governance_config unchanged	MEDIUM
CITE-EXT + WINRATE-FIX	Extend cited_target_kind to {reference_item, source_document, concept} + activate q_a_pair; re-point writers/readers + both win-rate consumers; resolve the dual outcome-source bug first	EXPENSIVE (consumer-contract)
EXTRACT-REPARENT + QA-REPARENT	Entities STAY in DB; re-parent content_chunks/entity_mentions/entity_relationships + q_a_extractions to source_document_id	CHEAP
QA-DBONLY	Retire the {59.x} __qa__/*.md sidecar round-trip; keep q_a_pairs.source_document_id as DB provenance	CHEAP
Drops + forms re-point	Drop content_item_workspaces + content_templates; re-point forms matching off content_items to q_a_pairs/RI/SD	CHEAP / MEDIUM

Quality bars inherited by every downstream spec/Executor: semantic design tokens only; UK English; auth.success + authFailureResponse(auth); getAuthorisedClient() / sb() / tryQuery() Supabase safety; no barrel re-exports (direct file imports); TanStack Query only; bun run test (never bun test); behaviour-first tests (test-philosophy.md).