Pre-reingest schema + deferral-review findings (S379)

Purpose. Durable capture of the S379 review of deferred items (V11-3/4, V11-11/12, C-4) for possible v1 promotion, plus the schema-correctness questions Liam raised about the ID-45 re-ingest. Written so this context is not lost in future sessions. Feeds: v1.1 planning, ID-71/104 AI-tooling, the Procurement domain build, and {45.3} TECH.

Reading caveat (Liam). Almost all referenced design docs are pre-cocoindex: read them for INTENT, not implementation approach. The original client spec was bids-focused; the platform has evolved well beyond that. Weigh post-pivot architecture (cocoindex-write-model, the current schema, id-45/id-63 specs) for implementation, the older docs for what was wanted.

0. Governing priority (Liam, S379)

“There isn’t a rush to re-ingest — getting the schema right is considerably more important.” Do not rush ID-45. Get the canonical data model (Q&A, layers, workspace linkage, UC8) right first, so re-ingest writes into the correct shape and we don’t build interim workarounds then rework.

ID-45 is a ONE-TIME event. Its purpose: take the client content we already have and run the pipeline fully on those files as the client’s source files for the first time. The client Supabase production currently holds content_items from earlier platform development; once ID-45 completes, that fresh dataset becomes the client’s starting dataset, with no further plans to full-re-ingest in the same way. After ID-45, the ongoing pipeline does incremental UPSERTs (day-to-day file changes on the local file server) — it never truncates. “Files are the source of truth; the DB is derived.”

Terminology note: my earlier “re-ingest #2” meant a future full INV-6 truncate-and-rewalk (not planned). It is NOT the ongoing incremental ingestion.

1. Q&A pairs / markdown sidecar (V11-3 + V11-4) — promote to spec-and-align now

Decision direction (Liam): do the investigation + spec + alignment now, before ID-45, so the Q&A canonical form is the surface it will take long-term (the markdown sidecar) — not a DB-only interim that gets reworked a few sessions later when v1.1 / procurement land.

What the review found (corrects an earlier “gold-plating” verdict)

The sidecar was always the INTENDED canonical form. decision-graph.md (Q2.9/Q3.2, the S233 “SIDECAR-REOPENED” thread) + cocoindex-write-model.md (files-are-truth) + the shipped lib/edit-intent/write-back.ts (file-first write-back for content_items) all establish: file = canonical, DB row = derived projection. A Q&A pair was meant to “end up as the markdown sidecar.”
Introducing the sidecar RESOLVES multiple-representations, it does not add a third. Today: q_a_extractions (derived cache) + q_a_pairs (curated golden record, DB-only). With the sidecar: one canonical file → two derived DB projections (extraction cache, then UC5-promoted pair). The earlier “third representation” framing was an inversion.
The “data-loss-at-truncate” risk is a NON-issue in practice. A clean ID-45 truncates q_a_pairs + q_a_extractions (INV-6 hard gate), and human UC6 edits / manually-authored (curated_explicit) pairs have no source-file home — BUT q_a_pairs live count is currently 0, and ID-45 is one-time, so nothing is lost. The ongoing incremental pipeline never truncates q_a_pairs.
The real, durable driver (Liam’s argument): in a files-are-truth system, curated Q&A that lives only in the DB means the canonical source (the files) is incomplete, so the AI tooling (ID-71) never sees curated knowledge in canonical form. Getting the canonical Q&A model right now feeds ID-71/104 with the canonical data it needs for highest-quality outputs, and avoids build-then-rework.

Crossover / why now

ID-71/104 (next-session impl) consume q_a_pairs; they should ground on the canonical Q&A surface. v1.1 is only a few sessions out and closely aligns.
Procurement (the prioritised next domain) owns the UC5/UC6/UC8 Q&A flywheel — the only domain that exercises capture→approve→promote.
Feeds ID-59.15 (UC8 cross-workspace dedup) — resolve the Q&A canonical model and UC8 together (dedup semantics change once pairs are canonical files). Currently {59.15} is deferred pending Spike#10 substrate + UC8 ratification — fold into this spec effort.

Action

Author a spec (likely a new Task, or reopen V11-3/4 into v1 scope) covering: the sidecar-emit-on-promotion model, the two-tier process_sidecar_file(file) / extract_q_a(content_text) fn shape (the COCO.10 idempotency gate, 05-qa-flow.md §5.2), the UC1-Candidate-A round-trip, lifecycle-columns-not-round-tripped classification, and the UC8 dedup tie-in. Sequence before ID-45 {45.3} TECH so re-ingest writes into the right model. Reference docs in §6.

2. Layers (C-4) — data model is CORRECT; only the read-side nested view is missing

The flat per-item layer tag (content_items.layer, 4 keys: sales_brief, bid_detail, company_reference, research) is the intended model and matches both the original client brief AND the new ontology CV. The new ontology layers CV (ID-63, DONE/S282) deliberately did not change the layer model — id-63 was a pre-reingest gate for content_type / form_type / domain / subtopic (its {63.11} NOT-NULL + unclassified sentinel migration), not layer.
Liam’s “current layers are wrong / one record transformed by audience” instinct points at a real read-side gap: the nested audience view (Sales-Brief on top, Bid-Detail nested) — gap-tracker §B.1, spec p0-layer-nesting-spec.md is genuinely unwritten. But this is a zero-row, render-layer change, safe any time, no re-walk → NOT a re-ingest gate. Re-ingest writes flat layer values (NULL-from-walk + post-hoc inference for content; 'research' constant for references) that any future nested view reads without re-walking.
C-4 decision to lock (cheap, prevents WP6 rework): the nested view composes on the existing guide_sections.parent_section_id (already in schema) — do NOT add a layer_priority_order column. Likely small touch: promote metadata.topic_id → a real indexable content_items.topic_id column (the audience-grouping key is currently unindexed JSONB) — additive, not a re-ingest gate.

Verdict: C-4 stays deferred; not a pre-reingest schema gate. The depth tabs (Summary/In-Depth/Takeaways/Original) are a different axis from the audience layer axis — both already correctly separated in schema.

3. ID-69 — workspace-agnostic canonical content (verify assignment-survival at cutover)

Liam’s position: should be treated as a pre-re-ingest gate (ledger currently done).
Facts: ID-69’s core model is ratified — content is workspace-AGNOSTIC (content_items has no workspace_id, BI-1); content links to workspaces via the M2M content_item_workspaces junction, populated by user action, not the pipeline (by design — this is the “one canonical record, reused across workspaces” model). ID-45 already treats content as workspace-agnostic (id-45 RESEARCH/PRODUCT: validates content_items count with no workspace_id). ID-69 task closed S371 as a tests/readiness task (no feature code).
The genuine open question for {45.3} TECH: the junction is NOT pipeline-populated, so what happens to existing content_item_workspaces rows (existing workspace assignments) across the ID-45 truncate/cutover? Is content_item_workspaces on the truncate list or the retain list? If truncated and not carried, existing assignments are lost. Confirm in {45.3} TECH: does ID-45 need a carry/repopulation step for existing workspace assignments, or is assignment purely forward (fresh user action post-cutover)? This is the schema-correctness item behind Liam’s “ID-69 should be a gate” — treat as a {45.3} verification item, not necessarily a task reopen.

4. Form supersession (V11-11 + V11-12) — defer now, revisit in the Procurement build

Stays deferred for v1: the supersession columns live outside FORM_TEMPLATES_SCHEMA and re-ingest never writes them (no DDL on that path); the procurement_workspaces satellite is still a bare 4-column stub (6 ratified feature columns unbuilt) — the DP-10 sequencing trigger (“when procurement-workspaces firm up”) has not fired.
But Procurement is now the prioritised next domain — so V11-11/12 re-enter scope when procurement firms up. The dependency is the {56.12} folder-drop “new revisions arrive as new files” contract. Revisit during the Procurement build, not before.

5. Prioritisation — Procurement is the aligned next domain (Liam: agreed)

ID-71 Waves 2/3 build the PROCUREMENT/forms AI surface (the bid→forms rename confirms it). Sales (O5) + proposals (O2) AI are explicitly deferred OUT of ID-71. Procurement is the only domain with the UC5/UC6/UC8 Q&A flywheel, has the most mature spec (ratified-S242, 14-task map), and is the first ID-104 eval-contract consumer.
Guides + Sales-Proposals are reserved-seat shells (4-column satellites). Land their migrations cheaply alongside the procurement T2 migration, but don’t invest in their feature surfaces yet. Sales-proposals is thinnest (no PLAN; Phase-3 research unratified).
Order: ID-45 spine (the gate, but unrushed — schema-correct first) → ID-111 + ID-71 W2/3 (next session, ultracode) → Procurement domain build → later own-Tasks for Sales-Proposals Phase 3 + Guides section-alignment + the layer nested-view.

6. Net pre-ID-45 schema worklist (do before {45.3} TECH / the run)

Q&A canonical/sidecar model (V11-3/4 + UC8/59.15) — spec + align so re-ingest writes the long-term canonical shape. (§1)
ID-69 junction survival — confirm whether existing content_item_workspaces assignments need a carry/repopulation step at cutover. (§3)
C-4 layer — NOT a gate; lock the parent_section_id (no layer_priority_order) decision; optionally promote metadata.topic_id → real column. (§2)
(Done) ID-63 domain/subtopic CV — already a landed pre-reingest gate.

Does this change the next-session ID-111 / ID-71 focus? No hard change. ID-111 (reference_items UI) is orthogonal. ID-71 W2 is AI infra (grounding shapes + born-evaluable contracts); W3 is procurement/forms reads + propose-writes — flag only that ID-71’s Q&A grounding should anticipate the sidecar-canonical direction (don’t hard-wire DB-only q_a_pairs as the eternal canonical source). The Q&A-sidecar spec lands before ID-45 anyway.

7. Reference docs (read for INTENT; platform has evolved)

Doc	For
`themes/canonical-pipeline/phase-0-investigation/0.9-edit-flow-investigation.md`	Liam-flagged: incredibly valuable — Q&A edit-flow + sidecar intent (§6.5.1)
`themes/canonical-pipeline/reference/decision-graph.md`	Q&A intended architecture; sidecar thread (Q2.9/Q3.2, S233 reopen)
`themes/canonical-pipeline/reference/cocoindex-write-model.md`	files-are-truth / canonical-store model
`05-qa-flow.md` (phase-0)	sidecar materialisation design §4–5; COCO.10 idempotency gate
`ontology/03-layer-vocabulary.md` + `specs/wp6-ontology-harness/investigations/INV-1-layer-vocabulary.md`	layer CV (4 keys, 3 orthogonal axes); §B.1 nesting intent
`specs/id-63-ontology-cv-integration/`	the ontology CV pre-reingest gate (DONE) — excludes layer
`specs/id-45-full-corpus-reingest/{RESEARCH,PRODUCT}.md`	INV-6 truncate; workspace-agnostic; what re-ingest writes
`specs/id-69-client-corpus-onboarding/`	canonical-record / workspace-agnostic model (BI-1)
`runbooks/...retention-export-relink-plan.md` ({64.7})	RE-WALK / MIGRATE-CARRY durability model; INV-23 promotion (flagged unowned)
client INTENT: `kh-client-content-archive/docs/client-documentation-corpus/.../Product_KB_Dev_Brief.md` + `kb-hub-gap-analysis-action-tracker.md`	original Phew brief (bids-focused) + gap tracker §B.1
`specs/{procurement,product-guide,sales-proposal}-workspaces/`	domain specs (procurement most mature)

Authored S379 (2026-06-19) at Liam’s request, post ID-112 landing, ahead of the ID-111 + ID-71 (ultracode) session.