OKF / Record-Model Reframe — v2 (Concept ≠ Record)

OKF / Record-Model Reframe — v2 (Concept ≠ Record)

Supersedes: okf-corpus-reframe-synthesis.md (the “prior synthesis”). That doc was right on the big shape (two layers, displace-don’t-delete, RAG-reduced-not-removed, M2M drop) but it carried one misframe the owner has now corrected: it treated the OKF bundle as a place a Canonical-owned record id would live, and it stopped at two layers. This v2 sharpens to three layers and resolves the concept ≠ record split cleanly.

Audience: Product owner (non-developer). Plain-English first, file:line evidence inline. Grounded against (this round): Google’s actual OKF SPEC.md + the three shipped bundles (ga4 / stackoverflow / crypto_bitcoin) + the reference_agent two-pass source; Canonical’s flow.py uuid5 seed code; the full content_items 67-column DDL + 11 FK constraints in squash_baseline.sql; ast-dataflow column-writes; the id-71 persona/use-case strawman; and the real first-client corpus at /Users/liamj/Documents/development/local-fs-client/corpus (33 files, ~86k tokens, Phew Design Ltd). Date: 2026-06-27.

---

0. The headline (read this first)

Your sharpening is correct on every point, and the ground truth backs it harder than the prior synthesis did.

1. OKF is a knowledge MAP, not a record store — confirmed verbatim against Google’s spec (“the metadata, context, and curated insight that surrounds data”; non-goals explicitly disclaim storage/query infra). Every shipped concept is a table / dataset / metric / join / enum that points at its data via resource + # Citations. Google never mints a concept per row. A Q&A pair would never be an OKF concept. Your instinct is the spec’s design intent.

2. There are THREE layers, not two. The prior synthesis’s “raw provenance substrate” actually splits into the client’s raw sources (which stay with the client, the authoritative provenance anchor) and the Canonical DB records the pipeline derives from them. Plus the OKF concept bundle on top. L-raw / L-records / L-concepts. (§1)

3. The identity question splits in two, and the prior “D5 Canonical-owned frontmatter id” was the wrong seat. Concept-identity is already the file path (spec + tooling: no id key exists in any of the three shipped bundles). Record-identity is a durable DB uuid seeded on the source — and your URL branch already does this correctly; only the local-fs branch is the offender, seeding on rel_path. Stamping a Canonical PK into a client-owned bundle pollutes a portable artifact and is redundant. D5 is superseded (D5-v2). (§2)

4. content_items can be FULLY eliminated as a table. Bottom-up from the real corpus and from every persona/use-case, nothing needs a generic “content” bucket. Its 9 pipeline-written columns re-home onto source_documents + content_chunks; its ~58 IMS-fork columns drop or move; its Q&A columns are vestigial. The one irreducible thing is a column family (per-document governance / freshness / lifecycle) that lands on source_documents. (§3, §4)

5. This is a CODE refactor, not a data migration. Full-replace re-ingest runs on a new empty preview branch (owner #6); prod content_items=33 are E2E debris with 0 chunks (pipeline never completed a prod write); Phew’s 631 are legacy pre-cocoindex rows the re-ingest replaces. Zero prod data lock-in. Pre-launch is the cheapest possible moment. (§7 of clarifications, confirmed)

6. Split into Tasks along the layer seam. L-concepts+producer is one Task; the L-records refactor (content_items decomposition + record-identity seed) is another and is the irreversible-decision owner; ID-127 {127.4} shrinks to a minimal Platform-DB fixture. (§7)

---

1. The three-layer model

The single most important reframe: there are three layers, with distinct owners, and they reference each other by provenance pointers, never by duplication.

┌──────────────────────────────────────────────────────────────────────────────┐
│ L-raw  — THE CLIENT'S RAW SOURCES            owner: THE CLIENT (stays put)      │
│   local-fs folders, Notion, Gmail, the .docx/.pdf originals, council DPIAs,     │
│   the live website. The authoritative provenance / audit / citation anchor.    │
│   Canonical does NOT host this. It is the ground truth a claim traces back to.  │
└───────────────▲────────────────────────────────────────────────────────────────┘
                │  cocoindex pipeline derives ↓ (delta-only, content-hash memoised)
                │  records POINT BACK to L-raw via source locator / "Source of Information"
┌───────────────┴────────────────────────────────────────────────────────────────┐
│ L-records — CANONICAL DB RECORDS              owner: CANONICAL (operated)         │
│   source_documents (provenance substrate + version chain + RECORD-IDENTITY SEED) │
│   q_a_pairs (reusable answers)   reference_items (external evidence, one-shot)    │
│   content_chunks (retrieval grain)   entities + entity_mentions (ontology)        │
│   citations (polymorphic)   a dedicated embeddings store                          │
│   op_id stamps each row with which pipeline RUN wrote it (audit, not identity).   │
│   THIS is where high-cardinality, churny rows live. NOT in OKF.                   │
└───────────────▲────────────────────────────────────────────────────────────────┘
                │  2nd-pass enrichment agent distills ↓ (Pass-1 from L-records, Pass-2 from L-raw)
                │  concepts POINT to records via `resource:` URI + `# Citations` (record uuids)
┌───────────────┴────────────────────────────────────────────────────────────────┐
│ L-concepts — THE OKF CONCEPT BUNDLE          owner: THE CLIENT (private git repo) │
│   A distilled KNOWLEDGE MAP: ~30-50 .md concept files for THIS client —          │
│   topic concepts (GDPR, Encryption, BCDR…), product concepts (Audit System, LMS),│
│   certification concepts (ISO 27001, Cyber Essentials+), the company concept,    │
│   client/case-study concepts, metric/dataset/schema/playbook concepts.           │
│   index.md (progressive disclosure) + log.md (change history).                    │
│   Concept-identity = the file path. Concepts CITE records; they never STORE them. │
│   A Q&A pair does NOT appear here. Fits one context window (~86k tokens) → load   │
│   wholesale; keep a vector index only for the long tail.                          │
└──────────────────────────────────────────────────────────────────────────────┘

What lives in each, who owns it, how they reference each other

	L-raw	L-records	L-concepts
Contents	Client’s authoritative source files (docx/pdf/md, Notion, Gmail, live site, council DPIAs/RAs)	source_documents, q_a_pairs, reference_items, content_chunks, entities/entity_mentions, citations, embeddings	OKF .md concept files (topics, products, certs, company, case studies, metrics, datasets, schemas, playbooks) + index.md + log.md
Owner	The client (never extracted)	Canonical (operated DB)	The client (private git repo)
Mutability	Authoritative; client edits at will	Derived; full-replace re-ingest rebuilds it	Curated derivation; enrichment agent + human edits via PR
Identity	The native source locator (path, URL, message-id)	Durable DB uuid seeded on the source	The file path (concept-id = path − .md)
Cardinality	Whatever the client has	High (one row per doc/chunk/Q&A/reference)	Low (one concept per named asset, ~30-50)
References	— (the anchor)	Points up to L-raw via source locator (source_url, source_documents.storage_path, the corpus “Source of Information” column)	Points down to L-records via resource: URI + # Citations (record uuids); points across to other concepts via untyped md links

Key references between layers (the only legitimate cross-layer pointers):

• L-records → L-raw (provenance): reference_items.source_url, source_documents.storage_path + content_hash, and the per-Q&A “Source of Information” value (DDAT DPIA, Wiltshire, Slough IS RA — observed verbatim in the real corpus). These are the audit trail back to the client’s ground truth.
• L-concepts → L-records (citation): a concept’s resource: field is “a URI that uniquely identifies the underlying asset” (SPEC §4.1). For us that is a canonical://<table>/<uuid> URN (or a deep-link API URL, or a citations-table row reference — scheme is ours to define, see §2.4). This — and # Citations — is the only place a Canonical record id touches the client-owned bundle, and it touches it as a pointer value, never as the concept’s identity key.
• No layer duplicates another. L-concepts does not copy rows; it maps and cites them. L-records does not copy source bytes beyond what extraction requires for provenance. This is what makes the “we don’t extract your value / you keep your canonical files” positioning literally true.

Why three and not two: the prior synthesis fused L-raw and L-records into one “raw provenance substrate.” But L-raw stays with the client and is authoritative; L-records is Canonical’s derived projection and is disposable (full-replace re-ingest rebuilds it). Conflating them is what made “is the git bundle the canonical store?” feel like a live fork. It isn’t: L-raw is canonical, L-records is derived, L-concepts is a curated derivation of L-records+L-raw. Two of the three layers are client-owned; only the middle is Canonical-operated.

---

2. The identity split, resolved (supersedes D5)

The prior synthesis had one identity recommendation — D5: “adopt a Canonical-owned frontmatter id.” The owner suspected this was wrong. It is wrong, and here is the clean resolution.

There is no single “identity” question. There are two, in two different layers.

2.1 Concept-identity = the OKF file path (client-owned; a Canonical id here is WRONG)

• It is the path, by spec. SPEC §2: “The path of the concept’s file within the bundle, with the .md suffix removed. For example, tables/users.md has concept ID tables/users.” The reference tooling agrees: bundle/paths.py:path_to_concept_id() derives it purely from the relative path; bundle/document.py REQUIRED_FRONTMATTER_KEYS = ('type','title','description','timestamp') — no id key. Across all three shipped bundles there is zero id field in any frontmatter.
• This is correct for concepts because concepts are stable named things. A rename genuinely IS a new concept (a navigational identity, like a wiki page title). Path-as-identity is fragile only for high-churn RECORDS — and records don’t live here.
• A Canonical-owned id in the bundle is wrong on three counts: (1) redundant — the path already identifies the concept; (2) pollution — the bundle is client-owned, portable, vendor-neutral (README: “universal, vendor-neutral format”); stamping a Canonical-namespaced PK into every concept inverts ownership and contradicts the “you keep your canonical files” positioning; (3) wrong layer — it solves a record problem inside the concept layer. It is spec-legal (SPEC §4.1 permits custom keys) but legal ≠ correct.

Verdict: D5 is superseded. Do NOT stamp a Canonical record id into concept frontmatter.

2.2 Record-identity = a durable DB uuid seeded on the SOURCE (not the OKF path)

The thing that needs a durable, location-independent id is the record, and it lives in the DB, not in the bundle. The correct pattern already exists in your pipeline — it is just applied inconsistently:

• URL branch (CORRECT): flow.py:2940-2941 seeds on source identity — source_document_id = uuid5(_KH_PIPELINE_DOC_NS, f"sd:{item.url}"), reference_item_id = uuid5(… f"ri:{item.url}"), with UNIQUE(source_url) enforcing idempotency (flow.py:1272,3486). Re-ingesting the same URL is a no-op. This is exactly right.
• Local-fs branch (THE OFFENDER): flow.py:2111-2112,2270,2529,2568 seeds on the file path — sd:{rel_path}, ci:{rel_path}, qa:{rel_path}:{idx}. This is the path-coupling you want to escape — and the real corpus proves why it’s fatal: bid-library Q&A masters carry “Merged From: Q1, Q2, Q3…” and migrate across files and versions (v4.2→v4.4, “Entries Moved to Company-Wide Library”). A canonical answer changes file, grouping, and version over time; path-as-identity orphans it on every re-merge.

2.3 The concrete record-identity seed recipe

The namespace is already tenant/pipeline-scoped (_KH_PIPELINE_DOC_NS, flow.py:1640). The seed value must change from the file path to a durable source identity. Per table:

Record	Durable PK seed (the identity)	Version-thread key (NOT the PK; used to set parent_id)	Rationale
source_documents	uuid5(tenant_ns, "sd:" + content_hash) for local-fs; "sd:" + url for URL (already so)	tenant-scoped logical locator (workspace_id + canonical relative path, or the URL)	Per-version-stable + idempotent on identical re-ingest; an edited file is a new version (new hash) threaded onto the prior via the logical locator → parent_id. Matches the existing content_hash + version + parent_id columns.
reference_items	uuid5(tenant_ns, "ri:" + source_url) (already correct)	— (one-shot, never re-extracted)	UNIQUE(source_url) already gives idempotency.
q_a_pairs	uuid5(tenant_ns, "qa:" + normalised_question_hash) at corpus level	(dedup-merge maps many raw entries → one master)	Matches the corpus’s own dedup-merge model (“44 masters from 123 Q&As”). Decouples a canonical answer from whichever library file currently holds it. scope_tag[] handles workspace relevance (no junction). Decision flag: merge-cardinality (many questions → one master) means the master’s seed may need to be a stable merge-anchor rather than a single question hash — ratify in Task B.
content_chunks	uuid5(tenant_ns, "chunk:" + source_document_id + ":" + position)	—	Re-parented to source_document_id (today it FKs content_item_id with no enforced FK, so this is a clean change).

The principle: seed record-identity on the durable source assertion, never on the OKF concept path. The OKF path is concept-identity only. A concept points out to the record uuid; the record uuid never lives in the client’s repo as an identity key.

Concrete pipeline change: migrate the local-fs seed at flow.py:2111-2112,2529,2568 from {rel_path} to the source-content/locator seeds above, mirroring the already-correct URL branch. Pre-launch-cheap (full-replace re-ingest, no prod lock-in).

2.4 Where a Canonical record-id legitimately appears in the bundle

Only as a pointer value, in the spec-sanctioned slots:

• resource: frontmatter → e.g. resource: canonical://q_a_pair/<uuid> or canonical://source_document/<uuid> (SPEC §4.1: “a URI that uniquely identifies the underlying asset the concept describes”).
• # Citations body section → the record uuids a concept’s claims are sourced from (SPEC §8).
• Optionally a derived_from: custom key as a backlink.

This is categorically different from id: <uuid> as the concept’s frontmatter identity. The concept is its path; it cites record uuids. Open decision (Task A): pick the resource-URI scheme — canonical://<table>/<uuid> URN (recommended: stable, opaque, vendor-namespaced as a value not an identity) vs a deep-link API URL vs a citations-table row reference.

---

3. The content_items decomposition

content_items is a 67-column union bucket spanning 6+ unrelated concerns plus 3 vestigial roles — the IMS personal-knowledge-base fork artifact. The cocoindex pipeline (the canonical writer) writes only 9 of the 67 columns (flow.py:1178-1209); the other ~58 are written exclusively by the TS app/MCP/cron “manual KB management” surface (ast-dataflow column-writes confirms the recurring writer set: app/api/items/*, lib/ai/{classify,summarise}, lib/mcp/tools/{content,governance}, lib/queue/handlers/batch-reclassify, freshness/quality cron).

3.1 Column-by-column re-home table

Target legend: SD = source_documents · CC = content_chunks · EMB = new dedicated embeddings store · QA = q_a_pairs · RI = reference_items · DROP = delete (no home needed) · DERIVE = compute on read, do not store.

content_items column(s)	Family	Target	Notes
id	identity	becomes record-identity on each typed record	The durable uuid moves to SD/QA/RI, seeded on source (§2.3). No generic content id survives.
content	core	SD (extracted_text, already duplicated there) / CC	The byte/text body belongs to the document; chunks hold retrieval grain.
title	core	SD (filename / extracted title) / concept	Human title is a concept attribute; machine title stays on SD.
content_type	core	SD classification / concept type	The CHECK enum mixes 3 things: q_a_pair→QA; article/blog/pdf/note/document→raw SD or RI; case_study/policy/certification/methodology/capability/product_description→L-concepts (these are distilled concepts, not rows).
embedding, embedding_model	embedding	EMB (+ CC already carries one)	Re-parent content_chunks to SD; EMB is the polymorphic store that also indexes concepts.
op_id	audit	SD/CC/QA/RI (already present on these)	Per-run stamp; keep per-record; never a concept attribute (§5, op_id).
source_document_id	provenance	KEEP as the L-raw→L-records link	Lineage anchor; every typed record carries it.
primary_domain, primary_subtopic, secondary_domain, secondary_subtopic	classification	SD	RI already has primary_domain/subtopic/layer; the document is the unit classified.
classification_confidence/_reasoning/_model, classified_at, ai_keywords, suggested_title, summary, summary_data	classification	SD	Per-document classification metadata.
content_text_hash	identity/dedup	SD (content_hash, GENERATED)	Already exists on SD; feeds the record-identity seed (§2.3).
quality_score, quality_score_updated_at, previous_quality_score	quality	SD or DROP	Keep current quality_score on SD if used; drop the history columns.
citation_count	quality	DERIVE	Rollup over citations; do not store.
freshness, freshness_checked_at, previous_freshness, lifecycle_type, expiry_date, next_review_date, review_cadence_days, captured_date	freshness/lifecycle	SD (+ slim queryable lifecycle projection for concepts with expiry)	No peer table has these today — this is the irreducible column family (see §3.3).
governance_review_status, governance_review_due, reviewer_id, verified_at, verified_by, content_owner_id, publication_status	governance	SD	SD already has the closest lifecycle (status, archived_at/by); governance is per-document.
archived_at, archived_by, archive_reason, dedup_status, superseded_by	archive/dedup	SD (already has archived_at/by); superseded_by → DROP	Full-replace re-ingest removes in-DB supersession need; SD parent_id chain covers versions.
answer_standard, answer_advanced	Q&A vestige	QA	Duplicate q_a_pairs.answer_standard/advanced. NB the real corpus shows the dual-answer need is real (Standard vs Advanced product tier, not detail level) — model as scoped variants on QA, not as these two fixed columns.
source_url	provenance	RI (drop from content_items per owner #5)	Canonical external-URL home is reference_items.source_url (NOT NULL, UNIQUE-keyed).
ingestion_source, metadata (jsonb), layer	provenance/meta	SD	Carried per-document.
platform, author_name, starred, thumbnail_url, user_tags, priority, brief, detail, reference, source_domain, file_path, source_file	IMS vestige	DROP	Social/bookmark heritage of the personal-KB fork. No pipeline writer, no consumer contract.

3.2 The FK re-home plan (11 enforced + 2 unenforced)

FK referencing content_items.id	Action	Target / note
content_chunks.content_item_id (unenforced)	RE-PARENT	→ source_document_id. No enforced FK exists → clean code change. Becomes the chunk+embedding model under SD.
entity_mentions.content_item_id (unenforced)	RE-PARENT	→ source_document_id (entities extracted from a doc).
source_document_diffs.affected_content_item_id (SET NULL)	RE-PARENT	→ source_document_id. id-117 already diffs documents; natural.
classification_disputes (CASCADE)	RE-PARENT	→ SD (classification lives there).
ingestion_quality_log (CASCADE)	RE-PARENT	→ SD (quality lives there).
entity_relationships.source_item_id (SET NULL)	RE-PARENT	→ source_document_id. (See §6 open Q: if entities become concepts, this graph may leave the DB — larger than a re-home.)
feed_articles.content_item_id (SET NULL)	RE-PARENT	→ reference_item_id (feed = reference layer). Verify feed_articles writers first.
verification_history (CASCADE)	MOVE with governance	→ SD, or DROP if git/OKF history suffices.
content_history (SET NULL)	DROP or move	git-versioned OKF + SD parent_id chain likely replaces it.
read_marks (CASCADE)	DROP	IMS reading-tracker.
content_item_workspaces (CASCADE)	DROP	Already ratified (prior decision (b)); query-time scope_tag overlap replaces it.
content_items.superseded_by self-FK (SET NULL)	DROP	Full-replace re-ingest removes in-DB supersession.
citations.cited_content_item_id (CASCADE)	HARD — extend the contract	The load-bearing one. See §3.4.

3.3 The irreducible residual (a column family, not a table)

After redistribution, no residual NEEDS a content_items-shaped generic table. The only thing with no current typed home is the per-document governance / freshness / quality / dedup / lifecycle family. Its correct landing zone is source_documents, which already carries status, archived_at/by, content_hash, and a parent_id version chain — the closest existing lifecycle surface. reference_items is deliberately write-policy-free by design (ID-75 BI-16) and stays so; q_a_pairs already carries publication_status/superseded_by/valid_from/valid_to.

Open sub-decision (Task B): land this family directly on source_documents, or in a slim 1:1 source_document_governance sidecar (keeps SD lean; mirrors the “concepts point to records” idea where some lifecycle could even live as OKF frontmatter for human-curated cadence). Note: a scheduled “what cert expires in 60 days” sweep (Priya / O6) needs a queryable column — a .md frontmatter value alone cannot power it. So lifecycle concepts (certs, policies) need a queryable lifecycle row keyed by record-identity in addition to their concept.

3.4 The one hard re-home: citations

citations is already its own polymorphic table (good), but cited_target_kind is an ENUM of only {content_item, q_a_pair} (q_a_pair currently DORMANT v1, bl-74) with a hard CHECK (citations_cited_one_of_chk). Live consumer: cite_content (lib/mcp/tools/procurement.ts). Eliminating content_items requires:

1. Extend the enum + columns + CHECK to {q_a_pair (activate), reference_item, source_document, concept}.
2. For DB-record targets, the citation target id = the new record-identity uuid (§2.3) — or cite_content loses its anchor.
3. For concept targets, the citation target = the concept path (concept-identity).

This is the single consumer-contract-load-bearing migration and must be ratified with the record-identity seed. It is the reason Task B owns the irreversible decisions.

3.5 Can content_items be fully eliminated? YES.

After §3.1–§3.4: pipeline core → source_documents + content_chunks; IMS columns → DROP; Q&A columns → q_a_pairs; URL → reference_items; the governance/lifecycle family → source_documents; distilled content_types → L-concepts. No role remains that the typed records + concepts cannot absorb. The table is eliminated, not renamed, not displaced. (This is a step beyond the prior synthesis’s “displace, don’t delete” — the owner’s concept≠record refinement plus the full-replace re-ingest is what makes full elimination cheap and correct now.)

---

4. The minimal record model (derived from personas)

Derived bottom-up from the id-71 persona/use-case strawman (O1–O9) and the real corpus, the smallest typed-entity set in L-records that satisfies every persona — with no generic content_item:

#	Entity	Role	Persona / use-case it serves
1	source_documents	Provenance substrate + version lineage + record-identity seed + governance/lifecycle host	O3 trust/diff (Rachel), audit anchor for every answer’s citation (James/Tom O1/O7), the L-raw→L-records boundary
2	q_a_pairs	Reusable answers — the bid/form flywheel (corpus-level scope_tag)	O1/O2/O7 answer + assemble (James, Tom, David, Emma); the dual Standard/Advanced variant need is real (corpus)
3	reference_items	External evidence, one-shot, citeable, deliberately governance-free	O5 signal (Maya), O2 evidence (David/Priya), the long-tail retrieval target
4	content_chunks	Retrieval grain, re-parented to source_document_id	O1 fuzzy <30s answer (James/Tom) — RAG reduced, not removed
5	dedicated embeddings store	Polymorphic (owner_kind,owner_id,model,vector); replaces scattered embedding columns AND indexes OKF concepts	every semantic-search use-case + concept-map traversal; the owner’s “dedicated embeddings model”
6	citations	Polymorphic; cited_kind extended to {q_a_pair, reference_item, source_document, concept}	O1/O3 provenance (every persona); cite_content contract
7	entities + entity_mentions	Ontology grounding graph	O5 signal triggers (David), grounding answers in who/what we know
8	governance/lifecycle facet	Distributed onto SD (+ slim queryable projection for lifecycle concepts)	O6 where-exposed / O6 cert-expiry sweep (Priya/Rachel), publication state

Computed, not stored: O6 “where-are-we-exposed”, freshness coverage, “what’s in queue”, O8 guide coverage/gaps — all aggregates over governance facets on the typed records, not a stored content type. The OKF concept layer is what makes coverage/gaps computable (the producer measures which concepts populate which guide sections).

Complementary (unchanged): guides/guide_sections (a guide_section is a concept-backed view), form_templates/form_responses, change_reports, feed_articles, workspaces + 1:1 satellites.

Headless-agent parity (HC-3): the non-human goose actor consumes the identical surface. Under this model an agent “creating content” = writing a source file (→ source_document) or proposing a concept edit (→ OKF bundle PR). There is no path where an agent needs a generic content_item create verb — confirmed against the id-71 OQ-1 resolution.

Real-corpus validation: the Phew corpus resolves into exactly three record shapes — Q&A pairs, source_documents (the docx/pdf binaries), and URL references — plus a hand-built proto-OKF concept map (the “BID RESPONSE TOPIC INDEX”, ~30 topics with usage runbooks sitting on top of provenance-tagged Q&A masters). The client has already performed the concept≠record split by hand.

---

5. Direct answers to every owner question

Q1 — Is OKF a knowledge map, not a record store? (confirm/correct) CONFIRMED, definitively. SPEC §1: OKF represents “the metadata, context, and curated insight that surrounds data and systems”; its non-goals explicitly disclaim “storage, serving, or query infrastructure” and say it “references” domain schemas, “it does not subsume them.” Google’s actual shipped concept types are BigQuery Table, BigQuery Dataset, and Reference (sub-typed by tag into metrics carrying SQL formulas, joins carrying ON-clauses, and enum/lookup docs). Granularity is always per named asset, never per row. Your “tables, datasets, metrics, playbooks, runbooks” list is the spec’s design intent, confirmed against ground truth.

Q2 — Concept ≠ record? CONFIRMED. A concept is “a single unit of knowledge … one markdown document” describing a named asset or abstract idea. A data record (Q&A pair, reference_item, ingested document) is a high-cardinality DB row a concept points to via resource + # Citations. A Q&A pair would never be an OKF concept — it would never appear in a bundle. Confirmed by the spec definition, every shipped example (per-table/per-metric, never per-row), AND your real corpus (the bid library already separates a topic index from the Q&A entries).

Q3 — Is a Canonical id namespace inside the client-owned bundle wrong? What’s the correct seed? YES, wrong seat — supersede D5. Concept-identity is already the path (spec + tooling: no id key in any of the three bundles). A Canonical PK in client-owned frontmatter is redundant, pollutes a vendor-neutral artifact, and solves a record problem in the concept layer. The correct record-identity seed = a durable tenant-namespaced DB uuid seeded on the SOURCE — uuid5(_KH_PIPELINE_DOC_NS, "sd:"+content_hash) for local-fs documents, "ri:"+source_url for references (already correct), "qa:"+question_hash corpus-level for Q&A. Your URL branch already does this right (flow.py:2940-2941); the local-fs branch is the offender (seeds rel_path at flow.py:2111-2112,2529,2568) and must migrate. The Canonical record-id may appear in the bundle only as a resource:/# Citations pointer value, never as concept identity (§2.4).

Q4 — Column redistribution: embeddings / classification / freshness / governance.

• Embeddings: CONFIRM redistribution. Already per-record (reference_items.embedding, q_a_pairs.question_embedding); content_chunks already IS a chunk-embedding model. Re-parent chunks to source_document_id (no enforced FK → clean), add a dedicated polymorphic embeddings store that also indexes concepts; content_items.embedding/embedding_model drop.
• Classification: CONFIRM, host = source_documents (the document is the unit classified), not q_a_pairs. RI already carries primary_domain/subtopic/layer.
• Freshness: CONFIRM, host = source_documents (+ a queryable lifecycle projection for concepts with expiry). No peer table has these today — this is the irreducible column family.
• Governance: CONFIRM, host = source_documents (it already has status, archived_at/by).

Q5 — op_id relevance? (what writes it, what it references) op_id is cocoindex’s per-flow run id — a uuid v4 minted at flow construction (FlowRunMeta, flow_context.py), written on every row a single pipeline run produces (content_items, source_documents, reference_items, content_chunks, entity_mentions, q_a_extractions, entity_pair_resolutions) and on pipeline_runs itself; it references pipeline_runs.op_id by logical round-trip (lib/pipeline/record-run.ts:86-89) — not a hard FK (no REFERENCES anywhere). It is provenance/audit-correlation, explicitly NOT a PK seed (flow.py:1631-1640 warns that seeding PKs on op_id would mint a new PK every run and break re-ingest idempotency). Two real uses: (1) Stage-5 op_id-scoped UPDATEs for cross-run entity resolution/dedup (pair_resolver.py); (2) rollback-by-operation + forensic log/webhook correlation. Relevance to the reframe: keep it per-record on the L-records typed tables (it already exists on SD/RI); it has no identity role and no place as an OKF concept attribute. Do not conflate op_id (run id) with record-identity (source-seeded uuid).

Q6 — Does content_items need to exist? NO. It can be fully eliminated as a table (§3.5). Every persona/use-case data need maps to a typed record, an OKF concept, a guide_section view, or a computed aggregate. The 9 pipeline columns re-home to SD+CC; the ~58 IMS columns drop/move; Q&A columns are vestigial; the only irreducible thing is the per-document governance/lifecycle column family, which lands on source_documents.

Q7 — Is the {127.4} synthetic corpus still needed given the real corpus? Mostly no — shrink it drastically. {127.4} targets local-fs-platform/corpus for the Platform-DB pipeline standup (a different destination from the real client corpus; Platform DB must not hold real client data). It is currently empty scaffolding. Keep only a minimal deterministic fixture (root .kh-workspace-map.json + workspace-map + 2-3 edge shapes) for Platform-DB CI/smoke. Use the REAL Phew corpus (local-fs-client/corpus, ~86k tokens, md+docx+pdf, real provenance metadata) as the authoritative ingestion-validation input for ID-45 on the empty preview branch — it exercises ≥3 record shapes and 3 formats far better than any synthetic could. Do not invest in an elaborate synthetic corpus.

---

6. Revised decision register

Reversibility: IRREVERSIBLE (frozen at ID-45 first write) / EXPENSIVE (large change) / CHEAP (reversible pre-launch). “Code refactor” throughout — full-replace re-ingest means no data migration (owner #6/#7).

#	Decision	Recommendation	Reversibility	Status
D5-v2	Record-identity seed	SUPERSEDES D5. Do NOT stamp a Canonical id into the client-owned bundle. Concept-identity = OKF path. Record-identity = durable tenant-namespaced DB uuid seeded on the SOURCE (content_hash/source_url/question_hash), migrating the local-fs branch off rel_path. Canonical record-id appears in the bundle only as a resource:/citation pointer.	IRREVERSIBLE / pre-launch-free. On the ID-45 critical path.	NEW — ratify
3-LAYER	L-raw / L-records / L-concepts	Ratify the three-layer model (§1): L-raw client-owned & authoritative (stays put); L-records Canonical DB (derived, disposable, full-replace); L-concepts client-owned OKF git bundle (curated derivation). Concepts cite records; no layer duplicates another.	EXPENSIVE to reverse once built.	NEW — ratify (refines prior (e) two-layer)
CI-DECOMP	Eliminate content_items	Fully eliminate the table (§3). 9 pipeline cols → SD+CC; IMS cols → DROP; Q&A → QA; URL → RI; governance/lifecycle family → SD; distilled content_types → L-concepts.	EXPENSIVE as code, but ZERO data migration pre-launch.	NEW — supersedes prior (a) “keep for v1”
CITE-EXT	Extend citations.cited_target_kind	Extend enum+CHECK to {q_a_pair (activate), reference_item, source_document, concept}; tie record targets to the D5-v2 uuid, concept targets to the path.	Contract-load-bearing; ratify WITH D5-v2.	NEW — ratify
(b)	Drop content_item_workspaces M2M	RECONFIRMED. Query-time scope_tag overlap; cancel ID-69’s v1.1 M2M-writer.	CHEAP (zero ingest writers).	reconfirmed
SRC-URL	Drop source_url from content_items only	FOLDED IN (owner #5). Stays on reference_items (NOT NULL, UNIQUE). Moot once the table is eliminated, but explicit: external-URL provenance lives on RI.	CHEAP.	reconfirmed
FULL-REPLACE	Full-replace re-ingest on empty preview branch	CONFIRMED (owner #6). ID-45 builds a new empty DB then cuts over. No prod data lock-in → content_items redesign is a code refactor.	n/a (mechanism)	confirmed fact
EMB-STORE	Dedicated polymorphic embeddings store	Build the owner’s “dedicated embeddings model” (owner_kind,owner_id,model,vector); also indexes OKF concepts. Re-parent content_chunks to SD.	CHEAP-MEDIUM (additive).	NEW — ratify
GOV-HOST	Governance/lifecycle family → source_documents	Land per-document governance/freshness/lifecycle on SD (or a 1:1 sidecar — sub-decision). Add a queryable lifecycle projection for expiry-bearing concepts.	MEDIUM.	NEW — ratify
PRODUCER	CocoIndex/agent OKF producer (2-pass)	Pass-1 source adapter over Canonical L-records (net-new, pluggable: sources/base.py); Pass-2 web-enrichment loop + gates (lift-and-shift). Reuse the ontology validator gate.	Decision CHEAP; build EXPENSIVE.	carried (was bl-367 / (f))
{127.4}	Shrink synthetic corpus	Minimal Platform-DB fixture only; real Phew corpus validates ID-45.	CHEAP.	NEW — supersedes prior {127.4} scope
RAG	Reduce, don’t remove	Keep a vector index over the OKF bundle + the long tail; the curated bundle fits one context window (~86k tokens for Phew) so load wholesale where possible.	n/a	reconfirmed

Superseded / dropped from prior register: D5 (false-premise frontmatter id) → D5-v2; prior (a) “keep content_item for v1, displace post-v1” → CI-DECOMP full elimination; prior (e) two-layer → 3-LAYER. The workspace→activity rename stays dropped (cosmetic, fights ratified naming).

---

7. Task breakdown

The owner approved splitting. The natural seam is the layer boundary plus the fixture rescope. Three Tasks; Task B owns the irreversible decisions.

Task A — OKF Concept Layer + Producer (L-concepts)

Scope: the client-owned OKF git bundle for Phew; the 2-pass enrichment agent (Pass-1 source adapter over Canonical’s typed DB — net-new, the only bespoke piece; Pass-2 web-enrichment loop + 4 gates — lift-and-shift from reference_agent); concept schema (type discriminated by tags: per Google’s pattern); index.md/log.md; the resource: URI scheme (the §2.4 open decision); a vector index over the bundle. Produces ~30-50 concepts mirroring the corpus’s existing topic index + products

• certs + company + clients. Owns: the resource-URI scheme; concept-granularity (what is a “table/metric” equivalent in this domain — a product? a topic? a metric?) informed by the id-71 strawman. Depends on: Task B’s record-identity seed (the producer points resource:/citations at record uuids) and L-records schema. Producer scaffolding can start in parallel; the pointer contract waits on B.

Task B — Record-model refactor / content_items decomposition (L-records) — IRREVERSIBLE-DECISION OWNER

Scope: eliminate content_items (§3); add the governance/lifecycle family to source_documents; build the dedicated embeddings store; re-parent content_chunks + entity_mentions to source_document_id; re-home the 11 FKs (§3.2); extend citations.cited_target_kind (§3.4); migrate the local-fs record-identity seed off rel_path to source-content/locator seeds (D5-v2); drop the M2M coupling (b) and source_url from content_items (#5). Owns the irreversible decisions: D5-v2 (record-identity seed) and CITE-EXT (citation-target contract) — both freeze at ID-45’s first write. Depends on: nothing upstream (it’s the foundation). Use GitNexus impact before touching the ~191 TS / 109 Python call sites; ast-dataflow to pin the consumer re-home.

Task C — Rescope ID-127 {127.4}

Scope: shrink {127.4} to a minimal deterministic Platform-DB fixture (root manifest + workspace-map + 2-3 edge shapes); document that the real Phew corpus is ID-45’s ingestion-validation input. Small, independent. Depends on: nothing; can land anytime.

Sequencing & dependencies

        ┌─────────────────────────────────────────────┐
        │ Task B (record-model + D5-v2 + citations)    │  ← irreversible-decision owner
        │   MUST precede ID-45's first write            │
        └───────────────┬─────────────────────────────┘
                        │ provides record-identity + L-records schema + resource targets
                        ▼
        ┌─────────────────────────────────────────────┐
        │ Task A (OKF concept layer + 2-pass producer) │  (scaffolding can start in parallel)
        └─────────────────────────────────────────────┘

  Task C (rescope 127.4) ── independent, anytime
  ID-45 (Phew re-ingest, v1 gate) ── re-opens {45.3}/{45.4} to encode the D5-v2 seed; runs on the
                                     empty preview branch; consumes the REAL corpus + Task B's seed
  goose G4 (current branch) ── NOT blocked: headless consumes the same MCP verbs (find/get/cite);
                               the substrate swap is invisible as long as the verb shapes hold.
                               Proceeds in parallel.

The single sequencing constraint that bites: Task B (specifically D5-v2 + CITE-EXT) must be ratified and landed before ID-45’s first permanent write, because the re-ingest mints the first rows whose identity grain then hardens. Everything else (Task A producer build, Task C, goose G4) runs in parallel.

---

8. Biggest risks

1. Citation contract break during content_items elimination (HIGH). cite_content is the load-bearing consumer; the cited_target_kind enum+CHECK only knows {content_item, q_a_pair}. If the enum extension and the record-identity uuid land out of step, every citation loses its anchor. Mitigation: ratify CITE-EXT with D5-v2 in Task B; tie citation targets to the new uuid from day one; do it pre-launch when no prod citations exist.

2. Record-identity seed chosen wrong, then frozen at ID-45 (HIGH, irreversible). Content-hash seed makes a moved+edited file look new; a logical-locator seed is reparenting-fragile; Q&A merge-cardinality (many entries → one master) complicates a single question-hash. Mitigation: ratify the §2.3 recipe (per-version hash PK + logical-locator version-thread) in Task B; resolve the Q&A merge-anchor sub-decision before ID-45; this is the one decision that cannot wait.

3. The Pass-1 source adapter is the only net-new producer piece — and the concept-granularity call is a product decision (MEDIUM). Deciding which typed tables become which concept types (and what the “table/metric” equivalent is in the procurement/sector-intel domain) is a product call, not a lift-and-shift. Mitigation: drive it from the id-71 persona/use-case strawman and the corpus’s existing hand-built topic index (the client already drew the concept map).

4. Governance-as-frontmatter vs governance-as-column (MEDIUM). A .md frontmatter value cannot power a scheduled expiry sweep (Priya/O6 needs a queryable date). If lifecycle is pushed entirely into concept frontmatter, the expiry sweep breaks. Mitigation: GOV-HOST keeps a queryable lifecycle row on source_documents (or a projection) for expiry-bearing concepts.

5. Entity graph relocation is larger than a FK re-home (MEDIUM, unverified). If entities become OKF concepts, entity_relationships/entity_mentions may leave the DB entirely — not a simple re-parent. Mitigation: scope this explicitly in Task B; for v1, re-parent entities to source_document_id and defer the “entities-as-concepts” question.

6. RAG-removal over-reach (MEDIUM, carried). “OKF removes RAG” is false; it reduces it for the curated layer. Dropping the vector index breaks the O1 <30s fuzzy answer and long-tail retrieval. Mitigation: keep the vector index (EMB-STORE indexes both records and concepts); the Phew bundle fits one window so load wholesale where it helps, but never assume that generalises.