Content-Model Invariants — PRODUCT

Content-Model Invariants — PRODUCT

Status: [CURRENT-CANONICAL] — NEW-S242. Sibling spec to docs/specs/0.9-canonical-pipeline/PRODUCT.md. Scope: Q1.3-Q1.N content-model invariants beyond the P-1 + P-2 baseline in canonical-pipeline §1. Built per implementation-readiness audit P6 recommendation (docs/specs/core-docs-pathway-assessment/implementation-readiness-audit-s240.md §P6) — “the per-flow content model invariants (what a content_item looks like after ingest, what fields are populated, what chunking boundary invariants apply) are not yet specified from the user perspective”.

How to use this doc

This file holds user-perspective invariants for a content_items row after the canonical pipeline has ingested it. The “user” of this surface is whoever reads content_items downstream — the MCP server surfacing rows to Claude, the search RPCs (hybrid_search, search_content_chunks), the governance UI, and any other consumer of corpus content. The invariants describe what such a consumer can rely on without prescribing the cocoindex flow internals (those live in docs/themes/canonical-pipeline/intended-architecture/02-data-flow.md §3, the WRITE-side docs/specs/cocoindex-flow-scaffolding/TECH.md, and the companion TECH.md).

Invariants are numbered (C-1, C-2, …) so the companion TECH.md can reference them by ID. Each invariant either carries explicit ratification or — for invariants gated on a pending spike — carries a §5.3-format gap flag rather than an inline TBD.

This spec builds on (does not restate) canonical-pipeline P-1 (external-folder canonical, re-ingest within polling cadence) and P-2 (the cocoindex flow is the recurring runtime ingest path; same-run queryability via MCP). (P-2’s original ExtractByLlm framing is superseded: ExtractByLlm is a cocoindex 0.3.x symbol ABSENT in the pinned 1.0.3 — the canonical 1.x extraction pattern is Path A, a custom @coco.fn wrapping the Anthropic SDK directly per docs/specs/cocoindex-flow-scaffolding/TECH.md §P-6 + sequencing §2.5.) Forward-refs to those invariants are by spec+section.

Source-of-truth pointers

• docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md — the S272 v1 master / source-of-truth precedence (where any older spec/doc conflicts, the sequencing doc wins). §1 current state; §2.1 pipeline stages; §2.5 embedder (LiteLLMEmbedder); §4 ID-49 remainder.
• docs/specs/0.9-canonical-pipeline/PRODUCT.md §1 — P-1 + P-2 baseline (Q1.1 + Q1.2 ratifications).
• docs/themes/canonical-pipeline/intended-architecture/02-data-flow.md §3 — 6-stage cocoindex topology (source walk → binary conversion → LLM extraction → embedding → entity resolution → Postgres UPSERT). §5 — op_id hybrid pattern. (NB: §3.1’s ExtractByLlm extractor naming and the openai/-prefixed embedder string are superseded — see sequencing §2.5: extraction is Path A @coco.fn per S256; embedder is LiteLLMEmbedder("text-embedding-3-large", dimensions=1024).)
• docs/themes/canonical-pipeline/intended-architecture/05-qa-flow.md §1-§3 — Q&A two-tier model; q_a_pair migration framing.
• supabase/types/database.types.ts — canonical schema source of truth (generated; never hand-edited). Consume row/enum shapes via Tables<'content_items'> / Tables<'content_chunks'> etc. Replaces the retired SCHEMA-QUICK-REFERENCE.md (ID-47.9 — see docs/reference/documentation-inventory.md). content_items column inventory + content_items_valid_content_type CHECK enum live here.
• docs/reference/entity-type-taxonomy-spec.md §1-§2 — entity classification taxonomy; universal inclusion/exclusion rules.
• lib/content/chunking.ts — heading-based chunking implementation (chunkByHeadings, MIN_CHUNK_CHARS, MIN_DOCUMENT_CHARS). Scope note (S276 amend round 2 per OQ-CMI-56-1 (c) + OQ-CMI-56-3 (i)): this app-side chunker is RETIRED in v1; content_chunks writes are owned by the cocoindex pipeline’s new chunking stage using cocoindex.ops.text.RecursiveSplitter (budget-driven via chunk_size / chunk_overlap, NOT heading-based — see amended C-10..C-14). The retired constants MIN_DOCUMENT_CHARS (500) and MIN_CHUNK_CHARS (100) do NOT translate 1:1: chunk_size is a TECH-side configuration parameter ({56.3} ratifies; may be set >=500 to preserve C-10 single-row UX for short docs), and min_chunk_size is RecursiveSplitter’s native parameter defaulting to chunk_size / 2, not a fixed 100-char floor. The file may still be referenced as the historical source-of-truth for the retired heading-split behaviour even after the executable code is removed under {56.5+} impl Subtasks. Cross-ref docs/specs/cocoindex-flow-scaffolding/TECH.md for the WRITE-side chunking-stage spec (to be amended under a separate {56.3}-adjacent tech wave).
• docs/specs/form-extraction/PRODUCT.md OQ-52-UI-UPLOAD-TENSION ratification (S274) — the directional precedent for OQ-CMI-56-1 (c). “Thin UI front-end drops file into resolved folder→workspace path so the same pipeline write applies — no second, independent writer.” Applies here too: the 8 app-side regenerateChunks callers retire in favour of the folder-drop wrapper feeding cocoindex’s chunking stage. Also see form-extraction/PRODUCT.md §OQ-52-WORKSPACE_WRITE (R3 Option A) — form_templates is a SEPARATE table from content_items (out of content_items scope per §Out of scope (v1)), but the write-ownership precedent is the same.
• docs/themes/canonical-pipeline/spike-findings/0.9-spike-S1-cocoindex-schema-coupling.md — Scenario A (managed_by="user") confirmed at source-code level. Superseded as a gating spike by docs/specs/cocoindex-flow-scaffolding/{RESEARCH,TECH}.md (S265 re-grounding against cocoindex 1.0.3 — live PROBE PASS), which proved the declare_row write path end-to-end and resolved the schema-coupling question.
• docs/specs/cocoindex-flow-scaffolding/{RESEARCH,TECH}.md — the WRITE-side companion (cocoindex flow scaffolding, T8/ID-28). RESEARCH §R0-§R4 (reactive declare_row write path + op_id-as-row-field + memo semantics); TECH §P-2 (flow scaffolding), §P-4 (op_id migration — LANDED), §P-6 (Path A extraction). This content-model spec owns the READ contract only and references this for the WRITE path.
• docs/specs/canonical-pipeline-implementation-plan/PLAN.md §4.1 T1.4 + §4.6 T6 + §4.7 T7 + §4.8 T8 — implementation task hooks.
• docs/specs/core-docs-pathway-assessment/S239-still-open-consolidation.md S241 closure addendum row 5 — Q&A migration 1-year retention RATIFIED-S241.
• CLAUDE.md gotchas: content_items.content_text_hash GENERATED ALWAYS; summary (not ai_summary) on content_items; entity classification false-positive framing; classifyContent userId UUID rule; pipeline service-account UUID.

Audience

Code authors writing readers over content_items (MCP tool handlers, search RPCs, governance jobs, UI hooks); reviewers verifying ingest-side compliance; agents consuming the corpus via MCP. Forward consumers of q_a_extractions and q_a_pairs rows are governed by docs/themes/canonical-pipeline/intended-architecture/05-qa-flow.md not this spec, except where the content_items.content_type='q_a_pair' migration boundary (T7) intersects.

Summary

After the canonical pipeline (per docs/specs/0.9-canonical-pipeline/PRODUCT.md P-1 + P-2) ingests a source file, every resulting content_items row satisfies a fixed set of population, chunking, source-attribution, embedding, content-type, and entity-classification invariants. A reader of any content_items row produced by the pipeline can rely on these invariants without inspecting the source flow, the source connector, or the application type. Invariants for the Q&A migration boundary (content_type='q_a_pair' rows already in the corpus pre-T7) carry explicit transitional semantics rather than being silently absorbed by the new pipeline.

Behaviour

Field-population invariants

C-1 When the pipeline writes a new content_items row, the row satisfies the schema-level NOT NULL contract: id (uuid, default gen_random_uuid()), title (text), content (text), content_type (varchar with CHECK enum — see C-30), created_at (timestamptz, default now()), publication_status (text, default 'published'), citation_count (int, default 0), and starred (boolean, default false) are all populated. Per the content_items Row shape in supabase/types/database.types.ts (Tables<'content_items'> — the NOT NULL columns appear without the | null union). A reader can SELECT these fields on any row without null-handling.

C-2 When the pipeline writes a content_items row, the system does NOT supply a value for content_text_hash. The column is GENERATED ALWAYS and Postgres computes md5(normalised content) on insert/update; any explicit write value is rejected with cannot insert a non-DEFAULT value into column "content_text_hash". Per CLAUDE.md “Supabase” gotcha (content_items.content_text_hash is GENERATED ALWAYS). A reader can rely on content_text_hash matching md5(normalised content) for any row that has it populated; rows pre-dating the trigger may have NULL.

C-3 When the pipeline classifies a content_items row, the row carries AI-derived classification fields populated from the same pipeline run that wrote the row: primary_domain, primary_subtopic, classification_confidence (0-1 numeric, CHECK enforced), classified_at (timestamptz), classification_reasoning (text), and classification_model (text). secondary_domain and secondary_subtopic MAY be NULL when the classifier did not surface a secondary classification. A reader filtering on classified_at IS NOT NULL selects rows that have been through classification; classified_at IS NULL indicates either pre-pipeline rows or rows still in flight.

C-4 When the pipeline computes a summary for a content_items row, the summary is written to the summary column. This column is NOT named ai_summary — that name is reserved for feed_articles.ai_summary and is intentionally separate per CLAUDE.md “Data & Architecture” gotcha (“content_items.summary (not ai_summary)”). A reader looking for AI-generated summary text on a content_items row reads summary. Structured per-field summary data (key points, quotes, etc.) lives in summary_data (jsonb) when populated.

C-5 When the pipeline writes a content_items row, the row carries lineage attribution: ingest_source (text, one of the canonical values per migration 20260428174512 COMMENT — manual, url_import, upload, upload_autosplit, mcp_create, rss_feed, bid_outcome_integration, python_url, python_markdown, qa_import, batch_reclassify) and created_by (uuid). For pipeline-originated rows (cocoindex flow per docs/themes/canonical-pipeline/intended-architecture/02-data-flow.md §3), created_by is the pipeline service-account UUID (a0000000-0000-4000-8000-000000000001) per CLAUDE.md “Data & Architecture” gotcha (“classifyContent userId must be a UUID; pipeline service account UUID, never literal strings”). A reader can route on ingest_source to distinguish manual entries from pipeline output without inspecting created_by.

C-6 When the pipeline writes a content_items row, the row’s freshness state is initialised to 'fresh' (the column default). The pipeline does NOT update freshness on subsequent re-ingest of the same source — per docs/themes/canonical-pipeline/intended-architecture/02-data-flow.md §9.1 “two separate substrates kept: cocoindex’s freshness = ingest-latency metric (operational); KH’s content_items.freshness enum = governance freshness (fresh/aging/stale/expired)”. Per §9.3 “re-ingesting a stale record does not make it fresh”. A reader treating freshness as the governance-quality signal is correct; reading it as ingest-latency is incorrect.

C-7 When the pipeline writes a content_items row, the row’s metadata jsonb column carries the structured side-channel for non-promoted fields. Columns that have been promoted out of metadata in past sessions (source_file, layer, starred — all present as first-class columns in the content_items Row of supabase/types/database.types.ts) are populated as first-class columns; the pipeline does NOT duplicate these into metadata. A reader looking for layer value reads the column, not metadata->>'layer'.

Chunking-boundary invariants

Scope note (S276 amend round 2 — supersedes S273 scope clarification per OQ-CMI-56-1 (c) ratification S276 + OQ-CMI-56-3 ratification S276 (i) cocoindex.ops.text.RecursiveSplitter): per Liam ratification S276 (round 2), the chunking writes described by C-10..C-14 are produced by the cocoindex pipeline’s new chunking stage, which uses cocoindex.ops.text.RecursiveSplitter (cocoindex-native, AST-aware via tree-sitter, budget-driven via chunk_size + chunk_overlap, NOT heading-based — see docs/specs/cocoindex-flow-scaffolding/TECH.md for the WRITE-side spec; per-call chunk_size / chunk_overlap values are ratified at {56.3} TECH amend). The S273 “two paths” framing is RETIRED: “the pipeline” in C-10..C-14 now means the cocoindex flow, not the retired app-side chunker (lib/content/chunking.ts + lib/content/chunk-store.ts). A reader still reads content_chunks rows (via the search_content_chunks RPC) for chunk-granular retrieval; the row’s provenance is now cocoindex (op_id stamped per C-13 — the load-bearing TECH-side requirement for {56.3}). A reader querying document-granular content still reads content_items (via hybrid_search / search_content). The full client corpus is re-ingested as part of the canonical-pipeline implementation, so no in-place migration of existing content_chunks rows is required (Liam-ratified S276). Critical semantic change vs the retired chunker: chunk boundaries are content-budget-driven, NOT semantic — a reader cannot rely on chunk boundaries aligning with heading boundaries (see amended C-11 and C-13’s NULL heading-derived-columns semantic).

C-10 When the cocoindex chunking stage (cocoindex.ops.text.RecursiveSplitter per OQ-CMI-56-3 ratified S276) ingests content whose total length is below the chunk_size budget configured for the stage (per {56.3} TECH; the retired 500-character threshold becomes a TECH-side configuration parameter, not a PRODUCT invariant), the content is stored as a single content_chunks row (position 0). A reader querying for a “short document” sees one chunk row, not many. The original “no splitting below 500 chars” UX from the retired app-side chunker (lib/content/chunking.ts:38 + :102, retiring under {56.5+} impl Subtasks) can be preserved by {56.3} TECH ratifying chunk_size >= 500 — the actual chosen value is recorded in TECH §2.x.

C-11 When the cocoindex chunking stage ingests content longer than the configured chunk_size, it splits the content into multiple content_chunks rows budget-bounded by chunk_size bytes with chunk_overlap bytes of overlap between adjacent chunks per cocoindex.ops.text.RecursiveSplitter semantics (.split(text, chunk_size, *, min_chunk_size=None, chunk_overlap=None, language=None) -> list[Chunk]; OQ-3 import-and-call check in TECH §6 at cocoindex==1.0.3). A reader CANNOT rely on chunk boundaries aligning with heading boundaries — chunk boundaries are content-budget-driven, not semantic. This is a deliberate semantic change vs the retired chunkByHeadings H2-default / H1-fallback behaviour (lib/content/chunking.ts:44-53, retiring under {56.5+} impl Subtasks). RecursiveSplitter is AST-aware via tree-sitter when the optional language argument is supplied; the KH markdown / plain text path does NOT require it. The chunk count for a chunked document is governed by len(content) / (chunk_size - chunk_overlap) approximately, not by heading count.

C-12 When the cocoindex chunking stage emits chunks, the short-trailing-chunk policy is governed by RecursiveSplitter’s native min_chunk_size parameter (defaults to chunk_size / 2 per the empirically verified signature; {56.3} TECH may ratify an explicit override or accept the default). The retired app-side MIN_CHUNK_CHARS=100 floor (lib/content/chunking.ts:172-216, retiring under {56.5+} impl Subtasks) is NOT preserved — the new minimum is a function of the ratified chunk_size, not a fixed 100-char absolute. A reader CAN rely on no content_chunks row being strictly below the ratified min_chunk_size (TECH §2.x) except for the C-10 single-row case (whole document was below chunk_size) or rows pre-dating the cocoindex chunking stage’s first production run.

C-13 When the cocoindex chunking stage writes a chunk, it writes a content_chunks row persisted in the dedicated content_chunks table (verified against supabase/types/database.types.ts). The schema-level invariant — columns exist, FK to content_items — STAYS unchanged: position (int, NOT NULL, 0-indexed), content_item_id (uuid, FK → content_items.id, content_chunks_content_item_id_fkey), content (text), char_count (int), word_count (int), embedding (vector(1024), nullable), created_at / updated_at (timestamptz). op_id is a NEW required column (currently ABSENT from content_chunks per supabase/types/database.types.ts; the P-4 migration supabase/migrations/20260521203414_t8_op_id_propagation.sql added op_id to content_items / q_a_extractions / source_documents / pipeline_runs only). The {56.3} TECH amend MUST add a new migration extending op_id to content_chunks (ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS op_id uuid NULL + partial B-tree index WHERE op_id IS NOT NULL matching the P-4 pattern); this is a NEW TECH-side requirement surfaced by the OQ-CMI-56-1 (c) ratification, not a property of the live schema. The semantic content of the heading-derived columns CHANGES under OQ-CMI-56-3 ratification (S276): heading_text (text, nullable), heading_level (int, nullable), heading_path (text[]), and parent_chunk_id (uuid, nullable — self-referential FK) are NULL on every chunk row the cocoindex stage emits — RecursiveSplitter’s budget-split algorithm does not preserve semantic heading boundaries, so no heading-derived value exists to stamp. A reader querying these columns sees NULL on cocoindex-emitted chunks; rows from any prior population path (e.g. q_a_pair soft-archive or pre-cutover backfill rows surviving in the corpus) MAY retain populated values. Disposition of these columns — keep as nullable / drop / re-purpose — is pending {56.3} TECH Planner + Liam decision (see RESEARCH §6 OQ-CMI-56-4). [GAP-CMI-001 RESOLVED, S273; provenance reframed S276 round 1; heading-derived-column semantic flagged S276 round 2]: the persistence target is a dedicated content_chunks table (dedicated columns), not a metadata jsonb side-channel and not columns on content_items. S276 amend per OQ-CMI-56-1 (c) + OQ-CMI-56-3 (i): the row is written by the cocoindex flow via declare_row (cross-ref docs/specs/cocoindex-flow-scaffolding/TECH.md for the WRITE-side spec), with op_id stamped on every emitted content_chunks row once the {56.3} migration lands (cross-ref C-21 for the parent-row pattern). A content_chunks row’s op_id correlates back to the cocoindex run that produced it; SELECT * FROM pipeline_runs WHERE op_id = X returns the run rollup that produced both the parent content_items row and the chunk rows under it. The C-21 memo refinement applies: a no-op re-ingest (unchanged source bytes) does NOT re-stamp op_id on the chunk rows, just as it does not re-stamp the parent content_items row. A reader queries chunk-content + position metadata directly from content_chunks (or via the search_content_chunks RPC). RPC consumer-safety note (S276 round 2): the current search_content_chunks Returns shape declares heading_text: string, heading_level: number, heading_path: string[] as NON-NULLABLE in the generated TypeScript types (supabase/types/database.types.ts lines 4404-4406, derived from the SQL function’s declared signature). Once cocoindex emits chunks with these columns NULL, typed consumers will hit strict-null violations. The {56.3} TECH amend MUST include updating the search_content_chunks SQL function signature to declare these three columns as nullable (heading_text text / heading_level integer / heading_path text[] with NULL acceptance), plus a regeneration of database.types.ts via the standard supabase gen types command per CLAUDE.md.

C-14 — RETIRED-S276 (round 2). The original invariant guarded against fenced-code-block heading-looking lines (e.g. # comment inside triple-backtick) being treated as heading boundaries by the retired chunkByHeadings marked-lexer (lib/content/chunking.ts:16-18, retiring under {56.5+} impl Subtasks). Under OQ-CMI-56-3 ratified (i) cocoindex.ops.text.RecursiveSplitter, the chunking stage does NOT look for heading lines at all — boundaries are budget-driven (C-11). The code-fence concern is moot. If {56.3} TECH ratifies the optional language argument on RecursiveSplitter.split() for any KH content path (e.g. markdown with tree-sitter-markdown grammar, or code paths), tree-sitter AST-aware splitting becomes a TECH-side property of the op call — not a separate PRODUCT invariant.

Source-attribution invariants

C-20 When the pipeline writes a content_items row, the row’s source_document_id column points to the source_documents row representing the original file in the external folder — per canonical-pipeline P-1 (external-folder canonical) + docs/themes/canonical-pipeline/intended-architecture/02-data-flow.md §2.1 + §2.2 (cocoindex source-key + content-hash idempotency). For pipeline rows, this FK is populated; for manual / non-pipeline rows, it MAY be NULL. The FK is ON DELETE SET NULL per schema, so a reader sees content_items rows persist even if their source_documents row is deleted (the lineage breaks but the corpus row remains).

C-21 When the pipeline writes a content_items row, the row carries an op_id (uuid, nullable) that correlates back to the cocoindex per-flow run that produced it — per docs/themes/canonical-pipeline/intended-architecture/02-data-flow.md §5.1 (N7 hybrid op_id pattern). [GAP-CMI-002 RESOLVED, S273]: the content_items.op_id column has LANDED (present in supabase/types/database.types.ts content_items Row; the migration is cocoindex-flow-scaffolding/TECH.md §P-4, which also added op_id to q_a_extractions, source_documents, and pipeline_runs with B-tree indexes). The value is KH-generated (run_op_id = uuid4() in app_main()), NOT cocoindex-emitted, and is written as a plain declare_row row field per cocoindex-flow-scaffolding/RESEARCH.md §R9 (correcting the older “cocoindex emits op_id” framing). A reader filtering on op_id = X selects exactly the rows written by a single cocoindex pipeline run, and SELECT * FROM pipeline_runs WHERE op_id = X returns that run’s rollup row (the Inv-12 round-trip). Memoisation refinement (cocoindex-flow-scaffolding RESEARCH §R4): op_id records “the run that produced or updated this row”, not “the most recent run that scanned it” — because the per-item fn is @coco.fn(memo=True), a no-op re-ingest (unchanged source bytes) is skipped and the row’s op_id is NOT re-stamped; only a full_reprocess=True run (e.g. the initial full-corpus ingest) re-stamps every row uniformly.

C-22 When the pipeline writes a content_items row, the governed-write audit substrate records the operation via the trigger-driven pattern per docs/themes/canonical-pipeline/intended-architecture/02-data-flow.md §5.1-§5.2 — recording the full operation including the role that performed it, distinct from created_by (which records only the user attribution). v1 substrate note (cocoindex-flow-scaffolding/TECH.md §P-5, P-OQ1): at v1 the audit-observability path for pipeline-driven writes is structured RAISE LOG lines on the Cloud Run sidecar log surface, NOT a populated audit_log table — audit_log table population is [DEFERRED-v1.1]. So a reader auditing “who wrote this row” at v1 uses the op_id correlation (C-21) plus the structured-log surface; the trigger-driven audit_log table contract is the v1.1 target. (The op_id columns + per-failure logs already cover provenance forensics — see C-21.) S276 amend: the Stage-5 entity-resolution UPDATE phase (docs/specs/stage-5-entity-resolution/PRODUCT.md Inv-1, Inv-2) deliberately breaks the managed_by=USER row-only contract — a flow-scope post-fan-out UPDATE pass that issues UPDATE entity_mentions SET canonical_name = … is platform-owned, not a declare_row write. The trigger-driven audit substrate still records the UPDATE; no v1 substrate change. Stage-5 UPDATEs are op_id-scoped (Stage-5 Inv-5), so a reader correlating entity_mentions.op_id to a pipeline_runs row still gets the round-trip; the memo-respecting op_id semantic (Stage-5 Inv-7) means Stage-5 only re-stamps op_id on rows whose canonical_name it actually changes.

C-23 When a content_items row references a source_documents row whose workspace_id is NULL (admin-shared binary per Q-OQR1-09; docs/themes/canonical-pipeline/intended-architecture/02-data-flow.md §2.2), the workspace-relevance routing of the content_items row falls through to its own content_item_workspaces junction assignments. A reader filtering “show me documents this workspace can see” uses the content_item_workspaces junction, not the source document’s workspace_id, for nullable-source-workspace cases.

Embedding invariants

C-30 When the pipeline writes a content_items row, the row’s embedding column (vector(1024)) is populated via LiteLLMEmbedder("text-embedding-3-large", dimensions=1024) per docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md §2.5 + §4 (ID-49.2). The model id is the bare OpenAI string (text-embedding-3-large; litellm auto-routes text-embedding-* to OpenAI, so no openai/ routing prefix is needed) and the dimensions=1024 argument truncates the 3072-d native output to the vector(1024) column. (This supersedes the older LiteLLMEmbedder("openai/text-embedding-3-large") no-dimensions framing in 02-data-flow.md §3.1 and the S265 cocoindex-flow-scaffolding/TECH.md “KH-owned embedder” sketch — per sequencing §2.5, which reverses OQ-B and confirms LiteLLMEmbedder ships in cocoindex 1.0.3.) The embedding model identifier is recorded in embedding_model (text) and token cost in embedding_tokens (int). A reader can rely on cosine-similarity queries (the hybrid_search / search_content RPCs in supabase/types/database.types.ts) returning meaningful results on any row whose embedding IS NOT NULL. (OQ-3 empirical check, 28/05/2026, cocoindex==1.0.3: from cocoindex.ops.litellm import LiteLLMEmbedder → PRESENT, signature (model: str, **kwargs) -> None accommodates the dimensions=1024 keyword.)

C-31 When the pipeline re-ingests a content_items row whose content has not changed (cocoindex content-hash idempotency hit per docs/themes/canonical-pipeline/intended-architecture/02-data-flow.md §2.2 — @coco.fn(memo=True)), the embedding is NOT recomputed: the memoised per-item fn is skipped entirely, so no fresh declare_row fires, the existing embedding value is preserved, and embedding_tokens is not double-counted. A reader cannot infer “this row was re-ingested” from embedding presence alone — and per the C-21 memo refinement, op_id likewise is NOT re-stamped on a no-op cycle (it records the run that last materially produced/updated the row).

C-32 When the pipeline writes a content_items row for an unembeddable content type (e.g. a binary placeholder row referenced for lineage only, or a row in a flow that does not include the embedding stage), embedding MAY be NULL. Search RPCs (hybrid_search, search_content_chunks in supabase/types/database.types.ts) skip rows with embedding IS NULL in semantic-similarity branches. A reader filtering on embedding IS NOT NULL selects the searchable corpus.

C-33 When the pipeline writes a content_items row that supersedes a prior row (UC1/UC4 write-back per docs/themes/canonical-pipeline/phase-0-investigation/0.9-edit-flow-investigation.md §6 + the content_items.superseded_by column in supabase/types/database.types.ts), the new row carries its own freshly-computed embedding; the superseded row’s embedding remains intact for history/restore paths. A reader doing semantic search against current corpus (include_superseded=false, the default per S186 WP-B.1) sees only the new row.

C-34 When the pipeline ingests a Q&A pair row pre-migration (i.e. a content_items row with content_type='q_a_pair' written before the T7 data migration completes per PLAN.md §4.7 + docs/themes/canonical-pipeline/intended-architecture/05-qa-flow.md §1.2 — 0 of 395 prod q_a_pair rows have workspace assignment), the row carries an embedding over its content text per the standard pipeline. Post-T7 migration, new Q&A pairs flow to q_a_pairs with question_embedding over question_text + alternate_question_phrasings per docs/themes/canonical-pipeline/intended-architecture/05-qa-flow.md §2.1 — a different embedding shape because the Q&A retrieval target is the question, not the answer. A reader during the transition window must consult content_type to know which shape applies.

Content-type taxonomy invariants

C-40 When the pipeline writes a content_items row, content_type is one of the values enforced by the content_items_valid_content_type CHECK constraint (verified 28/05/2026 against the live migration): article, blog, pdf, note, research, other, q_a_pair, case_study, policy, certification, compliance, methodology, capability, product_description, document. Any other value is rejected by Postgres at insert/update time. A reader can pattern-match on the enum without defensive handling of out-of-band values.

C-41 When the T7 data migration completes per PLAN.md §4.7 (P-70: 395 prod content_type='q_a_pair' rows migrate to q_a_pairs with origin_kind='imported_legacy'), new ingestion does NOT write content_items rows with content_type='q_a_pair'. Q&A pairs are first-class q_a_pairs rows from that point forward; content_items.content_type='q_a_pair' describes only pre-migration rows that survive in soft-archived state. The soft-archive retention window is 1 year per S239-still-open-consolidation.md S241 closure addendum row 5 + PLAN.md §4.7 acceptance criteria — hard-delete soft-archived rows 1 year post-migration. A reader during the retention window MAY see soft-archived q_a_pair content_items; reading their content for production-Q&A purposes is incorrect (the canonical record lives in q_a_pairs).

C-42 When the T7 data migration runs, the soft-archive transformation on the source content_items row is one of: setting archived_at (timestamptz) or changing content_type to q_a_pair_archived — per PLAN.md §4.7 acceptance criteria “either flag column archived_at set OR content_type changed to q_a_pair_archived per P-70 target state ratification”. A reader of post-migration soft-archived rows checks both signals (archived_at IS NOT NULL OR content_type = 'q_a_pair_archived') to identify a migrated-and-archived row.

C-43 When a reader queries content_items for active corpus content, the default filter excludes publication_status='archived' rows (per the publication_status enum: draft / in_review / published / archived). Search RPCs (hybrid_search, search_for_bid_response, search_content_chunks in supabase/types/database.types.ts — each carries a visibility_filter?: string Arg) accept a visibility_filter argument with three values: 'default' (published only), 'all' (drafts + in_review + published, excludes archived), 'admin' (all four states). A reader that wants to include soft-archived q_a_pair rows during the T7 retention window MUST request visibility_filter='admin' explicitly.

Entity-classification invariants

C-50 When the pipeline runs entity extraction over a content_items row, the extracted entity mentions satisfy the universal inclusion/exclusion rules per docs/reference/entity-type-taxonomy-spec.md §2 — every candidate must pass: Rule 1 (Named Entity Test — proper nouns / specific named things), Rule 2 (External Reference Test — independently lookable-up), Rule 3 (Policy/Procedure/Plan Rule — internal-document suffixes excluded), and the remaining universal rules. A reader of entity mentions on a content_items row sees only those that survived universal filtering.

C-51 The classification failure mode is false positives, not type errors. Per CLAUDE.md “Data & Architecture” gotcha: “Entity classification: false positives, not type errors. The problem is extracting non-entities (policies, generic concepts, job titles), not mistyping real ones.” Per entity-type-taxonomy-spec.md §1 source-of-truth framing, the prior misclassification root cause was 79% of framework entities being wrong because the prompt extracted abstract concepts as entities. The invariant: when the pipeline classifies an entity mention, it has already rejected candidates that fail the Named Entity Test (C-50); a reader can treat surviving classifications as type-tagged real entities, not as “maybe-entities” requiring re-validation. Validation failures upstream of the classification surface as zero extracted entities for that span, not as a mis-typed entity row.

C-52 When the pipeline extracts entity mentions from a content_items row whose source explicitly names an internal organisational artefact (e.g. “Phew Information Security Policy”, “Staff Onboarding Procedure”), no entity row is emitted for that artefact — per Rule 3 + Rule 2 + entity-type-taxonomy-spec.md §2 examples. A reader of the entity-mention table for that content item sees only the externally-referenceable entities (e.g. “ISO 27001” but not “Phew Information Security Policy”); absence of an entity row for an internal artefact is the correct behaviour, not a missed extraction.

C-53 When the pipeline classifies a content_items row’s primary domain + subtopic (per C-3), the classification is over the content as a whole; entity-mention extraction (per C-50-C-52) is a separate operation producing rows in a separate table. A reader cannot infer the row’s classification from its entity mentions or vice versa; the two surfaces inform each other but are not derivable from one another at v1.

C-54 (S276 — Stage-5 cross-cut amend per RESEARCH §3 + OQ-CMI-56-2 default (a).) When a reader queries entity_mentions.canonical_name, the value is stable only AFTER pipeline_runs.status='completed' for the run that produced or updated the row (op_id matched). Within the in-flight window, the per-document phase has written a default canonical (deterministic per-document normalisation per canonicalise_entity_name); the Stage-5 UPDATE phase MAY rewrite that value to a cross-document canonical before the run completes. A reader reading mid-flight cannot rely on cross-document consistency. Cross-ref docs/specs/stage-5-entity-resolution/PRODUCT.md Inv-1 (Stage-5 runs as a flow-scope post-fan-out resolution stage), Inv-3 (canonical_name freshness on successful run), Inv-5 (Stage-5 UPDATEs ONLY rows whose op_id equals the current run’s op_id), Inv-7 (memo-respecting op_id semantic — Stage-5 only re-stamps op_id when it materially changes canonical_name).

v1 scope boundaries

C-60 When a content_items row is written by the v1 pipeline, the row is workspace-assigned via content_item_workspaces junction rather than a direct content_items.workspace_id column — per canonical-pipeline §2 P-10 (source_documents Option α) + docs/themes/canonical-pipeline/intended-architecture/04-workspace-types.md. A reader filtering by workspace JOINs through content_item_workspaces, never reads a workspace_id from content_items itself.

C-61 When a content_items row is written by the v1 pipeline, the row’s parent_id (self-referential FK on content_items for hierarchical content) is NULL unless the flow explicitly stitches hierarchy. (Chunk-to-document hierarchy is carried separately: content_chunks.content_item_id links a chunk to its parent document, and content_chunks.parent_chunk_id links a chunk to its parent chunk — see C-13. content_items.parent_id is for inter-document hierarchy, not chunk hierarchy.) Hierarchical-stitching policy is per-flow and is NOT a universal v1 invariant; consult per-flow specs (e.g. docs/themes/canonical-pipeline/intended-architecture/02-data-flow.md per-stage detail). A reader cannot rely on content_items.parent_id being populated for every document at v1.

C-62 When a content_items row enters the corpus via the v1 pipeline, the row is NOT auto-promoted from draft → published state. publication_status defaults to 'published' per schema (C-1), but flows that need a review queue (e.g. UC5 Q&A promotion per PLAN.md §4.9 T9) write 'draft' or 'in_review' explicitly. A reader looking for a “promotion state” surface reads publication_status, not a separate workflow column.

v1.1 candidates (DEFERRED — not blocking)

• [DEFERRED-v1.1] Per-content_type column-population invariants (e.g. pdf rows MUST have file_path populated) — currently no enforcement beyond CHECK enum at v1.
• [DEFERRED-v1.1] Embedding refresh policy on classification re-run (whether re-classifying a row also re-embeds) — currently independent at v1; v1.1 may codify the coupling.
• (S276 round 2 — was [DEFERRED-v1.1]) RESOLVED in v1 per OQ-CMI-56-1 (c) ratification S276 + OQ-CMI-56-3 (i) ratification S276 round 2. The cocoindex pipeline writes content_chunks rows via the new chunking stage using cocoindex.ops.text.RecursiveSplitter (budget-driven via chunk_size / chunk_overlap, NOT heading-based); the app-side chunker retires. See C-10..C-14 amended (C-14 retired-S276 round 2); lib/content/chunking.ts + lib/content/chunk-store.ts Scope note in §Source-of-truth pointers; per-call chunk_size / chunk_overlap / min_chunk_size configuration is the residual {56.3} TECH decision. No data migration required — full corpus re-ingest is part of the canonical-pipeline implementation. New [GAP-CMI-004] flagged for content_chunks heading-derived columns disposition under budget-split (default keep-nullable-legacy unless Liam intervenes).

---

Open questions

Open content-model items, retaining the construction-guide §5.3 format (category + sources checked + prerequisite work + interim semantics) under the [GAP-CMI-NNN] ID family. These are NOT inline TBDs — they are explicit gap flags. Spec readers can act on or wait for resolution; no invariant in §Behaviour depends on an unflagged gap.

[GAP-CMI-001] — Chunk-position metadata persistence target — RESOLVED (S273)

Category: Tech-spec needed (was gated on Spike #1 outcome).

Resolution: The persistence target is a dedicated content_chunks table with dedicated columns — NOT a metadata jsonb side-channel and NOT columns on content_items. Verified against supabase/types/database.types.ts: the content_chunks Row carries id, content_item_id (FK → content_items.id, content_chunks_content_item_id_fkey), content, heading_level, heading_path (text[]), heading_text, position (NOT NULL), parent_chunk_id (self-ref FK), char_count, word_count, embedding (vector(1024)), created_at, updated_at. The search_content_chunks RPC reads from this table (returning the same chunk-position fields + chunk_id + similarity + item-level fields). The “dedicated columns vs jsonb” question that Spike #1 posed is answered by the live schema; the cocoindex schema-coupling concern (managed_by="user") was further re-grounded and proven by cocoindex-flow-scaffolding/{RESEARCH,TECH}.md (S265, live PROBE PASS). See C-13.

Two-pipelines nuance — SUPERSEDED-S276: the S273 “two paths” framing — app-side chunker (lib/content/chunking.ts → content_chunks) vs cocoindex (document-granular, no chunking) — is RETIRED per OQ-CMI-56-1 (c) ratification (S276). The chunking-and-cocoindex question is settled: the cocoindex pipeline writes content_chunks rows via its new chunking stage; the app-side chunker retires. See amended C-10..C-14; §Source-of-truth pointers lib/content/chunking.ts Scope note; §v1 scope boundaries v1.1-candidates entry resolved-in-v1. The canonical-pipeline-sequencing.md §2.5 “no chunking stage by design … SplitRecursively intentionally unused” framing is also superseded — flagged for orchestrator dispatch under a separate canonical-pipeline-sequencing edit Subtask (NOT {56.2} scope; see §Out of scope (v1)).

Sources checked:

• supabase/types/database.types.ts (content_chunks Row + search_content_chunks RPC — the resolved persistence target)
• lib/content/chunking.ts:23-32 (app-side ContentChunk shape)
• docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md §2.5 (no cocoindex chunking stage by design)
• docs/specs/cocoindex-flow-scaffolding/{RESEARCH,TECH}.md (S265 re-grounding; managed_by=ManagedBy.USER declare_row write path proven — supersedes the Spike #1 gating)
• docs/themes/canonical-pipeline/spike-findings/0.9-spike-S1-cocoindex-schema-coupling.md (Scenario A source-code confirmation — now superseded by the flow-scaffolding re-grounding)

[GAP-CMI-002] — content_items.op_id column landing window — RESOLVED (S273)

Category: Migration-pending (was a bridging window for an inflight schema change).

Resolution: The content_items.op_id column has LANDED (uuid, nullable) — verified present in the content_items Row of supabase/types/database.types.ts. The landing migration is cocoindex-flow-scaffolding/TECH.md §P-4 (<timestamp>_t8_op_id_propagation.sql), which adds op_id to all four correlated tables — content_items, q_a_extractions, source_documents, pipeline_runs — each with a partial B-tree index (WHERE op_id IS NOT NULL). T2 (ID-37) and T8 (ID-28) are both done per sequencing §3-§4. The column-landing slot that was STILL-OPEN at S242 spec time (it was not enumerated in the T2 §11.3 scope) was resolved by being absorbed into T8’s op_id-propagation slice rather than T2 — exactly the “column added in T2 if not present … verify present after T2” fallback the T8 acceptance criterion anticipated. C-21 is therefore now load-bearing (not bridging).

Write-mechanism correction (important): op_id is KH-generated (run_op_id = uuid4() in app_main(), bound via FLOW_META_CTX and read inside ingest_file as current_flow_meta().op_id), written as a plain declare_row row field — cocoindex does NOT emit op_id (correcting the older 02-data-flow.md §5.1 “cocoindex emits a stable op_id” phrasing per cocoindex-flow-scaffolding/RESEARCH.md §R9). There is no bind_target(op_id=) / flow["op_id"] in cocoindex 1.0.3 (both ABSENT). The memo refinement (RESEARCH §R4) is folded into C-21: a no-op re-ingest does not bump op_id.

Read mechanism: a reader correlates by content_items.op_id = X (or any of the sibling tables’ op_id) and round-trips to pipeline_runs via SELECT * FROM pipeline_runs WHERE op_id = X. Note the v1 audit_log table is [DEFERRED-v1.1] (cocoindex-flow-scaffolding/TECH.md §P-5) — so at v1, op_id (not audit_log) is the primary write-correlation handle (see C-22).

Sources checked:

• supabase/types/database.types.ts (content_items.op_id PRESENT; also q_a_extractions/source_documents/pipeline_runs)
• docs/specs/cocoindex-flow-scaffolding/TECH.md §P-4 (the landing migration + per-row stamping) + RESEARCH.md §R4 (memo semantics) + §R9 (KH-generated, not cocoindex-emitted)
• docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md §3-§4 (T2/ID-37 + T8/ID-28 done)
• docs/themes/canonical-pipeline/intended-architecture/02-data-flow.md §5.1 + §5.3 (hybrid pattern; the “cocoindex emits op_id” phrasing is corrected by the flow-scaffolding re-grounding)

[GAP-CMI-003] — Per-application-type content-model differences

Category: Scope-boundary clarification (deferred to per-feature specs).

What is open: The 6 baseline application_types (procurement, intelligence, sales_proposal, product_guide, competitor_research, training_onboarding per docs/themes/canonical-pipeline/intended-architecture/04-workspace-types.md §3.2 + Q-OQR1-03) may surface application-type-specific content-model behaviour — e.g. procurement-flow rows capturing form_type discriminators per docs/themes/canonical-pipeline/intended-architecture/02-data-flow.md §1 (application-type-coupling paragraph: per-application-type differences emerge from the edit_intent and extraction output shape, not from separate flow branches). (NB: the substrate doc’s ExtractByLlm extractor naming is superseded by Path A @coco.fn per cocoindex-flow-scaffolding — the application-type-coupling logic is unaffected.) This spec covers the data-flow substrate (cocoindex → Cloud Run → Postgres) which is application-type-agnostic at the ingest level. Per-application-type schema details (e.g. procurement satellite-table columns) are owned by per-feature specs (e.g. docs/specs/procurement-workspaces/{PRODUCT,TECH}.md per PLAN.md §5).

Sources checked:

• docs/themes/canonical-pipeline/intended-architecture/02-data-flow.md §1 (application-type coupling is mention-only at the data-flow substrate)
• docs/specs/canonical-pipeline-implementation-plan/PLAN.md §5 (procurement-workspaces NEW spec required for the 6-column scope)
• docs/themes/canonical-pipeline/intended-architecture/04-workspace-types.md (application-type satellites)

Prerequisite work: Per-feature specs — procurement-workspaces (T4.7 per PLAN.md §5), plus the 5 reserved satellite seats (RWS spec — docs/specs/reserved-workspace-seats/PRODUCT.md S-1..S-8). This sibling content-model-invariants spec stays application-type-agnostic by design.

What the spec does in the meantime: Invariants C-1 through C-62 hold across all 6 application types. Per-application-type differences (form_type discriminators, application-specific column populations) are forward-referred to feature specs, not pre-empted here.

[GAP-CMI-004] — content_chunks heading-derived columns disposition under RecursiveSplitter — NEW S276 round 2 — RESOLVED-(a) S286

Category: Schema-disposition decision (raised by OQ-CMI-56-3 ratification (i) cocoindex.ops.text.RecursiveSplitter budget-split semantics; tracked also as RESEARCH §6 OQ-CMI-56-4 packet for Liam).

Resolution (S286 — ratified (a)): Liam ratified disposition (a) keep nullable legacy directly in session S286 (OQ decision oq-c1a83ca097293fb6). The content_chunks columns heading_text, heading_level, heading_path, and parent_chunk_id STAY in the schema and are NULL (heading_path → DB default '{}') on every chunk row the cocoindex stage emits (per amended C-13).

This matches the as-built {56.8} chunking stage in sibling worktree id56-chunk @ commit 3d5870ed: scripts/cocoindex_pipeline/flow.py:1306 runs a bare RecursiveSplitter() budget-split (no language=/AST argument), and flow.py:1329-1333 DELIBERATELY OMITS the four heading columns from the declared row — so heading_text / heading_level / heading_path / parent_chunk_id are NULL/'{}' on every emitted chunk.

Typed-consumer safety — the search_content_chunks RPC Returns nullability (TECH “Migration 2”, TECH.md:162-181 under §2.Y) and the {56.10} consumer fix-up of lib/mcp/tools/search.ts / lib/mcp/tools/content.ts / lib/mcp/formatters/search.ts — is handled in those own Subtasks/lanes, NOT in {56.15}. No schema-drop migration is required under (a); the {56.4} PLAN carries NO column-drop Subtask.

Retaining the columns is the precondition for a possible future AST heading-population route. That route is NOT being dropped to v1.1 — it is escalated as a NEW spike Subtask (go/no-go on AST heading population) tracked as OQ oq-fedd4fc673da6349.

Disposition is grounded in the full audit + §7 recommendation in docs/specs/id-56-content-model-invariants/heading-cols-disposition-RESEARCH.md.

The historical option text is retained below for traceability:

• (a) Keep as nullable legacy [RATIFIED S286] — columns stay in schema; NULL on cocoindex-emitted rows; rows from prior population paths (e.g. q_a_pair soft-archive) MAY retain populated values; search_content_chunks RPC Returns continue to surface them; consumers tolerate NULL.
• (b) Drop the columns [not chosen] — would have required a schema migration plus coordinated RPC + Zod + consumer call-site updates.
• (c) Re-purpose for AST-aware boundaries [not chosen for v1 — under spike, see OQ oq-fedd4fc673da6349] — possible future AST heading-population route, gated on the escalated spike Subtask.

Sources checked:

• supabase/types/database.types.ts (content_chunks Row + search_content_chunks RPC Returns)
• lib/mcp/tools/search.ts:575-645 (MCP search_content_chunks tool — consumer of heading_text / heading_level / heading_path)
• lib/mcp/formatters/search.ts:184 (Zod schema for search_content_chunks response shape)
• scripts/cocoindex_pipeline/flow.py:1306 + flow.py:1329-1333 (sibling worktree id56-chunk @ 3d5870ed — as-built {56.8} budget-split + heading-column OMIT)
• docs/specs/id-56-content-model-invariants/heading-cols-disposition-RESEARCH.md (disposition audit + §7 recommendation)
• RESEARCH §6 OQ-CMI-56-4 packet + §7 OQ-CMI-56-3 ratification block

Settled v1 behaviour: PRODUCT C-13 (amended S276 round 2) records the NULL-on-emit semantic explicitly. Under ratified (a), readers see NULL heading-derived values; search_content_chunks Returns continue to surface the columns; no schema change is required at v1.

---

Out of scope (v1)

• Per-cocoindex-flow content-model invariants (different flows for procurement vs sales_proposal vs intelligence) — covered by per-feature specs and per-stage tech docs per [GAP-CMI-003].
• Q&A migration mechanics (T7) — see PLAN.md §4.7 and the soft-archive runbook (Spike #8 deliverable).
• q_a_pairs and q_a_extractions content-model invariants — owned by docs/themes/canonical-pipeline/intended-architecture/05-qa-flow.md §2-§3.
• Embedding cost projection / prompt-cache verification — see PLAN.md §4.13 T13.
• Concurrent-edit edit_intent arbitration — see docs/themes/canonical-pipeline/intended-architecture/02-data-flow.md §8.3 gap flag + docs/specs/concurrent-edit-intent-arbitration/TECH.md (LANDED, S244 per sequencing §7).
• Form-template ingest (form_templates + form_template_fields) is owned by docs/specs/form-extraction/ (Task ID-52); its rows do NOT live in content_items and the form-extraction WRITE path is out of scope of this spec. C-N invariants here describe content_items and content_chunks only. (Cross-link only — the form-extraction R3 Option A pipeline-owned write is the spiritual precedent for the OQ-CMI-56-1 (c) ratification per §Source-of-truth pointers.)
• (S276 round 2 — was S276 amend; now resolved.) Algorithm choice for the cocoindex chunking stage is RATIFIED-S276 (i) cocoindex.ops.text.RecursiveSplitter per OQ-CMI-56-3 (see RESEARCH §7). Per-call chunk_size / chunk_overlap / min_chunk_size values are the residual TECH-side configuration decisions for {56.3} Planner (tracked in RESEARCH §7 OQ-CMI-56-3 implications), not deferred questions blocking dispatch. The three non-chosen candidates ((ii) SeparatorSplitter, (iii) ekimetrics/adaptive-chunking v1.1 research-grade, (iv) port chunkByHeadings as custom @coco.fn) remain out of scope.
• (S276 amend.) Sequencing-doc amendment — docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md §2.5 (“no chunking stage by design … SplitRecursively intentionally unused”) needs editing to reflect the OQ-CMI-56-1 (c) ratification. NOT part of {56.2} PRODUCT amend; flagged for orchestrator dispatch under a separate canonical-pipeline-sequencing edit Subtask.

---

End of PRODUCT spec. Implementation references in ./TECH.md — the consumer READ-contract companion (authored S273; the WRITE side lives in docs/specs/cocoindex-flow-scaffolding/TECH.md).