Content-Model Invariants — TECH

Content-Model Invariants — TECH

§1. Context

This is the consumer READ contract for content_items (and its satellite content_chunks) after the canonical pipeline has written a row. It exists so a downstream reader — the MCP server, the search RPCs, the governance UI, a TanStack-Query fetcher — knows which column or RPC to read to satisfy each numbered invariant in ./PRODUCT.md, without inspecting the cocoindex flow internals.

Ownership split (load-bearing):

• WRITE side — how rows are produced (the cocoindex 6-stage flow, declare_row shape, op_id stamping, managed_by=ManagedBy.USER, the migration that landed op_id, Path A extraction, the embedder call) — is owned by docs/specs/cocoindex-flow-scaffolding/TECH.md (§P-2 flow scaffolding, §P-3 adapters, §P-4 op_id migration + stamping, §P-5 audit substrate, §P-6 Path A extraction) and grounded in docs/specs/cocoindex-flow-scaffolding/RESEARCH.md (§R0–§R9, live PROBE PASS). This spec does not restate the write path.
• READ side — this document. Per-invariant-family: the live schema column(s) / RPC the consumer reads.

Single ingest path (post-OQ-CMI-56-1 (c) S276 — supersedes the prior two-paths framing): the cocoindex pipeline is the sole writer of BOTH whole-document content_items rows AND chunk-granular content_chunks rows. The app-side chunker (lib/content/chunking.ts + lib/content/chunk-store.ts) is RETIRED in v1; the eight regenerateChunks call sites enumerated in RESEARCH §2.1 (app/api/upload/route.ts:697; app/api/ingest/url/route.ts:228; app/api/items/route.ts:257; app/api/items/[id]/route.ts:627,833; lib/mcp/tools/governance.ts:521; lib/mcp/tools/content.ts:662; lib/ingest/markdown-orchestrator.ts:78; scripts/backfill-chunks.ts:28) are rewired to a thin folder-drop wrapper that drops the file into the resolved folder→workspace path so the cocoindex flow picks it up — directional precedent: form-extraction docs/specs/form-extraction/PRODUCT.md §OQ-52-UI-UPLOAD-TENSION (S274). Chunk-granular retrieval lives at content_chunks (via search_content_chunks); document-granular at content_items (via hybrid_search / search_content). Both row classes carry the same op_id correlation handle once the content_chunks.op_id migration lands (§2.Y NEW SCHEMA MIGRATIONS below).

Canonical schema source: supabase/types/database.types.ts (generated; never hand-edited; consume via Tables<'content_items'> / Tables<'content_chunks'>). The legacy SCHEMA-QUICK-REFERENCE.md is retired (ID-47.9).

Relevant files / surfaces (with the line refs a reader needs):

• supabase/types/database.types.ts — content_items Row (op_id, embedding, superseded_by, archived_at, classification fields, summary/summary_data, layer/source_file/starred first-class), content_chunks Row + its content_chunks_content_item_id_fkey, pipeline_runs Row (op_id, items_created), and the RPCs hybrid_search / search_content / search_content_chunks / search_for_bid_response / q_a_search / q_a_get_verbatim.
• lib/content/chunking.ts:16-18,23-32,38,44-53,102,172-216 (RETIRED in v1 per §1 single-path block above — historical reference only; file removed under {56.5+} impl Subtasks) — the app-side chunker (ContentChunk shape, MIN_DOCUMENT_CHARS, MIN_CHUNK_CHARS, determineSplitLevel, code-fence guard, merge step).
• lib/query/query-keys.ts + lib/query/fetchers.ts — where any TanStack-Query reader of these surfaces registers (per CLAUDE.md data-fetching gotcha; no SWR / raw fetch in hooks).
• lib/supabase/safe (sb() / tryQuery()) — the no-silent-failure access wrapper any feature-path reader uses.

§2. Proposed changes per invariant family

This is a read-contract spec, so “proposed changes” = the mapping a consumer relies on, not new write code. Each row maps a PRODUCT invariant family → the live READ mechanism. Where a real gap or v1 caveat remains it is flagged; otherwise the mechanism is live today.

§2.1 Field-population (C-1 … C-7)

Inv	Reader reads	Live mechanism (verified database.types.ts)
C-1	NOT NULL columns on any row	content_items Row columns without the | null union: id, title, content, content_type, created_at, publication_status, citation_count, starred, dedup_status. Safe to SELECT without null-handling.
C-2	content_text_hash	Column is GENERATED ALWAYS — present on the Row but never written by a consumer; readers treat it as md5(normalised content). Pre-trigger rows MAY be NULL.
C-3	classification fields	primary_domain, primary_subtopic, secondary_domain, secondary_subtopic, classification_confidence, classified_at, classification_reasoning, classification_model (all nullable). hybrid_search surfaces classification_confidence + primary_domain + primary_subtopic in its Returns. Filter classified_at IS NOT NULL for classified rows.
C-4	summary text	summary (text) — NOT ai_summary. Structured per-field summary → summary_data (jsonb). hybrid_search Returns summary.
C-5	lineage	ingest_source (text, nullable — canonical value set per migration 20260428174512 COMMENT) + created_by (uuid, nullable; pipeline rows = service-account a0000000-0000-4000-8000-000000000001).
C-6	governance freshness	freshness (governance enum, default 'fresh') — NOT cocoindex ingest-latency. The cocoindex UPSERT does not touch it (02-data-flow.md §9).
C-7	promoted-out columns	source_file, layer, starred are first-class columns on the Row (not metadata->>'…'). Read the column.

No gap. All field-population columns are live.

§2.2 Chunking (C-10 … C-14) — cocoindex chunking stage → content_chunks (post-S276)

Inv	Reader reads	Live mechanism
C-10	short-doc single row	content_chunks — one row (position = 0) when content total bytes < the ratified chunk_size value (see §2.X RecursiveSplitter configuration; ratified value in §2.X may be set >=500 to preserve the retired MIN_DOCUMENT_CHARS=500 UX). chunk_size is a TECH-side configuration parameter under cocoindex RecursiveSplitter budget-split semantics, not a PRODUCT invariant.
C-11	budget-split chunks with overlap	content_chunks rows split byte-budget bounded by the ratified chunk_size with chunk_overlap bytes of overlap between adjacent chunks per cocoindex.ops.text.RecursiveSplitter per-call .split() semantics (§2.X). Boundaries are content-budget-driven, NOT semantic — a reader cannot rely on chunk boundaries aligning with heading boundaries. Chunk count for a chunked document ≈ len(content) / (chunk_size - chunk_overlap).
C-12	short-trailing-chunk policy	Governed by RecursiveSplitter’s native min_chunk_size parameter (defaults to chunk_size / 2 per V-11 signature pin; §2.X may ratify an explicit override). The retired app-side MIN_CHUNK_CHARS=100 fixed floor does NOT translate 1:1 — the new minimum is a function of the ratified chunk_size. A reader can rely on no content_chunks row strictly below the ratified min_chunk_size except for the C-10 single-row case or rows pre-dating the cocoindex chunking stage’s first production run.
C-13	chunk-position + provenance metadata	content_chunks dedicated columns (GAP-CMI-001 RESOLVED-S273; SUPERSEDED-S276 per OQ-CMI-56-1 (c) provenance reframe): content (text, NOT NULL), content_item_id (FK → content_items.id, content_chunks_content_item_id_fkey), position (int, NOT NULL, 0-indexed), char_count, word_count, embedding (vector(1024), nullable), created_at/updated_at — all PRESENT in supabase/types/database.types.ts. NEW required column — op_id (uuid, nullable) — currently ABSENT from the live content_chunks Row (the P-4 migration supabase/migrations/20260521203414_t8_op_id_propagation.sql added op_id to content_items / q_a_extractions / source_documents / pipeline_runs only). The {56.3} TECH amend mandates a new migration extending op_id to content_chunks per the P-4 pattern — see §2.Y NEW SCHEMA MIGRATIONS. Heading-derived columns are NULL on cocoindex-emitted chunks: heading_text (nullable in schema), heading_level (nullable), heading_path (text[]), parent_chunk_id (self-ref FK, nullable) — budget-split does not preserve heading boundaries, so no value to stamp. Rows from prior population paths (e.g. q_a_pair soft-archive) MAY retain populated heading values. Disposition is [GAP-CMI-004] (default keep-nullable-legacy per PRODUCT). Read directly or via search_content_chunks (Returns all of these + chunk_id + similarity + item-level fields item_title/item_content_type/item_primary_domain/item_primary_subtopic/item_suggested_title) — but the RPC’s Returns signature MUST be updated to declare heading_text/heading_level/heading_path as nullable before cocoindex-emitted NULLs reach typed consumers; see §2.Y NEW SCHEMA MIGRATIONS.
C-14	RETIRED-S276 round 2	The original code-fence guard was needed because heading-detection-via-regex (retired chunkByHeadings) could misfire on # comment lines inside triple-backtick fences. Budget-split with RecursiveSplitter does not look for headings, so the code-fence concern is moot. If {56.3} (or a later TECH amend) ratifies the optional language parameter on .split() for markdown / code paths, tree-sitter-aware splitting becomes a TECH-side property of the op call (§2.X) — not a separate PRODUCT invariant.

GAP-CMI-001 — SUPERSEDED-S276 (was RESOLVED-S273; provenance reframed per OQ-CMI-56-1 (c)). Persistence target = dedicated content_chunks table (not metadata jsonb, not content_items columns). The “dedicated columns vs jsonb” question Spike #1 posed is answered by the live schema; the cocoindex schema-coupling concern (managed_by="user") was further re-grounded by cocoindex-flow-scaffolding/{RESEARCH,TECH}.md (S265, live PROBE PASS). Cross-ref PRODUCT §Chunking-boundary scope note round 2. Implementation correction surfaced to readers: the live column is parent_chunk_id (a self-ref FK to a chunk row) — the PRODUCT spec’s earlier parent_position (integer index) framing is corrected in C-13.

[GAP-CMI-004] heading-derived-column disposition (NEW S276 round 2 — RESEARCH §6 OQ-CMI-56-4). Under RecursiveSplitter budget-split, heading_text / heading_level / heading_path / parent_chunk_id are NULL on every cocoindex-emitted chunk. Default disposition (a) — keep as nullable legacy — applies unless Liam intervenes at the {56.3} Checker report. Under (a) the columns stay in the schema, the search_content_chunks RPC Returns nullability update (§2.Y) is required for typed-consumer safety, and no column-drop migration is needed. Under (b) — drop the columns — a coordinated schema migration + RPC + Zod + consumer-call-site retirement is needed; surface as an additional {56.4} PLAN Subtask if ratified.

v1 caveat (DEFERRED-v1.1 — RESOLVED-in-v1 per OQ-CMI-56-1 (c) S276): the question of whether the cocoindex pipeline should additionally emit content_chunks rows is RESOLVED — cocoindex IS the writer of content_chunks rows under the new chunking stage. No data migration of existing content_chunks is required (Liam-ratified S276 — full client corpus is re-ingested as part of the canonical-pipeline implementation).

§2.3 Source-attribution (C-20 … C-23)

Inv	Reader reads	Live mechanism
C-20	source lineage FK	content_items.source_document_id (uuid, nullable; FK ON DELETE SET NULL) → source_documents. NULL for manual/non-pipeline rows.
C-21	run correlation	content_items.op_id (uuid, nullable) — LANDED (GAP-CMI-002 resolved; verified in the Row). Round-trip: SELECT * FROM pipeline_runs WHERE op_id = <value> (pipeline_runs.op_id present). See §2.7 for the memo semantics that govern what op_id means.
C-22	write audit	v1 substrate = structured RAISE LOG lines on the Cloud Run sidecar log surface; the audit_log table is [DEFERRED-v1.1] per cocoindex-flow-scaffolding/TECH.md §P-5 (P-OQ1). So at v1 the primary write-correlation handle is op_id (C-21), not an audit_log SELECT. (Stage-5 UPDATE phase deliberately breaks managed_by=USER contract; cross-ref docs/specs/stage-5-entity-resolution/PRODUCT.md Inv-2.)
C-23	nullable-source-workspace routing	When source_documents.workspace_id is NULL, route workspace relevance through the content_item_workspaces junction (see C-60), not the source doc’s workspace_id.

GAP-CMI-002 — RESOLVED. content_items.op_id (and q_a_extractions.op_id, source_documents.op_id, pipeline_runs.op_id) all landed via cocoindex-flow-scaffolding/TECH.md §P-4’s migration (<timestamp>_t8_op_id_propagation.sql), each with a partial B-tree index WHERE op_id IS NOT NULL. WRITE mechanism (do not restate, reference): op_id is KH-generated (uuid4() in app_main()) and written as a plain declare_row field — cocoindex does NOT emit it (RESEARCH §R9). The bridging text in the prior PRODUCT (audit_log / pipeline_runs fallback during the pre-T2 window) is now historical — the column is live.

§2.4 Embedding (C-30 … C-34)

Inv	Reader reads	Live mechanism
C-30	document embedding	content_items.embedding (vector(1024), nullable) + embedding_model + embedding_tokens. WRITE-side embedder = LiteLLMEmbedder("text-embedding-3-large", dimensions=1024) per canonical-pipeline-sequencing.md §2.5 + ID-49.2 (see §Verification for the OQ-3 import check). Read via cosine-similarity RPCs hybrid_search / search_content.
C-31	re-ingest is idempotent	embedding preserved (not recomputed) on a no-op re-ingest; embedding_tokens not double-counted. Memo-hit signal: pipeline_runs.items_created empty for that run (cocoindex-flow-scaffolding/TECH.md §P-3 Inv-4 test).
C-32	searchable subset	embedding IS NOT NULL selects the searchable corpus; the cosine RPCs skip NULL-embedding rows in their similarity branch.
C-33	supersession	content_items.superseded_by (uuid, nullable). New row carries its own fresh embedding; superseded row’s embedding intact. Current-corpus search uses include_superseded = false (the default on hybrid_search / search_for_bid_response).
C-34	Q&A embedding shape (transition)	Pre-T7: content_type='q_a_pair' rows carry embedding over content. Post-T7: Q&A pairs are q_a_pairs rows whose question is the retrieval target — read via q_a_search (Returns embedding_score + fulltext_score over question_text) / q_a_get_verbatim (Returns alternate_question_phrasings, answer_standard, answer_advanced, origin_kind). Consult content_type to know which shape applies.

No gap. embedding column + cosine RPCs are live. The embedder value (C-30) is a WRITE-side concern landing under ID-49.2 (cocoindex-flow-scaffolding); the READ contract (a populated vector(1024)) is unaffected by which embedder produces it.

§2.5 Content-type taxonomy (C-40 … C-43)

Inv	Reader reads	Live mechanism
C-40	content_type enum	content_items.content_type (text), constrained by content_items_valid_content_type CHECK — 15 values (verified against the live migration; enumerated in C-40). Pattern-match without out-of-band defence.
C-41	post-T7 q_a_pair semantics	After the T7 migration, no new content_items row carries content_type='q_a_pair'; surviving ones are soft-archived (1-year retention). Canonical Q&A lives in q_a_pairs.
C-42	migrated-and-archived detection	Check both archived_at IS NOT NULL OR content_type = 'q_a_pair_archived' (content_items.archived_at is a live column).
C-43	visibility filtering	publication_status enum (draft/in_review/published/archived). Only hybrid_search, search_content_chunks, and search_for_bid_response carry a visibility_filter?: string Arg ('default'/'all'/'admin'). Note (verified): search_content does NOT carry visibility_filter — a reader needing archived-row visibility must use one of the three filter-aware RPCs, not search_content. To see soft-archived q_a_pair rows during the retention window, request visibility_filter='admin'.

No gap. CHECK enum + archived_at + the three filter-aware RPCs are live. The only correction vs the prior PRODUCT prose is the explicit note that search_content lacks visibility_filter.

§2.6 Entity-classification (C-50 … C-54)

These invariants are behavioural (what survives universal entity-filtering), not column-shape contracts — the read surface is the entity-mention table, and the invariant is about absence (no row for an internal artefact is correct, not a miss).

Inv	Reader relies on	Live mechanism
C-50	filtered mentions	Entity mentions on a row have already passed the universal inclusion/exclusion rules (entity-type-taxonomy-spec.md §2).
C-51	false-positive failure mode	A surviving classification is a real entity (not a “maybe”); validation failure surfaces as zero entities for a span, not a mistyped row.
C-52	internal-artefact exclusion	No entity row for an internal org artefact (Rule 3) — absence is correct.
C-53	classification ≠ entities	content_items primary-domain/subtopic classification (C-3) and entity mentions are separate surfaces; neither is derivable from the other at v1.
C-54	mid-flight canonical_name freshness	Reader reads entity_mentions.canonical_name AFTER pipeline_runs.status='completed' AND op_id match. Live mechanism: in-flight reads see the per-document default (deterministic canonicalise_entity_name per Stage-5 Inv-3); post-completion reads see the Stage-5 cross-document canonical UPDATE per Stage-5 Inv-1 + Inv-5. Stage-5 only re-stamps op_id on rows whose canonical_name it materially changes (Stage-5 Inv-7). Cross-ref docs/specs/stage-5-entity-resolution/PRODUCT.md Inv-1/Inv-3/Inv-5/Inv-7.

No gap at the read-contract level. The entity-mention WRITE path (Path A extract_entity_mentions + Stage-5 entity resolution) lives in cocoindex-flow-scaffolding/TECH.md §P-6 + §P-2 (Stage 5 resolve_entities, which requires faiss-cpu pinned — a WRITE-side dependency gap tracked there / sequencing §7, not a READ-contract concern). A reader of mentions sees only the filtered, resolved set.

§2.7 v1-scope (C-60 … C-62) + the op_id memo refinement

Inv	Reader reads	Live mechanism
C-60	workspace assignment	content_item_workspaces junction (NOT a content_items.workspace_id column — there is none). JOIN through the junction.
C-61	hierarchy	content_items.parent_id (self-ref FK) for inter-document hierarchy; NULL unless a flow stitches it. Chunk-to-document hierarchy is separate: content_chunks.content_item_id + content_chunks.parent_chunk_id (C-13).
C-62	promotion state	publication_status (default 'published'); review-queue flows write 'draft'/'in_review' explicitly. No separate workflow column.

op_id memo refinement (governs C-21 + C-31 — surfaced from cocoindex-flow-scaffolding/RESEARCH.md §R4): because the per-item ingest fn is @coco.fn(memo=True), op_id records “the run that produced or updated this row”, NOT “the most recent run that scanned it”. A no-op re-ingest (unchanged source bytes) is skipped — the row’s op_id is NOT re-stamped and its embedding is NOT recomputed. Only a full_reprocess=True run (e.g. the initial full-corpus ingest) re-stamps every row uniformly. A reader doing provenance forensics on op_id must read it with this semantics: equal op_id across rows ⇒ same producing run; a row’s op_id lagging the latest run ⇒ that row was unchanged since, not missed.

Chunk-row memo semantics (post-content_chunks.op_id migration — §2.Y): the same @coco.fn(memo=True) guarantee extends to chunk rows. A no-op re-ingest does NOT re-stamp op_id on chunk rows, and chunks for unchanged source bytes are not re-emitted (the parent per-item fn is skipped wholesale, so the chunking-stage @coco.fn does not fire for that source). A reader correlating chunk-row op_id to its parent content_items.op_id SHOULD find them equal for the same run; cross-run drift on chunk-row op_id without parent drift indicates either a full_reprocess=True run boundary or a chunking-stage-only re-run (out of v1 scope — the chunking stage is wired inside the per-item fn).

§2.X RecursiveSplitter chunking stage configuration (NEW S276 per OQ-CMI-56-3 (i))

The cocoindex chunking stage that owns content_chunks writes uses cocoindex.ops.text.RecursiveSplitter at cocoindex==1.0.3 (V-11 signature pin recorded in §6).

Symbol contract (V-11 verified):

• Constructor — (*, custom_languages: list[CustomLanguageConfig] | None = None) -> None. KH default-instantiates; custom_languages not required for the markdown / plain text path.
• Per-call — .split(text: str, chunk_size: int, *, min_chunk_size: int | None = None, chunk_overlap: int | None = None, language: str | None = None) -> list[Chunk]. chunk_size is a target in BYTES (not characters). min_chunk_size defaults to chunk_size / 2 when not supplied. chunk_overlap defaults are not surfaced via signature; if not supplied the splitter applies its own internal default — {56.3} ratifies an explicit value below for determinism. language enables tree-sitter syntax-aware splitting; the KH markdown / plain text path does NOT supply it.
• Per-Chunk shape — the per-call return is list[Chunk] where each Chunk carries text: str, start: TextPosition, end: TextPosition (frozen @dataclass(slots=True); verified against the INSTALLED cocoindex==1.0.3 surface at cocoindex/resources/chunk.py:26-36). start / end are TextPosition OBJECTS, NOT ints. Each TextPosition is byte_offset: int, char_offset: int, line: int, column: int (cocoindex/resources/chunk.py:8-22). byte_offset and char_offset DIVERGE on multibyte text — a chunk whose preceding text contains any non-ASCII (UTF-8 multibyte) characters will have byte_offset > char_offset at that boundary. Any consumer code that treats chunk.start / chunk.end as plain ints (e.g. chunk.start used as a character index, or int(chunk.end) - int(chunk.start) for a length) is WRONG — it must read the explicit field: chunk.start.char_offset for a Python string index, chunk.start.byte_offset for a byte index. KH content_chunks writes derive char_count from len(chunk.text) (not from offset arithmetic), so the offset divergence does not affect the v1 row shape — but the spec records it so the {56.8} impl never mis-uses the positions. The prior [GAP-CMI-TECH-001] “start: int, end: int” framing was an empirical error corrected here (see §6 verification block).

Ratified configuration (RECOMMENDED defaults — flagged for Liam ratification at {56.3} Checker report as OQ-CMI-56-5):

• chunk_size — RECOMMEND 2000 bytes (~500 tokens at the 4-bytes-per-token rule-of-thumb; close to the retired MIN_DOCUMENT_CHARS=500 short-doc threshold’s spiritual replacement; well within text-embedding-3-large’s 8192-token context limit). Flagged for Liam.
• chunk_overlap — RECOMMEND 200 bytes (10% of chunk_size; balances retrieval recall vs token cost). Flagged for Liam.
• min_chunk_size — RECOMMEND accept the chunk_size / 2 default (no explicit override); preserves spec ambiguity-budget. Flagged for Liam.
• language — KH markdown / plain text path does NOT supply; tree-sitter AST-awareness deferred to v1.1 if attractive (see C-14 RETIRED note).

Write integration: the chunking stage @coco.fn calls RecursiveSplitter().split(content, chunk_size=2000, chunk_overlap=200), then for each Chunk writes a content_chunks row via declare_row(row={...}) with op_id stamped from current_flow_meta().op_id (cross-ref cocoindex-flow-scaffolding/TECH.md §P-2 declare_row pattern + §P-4 op_id pattern; the chunking stage @coco.fn is itself wired inside the per-item ingest_file body so the parent memo cascade applies — see §2.7 chunk-row memo semantics). Heading-derived columns (heading_text, heading_level, heading_path, parent_chunk_id) are NOT supplied — they default to NULL per the live schema (and per [GAP-CMI-004] disposition (a) keep-nullable-legacy).

Read integration: unchanged — search_content_chunks RPC + direct content_chunks SELECT, subject to the RPC signature nullability update in §2.Y.

§2.Y Schema migrations required (NEW S276)

Two coordinated migrations follow from the OQ-CMI-56-1 (c) + OQ-CMI-56-3 (i) ratification. Both MUST land BEFORE the cocoindex chunking stage produces production rows.

Migration 1 — content_chunks.op_id add (extends P-4 pattern). Mirror the P-4 reference migration supabase/migrations/20260521203414_t8_op_id_propagation.sql (cross-ref cocoindex-flow-scaffolding/TECH.md §P-4). New migration filename: <timestamp>_id56_content_chunks_op_id.sql. DDL:

SET search_path = public, extensions;

ALTER TABLE public.content_chunks
  ADD COLUMN IF NOT EXISTS op_id uuid NULL;
CREATE INDEX IF NOT EXISTS idx_content_chunks_op_id
  ON public.content_chunks (op_id)
  WHERE op_id IS NOT NULL;
COMMENT ON COLUMN public.content_chunks.op_id IS
  'Cocoindex per-flow op_id stamped by the chunking stage; ID-56.3 (extends docs/specs/cocoindex-flow-scaffolding/TECH.md §P-4 pattern to content_chunks per OQ-CMI-56-1 (c) S276).';

Regenerate database.types.ts post-migration per CLAUDE.md (/opt/homebrew/bin/supabase gen types typescript --project-id rovrymhhffssilaftdwd --schema public > supabase/types/database.types.ts). Verification step in §6: post-regen, the content_chunks Row shows op_id: string | null.

Migration 2 — search_content_chunks RPC signature nullability. The current search_content_chunks SQL function declares its Returns columns heading_text text, heading_level integer, heading_path text[] as NON-nullable (verified at supabase/types/database.types.ts:4404-4406 — generated as heading_text: string, heading_level: number, heading_path: string[]). Under cocoindex chunking these columns are NULL on every emitted row; typed consumers will hit strict-null violations. The migration updates the function’s RETURNS TABLE (...) clause to declare these three columns as nullable. Concrete shape:

SET search_path = public, extensions;

-- Find current definition first:
--   SELECT pg_get_functiondef('search_content_chunks(...)'::regprocedure);
-- Then redeclare the function with the RETURNS TABLE clause updated to:
--   ...
--   heading_text text,         -- nullable per SQL default
--   heading_level integer,     -- nullable per SQL default
--   heading_path text[],       -- nullable per SQL default
--   ...
-- (SQL columns are nullable by default; the existing declaration may use NOT NULL
-- inline or COALESCE the columns to non-null defaults — remove either to allow NULL.)

The concrete SQL is a mechanical exercise for the impl Subtask ({56.5+} scope per PLAN); this spec records the requirement. Regenerate database.types.ts post-migration; verification step in §6: post-regen, the search_content_chunks Returns shows heading_text: string | null, heading_level: number | null, heading_path: string[] | null.

Consumer-side fix-up scope (NOT migrations, but follow-on impl): the typed consumers of search_content_chunks Returns — lib/mcp/tools/search.ts:575-645, lib/mcp/tools/content.ts:182, lib/mcp/formatters/search.ts:184 (the Zod schema for the response shape) — MUST be updated to accept the new nullable types and to handle NULL heading-derived values gracefully (e.g. UI surfaces displaying heading_path show an empty path or “(unstructured chunk)” placeholder). bun run build will surface every type-strict consumer; the {56.4} PLAN should include a dedicated consumer-audit Subtask.

DDL discipline reminder (CLAUDE.md): both migrations land via supabase migration new + supabase db push only — never via MCP execute_sql or mcp__supabase__apply_migration. The pre-push checks (cat supabase/.temp/project-ref to verify linked project; relink if drift) apply to both.

§3. Testing and validation

This spec is a read contract; validation is the set of acceptance checks a Checker runs to confirm each invariant family’s read mechanism behaves as mapped. Most write-side acceptance tests already live in cocoindex-flow-scaffolding/TECH.md §2.10 + §P-* — referenced here, not duplicated.

PRODUCT inv	READ-contract acceptance check
C-1	Type-level: Tables<'content_items'> shows the C-1 columns without | null. Runtime: a pipeline-written row SELECTs those columns non-null.
C-2	Attempting an explicit content_text_hash write is rejected (cannot insert a non-DEFAULT value); a written row’s hash equals md5(normalised content).
C-3	A classified row has classified_at IS NOT NULL and the six classification fields populated; hybrid_search Returns surface classification_confidence/primary_domain/primary_subtopic.
C-10/C-11/C-12 (round 2)	Runtime: a re-ingested 5000-byte source produces ~3 chunks (5000 / (2000 − 200) ≈ 2.78) all sized 2000 bytes ± min_chunk_size; the last chunk MAY be smaller, bounded below by the ratified min_chunk_size (default chunk_size / 2 = 1000 per V-11). Type-level: Chunk.text (from the V-11 per-call return) is a non-empty str for every emitted chunk.
C-13 (round 2 — schema-side)	Runtime: a pipeline-produced content_chunks row has non-null op_id POST-Migration 1 (§2.Y); SELECT * FROM pipeline_runs WHERE op_id = <chunk.op_id> returns the same rollup as WHERE op_id = <parent_content_items.op_id>. Type-level: Tables<'content_chunks'> shows `op_id: string
C-13 (round 2 — heading-cols semantic)	Runtime: a cocoindex-emitted content_chunks row has NULL on heading_text / heading_level / heading_path / parent_chunk_id (default [GAP-CMI-004] (a) keep-nullable-legacy). A typed consumer reading those columns handles NULL without throwing.
C-14	RETIRED — no acceptance check (row removed; invariant retired).
C-54 (Stage-5 cross-cut)	Cross-ref docs/specs/stage-5-entity-resolution/TECH.md Stage-5 integration tests (resolve-then-rewrite cycle; pipeline_runs.status='completed' gate). Acceptance check on entity_mentions.canonical_name stability: assert mid-flight reads MAY differ from post-completion reads for the same row, and post-completion reads are stable for the duration of op_id equality.
C-21 / GAP-CMI-002	A pipeline-produced row has non-null op_id; SELECT * FROM pipeline_runs WHERE op_id = <row.op_id> returns exactly one rollup row. (Cross-ref cocoindex-flow-scaffolding/TECH.md §P-4 Inv-11/Inv-12 integration tests.)
C-31 + §2.7 memo	Re-ingest unchanged → pipeline_runs.items_created empty for that run AND the row’s op_id unchanged AND embedding unchanged. (Cross-ref cocoindex-flow-scaffolding/TECH.md §P-3 Inv-4 memo-hit test.)
C-30	A pipeline-embedded row has embedding IS NOT NULL of dimension 1024 and embedding_model recorded; a cosine query returns it. (Embedder identity verified empirically in §Verification.)
C-33	Superseding write creates a new row with its own embedding; hybrid_search default (include_superseded=false) returns only the new row.
C-40	An out-of-enum content_type insert is rejected by content_items_valid_content_type.
C-42	A migrated-and-archived row satisfies archived_at IS NOT NULL OR content_type='q_a_pair_archived'.
C-43	search_content has no visibility_filter Arg (type-level); the three filter-aware RPCs do, and 'admin' surfaces archived rows.
C-60	No workspace_id column on Tables<'content_items'>; workspace membership resolves only via content_item_workspaces.

Test philosophy: these checks verify real behaviour (a row’s columns / an RPC’s Returns / a CHECK rejection), not implementation internals — per docs/reference/test-philosophy.md. Run the TypeScript suite with bun run test (not bun test); the cocoindex integration suite (op_id round-trip, memo-hit) runs under bun run test:integration per the flow-scaffolding spec.

§4. Risks and mitigations

• Reading op_id with flow-scope semantics (the trap §2.7 closes). A reader that assumes “every run stamps every touched row” will mis-attribute unchanged rows. Mitigation: §2.7 states the memo semantics explicitly; the C-21/C-31 acceptance checks assert the no-op-no-restamp behaviour.
• audit_log-table assumption at v1. C-22 readers expecting a populated audit_log table will find structured logs instead (table population DEFERRED-v1.1). Mitigation: C-22 + §2.3 redirect to op_id as the v1 correlation handle.
• search_content visibility gap. A reader using search_content for archived-row visibility gets none. Mitigation: §2.5 C-43 names the three filter-aware RPCs explicitly.
• Schema drift. Every column/RPC here is from generated database.types.ts, CI-guarded by supabase-types-parity; if a future migration renames a column, regenerate types and re-verify this table.
• RecursiveSplitter byte-budget config drift (NEW S276). The ratified chunk_size=2000 / chunk_overlap=200 defaults in §2.X are RECOMMENDED but un-empirically-tuned for KH’s UK procurement corpus. Retrieval-quality regression risk is real: under-sized chunks fragment semantic context; over-sized chunks dilute cosine similarity. Mitigation: post-migration retrieval-quality eval Subtask (a/b comparison vs the retired heading-split baseline on a representative corpus sample); scope is {56.4} PLAN. Surface as an OQ if Liam wants a pre-impl pilot rather than a post-impl eval.
• search_content_chunks RPC consumer breakage (NEW S276). Typed consumers reading the RPC pre-Migration 2 (§2.Y) will get strict-null violations on cocoindex-emitted chunks (heading_text / heading_level / heading_path are NULL but the generated types declare them non-nullable). Mitigation: Migration 2 (RPC signature update) + database.types.ts regen + bun run build all-callers fix Subtask MUST land BEFORE the cocoindex chunking stage produces production rows. Order: Migration 1 (content_chunks.op_id add) → Migration 2 (RPC nullability) → types regen → consumer-audit Subtask → chunking-stage @coco.fn wire-up.

§5. Cross-references

• ./PRODUCT.md — the numbered invariants this contract reads against.
• ./RESEARCH.md §7 ratification log (commit bc5018f5) — OQ-CMI-56-1 (c) reverse split + OQ-CMI-56-3 (i) cocoindex.ops.text.RecursiveSplitter + OQ-CMI-56-2 default (a) Stage-5 cross-cut.
• docs/specs/cocoindex-flow-scaffolding/TECH.md — the WRITE side (flow scaffolding, op_id migration §P-4 — the pattern Migration 1 §2.Y mirrors; Path A extraction §P-6; audit substrate §P-5). The authoritative write contract; this spec defers to it. The new cocoindex chunking stage @coco.fn is a cocoindex-flow-scaffolding-side amend (NOT {56.3} scope) — flagged for orchestrator dispatch under a separate write-side wave.
• docs/specs/cocoindex-flow-scaffolding/RESEARCH.md — §R4 (memo vs op_id) + §R9 (KH-generated op_id) ground §2.7.
• docs/specs/form-extraction/PRODUCT.md §OQ-52-UI-UPLOAD-TENSION (S274) — directional precedent for the OQ-CMI-56-1 (c) ratification (“thin UI front-end drops file into resolved workspace folder”). Applied 1:1 to the eight regenerateChunks caller sites enumerated in §1 single-path block.
• docs/specs/stage-5-entity-resolution/PRODUCT.md — Inv-1 (Stage-5 runs flow-scope post-fan-out), Inv-2 (deliberately breaks managed_by=USER), Inv-3 (canonical_name freshness on completed run), Inv-5 (op_id-scoped UPDATEs), Inv-7 (memo-respecting op_id semantic). Ground C-22 amend + new C-54.
• docs/specs/stage-5-entity-resolution/TECH.md — Stage-5 integration tests reused as cross-references for C-54 acceptance check.
• docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md — the v1 master (§2.5 embedder + §4 ID-49 remainder). [FLAG-S276] §2.5 “no chunking stage by design / SplitRecursively intentionally unused” framing needs amending post-{56.3} TECH ratification — NOT {56.3} scope; surface to Workflow Curator for a separate sequencing-doc amend dispatch.
• docs/themes/canonical-pipeline/intended-architecture/02-data-flow.md — §3 stage topology, §5 op_id hybrid, §9 freshness substrates.
• supabase/types/database.types.ts — canonical schema (every column/RPC named here).
• supabase/migrations/20260521203414_t8_op_id_propagation.sql — the P-4 reference migration whose pattern Migration 1 (§2.Y) extends.
• CLAUDE.md gotchas applied: DDL via supabase migration new + supabase db push only (NEVER MCP execute_sql / apply_migration); SET search_path = public, extensions on new PL/pgSQL fns; JSON.stringify(embedding) for vector RPC params; getAuthorisedClient() discriminated union (auth.success not auth.authorised; authFailureResponse(auth) helper); content_text_hash GENERATED ALWAYS (C-2); summary not ai_summary (C-4); sb()/tryQuery() no-silent-failure access; TanStack Query exclusively for hook reads; no barrel re-exports; UK English; bun run test not bun test.

§6. Verification

OQ-3 pre-ratification empirical import-and-call checks against the external-library symbols and live-schema surfaces this spec’s READ contract + S276 amend depend on:

Date	Pinned version	Symbol path checked	Result
28/05/2026	cocoindex==1.0.3 (requirements.txt:38)	cocoindex.ops.litellm.LiteLLMEmbedder	PRESENT. from cocoindex.ops.litellm import LiteLLMEmbedder resolves; inspect.signature = (model: str, **kwargs) -> None. The dimensions=1024 argument in LiteLLMEmbedder("text-embedding-3-large", dimensions=1024) (sequencing §2.5) is accepted via **kwargs.
28/05/2026	cocoindex==1.0.3 (requirements.txt:38)	cocoindex.ops.text.RecursiveSplitter (NEW S276 — OQ-CMI-56-3 (i) ratified op)	PRESENT per RESEARCH §5 V-11. Constructor `(*, custom_languages: list[CustomLanguageConfig]
29/05/2026	cocoindex==1.0.3 (installed at ~/Library/Python/3.14/lib/python/site-packages/cocoindex)	cocoindex.resources.chunk.Chunk + .TextPosition (S278 RE-PIN — corrects the S276 Checker start: int, end: int shape)	CONFIRMED — positions are TextPosition OBJECTS, NOT ints. cocoindex/resources/chunk.py:26-36: @dataclass(frozen=True, slots=True) class Chunk with text: str, start: TextPosition, end: TextPosition. cocoindex/resources/chunk.py:8-22: @dataclass(frozen=True, slots=True) class TextPosition with byte_offset: int, char_offset: int, line: int, column: int. cocoindex/ops/text.py:183-209 _convert_chunk populates byte_offset (from raw start_byte / end_byte) and char_offset (from raw start_char_offset / end_char_offset) as SEPARATE fields → they DIVERGE on multibyte text. Consumer correctness: read chunk.start.char_offset for a Python str index and chunk.start.byte_offset for a byte index; never treat chunk.start / chunk.end as a bare int (see §2.X line 129). No KH consumer code exists yet (the chunking-stage @coco.fn is unbuilt {56.8}), so this is a pre-build spec correction with zero current-code blast radius.
28/05/2026	live schema	content_items.op_id (GAP-CMI-002)	PRESENT in supabase/types/database.types.ts content_items Row (`op_id: string
28/05/2026	live schema	content_chunks table + columns (GAP-CMI-001)	PRESENT — content_chunks Row carries heading_text/heading_level/heading_path/position/parent_chunk_id/char_count/word_count/embedding/content_item_id (FK content_chunks_content_item_id_fkey → content_items). search_content_chunks RPC reads it.
28/05/2026	live schema	content_chunks.op_id (NEW S276 — Migration 1 §2.Y target)	ABSENT as of TECH amend. Verified at supabase/types/database.types.ts:510-524 — the content_chunks Row does NOT show op_id. The P-4 migration (supabase/migrations/20260521203414_t8_op_id_propagation.sql) added op_id to content_items / q_a_extractions / source_documents / pipeline_runs only. Migration 1 (§2.Y) extends the P-4 pattern to content_chunks. Post-Migration 1, re-verify PRESENT in Tables<'content_chunks'> post database.types.ts regen.
28/05/2026	live schema	search_content_chunks Returns nullability (NEW S276 — Migration 2 §2.Y target)	NON-NULLABLE on heading cols as of TECH amend. Verified at supabase/types/database.types.ts:4404-4406 — Returns declare heading_text: string, heading_level: number, heading_path: string[] as non-nullable. Migration 2 (§2.Y) updates the function signature to declare the three columns nullable. Post-Migration 2, re-verify nullable in search_content_chunks Returns post database.types.ts regen.
28/05/2026	live schema	content_items_valid_content_type CHECK (C-40)	PRESENT — 15-value enum matches C-40 exactly (verified against the live migration).

Companion-pin note (informational, not a READ-contract blocker): Stage-5 entity resolution requires faiss-cpu (not yet in requirements.txt — sequencing §7); this is a WRITE-side dependency gap owned by cocoindex-flow-scaffolding, and does not affect the entity-mention READ contract (§2.6).

[GAP-CMI-TECH-001] RE-PINNED-S278 (corrects the S276 Checker shape): the per-Chunk field shape was re-verified against the INSTALLED cocoindex==1.0.3 package surface as exactly text: str, start: TextPosition, end: TextPosition — NOT the start: int, end: int the {56.3} Checker recorded at S276. start / end are TextPosition objects (byte_offset: int, char_offset: int, line: int, column: int), and byte_offset / char_offset DIVERGE on multibyte text (see §2.X line 129 for the consumer-correctness note). Cited package paths: cocoindex/resources/chunk.py:26-36 (@dataclass(frozen=True, slots=True) class Chunk) + cocoindex/resources/chunk.py:8-22 (class TextPosition) + cocoindex/ops/text.py:154-180 (RecursiveSplitter.split(...) -> list[Chunk]) + cocoindex/ops/text.py:183-209 (_convert_chunk populating both offsets from the raw Rust chunk). The {56.5+} impl Subtask must still pin shape empirically at impl time as a defensive check (cocoindex version pin drift); the {56.4} PLAN authoring may treat this corrected shape as known.

---

End of TECH spec. READ-contract companion to ./PRODUCT.md; WRITE side in docs/specs/cocoindex-flow-scaffolding/TECH.md.