Content-Model Invariants — RESEARCH (revisit / S275-delta reconciliation)

Content-Model Invariants — RESEARCH (revisit / S275-delta reconciliation)

§0. Provenance + scope

Authored: S276 by a fresh task-planner instance (Q-PLANNER-2: first and only {56.1} Planner — successor {56.2} / {56.3} / {56.4} Planners run in independent contexts). Task: ID-56 — content-model-invariants PLAN authoring + S275-delta reconciliation + chunking strategy revisit. Subtask: {56.1} RESEARCH. Base commit: a2a6cdfe on cmux-worker-subo-id-56-research-a2a6cdfe. Status: NEW. This document is authored at S276 to reconcile the S273-ratified PRODUCT + TECH against S275 spec/ledger deltas, and to marshal evidence for the Liam-flagged TWO-paths-vs-SINGLE-path open question. Decision authority for the TWO-paths question is Liam’s (via OQ-escalation per docs/specs/oq-escalation/PRODUCT.md); this RESEARCH frames evidence, does not pre-empt.

Spec-dir convention note (CLAUDE.md ratification, S275): ID-56’s pre-existing dir docs/specs/content-model-invariants/ predates the ID-N-<slug>/ convention. It is NOT mass-migrated. RESEARCH is authored in place per the cmux brief.

Sources consumed (in full):

1. docs/specs/content-model-invariants/PRODUCT.md — ratified S273 (header <!-- RATIFIED-S242 -->; S273 reconciliation note inline).
2. docs/specs/content-model-invariants/TECH.md — ratified S273 (<!-- RATIFIED-S273 -->).
3. docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md — v1 master, refreshed S275.
4. docs/reference/task-list.json ID-56 record (deps ["28","49"], status spec_needed, no Subtasks yet).
5. lib/content/chunking.ts (220 lines) + lib/content/chunk-store.ts (~200 lines).
6. lib/ingest/markdown-orchestrator.ts:1-100,78 (the app-side full-lifecycle orchestrator that calls regenerateChunks).
7. lib/ai/embed.ts:4,42,93,108 (TS-side embedder — text-embedding-3-large truncated via Matryoshka to 1024 dims).
8. supabase/types/database.types.ts:509-571 (content_chunks table Row + Insert + FK), :4389-4416 (search_content_chunks RPC), :1725-1796 (form_templates table Row).
9. docs/specs/cocoindex-flow-scaffolding/{RESEARCH,TECH}.md (WRITE-side companion: RESEARCH §R0-§R9, especially §R1.10 ops survey; TECH §1, §P-2, §P-4, §P-6).
10. docs/specs/stage-5-entity-resolution/PRODUCT.md (Option B post-fan-out write; Inv-1 through Inv-13 examined for cross-cut into C-50/C-51/C-52).
11. docs/specs/form-extraction/{PRODUCT,TECH,PLAN}.md (R3 Option A folder→workspace; TECH §2.5/2.5a/2.6 schema migration; PRODUCT §OQ-52-UI-UPLOAD-TENSION; PLAN {52.M1b}).
12. docs/reference/task-list.json ID-52 Subtask id=18 ({52.M1b}) record.
13. WebFetch — https://cocoindex.io/blogs/index-code-base-for-rag/ (cocoindex’s own RAG example; see §2.4.1).
14. WebFetch — https://github.com/ekimetrics/adaptive-chunking (adaptive-chunking library; see §2.4.2).
15. Live empirical probes against cocoindex==1.0.3 (requirements.txt:35; see §5).

Out of scope of this RESEARCH:

• Authoring PRODUCT.md or TECH.md amendments (conditional {56.2} / {56.3} work for SEPARATE Planners per Q-PLANNER-2).
• Authoring PLAN.md ({56.4} work for a SEPARATE Planner via planning-and-task-breakdown).
• Editing canonical-pipeline-sequencing.md (only on separate Liam dispatch).
• Editing lib/content/chunking.ts or scripts/cocoindex_pipeline/flow.py (impl work; not this wave).
• Cross-Task touches on ID-28 / ID-37 / ID-49 / ID-52 / ID-53 specs.

---

§1. S275-delta reconciliation table

PRODUCT.md + TECH.md were ratified at S273 (sequencing-doc refresh). The following landings post-date that ratification per canonical-pipeline-sequencing.md §1 (“Last refreshed: S275”). For each, the verdict is AFFECTS (drives a PRODUCT/TECH amend in {56.2}/{56.3}) / NO-CHANGE (clarification only, no amend) / NEEDS-LIAM-RATIFICATION (Liam-judgment call) plus the invariants touched.

#	S275 delta	Source citation	Invariants potentially touched	Verdict	Rationale
1	Stage-5 Option B writes (entity-resolution; post-fan-out flow-scope UPDATE phase that deliberately breaks managed_by=USER row-only contract; entity_mentions.op_id-scoped UPDATEs only).	docs/specs/stage-5-entity-resolution/PRODUCT.md Inv-1, Inv-2, Inv-5, Inv-6, Inv-7, Inv-8, Inv-11, Inv-12; canonical-pipeline-sequencing.md §1, §2.1 row 5.	C-50, C-51, C-52 (entity-classification invariants); C-22 (governed-write audit); arguably C-21 (op_id semantic) via cross-table consistency.	AFFECTS (light). PRODUCT C-50/C-51/C-52 currently describe entity extraction (“the pipeline runs entity extraction over a content_items row, the extracted entity mentions satisfy …”). Stage-5 adds a SECOND write — the post-fan-out UPDATE phase that resolves cross-document canonical_name AFTER all entity_mentions rows are landed. C-50/C-51/C-52 are still true (they’re behavioural about what survives universal filtering), but a new C-53.1 / C-54 invariant should be added: “When a reader queries entity_mentions.canonical_name, the value MAY change between when the per-item phase writes the row and when Stage-5’s UPDATE phase completes; readers within the same pipeline-run window MUST treat canonical_name as stable only AFTER pipeline_runs.status='completed'”. Plus a C-22 caveat that the managed_by=USER row-only contract is now BROKEN deliberately for Stage-5 (PRODUCT.md S273 framing of “the governed-write audit substrate records the operation via the trigger-driven pattern” remains true — Stage-5 UPDATEs go through the same audit substrate). Single new invariant — surgical amend, not structural.
2	Stage-4 LANDED via ID-49.2 — LiteLLMEmbedder("text-embedding-3-large", dimensions=1024) live at flow.py:623-625. Split pgvector index ownership ratified PERMANENT S274 (pipeline writes vector(1024); migration owns HNSW).	canonical-pipeline-sequencing.md §1, §2.1 row 4, §2.5; docs/research/id49-2-trade-off-analysis-s274.md; PRODUCT.md C-30 cocoindex-flow-scaffolding/TECH.md S274 amendment.	C-30 (pipeline embedding model), C-31 (no-op re-ingest semantics), C-32 (NULL embedding fallback), C-33 (supersession + embedding refresh).	NO-CHANGE. PRODUCT.md C-30 already cites LiteLLMEmbedder("text-embedding-3-large", dimensions=1024) per sequencing §2.5 (verified at C-30 lines 84). The S275 PERMANENT-ratification of the split pgvector index ownership does NOT touch the READ contract — a reader of content_items.embedding cannot tell whether the HNSW index was migration-owned or cocoindex-owned. Optional clarification footnote in TECH §2.4 noting the cocoindex 1.0.6 upgrade-watch (informational only).
3	Form-extraction R3 Option A ratified (folder→workspace; pipeline-owned WRITE incl. instance row; retires analyse/route.ts via {52.15}).	docs/specs/form-extraction/PRODUCT.md Inv-1-Inv-26 + OQ-52-WORKSPACE_WRITE; PLAN.md {52.5} through {52.16}; sequencing §2.1 row 5 + §3 ID-50 cross-cut.	None directly on content_items invariants. Indirectly: C-60 (workspace assignment via content_item_workspaces junction) framing assumes documents are content_items; form_templates is a separate table on a DIFFERENT write path with its own workspace_id NOT NULL FK.	NO-CHANGE — but clarifying note recommended. form_templates and form_template_fields are separate tables (lines database.types.ts:1725-1796 for form_templates; the form-extraction analyse/route.ts retirement is unrelated to the chunker-bearing chunk-store.ts callers — see §2.1 below for the careful path inventory). The form-extraction R3 Option A ratification does NOT touch any C-N invariant. Recommendation: add a one-line scope-clarification note in PRODUCT §“Out of scope (v1)” reading “Form-template ingest (form_templates + form_template_fields) is owned by docs/specs/form-extraction/ (ID-52); its rows do NOT live in content_items and are out of scope of this spec”. This is a clarification, NOT an amend to any C-N.
4	M1b columns ({52.M1b} / id=18) — form_templates first-class form_type FK + deadline + issuing_organisation + evaluation_methodology.	docs/specs/form-extraction/PLAN.md:230-267; task-list.json 52.18 record; TECH §2.6d.	None. form_templates is not content_items.	NO-CHANGE. Same rationale as delta 3 — form_templates schema deltas do not touch this spec. Even the scope-clarification note in delta 3 covers this — no separate amend needed.
5	Fixture-staging (ID-49.10) — staging fixture infra + S266 corpus library + RFP form acquisition. Pending; gates ID-53.14.	canonical-pipeline-sequencing.md §4 row 49.10.	None on PRODUCT C-N invariants. May affect impl-Subtask testStrategy shape in {56.5+}.	NO-CHANGE (PRODUCT/TECH-side). Affects PLAN authoring only. {56.4} Planner should reference ID-49.10 in {56.5+} test-strategy lines where pipeline-corpus fixtures are needed; not a spec amend.
6	OQ-52-UI-UPLOAD-TENSION ratified — thin UI front-end drops file into resolved folder→workspace path so the same pipeline write applies; no second independent writer.	docs/specs/form-extraction/PRODUCT.md:429.	Indirect on the TWO-paths question (§2.2 below) — Liam has ALREADY ratified the “single-path” model for form_templates. The content-model-invariants TWO-paths question is essentially “extend the same answer to the wider content_items corpus?”.	NEEDS-LIAM-RATIFICATION (via §2 / §6.OQ-CMI-56-1). This S274 form-extraction ratification reveals the mental-model bias toward SINGLE-path. But that decision was ratified ONLY for form_templates; whether it extends to content_items + content_chunks is the open question §2 raises.

Tally: 1 AFFECTS (delta 1, Stage-5 — light, ~1-2 new invariants in PRODUCT + a paragraph in TECH); 4 NO-CHANGE (deltas 2, 3, 4, 5 — only an optional scope-clarification footnote for delta 3); 1 NEEDS-LIAM-RATIFICATION (delta 6 → §2’s TWO-paths-vs-SINGLE-path question).

If delta 1 is the only AFFECTS verdict: {56.2} PRODUCT amend is small (1-2 invariants under §Entity-classification or a new §“Stage-5 cross-document resolution” subsection). {56.3} TECH amend is a single new row in §2.6 + a paragraph in §2.7 (the op_id memo section already documents the read-window). Both can be authored together by the same dispatch wave (Q-PLANNER-2 still requires SEPARATE Planners for {56.2} and {56.3}).

---

§2. TWO ingest paths vs SINGLE ingest path (the crux of this RESEARCH)

§2.1 Current state — what the two paths actually are

PRODUCT.md §C-13 scope-clarification (S273) + TECH.md §1 “Two ingest paths” frame the live state:

Path 1 — Cocoindex pipeline (document-granular). Walks an external localfs folder, fans out per-file via coco.mount_each, runs Stage-2 binary→markdown conversion + Stage-3 Path-A LLM extraction + Stage-4 LiteLLMEmbedder + (Stage-5, in flight under ID-53) entity resolution, and writes a single whole-document content_items row per file via target.declare_row(...) (canonical-pipeline-sequencing.md §2.1; cocoindex-flow-scaffolding/RESEARCH.md §R0 + §R2 canonical shape). Does NOT chunk — canonical-pipeline-sequencing.md §2.5 (“no chunking stage by design … SplitRecursively intentionally unused”).

Path 2 — App-side ingest chunker. lib/content/chunking.ts (chunkByHeadings) → lib/content/chunk-store.ts (regenerateChunks) splits a stored content_items.content into rows of the dedicated content_chunks table (FK content_chunks.content_item_id → content_items.id), and embeds each chunk via lib/ai/embed.ts:108 (text-embedding-3-large, dimensions=1024 — same model + same vector(1024) shape as Path 1 — but via the OpenAI SDK directly, not via cocoindex’s LiteLLMEmbedder).

Live callers of regenerateChunks (the entry point to Path 2’s chunk-write):

Caller	Path	Purpose
app/api/upload/route.ts:697	App upload endpoint	Initial chunk generation post-upload.
app/api/ingest/url/route.ts:228	URL ingest endpoint	Initial chunk generation post-URL-fetch.
app/api/items/route.ts:257	Items create endpoint	Initial chunk generation on create.
app/api/items/[id]/route.ts:627,833	Items update + reclassify endpoints	Regeneration on content edit / reclassify.
lib/mcp/tools/governance.ts:521	MCP governance publish-from-draft tool	Regeneration when promoting draft → published.
lib/mcp/tools/content.ts:662	MCP content tool	Regeneration on MCP-driven content mutation.
lib/ingest/markdown-orchestrator.ts:78,@import	EP2 markdown-batch ingest (the full lifecycle orchestrator)	Initial chunk generation per file in batch ingest.
scripts/backfill-chunks.ts:28	One-off backfill	Historical-corpus chunk-population script.

Live readers of content_chunks / search_content_chunks:

Reader	Surface
lib/mcp/tools/search.ts:575-645	MCP search_content_chunks tool (#21 in the MCP tool catalogue).
lib/mcp/tools/content.ts:182	MCP content tool reads content_chunks for per-item chunk metadata.
lib/mcp/formatters/search.ts:184	MCP response-shape Zod schema for search_content_chunks.
supabase/types/database.types.ts:4389-4416	The search_content_chunks RPC itself — joins content_chunks + content_items and returns chunk-position metadata + item-level fields + visibility_filter support.

Important correction to a popular framing: the brief and prior continuation prompts say “upload and analyse routes” call the chunker. Verified empirically — they DO NOT. analyse/route.ts exists in two locations (app/api/intelligence/workspaces/[id]/flags/analyse + app/api/procurement/[id]/templates/[templateId]/analyse); neither calls regenerateChunks or chunkByHeadings. The procurement analyse/route.ts is the form-extraction template analyse route that ID-52.15 retires (form-templates path — unrelated to the chunker). The Path 2 entry surface is regenerateChunks called from the eight sites tabulated above; the chunker writes are correctly classed as “app-side ingest lifecycle” not “analyse routes”. {56.2} PRODUCT amend (if any) should correct C-13 scope-note prose accordingly (a one-line correction, not a structural change).

§2.2 Liam’s mental model — SINGLE path

Per cmux brief: “SINGLE path: UI = thin wrapper dropping document into localfs folder → cocoindex picks it up”. This is the same shape Liam ratified in form-extraction/PRODUCT.md §OQ-52-UI-UPLOAD-TENSION (S274): the interactive form-upload UI drops the file into the resolved folder→workspace path so the same pipeline write applies — no second, independent writer. The TWO-paths question for content_items + content_chunks is whether to extend that ratification.

Under SINGLE-path, the implication is: app-side chunker retires (Option (b)), OR cocoindex starts chunking too (Option (c)), OR the app-side UI is rewritten as a thin wrapper that drops files into the localfs folder so cocoindex picks them up (a special case of Option (b) where the chunker dies but per-chunk granularity also goes away).

§2.3 Friction of the status quo (Option (a))

Keeping both paths surfaces real costs already visible in the live code + specs:

1. Schema fork. content_chunks is populated ONLY by Path 2. A user who uploads a PDF via Path 2 gets search_content_chunks coverage; a user whose PDF lands in the cocoindex external folder gets content_items-only coverage. The MCP search_content_chunks tool returns ZERO rows for cocoindex-ingested documents at v1. PRODUCT.md §“v1.1 candidates” line 126 acknowledges this as a deferred question; the user-facing impact is asymmetric chunk-granular retrieval.
2. Embedding-input divergence. Path 1 embeds the WHOLE document (LiteLLMEmbedder over content_items.content); Path 2 embeds per-chunk text prefixed with heading_path.join(' > ') (lib/content/chunk-store.ts:32-39 buildChunkEmbeddingText). Search results from the two surfaces have systematically different ranking profiles (whole-doc cosine vs heading-prefixed chunk cosine). A reader using hybrid_search (Path 1 surface) vs search_content_chunks (Path 2 surface) gets different result orders for the same query against the same corpus.
3. Re-ingest semantics divergence. Path 1 is memoised (@coco.fn(memo=True); unchanged content → SKIP; embedding + op_id preserved per PRODUCT C-31 + C-21). Path 2 has no memo layer — every regenerateChunks call DELETES + re-INSERTs all content_chunks rows for the item (chunk-store.ts:173-177) and re-embeds every chunk. The “no-op re-ingest is free” invariant on Path 1 has no Path 2 counterpart.
4. Ownership ambiguity for the canonical record. Some content_items rows are written by Path 1 (cocoindex pipeline; op_id IS NOT NULL); others by Path 2 (ingest_source IN {'upload','url_import','manual','mcp_create','python_url','python_markdown','batch_reclassify','qa_import',...}; op_id IS NULL). PRODUCT C-21 is true for Path 1 rows; for Path 2 rows op_id IS NULL. The audit-forensics handle C-22 promotes (op_id) doesn’t exist for Path 2 rows. C-5’s ingest_source discriminator is the workaround but consumers must know to use it.
5. Two test suites for one behaviour family. Vitest tests cover Path 2 (__tests__/lib/content/chunk-store.test.ts, __tests__/lib/content/chunking.test.ts); cocoindex integration tests cover Path 1 (cocoindex-flow-scaffolding/TECH.md §P-3 Inv-4). Behaviour drift between them is invisible to either suite.

Costs of RETIRING one of the two paths:

• Retiring Path 2 (Options (b)/(c)) deletes ~220 lines of chunking.ts + ~200 lines of chunk-store.ts + 8 caller sites + ~5-8 test files. UI affordances for in-place edit / reclassify lose their “regenerate chunks immediately” path — they must instead trigger a cocoindex re-ingest, which (a) requires the content to land in the cocoindex external folder, (b) is asynchronous (fs-watch + polling cadence), (c) loses the synchronous user-visible feedback the UI gives today.
• Retiring Path 1 (no Liam framing for this — out of scope; cocoindex IS the canonical pipeline per 01-vision.md:44).
• Reverse split (Option (c) — cocoindex chunks) keeps Path 2’s read surface alive (content_chunks + search_content_chunks continue to exist; MCP tool unchanged) but moves the WRITE to cocoindex. Path 2’s app-side chunker dies; the UI’s synchronous “regenerate chunks now” path is replaced with “drop into folder; cocoindex picks up”. Same UX cost as retiring Path 2 cleanly.

§2.4 Cocoindex chunking sources — review

§2.4.1 cocoindex.io/blogs/index-code-base-for-rag/

Method: cocoindex’s own code-RAG example uses cocoindex.functions.SplitRecursively(language=file["extension"], chunk_size=1000, chunk_overlap=300) — AST-aware via Tree-sitter, integrated language-detection from file extension.

Embedder: SentenceTransformerEmbed(model="sentence-transformers/all-MiniLM-L6-v2") — bundled inside a @cocoindex.transform_flow() decorator. (NOT LiteLLMEmbedder; the code-RAG example uses a smaller MiniLM model. This is different from KH’s text-embedding-3-large choice.)

Storage: Postgres via cocoindex.storages.Postgres() with a vector_index on the embedding column using COSINE_SIMILARITY metric. Primary keys ["filename", "location"]; columns filename, location, code, embedding. No parent FK — the chunk row stands alone with the parent file identified only by filename + chunk position.

File types: ["*.py", "*.rs", "*.toml", "*.md", ".mdx"] — code + markdown + TOML. PDF/DOCX/XLSX NOT in scope of this example.

Granularity: AST-aware via Tree-sitter; chunk_size=1000 chars + chunk_overlap=300 chars.

Scope of the flow: chunk-only embedding. The flow chunks then embeds; it does NOT also embed the whole document. This is the shape Option (c) would adopt for KH — chunk-write to content_chunks, NO content_items.embedding writes (or it co-exists — see §2.5 below).

CRITICAL FINDING (empirically verified — see §5): cocoindex.functions.SplitRecursively is ABSENT in cocoindex==1.0.3 (the KH-pinned version). The 0.3.x → 1.0 restructure moved/renamed it. The 1.0.3 equivalents live at cocoindex.ops.text:

• cocoindex.ops.text.RecursiveSplitter(*, custom_languages: list[CustomLanguageConfig] | None = None) -> None — the spiritual successor, but signature has CHANGED. chunk_size and chunk_overlap are NOT positional or top-level keyword args in 1.0.3; they are presumably configured via the CustomLanguageConfig schema or per-call run-args (which need separate empirical verification — the constructor signature shows only custom_languages as a kwarg).
• cocoindex.ops.text.SeparatorSplitter(separators_regex: list[str], *, keep_separator: Literal['left', 'right'] | None = None, include_empty: bool = False, trim: bool = True) -> None — regex-driven, simpler but viable for heading-based splits.
• cocoindex.ops.text.CustomLanguageConfig — language configuration class (separate constructor).
• cocoindex.ops.text.detect_code_language — helper for inferring language from file extension.

Implication: any spec that cites the cocoindex.io blog verbatim would import a fictional symbol against cocoindex==1.0.3. This is exactly the S252 cocoindex ExtractByLlm fabrication trap the OQ-3 forcing function exists to catch (docs/research/cocoindex-1.0.3-extractbyllm-spec-reality-investigation.md). The blog is informational about the cocoindex chunking SHAPE; spec citations of the EXACT symbol must use the 1.0.3 names. This is a {56.2}/{56.3} Planner concern if Option (b) or Option (c) is ratified.

§2.4.2 ekimetrics/adaptive-chunking

Method: automatically evaluates multiple document-splitting strategies (recursive, page-based, LLM-regex, semantic clustering) using intrinsic quality metrics and selects the best per-document. Adaptive (per-doc, per-strategy) rather than fixed.

Inputs: PDFs (3 parser backends: Docling default + PyMuPDF lightweight + Azure Document Intelligence), Excel files, raw text. Tabular content via dedicated ExcelParser + block-integrity metrics that preserve paragraph / table / list boundaries.

Output: text chunks ONLY (no embeddings — embeddings are used internally for metric computation but not returned).

Package: pip install adaptive_chunking; key deps: spacy, jina-embeddings-v3, docling/pymupdf, optional maverick-coref.

License: MIT core. Caveats: maverick-coref optional dep is CC BY-NC-SA 4.0 (non-commercial — would block KH commercial use if enabled); pymupdf4llm optional is AGPL-3.0 or commercial (same KH AGPL-network-boundary issue as pullmd — see cocoindex-flow-scaffolding/TECH.md §1 O-Q3). Default install (Docling backend) avoids both.

Maintenance: active. 4 commits on main; 193 stars; 18 forks; LREC 2026 paper accepted. No release history — pre-release / pre-1.0; semver guarantees absent. Risk: KH would be an early adopter; API stability not guaranteed; no production usage testimonials in the README.

Integration: No native cocoindex / LangChain / LlamaIndex integration. Benchmarks against “LangChain recursive” as a baseline. Adoption requires KH to write the integration glue (wrap adaptive_chunking calls inside a @coco.fn if cocoindex-side, or inside chunk-store.ts if app-side).

Suitability for KH content shapes:

Format	Coverage
PDF	YES — 3 backends; Docling matches KH’s existing Stage-2 (PDF→markdown) cocoindex adapter.
DOCX	NOT explicitly named. The README highlights PDF + Excel + raw text. DOCX likely requires a pre-conversion step (DOCX→markdown via Docling, then route through raw text).
XLSX	YES — dedicated ExcelParser + block-integrity for tables.
Markdown	YES — raw text input.
Procurement-RFP scale	UNKNOWN. LREC 2026 paper benchmarks “technical / legal / sustainability reports” — overlapping but not identical to UK procurement RFPs. Empirical pilot needed before commitment.

Headline judgment: adaptive-chunking is a research-grade, single-purpose chunking library with no native pipeline integration story; adoption is an option-on-top of either Option (b) or Option (c). It would be a substitute for KH’s chunkByHeadings / cocoindex’s RecursiveSplitter. Adoption decision is decoupled from the TWO-paths-vs-SINGLE-path decision: it’s the “which chunking algorithm” sub-question, downstream of the “where does chunking happen” decision §2 frames.

§2.5 Three decision options — spelled out

Each option carries: affected invariants list (relative to S273-ratified PRODUCT/TECH), implementation cost rough sketch, risk vector. The decision IS Liam’s.

Option (a) — Keep both paths as-is

• Affected invariants: none structurally. PRODUCT.md C-13 scope-clarification + C-60 stand; the v1.1-deferred question on line 126 (whether cocoindex should additionally emit content_chunks) stays deferred. TECH §2.2 chunking-table stands.
• Implementation cost ({56.5+}): very low. The only PLAN-worthy work is (i) the §1 delta-1 Stage-5 amend (small, see §3 below); (ii) optional clarification footnotes for deltas 3 + 5; (iii) prose tightening on PRODUCT C-13 scope-note to correct the misattribution of analyse/route.ts as a chunker caller (see §2.1 above).
• Risk vector: none new — accept the §2.3 friction (asymmetric chunk-granular coverage; embedding-input divergence; ownership ambiguity; two test suites) as v1 cost. Defers the SINGLE-path question to v1.1.

Option (b) — Retire app-side chunker; cocoindex becomes SINGLE write path

• Affected invariants: STRUCTURAL. C-10..C-14 (chunking-boundary invariants) are entirely re-targeted at the cocoindex pipeline — the lib/content/chunking.ts source-code references become historical; the chunking algorithm becomes whichever cocoindex.ops.text.* op the new cocoindex chunking stage adopts (RecursiveSplitter or SeparatorSplitter — see §2.4.1). C-13’s chunk-row shape REMAINS (content_chunks table, same columns) IF cocoindex writes chunks to that table; OR is REWRITTEN entirely if the chunking stage is removed wholesale and content_items becomes the only granularity. The MCP search_content_chunks tool either continues to exist (cocoindex emits chunks) or is retired (cocoindex doesn’t chunk; chunk-granular retrieval goes away). PRODUCT §“v1.1 candidates” line 126 changes from “deferred” to “ratified — chunks happen in cocoindex”.
• Implementation cost ({56.5+}): HIGH if cocoindex chunks (full new stage: RecursiveSplitter/SeparatorSplitter op chosen + @coco.fn for chunk-row build + new mount_table_target(managed_by=USER) for content_chunks + integration tests + retirement of chunkByHeadings/chunk-store.ts + UI rewires for the 8 regenerateChunks call sites + backfill strategy for existing content_chunks rows). MEDIUM if cocoindex doesn’t chunk (only chunk-store.ts + chunkByHeadings retire; content_chunks table deprecates; search_content_chunks MCP tool retires; ~12-15 {56.5+} Subtasks).
• Risk vector: (i) cocoindex chunking stage needs declare_row semantics for chunk rows — viable per RESEARCH §R2 (TableTarget declare_row is row-shape-agnostic), but adds a SECOND mount_table_target to the flow and a per-chunk row-emit pattern not yet exercised. (ii) Memo semantics interact with chunks differently — if a document re-ingests but chunk-boundaries shift (e.g. heading edit moves the H2 split), the per-chunk op_id semantics need clear ratification (each chunk gets its own deterministic id? UUID5 of content_item_id + position? Stage-5 op_id semantics don’t apply). (iii) The 8 regenerateChunks call sites need replacement — most are synchronous post-write UX hooks (upload, reclassify, governance publish); async cocoindex re-ingest is a UX regression unless paired with a sync “kick the cocoindex flow” trigger. (iv) Existing content_chunks corpus needs migration (re-chunk under the new boundaries or accept boundary drift). (v) Recursive-fabrication risk — the cocoindex 1.0.3 chunking API surface is ops.text not cocoindex.functions; spec citations must use the 1.0.3 names and signatures, NOT the cocoindex.io blog’s symbols. (vi) Adaptive-chunking adoption is an open sub-question (orthogonal — see §2.4.2).

Option (c) — Reverse split — cocoindex chunks too; app-side chunker retires; UI thin wrapper

• Affected invariants: same scope as Option (b)‘s “cocoindex chunks” variant — C-10..C-14 re-targeted at cocoindex; content_chunks table retained; search_content_chunks MCP tool retained; UI rewires from synchronous regenerateChunks to asynchronous folder-drop. Additional invariant: C-13 v1.1-deferred question line 126 becomes “RESOLVED — cocoindex emits content_chunks rows”.
• Implementation cost ({56.5+}): HIGH — similar to Option (b)‘s “cocoindex chunks” variant. ~15-20 {56.5+} Subtasks.
• Risk vector: all of Option (b)‘s risks (i)-(vi), plus UI-rewire complexity (the 8 caller sites of regenerateChunks are different shapes — some are batched, some single-item, some MCP-tool-driven, some script-driven — folder-drop wrapper needs to cover all of them or some UI surfaces lose synchronous feedback).

§2.6 Open-Question packet for Liam (the OQ-escalation payload)

Per docs/specs/oq-escalation/PRODUCT.md shape (oq_id / question / context_ref / urgency / blocking):

oq_id: OQ-CMI-56-1
worker_id: subo-id-56-research
emitted_at: 2026-05-28T17:00:00Z
seq: 1
question: |
  For ID-56 content-model-invariants PLAN authoring, which of three decision
  options is ratified for the TWO-paths-vs-SINGLE-path question?

  (a) Keep both paths as-is. App-side chunker (lib/content/chunking.ts +
      chunk-store.ts) continues to own content_chunks writes for 8 caller
      sites; cocoindex remains document-granular (no chunking stage). v1.1
      defers the question. PLAN authoring is minimal (~3-5 Subtasks: Stage-5
      cross-cut + delta clarifications + C-13 prose tightening).

  (b) Retire app-side chunker. Cocoindex becomes the SINGLE ingest write
      path. Sub-choice: (b1) cocoindex chunks too (adds cocoindex.ops.text.
      RecursiveSplitter or SeparatorSplitter stage; chunks land in
      content_chunks; ~15-20 Subtasks); or (b2) cocoindex stays doc-granular
      and chunking goes away wholesale (content_chunks table deprecates;
      search_content_chunks MCP tool retires; ~12-15 Subtasks).

  (c) Reverse split. Cocoindex chunks too (writes content_chunks rows);
      app-side chunker retires; UI rewires as thin wrapper that drops files
      into folder→workspace path so cocoindex picks them up. Behaviourally
      equivalent to (b1) on the data side; differs on the UI rewire shape.

  RESEARCH evidence summary:
    - Friction of status quo (a): asymmetric chunk-granular coverage
      (cocoindex-ingested docs absent from search_content_chunks at v1);
      embedding-input divergence (whole-doc vs heading-prefixed chunks);
      re-ingest semantics divergence (memo on path 1, full delete-reinsert
      on path 2); ownership ambiguity (op_id IS NULL for app-side rows).
    - Friction of (b)/(c): cocoindex.functions.SplitRecursively ABSENT in
      cocoindex==1.0.3 — must use cocoindex.ops.text.RecursiveSplitter or
      SeparatorSplitter (signature differs); UI rewires for 8 caller sites
      of regenerateChunks (most synchronous); existing content_chunks corpus
      migration; recursive-fabrication risk on spec authoring.
    - Already-ratified precedent: OQ-52-UI-UPLOAD-TENSION (S274) ratified
      "thin UI front-end drops file into resolved folder→workspace path"
      for form_templates. Extending the same answer to content_items
      + content_chunks is consistent with that ratification's directional
      preference for SINGLE-path.
    - adaptive-chunking library (ekimetrics) is an orthogonal sub-question
      (which chunking algorithm) downstream of this decision; not on the
      critical path of OQ-CMI-56-1.

context_ref:
  - docs/specs/content-model-invariants/RESEARCH.md §2
  - docs/specs/content-model-invariants/PRODUCT.md §C-13 + §C-60 + v1.1 line 126
  - docs/specs/content-model-invariants/TECH.md §1 + §2.2
  - docs/specs/form-extraction/PRODUCT.md OQ-52-UI-UPLOAD-TENSION
  - docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md §2.5
urgency: high
blocking: true
  # BLOCKS {56.2} PRODUCT amend authoring + {56.3} TECH amend authoring +
  # {56.4} PLAN authoring + all {56.5+} impl Subtasks. Liam is the
  # decision authority; the parent Orchestrator surfaces this OQ and
  # routes Liam's ratification back to {56.2} dispatch.

---

§3. Cross-spec ratification chain — what {56.2} / {56.3} would amend

Conditional on the §1 verdicts above and the §2 Liam ratification.

Under §1 delta 1 (Stage-5 — AFFECTS, light)

PRODUCT.md amends:

• New invariant C-54 under §Entity-classification (or a new §Entity-resolution subsection): “When a reader queries entity_mentions.canonical_name, the value is stable only AFTER pipeline_runs.status='completed' for the run that produced or updated the row (op_id matched). Within the in-flight window, the per-document phase has written the per-document default canonical (deterministic per-doc normalisation per canonicalise_entity_name), and the Stage-5 UPDATE phase MAY rewrite that value to a cross-document canonical before the run completes. Readers reading mid-flight cannot rely on cross-document consistency.” Cross-ref Stage-5 Inv-1, Inv-3, Inv-5, Inv-7.
• One-line amend on C-22: “(Stage-5 UPDATE phase deliberately breaks the managed_by=USER row-only contract; cross-ref Stage-5 Inv-2. The trigger-driven audit substrate still records the UPDATE; no v1 substrate change.)”.

TECH.md amends:

• New row in §2.6 (entity-classification) mapping C-54 → read entity_mentions.canonical_name AFTER pipeline_runs.status='completed' AND op_id match.
• Footnote in §2.7 (op_id memo refinement) referring to Stage-5’s distinct memo semantics (Stage-5 UPDATE re-stamps op_id only when it changes canonical_name — entity_mentions Stage-5 Inv-7).

Amend shape: surgical, ~10 lines across two files. No new section. One PRODUCT invariant + one prose amend.

Under §1 delta 3 (Form-extraction — NO-CHANGE, optional clarification)

PRODUCT.md amends (optional, recommended):

• Append to §Out of scope (v1): “Form-template ingest (form_templates + form_template_fields) is owned by docs/specs/form-extraction/ (Task ID-52); its rows do NOT live in content_items and the form-extraction WRITE path is out of scope of this spec. C-N invariants here describe content_items and content_chunks only.”

TECH.md amends: none.

Amend shape: 2-line clarification. No invariant amend.

Under §2 OQ-CMI-56-1 (TWO-paths question)

If Liam ratifies (a):

• PRODUCT amends: light prose correction on C-13 scope-note (replace “upload / analyse routes” → “8 caller sites of regenerateChunks enumerated in TECH §2.2 caveat”; see §2.1 above). v1.1-deferred line 126 stands.
• TECH amends: §2.2 caveat extended to enumerate the 8 callers (optional; or just point to RESEARCH §2.1).
• Amend shape: prose tightening, ~5 lines.

If Liam ratifies (b) or (c):

• PRODUCT structural amend: re-write §Chunking-boundary subsection (C-10..C-14) to target cocoindex; rewrite C-13 v1.1 deferred line 126; add new invariant for chunk-row provenance / op_id semantics; potentially add new invariant for UI thin-wrapper drop-path (folder→workspace consistency with form-extraction).
• TECH structural amend: §1 two-paths framing rewritten to single-path; §2.2 chunking table rewritten to point at cocoindex.ops.text.RecursiveSplitter (or SeparatorSplitter) per §2.4.1 with signature corrected for cocoindex 1.0.3; §6 Verification block extended with OQ-3 empirical check on whichever cocoindex chunking op is chosen.
• Amend shape: structural — new §Chunking subsection in PRODUCT (~30-50 lines); new §2.2 in TECH (~30-50 lines); two new §6 OQ-3 verification rows.

Sequencing constraint: the OQ-CMI-56-1 ratification MUST land BEFORE {56.2} PRODUCT amend dispatch — otherwise the Planner has no scope. Per cmux brief Phase 1 dispatch cadence.

---

§4. PLAN.md prep notes (NOT the PLAN — for the {56.4} Planner)

{56.4} is authored by a SEPARATE fresh Planner via planning-and-task-breakdown. This section is input prep only — it sketches the impl-Subtask classes the {56.4} Planner will be decomposing into, conditional on the §1 verdicts + §2 Liam ratification.

Sibling-only dep constraint (§3.3 A6). All {56.5+} Subtasks MUST depend on other ID-56 siblings only. If decomposition surfaces a cross-Task dep (e.g. “{56.7} depends on {53.10}” because Stage-5 must land first), the {56.4} Planner MUST escalate per the workflow-orchestration skill — do NOT bend the constraint. Likely cross-Task touch points:

Potential cross-Task dep	Resolution
Stage-5 PRODUCT/TECH amend in this spec depends on ID-53 PRODUCT/TECH being final	Not a true dep — ID-53 PRODUCT/TECH already ratified S274; the amend is a READ-side cross-cut that references existing ID-53 invariants. No sibling-only violation.
Under (b)/(c) cocoindex chunking impl depends on Stage-4 (ID-49.2)	ID-49.2 is DONE per sequencing §4. The dep is satisfied at Task-level (ID-56 deps ["28","49"]); no Subtask-level violation.
Under (b)/(c) UI rewires touch ID-50 (OPS-T1 route rollout)	Coordination signal (per ID-52 pattern in form-extraction PLAN §3), NOT a sibling-only-dep violation. {56.4} Planner should document the coordination in PLAN §“Sequencing notes” and surface to Orchestrator.
Under (b)/(c) chunk-corpus migration touches existing content_chunks rows	Pure migration Subtask within ID-56; no cross-Task.

If any other cross-Task dep surfaces during {56.4}, escalate.

25-Subtask soft ceiling (§3.4 A7). Under (a): well within ceiling (3-5 Subtasks). Under (b): close to ceiling (12-20 Subtasks). Under (c): likely exceeds ceiling (15-25 Subtasks). If the {56.4} decomposition under (b)/(c) approaches 25, the {56.4} Planner SHOULD propose a Task split to Orchestrator (e.g. split into “ID-56 spec-amend Subtasks” + a new Task ID-57 “cocoindex chunking implementation”).

Impl-Subtask classes per decision branch

Under (a) keep both:

1. Spec amend Subtask — {56.5} PRODUCT amend (Stage-5 C-54 + scope-clarification note + C-13 prose tightening). Author by fresh {56.2} Planner.
2. Spec amend Subtask — {56.6} TECH amend (matching §2.6 row + §2.7 footnote + §2.2 caller enumeration). Author by fresh {56.3} Planner.
3. (Optional) Doc-side Subtask {56.7} — update canonical-pipeline-sequencing.md v1.1-candidates section to reflect the deferral being explicit-and-known rather than open. Only if Liam dispatches separately (per cmux brief out-of-scope guard).
4. (Optional) Test Subtask {56.8} — add a guard test against PRODUCT §“v1.1 candidates” prose to catch v1.1 deferred-question drift.

Under (b) cocoindex chunks (sub-option b1):

1. Spec amend Subtask — {56.5} PRODUCT structural amend (rewrite §Chunking).
2. Spec amend Subtask — {56.6} TECH structural amend (rewrite §1 + §2.2; new §6 OQ-3 row).
3. Impl Subtask — {56.7} cocoindex chunking stage spike (verify cocoindex.ops.text.RecursiveSplitter real signature for chunk_size/chunk_overlap config; produces a working @coco.fn chunker).
4. Impl Subtask — {56.8} new mount_table_target(managed_by=USER) for content_chunks + per-chunk row build.
5. Impl Subtask — {56.9} chunk row UUID strategy (UUID5 of content_item_id + position?) — depends on Stage-5 PairResolver / canonicalise_entity_name pattern.
6. Impl Subtask — {56.10} integration tests against fixture-staging (ID-49.10 corpus once landed).
7. Impl Subtask — {56.11} UI rewire for upload route — folder-drop pattern.
8. Impl Subtask — {56.12} UI rewire for URL ingest route — folder-drop pattern.
9. Impl Subtask — {56.13} UI rewire for items create + edit routes — folder-drop pattern (or sync trigger).
10. Impl Subtask — {56.14} UI rewire for MCP governance + content tools.
11. Impl Subtask — {56.15} migration of existing content_chunks corpus to new chunk boundaries.
12. Impl Subtask — {56.16} retire lib/content/chunking.ts + chunk-store.ts + 8 caller sites.
13. Impl Subtask — {56.17} retire script backfill-chunks.ts (no longer needed under cocoindex-owned chunks).
14. Impl Subtask — {56.18} regression suite — bun run test:integration covering Path-1-only chunk-search semantics.

Under (b) cocoindex stays doc-granular; chunking goes away (sub-option b2): 1-2 above + structural amend that retires content_chunks entirely; impl Subtasks for retiring search_content_chunks MCP tool + Zod schema + 8 caller sites + content_chunks table migration (DROP or archive).

Under (c) reverse split: essentially (b1) with the additional UI thin-wrapper-rewire Subtasks landed first; same total Subtask count.

Testing-substrate notes (delta 5 — fixture-staging)

{56.4} Planner: when (b) or (c) is ratified, the {56.10} (or equivalent) integration-test Subtask SHOULD depend on ID-49.10 fixture-staging being DONE. ID-49.10 is currently pending per sequencing §4; this is a Task-level dep (ID-56 deps ["28","49"] — ID-49 is closed done per sequencing §4 with 49.10 as carrying-forward). If 49.10 is still pending at {56.5+} dispatch time, escalate sequencing to Orchestrator.

---

§5. Verification log

OQ-3 empirical import-and-call checks for the external-library symbols this RESEARCH cites:

#	Date	Pinned version	Symbol path checked	Result
V-1	28/05/2026	cocoindex[postgres]==1.0.3 (requirements.txt)	cocoindex.functions.SplitRecursively (cited verbatim in the cocoindex.io code-RAG blog)	ABSENT. ModuleNotFoundError: No module named 'cocoindex.functions'. Symbol removed in the 0.3.x → 1.0 restructure (analogous to ExtractByLlm per docs/research/cocoindex-1.0.3-extractbyllm-spec-reality-investigation.md). Any {56.2}/{56.3} spec citation under Options (b)/(c) MUST use the 1.0.3 names below, not the blog’s symbol.
V-2	28/05/2026	cocoindex==1.0.3	cocoindex.ops.text.RecursiveSplitter	PRESENT. Signature `(*, custom_languages: list[cocoindex.ops.text.CustomLanguageConfig]
V-3	28/05/2026	cocoindex==1.0.3	cocoindex.ops.text.SeparatorSplitter	PRESENT. Signature `(separators_regex: list[str], *, keep_separator: Literal[‘left’, ‘right’]
V-4	28/05/2026	cocoindex==1.0.3	cocoindex.ops.text.CustomLanguageConfig + detect_code_language	PRESENT. Auxiliary; cited for completeness — adoption of either RecursiveSplitter or SeparatorSplitter does not strictly require these unless KH wants language-specific splitting.
V-5	28/05/2026	cocoindex==1.0.3	cocoindex.ops.litellm.LiteLLMEmbedder (C-30 dependency; verified at PRODUCT/TECH ratification S273)	PRESENT (re-verified). Signature (model: 'str', **kwargs: '_Any') -> 'None'. Accommodates LiteLLMEmbedder("text-embedding-3-large", dimensions=1024) per sequencing §2.5. No change since S273 verification.
V-6	28/05/2026	live schema	content_chunks Row + content_chunks_content_item_id_fkey + content_chunks_parent_chunk_id_fkey	PRESENT (database.types.ts:509-571). Columns: id, content_item_id, parent_chunk_id, position (NOT NULL), heading_level, heading_path (text[]), heading_text, char_count, word_count, embedding (vector(1024) — serialised as string), content, created_at, updated_at. Two FKs as named. Matches PRODUCT C-13 + TECH §2.2 exactly.
V-7	28/05/2026	live schema	search_content_chunks RPC	PRESENT (database.types.ts:4389-4416). Args: filter_content_item_id?, filter_overdue_review?, filter_review_due_within_days?, limit_count?, query_embedding (string), similarity_threshold?, visibility_filter?. Returns chunk fields + similarity + item-level fields. Matches TECH §2.2 C-13 row exactly.
V-8	28/05/2026	live schema	form_templates Row	PRESENT (database.types.ts:1725-1796). Columns prior to M1b: id, workspace_id (NOT NULL FK → workspaces), created_by (FK → user_profiles), name, filename, mime_type, file_size, storage_path, structure_path, status, ingest_source, description, field_count, mapped_count, created_at, updated_at. {52.M1b} adds form_type (FK form_types.key) + deadline + issuing_organisation + evaluation_methodology per task-list.json Subtask id=18 + PLAN {52.M1b} block at PLAN.md lines 230-267. NOT part of content_items — out of scope of this spec per §1 delta 3.
V-9	28/05/2026	live source	lib/ai/embed.ts:4,42,93,108	PRESENT. Path 2 embedder model = process.env.AI_EMBEDDING_MODEL ?? 'text-embedding-3-large'; MAX_EMBEDDING_CHARS = 24_000; dimensions=1024 via Matryoshka shortening. SAME model + SAME final vector dimension as Path 1 (cocoindex LiteLLMEmbedder), but via OpenAI SDK directly — confirms §2.3 friction-point 2 (different SDK path; potentially identical vectors per Matryoshka spec, but embedding-input prefixing differs).
V-10	28/05/2026	live source	lib/content/chunk-store.ts callers (8 sites enumerated §2.1)	PRESENT. Verified via grep against app/api, lib, scripts. The brief’s “upload + analyse routes” framing is INCOMPLETE — actual caller surface is 8 sites; analyse/route.ts is NOT among them (it’s the form-extraction template-analyse route, retired by ID-52.15). Correction surfaced for {56.2} PRODUCT C-13 prose.
V-11	28/05/2026	cocoindex==1.0.3	cocoindex.ops.text.RecursiveSplitter constructor + .split() (S276 amend round 2 — post OQ-CMI-56-3 ratification (i))	PRESENT, re-verified. Constructor (*, custom_languages: list[CustomLanguageConfig] | None = None) -> None. Per-call .split(text: str, chunk_size: int, *, min_chunk_size: int | None = None, chunk_overlap: int | None = None, language: str | None = None) -> list[Chunk]. Key semantics: chunk_size is target in BYTES (not characters); min_chunk_size defaults to chunk_size / 2 — native min-chunk-merge semantic, replaces retired app-side MIN_CHUNK_CHARS=100 floor; chunk_overlap optional in bytes; language enables tree-sitter syntax-aware splitting (markdown / plain text paths do NOT require it). Per-call configuration model means chunk_size / chunk_overlap ratification is per-flow, not constructor-baked — {56.3} TECH ratifies the values.

Verification discipline: every external-library symbol cited in §2.4 + §2.5 has an OQ-3 import-and-call check above. Every cited file:line in §1 / §2.1 / §2.4 was verified at S276 against the worktree base commit. No fabricated citations — if a citation was uncertain, it was either verified or omitted with a [GAP-CMI-RESEARCH-NNN] flag (none raised).

---

§6. Open Questions for Liam (full list — for OQ-escalation)

OQ-CMI-56-1 — TWO ingest paths vs SINGLE ingest path

See §2.6 above for the full packet. Urgency: high. Blocking: true. Authority: Liam. Blocks {56.2} / {56.3} / {56.4} dispatch.

OQ-CMI-56-2 — Scope of the Stage-5 cross-cut amend

Question: Should the Stage-5 cross-cut amend (per §3 under §1 delta 1) be:

• (a) appended to PRODUCT §Entity-classification as a new C-54 + a one-line amend on C-22?
• (b) elevated to a new top-level §“Cross-document entity resolution” subsection with C-54, C-55, C-56 carved out?
• (c) deferred to ID-53 PRODUCT (cross-link from there), keeping ID-56 PRODUCT silent on Stage-5?

Why it matters: the cross-cut is light, but the precedent affects how this spec relates to ID-53. The S273 PRODUCT/TECH ratification placed entity-classification (C-50/C-51/C-52) in this spec; Stage-5 lands in ID-53; the boundary is not yet codified.

Recommendation: (a) — minimal amend, preserves S273 ratification footprint. (b) is over-scoped; (c) leaves a stale C-50-C-52 frame in this spec when Stage-5 lands.

Urgency: medium. Blocking: false (the {56.2} Planner can default to (a) absent ratification; surface to Liam in {56.2} Checker report if needed). Authority: Liam, post-hoc OK.

OQ-CMI-56-3 — Chunking algorithm choice (RATIFIED — see §7)

Status: RATIFIED-S276 (i) cocoindex.ops.text.RecursiveSplitter. See §7 ratification block for the decision + implications. Was originally framed conditional on OQ-CMI-56-1 = b1 or c; now LIVE under the ratified (c) path.

The four sub-options at packet-emission time were:

• (i) cocoindex.ops.text.RecursiveSplitter (the spiritual successor to the blog’s SplitRecursively; needs chunk_size / chunk_overlap configuration verified at impl time) — RATIFIED.
• (ii) cocoindex.ops.text.SeparatorSplitter (regex-driven; simpler; viable for heading-based splits matching the current chunkByHeadings behaviour) — not chosen.
• (iii) ekimetrics/adaptive-chunking library (research-grade, pre-1.0, no native cocoindex integration; KH would be an early adopter; license caveats on optional deps) — not chosen; v1.1 candidate.
• (iv) port chunkByHeadings heading-split semantics into a custom @coco.fn (preserves current chunker behaviour; minimal user-visible change) — not chosen.

OQ-CMI-56-4 — Heading-derived content_chunks columns disposition (NEW, S276 amend round 2)

Formal OQ-INV-1 packet per docs/specs/oq-escalation/PRODUCT.md:

oq_id: OQ-CMI-56-4
worker_id: subo-id-56-research
emitted_at: 2026-05-28T17:30:00Z
seq: 4
question: |
  Under OQ-CMI-56-3 ratified (i) cocoindex.ops.text.RecursiveSplitter
  (budget-driven, NOT heading-based), the content_chunks columns
  heading_text / heading_level / heading_path / parent_chunk_id become
  NULL on every chunk row the new cocoindex chunking stage emits. What
  is the disposition?

  (a) Keep as nullable legacy. Columns stay in schema; NULL on
      cocoindex-emitted rows; rows from prior population paths (e.g.
      q_a_pair soft-archive) MAY retain populated values.
      search_content_chunks RPC Returns continue to surface them (RPC
      signature update still required per C-13 RPC consumer-safety
      note — heading cols become nullable in Returns). Consumers
      tolerate NULL. Recommended default if Liam does not intervene.

  (b) Drop the columns. Schema migration removes heading_text /
      heading_level / heading_path / parent_chunk_id from
      content_chunks. search_content_chunks RPC + Zod schema +
      consumer call-sites need coordinated retirement. Higher impact;
      cleaner schema; loses historical heading-context for
      soft-archived rows.

  (c) Re-purpose for AST-aware boundaries. If {56.3} ratifies the
      language parameter on RecursiveSplitter.split() for markdown /
      code paths, the tree-sitter-derived node-type could populate
      heading_text semantically (e.g. function name, section name).
      Speculative; out of scope for v1.
context_ref:
  - docs/specs/content-model-invariants/PRODUCT.md C-13 (round-2 amend)
  - supabase/types/database.types.ts content_chunks Row (lines 509-571)
  - supabase/types/database.types.ts search_content_chunks Returns (lines 4404-4406)
  - docs/specs/content-model-invariants/RESEARCH.md §7 OQ-CMI-56-3 ratification
urgency: low
blocking: false
  # Default to (a) absent Liam intervention; {56.3} TECH amend
  # proceeds either way. {56.4} PLAN Subtask shape differs only by
  # presence/absence of column-drop migration Subtask.

Why it matters: the C-13 PRODUCT invariant ({56.2} amend round 2) now records the heading-derived columns as NULL-on-emit. The schema-disposition decision affects whether {56.3} TECH carries a schema-migration row for column drop, or simply records the NULL semantic and leaves the columns alone. Affects {56.4} PLAN Subtask shape (extra “heading-derived columns disposition” Subtask under disposition (b) vs no extra Subtask under disposition (a)).

Recommendation: (a) — minimal coordinated impact; preserves soft-archived row semantics; defers schema cleanup to v1.1.

---

End of RESEARCH. The decision authority on OQ-CMI-56-1 is Liam’s. The {56.2} PRODUCT amend dispatch should NOT begin until OQ-CMI-56-1 is ratified. The {56.4} PLAN authoring sees a fundamentally different Subtask shape under each branch — §4 sketches the shapes; the actual decomposition is the {56.4} Planner’s call once spec amends (if any) are in place.

---

§7. Ratification log (post-Liam, S276)

OQ-CMI-56-1 — RATIFIED (c) reverse split

Date: 28/05/2026 (S276). Authority: Liam. Channel: OQ-escalation via parent (sub-O subo-id-56-research → S276 main session).

Decision: (c) reverse split — UX-first framing matching S274 OQ-52-UI-UPLOAD-TENSION 1:1. Substantively equivalent to (b1) on the data side (cocoindex emits content_chunks rows; app-side chunker lib/content/chunking.ts + lib/content/chunk-store.ts retires; UI rewires to thin folder-drop wrapper); spec narrative leads with the UI rewire / folder-drop pattern as the headline change, not the cocoindex stage addition.

Liam-supplied load-bearing constraint: NO data migration is required. The full client corpus will be re-ingested as part of the canonical-pipeline implementation. This removes the largest risk vector from §2.5 / §3 — no existing content_chunks corpus migration is needed; the path-1 re-ingest under the new chunking stage produces the canonical content_chunks rows.

Implications for {56.2} / {56.3} / {56.4}:

• {56.2} PRODUCT amend Planner (fresh, Q-PLANNER-2): scope of amend = retire the “app-side chunker” framing in §Chunking-boundary scope clarification (the S273 note above C-10); rephrase C-10..C-14 to describe cocoindex’s new chunking stage (the algorithm choice is {56.3} TECH); reframe [DEFERRED-v1.1] line 126 (“Whether the cocoindex pipeline should additionally write content_chunks rows”) as RESOLVED in this v1; add a new C-7x or amend C-13 to record the op_id stamping pattern on content_chunks rows (cocoindex emits them now); cross-link OQ-52-UI-UPLOAD-TENSION precedent in §Source-of-truth pointers.
• {56.3} TECH amend Planner (fresh, Q-PLANNER-2 separate instance): scope of amend = §2.2 chunking table rows C-10..C-14 are now cocoindex-served; new §2.x for the cocoindex chunking stage call (algorithm choice per OQ-CMI-56-3); §1 two-paths framing collapses to single-path; §6 Verification adds OQ-3 check on the chosen chunking op symbol; risk row added on “the cocoindex chunking op symbol is cocoindex.ops.text.RecursiveSplitter or SeparatorSplitter — cocoindex.functions.SplitRecursively is ABSENT in cocoindex==1.0.3 per V-1”.
• {56.4} PLAN authoring (fresh, Q-PLANNER-2 separate Planner instance via planning-and-task-breakdown): branch (c) Subtask shape from §4 applies — ~15-20 Subtasks across cocoindex chunking-stage addition · cocoindex content_chunks declare_row + op_id stamping · existing-corpus path NO MIGRATION (full re-ingest planned, so “delete and re-emit” is the pattern, not “rebuild in place”) · UI rewire for the 8 regenerateChunks caller sites + folder-drop wrapper · search_content_chunks RPC consumer audit (ensure post-cocoindex chunk semantics are compatible) · tests + integration tests + Stage-5 cross-cut amend (per §3 + OQ-CMI-56-2 default).

OQ-CMI-56-2 — DEFAULT (a), pending Liam ratification at {56.2} Checker report

Per RESEARCH §6 OQ-CMI-56-2 recommendation: {56.2} Planner defaults to (a) — append new C-54 to §Entity-classification + one-line amend on C-22 — preserving S273 footprint. Liam confirms or overrides post-{56.2} Checker.

OQ-CMI-56-3 — RATIFIED (i) cocoindex.ops.text.RecursiveSplitter

Date: 28/05/2026 (S276). Authority: Liam. Channel: OQ-escalation via parent (sub-O subo-id-56-research → S276 main session).

Decision: (i) cocoindex.ops.text.RecursiveSplitter — cocoindex-native, AST-aware via tree-sitter, budget-driven (chunk_size + chunk_overlap per-call configuration). Supersedes the §2.4.1 / §6 OQ-CMI-56-3 framing where four sub-options were live; (ii) SeparatorSplitter / (iii) ekimetrics adaptive / (iv) port chunkByHeadings are all NOT chosen.

Empirical re-verification at S276 amend round 2 (cocoindex==1.0.3): RecursiveSplitter is constructed with optional custom_languages kwarg only; the chunking parameters live on the .split(text, chunk_size, *, min_chunk_size=None, chunk_overlap=None, language=None) per-call method. chunk_size is a target in bytes (not characters). min_chunk_size defaults to chunk_size / 2 — native min-chunk-merge semantic; replaces the retired app-side MIN_CHUNK_CHARS=100 floor ({56.3} TECH may ratify a specific min_chunk_size value or accept the default). chunk_overlap is optional in bytes; default not surfaced via signature ({56.3} TECH ratifies). Language-aware splitting via tree-sitter is supported but optional (markdown / plain text inputs do not require it).

CHANGES current behaviour (load-bearing for downstream Planners): the retired chunkByHeadings was heading-based (H2-default split with H1 fallback) plus min-chunk merge; RecursiveSplitter is budget-based with overlap, NOT heading-based. A reader cannot rely on chunk boundaries aligning with heading boundaries — chunk boundaries are content-budget-driven (per chunk_size).

Implications for {56.2} PRODUCT (this S276 amend round 2):

• C-10..C-14 need substantive REWRITE, not just rephrase. The invariants change semantically — see PRODUCT.md commit ahead of this for the rewritten prose:
  • C-10 (rewrite). Short-doc single-row invariant reframed as a budget-driven property: when content total length < chunk_size (per {56.3} TECH ratified value; the retired 500-char threshold becomes a TECH-side configuration parameter, not a PRODUCT invariant). The original “no splitting below 500 chars” behaviour can be preserved by setting chunk_size >= 500 if Liam wants the same UX.
  • C-11 (rewrite — semantic change). Heading-split invariant REPLACED by a budget-split invariant: chunks bounded by chunk_size with chunk_overlap bytes overlap between adjacent chunks per RecursiveSplitter semantics. The OLD H2/H1 behaviour is GONE.
  • C-12 (reframe — absorbed by RecursiveSplitter native semantics). min_chunk_size default (chunk_size / 2) governs short-trailing-chunk policy. The 100-char floor is RETIRED; reframe to delegate to TECH §x for the exact min_chunk_size value.
  • C-13 (rewrite — significant). The schema-level invariant (columns exist, FK to content_items, op_id stamped) STAYS. The semantic content of the heading-derived columns CHANGES: heading_text / heading_level / heading_path / parent_chunk_id are NO LONGER populated by the cocoindex stage (no heading boundary preserved by budget-split). A reader sees NULL on cocoindex-emitted chunks. Disposition of these columns — drop / keep nullable-legacy / re-purpose — is pending {56.3} TECH Planner + Liam decision (see OQ-CMI-56-4 below).
  • C-14 (RETIRED-S276). Code-fence guard was needed because heading-detection-via-regex (retired chunkByHeadings) would misfire inside fenced code. Budget-split with RecursiveSplitter does not look for headings, so the code-fence concern is moot. AST-awareness via tree-sitter is a property of the op when language is supplied (KH markdown / plain text path does NOT require it), but it is not the same invariant.

Implications for {56.3} TECH (amend round 2 dispatch — separate fresh Planner):

• New §2.x for the RecursiveSplitter configuration: ratified chunk_size value (likely >=500 to preserve C-10 single-chunk UX for short docs); ratified chunk_overlap value (TBD-{56.3}); min_chunk_size either default (chunk_size / 2) or override.
• §2.2 chunking table rows C-10..C-14 entirely re-targeted (now cocoindex-served via budget split).
• §6 Verification adds OQ-3 import-and-call check on cocoindex.ops.text.RecursiveSplitter at cocoindex==1.0.3 (already empirically PASSED at S276 amend round 2 — record the verification block matching the V-2 / V-3 RESEARCH §5 row).
• §4 Risks row: “heading-derived columns become NULL on cocoindex-emitted chunks” — search_content_chunks RPC consumer audit needed; the RPC’s Returns currently surface heading_text / heading_level / heading_path, which will now be NULL on every cocoindex chunk. Consumer-side impact assessment is {56.3} Planner + {56.4} PLAN Subtask scope.

Implications for {56.4} PLAN (separate fresh Planner via planning-and-task-breakdown):

The Subtask shape under branch (c) per §4 expands:

• C-10..C-14 PRODUCT rewrite Subtask (THIS amend round 2 work counts as the spec amend — does not need a separate Subtask in PLAN).
• RecursiveSplitter configuration spike Subtask — determine chunk_size / chunk_overlap empirically via retrieval-quality eval against a corpus sample.
• Retrieval-quality eval Subtask — compare current heading-split retrieval (Path-2 baseline if still available pre-cutover) vs budget-split retrieval on a representative corpus sample (e.g. ID-49.10 fixture-staging corpus once landed).
• Heading-derived columns disposition Subtask — drop / keep-nullable-legacy / re-purpose, per OQ-CMI-56-4 ratification.
• search_content_chunks RPC consumer audit Subtask — MCP search_content_chunks tool (lib/mcp/tools/search.ts), MCP content tool reads at lib/mcp/tools/content.ts:182, MCP formatter Zod schema at lib/mcp/formatters/search.ts:184. Each caller’s tolerance for NULL heading-derived columns must be assessed; UI surfaces displaying heading_path may need adaption.

OQ-CMI-56-4 — Heading-derived columns disposition (NEW, S276 amend round 2)

Question: Under OQ-CMI-56-3 ratified (i) cocoindex.ops.text.RecursiveSplitter, the content_chunks heading-derived columns (heading_text, heading_level, heading_path, parent_chunk_id) become NULL on every chunk row the cocoindex stage emits (budget-split does not preserve heading boundaries). What is the disposition?

• (a) Keep as nullable legacy (default if Liam does not intervene). The columns stay in the schema as NULL on cocoindex-emitted rows; rows from prior population paths (e.g. q_a_pair soft-archive) MAY retain populated values. search_content_chunks RPC Returns continue to surface them. Consumers tolerate NULL.
• (b) Drop the columns. Schema migration to remove heading_text, heading_level, heading_path, parent_chunk_id from content_chunks. search_content_chunks RPC + Zod schema + consumer call-sites need coordinated retirement. Higher impact; cleaner schema; loses any historical heading-context for soft-archived rows.
• (c) Re-purpose for AST-aware boundaries. If {56.3} ratifies the language parameter on .split() for markdown / code paths, the tree-sitter-derived node-type could populate heading_text semantically (e.g. function name, section name) — but this is speculative and out of scope for v1.

Recommendation: (a) keep as nullable legacy — minimal coordinated impact; preserves soft-archived row semantics; defers schema cleanup to v1.1.

Urgency: low. Blocking: false (default to (a) absent Liam intervention). Authority: Liam, post-hoc OK at {56.3} Checker report.

Surface OQ-CMI-56-4 to Liam at {56.3} dispatch — non-blocking on the dispatch itself.