ID-69 — Client corpus & AI-readiness onboarding model: cross-workspace canonical ingest

ID-69 — Client corpus & AI-readiness onboarding model: cross-workspace canonical ingest

{69.1} RESEARCH artefact. This is the investigation phase of the spec-driven lifecycle: it scopes the problem, records verified current-state findings, lays out options with trade-offs, and surfaces decisions for Liam. It is not an implementation plan and does not author PRODUCT / TECH / PLAN — it recommends the downstream chain shape only. Liam ratifies this before {69.2} PRODUCT.

• Task: ID-69 “Client corpus & AI-readiness onboarding model — cross-workspace canonical ingest”.
• Current roadmap theme: 3 “Cocoindex canonical pipeline” (horizon: now, in_progress). A rebind recommendation is made in §10.
• Authored: S290 (31/05/2026), agent worktree off main (7af3d3fc).
• Status: Draft, pending ratification.

---

1. Problem framing

The platform vision Liam ratified this session: the corpus is a Wikipedia-style canonical record layer. A record exists ONCE, is kept accurate, and is reused with confidence across MANY workspaces and activities (procurement, sales, marketing, …). It is deliberately NOT organised by workspace. The pipeline is not pointed at a client’s entire document dump — the corpus is CLEAN and CURATED: content enters only once it has been created in an “AI-ready canonical form”. The real product value (deferral-register C-7, docs/themes/canonical-pipeline/reference/deferral-register.md:87) is helping SMBs restructure their data — plain files OR databases — into a shape AI agents can consume with low integration overhead.

This vision is already ratified at the architecture level: decision-graph Q2.1 (docs/themes/canonical-pipeline/reference/decision-graph.md:181) records “External folder canonical … DB = materialised index + curated derivations. Aligns with cocoindex outside-the-system model + Wikipedia one-golden-record + SMB data-fix product framing.” ID-69 is the feature-level binding of that already-ratified direction.

The central architectural gap this RESEARCH investigates and confirms (§4): the cocoindex ingest writes the canonical content_items rows but NEVER populates the content_item_workspaces many-to-many table. The cross-workspace association — the literal heart of the Wikipedia model — has a schema home but no ingest path. The single workspace_id the ingest does resolve is consumed only by the Path-B form-write, and only after the content_items row is already declared.

Use-case anchors

• Phew Q&A pairs (live). Phew, the first client, has already authored their own Q&A pairs — one live use case sitting on the canonical layer today.
• Phew product guides (next). Phew wants product guides next; these generalise to future clients. Both are derived from the underlying canonical data, not separately ingested dumps.

---

2. Method and scope of this investigation

All findings were derived from the repository at HEAD on main (7af3d3fc) on 31/05/2026:

• Code-intelligence orientation (§3) — gitnexus_query over the domain vocabulary, gitnexus_context on resolve_workspace, and ast-dataflow column-writes on content_item_workspaces to enumerate every writer.
• Direct reads of scripts/cocoindex_pipeline/flow.py, scripts/cocoindex_pipeline/workspace_resolver.py, scripts/cocoindex_pipeline/adapters.py, and lib/intelligence/pipeline.ts.
• Schema reads of supabase/migrations/20260416102457_pre_squash_reconciliation.sql.
• Ledger / theme reads: docs/reference/task-list.json (ID-45, ID-66), docs/reference/product-roadmap.json (themes 3/5/7/12), the canonical-pipeline deferral-register.md and decision-graph.md, and docs/specs/ID-68-…/RESEARCH.md.
• Empirical pin check (§9) — cocoindex==1.0.3 import-verified against the installed package.

Scope boundary. This RESEARCH scopes, characterises the gap, and recommends the downstream chain shape. It does not author PRODUCT / TECH / PLAN, does not write code, and does not set the theme rebind — it recommends it for Liam’s call.

---

3. Code-intelligence orientation (mandatory, cited verbatim)

3.1 gitnexus_query

gitnexus_query({query: "content_item_workspaces workspace resolution corpus ingest", repo: "knowledge-hub"}) returned zero execution flows (processes: [], process_symbols: []) for the canonical-corpus-to-workspace association concept. The definitions it surfaced are telling: the only workspace-link execution flow in the graph is the intelligence pipeline — lib/intelligence/pipeline.ts:ensureWorkspaceLink (L548–568) and storeAsContentItem (L576–734) — plus resolve_workspace and the cocoindex ingest_file / app_main symbols. There is no graph-level flow connecting cocoindex content_items writes to a content_item_workspaces write. The empty process set IS the finding: the cross-workspace ingest path does not exist as a traced flow.

3.2 gitnexus_context

gitnexus_context({name: "resolve_workspace", repo: "knowledge-hub"}) — verdict context: resolve_workspace (scripts/cocoindex_pipeline/workspace_resolver.py:144-185) has exactly one non-test caller: ingest_file (scripts/cocoindex_pipeline/flow.py). Every other incoming call is a unit test in scripts/tests/test_workspace_resolver.py. outgoing: {}, processes: []. Blast radius of the single production caller is the Path-B form-write block only (§4.3). This is LOW structural risk for an additive change but HIGH product significance — the symbol that resolves a workspace is wired to exactly one downstream write, and it is the wrong one for the canonical model.

3.3 ast-dataflow — who writes content_item_workspaces

bun scripts/ast-dataflow-cli.ts column-writes --table content_item_workspaces --column workspace_id (and --column content_item_id) returned an identical, exhaustive writer set (8 sites, none in the cocoindex pipeline):

File	Line	Method	Nature
lib/intelligence/pipeline.ts	565	insert	RSS/intelligence feed pipeline (production)
app/api/items/[id]/workspaces/route.ts	145, 170	insert	Manual UI/API association (production)
__tests__/integration/intelligence-golden-path.integration.test.ts	460, 849	insert	Test
e2e/fixtures/test-data-fixture.ts	243, 440	insert	Test fixture
e2e/helpers/data-factory.ts	246	insert	Test fixture

A cross-corpus grep for content_item_workspaces over scripts/**, lib/**, app/**, supabase/migrations/** confirms the cocoindex pipeline (scripts/cocoindex_pipeline/**) contains zero references to the table. The only scripts/ hits are audit-opaque-json-rpcs.ts (read) and cleanup-stale-test-artifacts.ts (cleanup list). Verdict: the canonical cocoindex ingest does not write content_item_workspaces.

---

4. Central finding — the cross-workspace ingest gap (verified)

4.1 The schema already supports the Wikipedia model

content_items has no workspace_id column. Records associate to workspaces via the many-to-many junction supabase/migrations/20260416102457_pre_squash_reconciliation.sql:3511:

CREATE TABLE IF NOT EXISTS "public"."content_item_workspaces" (
    "content_item_id" "uuid" NOT NULL,
    "workspace_id" "uuid" NOT NULL,
    "assigned_at" timestamp with time zone DEFAULT "now"(),
    "id" "uuid" DEFAULT "gen_random_uuid"()
);

FK cascades to both parents (…:5361 content_item_id → content_items, …:5366 workspace_id → workspaces, both ON DELETE CASCADE). RLS: ciw_insert allows admin/editor (…:5958); ciw_select is USING (true) (…:5962). The many-to-many shape is exactly the Wikipedia one-record-many-workspaces model. The PK is the composite (content_item_id, workspace_id) (legacy constraint name content_item_projects_pkey, per lib/intelligence/pipeline.ts:543-544). Only form_templates.workspace_id is NOT NULL and single-workspace — the Path-B activity output, correctly single-tenant.

4.2 Path A (canonical content) associates NO workspace today

ingest_file (scripts/cocoindex_pipeline/flow.py:1106-1435) declares the canonical rows: source_documents (…:1247), content_items (…:1266 — the record itself, with content_text, embedding, source_document_id, primary_domain, primary_subtopic), content_chunks (…:1322), q_a_extractions (…:1353), entity_mentions (…:1393). None of these declare blocks touches content_item_workspaces. The content_item_id is a deterministic uuid5(_KH_PIPELINE_DOC_NS, "ci:" + rel_path) (…:1240) — a stable canonical identity — but nothing links that identity to any workspace.

4.3 The single resolved workspace_id feeds only Path-B form-write

resolve_workspace(manifest, rel_path) is called at flow.py:1433 — after the content_items declare (…:1266) and gated behind an early return when no manifest is bound (…:1422-1424). Its result, workspace_id, is used solely to populate form_templates.workspace_id (…:1473, …:1542). It never reaches a content_item_workspaces write. The resolver itself is a clean, deterministic longest-prefix matcher (workspace_resolver.py:145-186) that maps ONE rel_path → ONE workspace_id. It resolves exactly one workspace per file — which is structurally the wrong cardinality for a canonical record that should associate to MANY workspaces.

4.4 The intelligence pipeline already does it correctly — the in-repo precedent

lib/intelligence/pipeline.ts is the working reference implementation of the Wikipedia model inside KH today. storeAsContentItem (L576-734) comments the contract explicitly (L589-593): “one content_items row per unique source_url, linked to many workspaces via content_item_workspaces. If the URL is already in the KB, attach the existing row to this workspace and skip creating a duplicate.” ensureWorkspaceLink (L549-568) pre-checks the junction and inserts the (workspace_id, content_item_id) pair idempotently. This is the exact behaviour the canonical cocoindex ingest is missing — a TypeScript precedent the cocoindex Python flow can mirror in shape (dedup-on-identity + idempotent junction upsert).

4.5 Verdict

CONFIRMED — the cross-workspace canonical association has a schema home (content_item_workspaces) but no ingest path in the cocoindex pipeline. The heart of the Wikipedia model is unimplemented on the canonical write path. The ingest SHOULD, after declaring the content_items row, declare one content_item_workspaces row per workspace the record belongs to — which means the ingest needs a way to express MANY workspaces per file, not the one-per-file the current manifest resolver yields (§6).

---

5. What the ingest SHOULD do (characterisation, not design)

The PRODUCT/TECH chain must decide the concrete mechanism, but this RESEARCH characterises the shape the gap implies:

1. Many-to-many association at declare time. After the content_items declare (flow.py:1266), declare one content_item_workspaces row per workspace the record serves. Idempotent on the composite PK (mirror the ensureWorkspaceLink pre-check / upsert idiom, §4.4) so a re-ingest re-stamps rather than duplicates — consistent with the deterministic uuid5 identity already used for every other row (flow.py:1234-1240).

2. A multi-workspace mapping, not single-prefix resolution. The current resolve_workspace is single-valued by design (one rel_path → one workspace_id, longest-prefix-wins, workspace_resolver.py:154-186). The canonical model needs a record to express MANY workspaces. Options for the TECH spec to weigh: (a) extend the manifest mapping to workspace_ids: [...] per prefix; (b) a separate canonical-association manifest distinct from the Path-B form manifest; (c) post-ingest association via the existing app/api/items/[id]/workspaces surface as a curation step. This is an open design choice, not settled here.

3. Manifest is currently MANDATORY and aborts the flow. A missing / unparseable .kh-workspace-map.json ABORTS the whole flow with a structured manifest_missing / manifest_invalid error (flow.py:1701-1725), not just Path B. So the canonical ingest is already coupled to a manifest at the root — but consumes it only for forms. The PRODUCT/TECH chain must decide whether the canonical-workspace mapping rides the same manifest or a separate one, and whether mandatory-manifest semantics should change for clean-corpus runs.

---

6. “AI-ready canonical form” as client-facing data-readiness guidance

deferral-register C-7 (deferral-register.md:87) binds two provisional items (PLAN.md §7 items 14/15) to “the onboarding feature spec / data-fix workstream”, gated on Liam ratification of decision-graph CX.28 and CX.29. Both are RESOLVED:

• CX.28 (decision-graph.md:236, RATIFIED OQ2 …:460): “folder connection REQUIRED at first-run for v1 … New onboarding flow needed: ‘connect your content folder’.” The platform requires external-folder configuration as a first step.
• CX.29 (decision-graph.md:237, RATIFIED OQ3 …:461): “SMB data-fix angle … critical framing for v1; dedupe-with-temporal as the sole v1 feature; remaining items review post-architecture.” Substrate ratified S230 as the S10 hybrid (cocoindex chunk-embedding + skill-seekers keyword co-confirmer).

What this means concretely for “AI-ready canonical form”:

• The product is explicitly a data-restructuring offer (C-7: “helping SMBs restructure their data — plain files OR databases — into a shape AI agents can use with low integration overhead”). “AI-ready” is a client-facing readiness bar, not just an internal pipeline contract.
• Files vs databases. The pipeline today accepts only a fixed binary/text set — .pdf/.docx/.xlsx/.html/.htm/.md/.markdown/.txt — and RAISES on anything outside it (adapters.py:74-77), with no include/exclude filter (§7, a real gap). Database sources (the “or databases” half of C-7) have no ingest path at all today. The readiness model must cover both, even if v1 scopes to files.
• Low integration overhead is the value proposition: the client hands over source material, the platform produces the clean canonical record. The v1 data-fix feature is dedupe-with-temporal (CX.29) — i.e. the canonical layer resolves “the same record appearing twice” so downstream activities trust one golden source.

The PRODUCT spec should turn “AI-ready canonical form” into a concrete, client-readable checklist (what enters the corpus, in what shape, and what the client must do to a plain-file or database source first), distinct from the internal ingest contract.

---

7. Curation / onboarding model — implications for ingest, identity, re-ingest

The “content enters only once, in canonical form, NOT a wholesale dump” principle has concrete ingest consequences:

• No-dump implies a filter the pipeline lacks. walk_dir is a single recursive walk of the source root (flow.py:1802-1806, recursive=True, live=True). Combined with the raise-on-unsupported adapter (adapters.py:74-77) and no include/exclude filter today, pointing the pipeline at a raw client folder would crash on the first unsupported file. A curated corpus needs either a clean staged folder (operator discipline) or an ingest-side allowlist/manifest of what is canonical. This is a v1 gap the PRODUCT/TECH chain must close.
• Identity is path-derived and fragile to moves. The per-document UUIDs are uuid5(namespace, "ci:" + rel_path) etc. (flow.py:1234-1240); moving a file changes rel_path and therefore re-ingests it as a NEW identity (flow.py:1234-1238 docstring). For a curated canonical layer this is a feature (stable path = stable record) but a hazard (reorganising the staged folder forks identities). Curation discipline must treat the staged layout as part of the canonical contract — this is the base-path drift concern (§8).
• Classification is LLM-driven, not path-driven. content_type / primary_domain come from the classifier (flow.py:1278-1279, …:1350), not the folder. So the curated folder layout does NOT encode taxonomy — taxonomy is derived. This decouples “where the file sits” from “what it is”, which is correct for the Wikipedia model but means workspace association cannot be inferred from content_type either; it must be explicit (§5.2).
• Re-ingest is idempotent by design. @coco.fn(memo=True) skips unchanged bytes (flow.py:1156-1158); deterministic PKs make a changed-bytes re-ingest an UPSERT, not a duplicate. The canonical-workspace association added per §5 must preserve this — idempotent junction upsert, not blind insert.

---

8. Intersection map — where ID-69 overlaps / hands off

Adjacent work	What it is	Overlap with ID-69	Hand-off boundary
ID-45 / T7 (task-list.json, status spec_needed, deps [28,41,36])	“T7 — Phew full-corpus reingest via cocoindex (all content types)”. Stage complete Phew corpus, run canonical flow E2E, validate all content types, promote Q&A extractions to q_a_pairs.	T7 is the execution of a full-corpus ingest; ID-69 is the model (cross-workspace association + AI-readiness + curation discipline) that T7 needs to be correct. T7 would today ingest content with NO workspace association (§4) — the exact gap ID-69 closes.	ID-69 defines the canonical-association + readiness model; T7 consumes it to actually reingest Phew. ID-69 should land before / gate T7 (or T7 absorbs ID-69’s outcome). Surface to Orchestrator: T7’s deps may need ID-69 added.
Base-path drift (the “bl-206” concern in the brief)	COCOINDEX_SOURCE_PATH defines the source root; rel_path is relative to it (flow.py:1647-1662, 1205). If the base path drifts between runs, every rel_path shifts and every record re-ingests as a new identity (§7).	Directly couples to ID-69’s identity/curation model: a stable, agreed base path IS part of the canonical contract. The manifest also lives at source_path/.kh-workspace-map.json (flow.py:1701) so base-path drift breaks manifest resolution too.	ID-69 should treat base-path stability as an invariant of the canonical contract. The infra side (mounting a stable corpus volume) is ID-66 (below).
ID-66 (task-list.json, in_progress)	On-prem pivot. {66.9} “mount LMDB persistent volume + corpus volume” (done); {66.15} “Datapath monitor — pipeline_runs row-arrival watch” (pending).	ID-66 provides the physical corpus location (the mounted volume that backs COCOINDEX_SOURCE_PATH) and the datapath monitor that observes ingest. ID-69 defines what lands there and how it associates.	ID-66 owns the volume + monitor infra; ID-69 owns the data model + curation discipline that runs on top. Clean hand-off; ID-69 should reference the ID-66 corpus volume as the canonical staging location.
ID-68 (docs/specs/ID-68-repo-visibility-ip-separation/RESEARCH.md, theme 12, horizon next)	Repo visibility / IP separation — proprietary + client content must leave the tracked tree before the repo goes public.	The canonical corpus is client-proprietary and must NOT live in the tracked repo. ID-68 wants client content OUT of the tree; ID-69’s external-folder-canonical model (Q2.1) is the architectural answer — the corpus lives in a mounted external folder (ID-66 volume), never committed.	ID-68 owns the scrub/relocation of any client content currently tracked; ID-69 confirms the canonical corpus belongs in the external folder by design, not the repo. Mutually reinforcing — cite ID-68 as the IP-protection rationale for external-folder-canonical.

---

9. Empirical verification (pre-ratification, Q-EX2)

This RESEARCH cites the external library cocoindex. Per the Q-EX2 forcing function, the cited symbols were import-verified against the pinned version.

Item	Value
Date	31/05/2026
Pinned version	cocoindex[postgres]==1.0.3 (requirements.txt)
Installed version	cocoindex 1.0.3 (python3 -c "import cocoindex; print(cocoindex.__version__)")
Symbols cited	cocoindex.connectors.localfs.walk_dir, cocoindex.connectors.postgres.mount_table_target, cocoindex.connectorkits.target.ManagedBy, cocoindex.ops.text.RecursiveSplitter
Result	PRESENT — these are the exact import paths the flow uses (flow.py:56-63, …:1311). Pin matches install. No drift.

Note: these symbols are cited as current-state description of the existing flow, not as new API the chain must call. The cocoindex 0.3.x → 1.0.x restructure that bit Q-EX2/S252 is already absorbed in this codebase (the flow is written against 1.0.3 throughout, e.g. flow.py:1119 “cocoindex 1.0.3 calls this component as …”). No ABSENT / SIGNATURE_DRIFT findings.

---

10. Recommended downstream spec-chain shape + theme-rebind call

10.1 Chain shape

Recommend the full chain: {69.2} PRODUCT → {69.3} TECH → {69.4} PLAN.

Rationale:

• PRODUCT is required. ID-69 carries genuine product-and-architectural ambiguity (the open design choice in §5.2: how a record expresses many workspaces) and a client-facing surface (the “AI-ready canonical form” readiness model + onboarding “connect your content folder” flow, CX.28). The behaviour the client sees and the invariants of the canonical layer both need numbered, testable statements.
• TECH is required. The work is cross-stack: Python cocoindex pipeline (junction write + multi-workspace mapping), Supabase (the content_item_workspaces write path + possibly manifest-schema changes), and the existing Next.js association API (app/api/items/[id]/workspaces). It mirrors an existing TypeScript precedent (§4.4) that the Python side must replicate in shape. This is exactly the “spans multiple subsystems” trigger for a tech spec.
• PLAN is warranted. Compound: the junction-write slice, the multi-workspace-mapping slice, the include/exclude-filter gap (§7), and the client-facing readiness/onboarding surface are distinct, chain-dependent slices. Decomposition will exceed a single ≤2h slice. Recommend a {69.4} PLAN after the PRODUCT+TECH pair ratifies.

One scoping caution for the chain: the brief’s vision spans both a narrow, shippable core (close the content_item_workspaces ingest gap — §4) AND a broad product framing (SMB data-restructuring, files-and-databases readiness — §6). The PRODUCT spec should explicitly separate the v1 core (canonical M2M ingest + file-source readiness, gating ID-45/T7) from the v1.1+ data-fix breadth (database sources, the full restructuring product). Folding the whole breadth into one Task risks the 25-Subtask ceiling and a Task-boundary problem.

10.2 Theme-rebind call

Recommend: do NOT rebind to theme 7. Keep the ID-69 ingest-gap CORE on theme 3, and surface the client-facing onboarding slice as a theme-5 dependant (preferred); if Liam wants a single Task, theme 5 “Ingestion UX” beats theme 7.

Reasoning:

• The brief proposes rebinding from theme 3 (“Cocoindex canonical pipeline”, horizon now, in_progress) to theme 7 (“Onboarding and UX polish”, horizon later). But theme 7’s scope boundary is explicit: “ingest-specific UX belongs to Procurement; … this theme owns onboarding + cross-cutting UX polish” and it is horizon later. Rebinding the ingest-gap core to a “later” theme would de-prioritise the exact gap that gates ID-45/T7.
• The closer fit for the client-facing half is theme 5 “Ingestion UX” (horizon next, pending), whose description is literally “the client-non-developer ingest path … a client admin uploads source material … result lands in the KB via the canonical pipeline”, with the scope boundary “the data shape it lands belongs to Knowledge base foundations; the pipeline belongs to Cocoindex canonical pipeline”.
• The clean split honours those scope boundaries: the content_item_workspaces ingest gap (§4) is canonical-pipeline data-shape work → stays theme 3 (horizon now), where it can gate T7. The “connect your content folder” + AI-readiness onboarding surface (§6, CX.28) is client-facing ingest UX → theme 5 (horizon next). Theme 7 is the wrong home for either half.
• Recommendation to the Orchestrator: keep the ID-69 core on theme 3; if Liam wants a single Task spanning both, prefer theme 5 over theme 7. Either way, flag that ID-45/T7’s dependency set ([28,41,36]) likely needs ID-69 added.

---

11. Open questions for Liam (ratify before {69.2} PRODUCT)

• OQ-69-1 (multi-workspace mapping mechanism). Should a canonical record’s many-workspace association ride (a) an extended manifest (workspace_ids: [...] per prefix), (b) a separate canonical-association manifest, or (c) post-ingest curation via app/api/items/[id]/workspaces? (§5.2)
• OQ-69-2 (v1 scope boundary). Is v1 the narrow ingest-gap core (M2M association + file-source readiness, gating T7), with database-source restructuring deferred to v1.1? (§10.1)
• OQ-69-3 (theme binding). Theme 3 core + theme 5 onboarding slice (this RESEARCH’s recommendation), single theme-5 Task, or the brief’s theme-7 rebind? (§10.2)
• OQ-69-4 (include/exclude filter). Should the curated-corpus discipline be enforced ingest-side (an allowlist/manifest so unsupported files are skipped not raised, §7) or operator-side (clean staged folder only)? (§7)
• OQ-69-5 (ID-45/T7 sequencing). Should ID-69 formally gate ID-45/T7 (added to T7’s dependencies), given T7 would otherwise ingest with no workspace association? (§8)

---

12. Provenance pointers

• Vision / Wikipedia-canonical framing: this session (Liam, S290); decision-graph Q2.1 (decision-graph.md:181).
• Cross-workspace gap: scripts/cocoindex_pipeline/flow.py:1106-1561, workspace_resolver.py:144-186, content_item_workspaces schema (…pre_squash_reconciliation.sql:3511), in-repo precedent lib/intelligence/pipeline.ts:540-734.
• AI-readiness / data-fix framing: deferral-register C-7 (…:87); decision-graph CX.28/CX.29 (…:236-237, RATIFIED …:460-461).
• Intersections: ID-45 (task-list.json task 45), ID-66 (task-list.json task 66 {66.9}/{66.15}), ID-68 (docs/specs/ID-68-repo-visibility-ip-separation/RESEARCH.md).
• Roadmap themes 3/5/7/12 (product-roadmap.json).

13. Changelog

Date	Session	Note
31/05/2026	S290	{69.1} RESEARCH authored. Verified the content_item_workspaces ingest gap (cocoindex writes content_items but never the M2M junction; intelligence pipeline is the in-repo precedent). Characterised AI-ready canonical form (CX.28/CX.29 RATIFIED). Mapped intersections (ID-45/T7, base-path drift, ID-66, ID-68). Recommended full PRODUCT→TECH→PLAN chain + theme-3-core / theme-5-onboarding split (NOT theme 7). Pending Liam ratification of OQ-69-1..5.