ID-69 — Client corpus & AI-readiness onboarding model: cross-workspace canonical ingest

ID-69 — Client corpus & AI-readiness onboarding model: cross-workspace canonical ingest

{69.2} PRODUCT artefact. This is the product-behaviour phase of the spec-driven lifecycle: it states what the canonical ingest and curation model must do, as numbered, testable Behavior invariants a Checker can verify against acceptance criteria. It does not decide implementation mechanism (that is {69.3} TECH) and does not decompose into Subtasks (that is {69.4} PLAN).

• Task: ID-69 “Client corpus & AI-readiness onboarding model — cross-workspace canonical ingest”.
• Predecessor: {69.1} RESEARCH.md (ratified, S290). Read it for the verified current-state gap analysis; this PRODUCT encodes the S291-ratified OQ answers (OQ-69-1..5) on top of those findings.
• Theme binding (OQ-69-3, ratified): theme 3 “Cocoindex canonical pipeline” (horizon now) is the CORE home; the client-facing onboarding slice rides theme 5 “Ingestion UX” as a dependant. Not theme 7.
• Authored: S291 (31/05/2026), agent worktree off local branch canonical-pipeline-setup.
• Status: Draft, pending ratification. Liam ratifies this before {69.3} TECH.
• Language: UK English throughout.

---

Summary

Close the canonical-association ingest gap so the Wikipedia-style corpus is real end-to-end: a single content_items record can be associated to many workspaces via an idempotent content_item_workspaces junction-write contract, keyed on the composite (content_item_id, workspace_id). v1 drives that contract operator-side (post-ingest curation via the existing app/api/items/[id]/workspaces surface); v1.1 adds an ingest-side declarative writer over the same, unchanged contract. v1 also establishes a file-source AI-readiness model and curated-corpus discipline so ID-45/T7 (Phew full-corpus reingest) can run on top of a correctly-associated canonical layer.

---

Problem

The platform corpus is a Wikipedia-style canonical record layer: a record exists once, is kept accurate, and is reused with confidence across many workspaces and activities (procurement, sales, marketing, …). It is deliberately not organised by workspace. RESEARCH §4 confirmed the central gap: the cocoindex canonical ingest (scripts/cocoindex_pipeline/flow.py) writes the content_items rows but never populates the content_item_workspaces many-to-many junction. The cross-workspace association — the literal heart of the Wikipedia model — has a schema home but no ingest path. The single workspace_id the ingest resolves (resolve_workspace, flow.py:1433) is consumed only by the Path-B form_templates write, after the content_items row is already declared.

Code-intelligence orientation (cited verbatim from RESEARCH §3, mandatory per .gitnexus/CLAUDE.md “Always Do”; this PRODUCT inherits the orientation its ratified predecessor ran against the same HEAD):

• gitnexus_query({query: "content_item_workspaces workspace resolution corpus ingest", repo: "knowledge-hub"}) returned zero execution flows (processes: [], process_symbols: []). The empty process set IS the finding — the cross-workspace ingest path does not exist as a traced flow. The only workspace-link flow in the graph is the intelligence pipeline (lib/intelligence/pipeline.ts:ensureWorkspaceLink L549-569, storeAsContentItem L577-734).
• gitnexus_context({name: "resolve_workspace", repo: "knowledge-hub"}) — resolve_workspace (workspace_resolver.py:144-185) has exactly one non-test caller: ingest_file (flow.py). Verdict: LOW structural risk for an additive change, HIGH product significance (it resolves a workspace and is wired to the wrong downstream write for the canonical model). Blast radius = the Path-B form-write block only.
• ast-dataflow column-writes --table content_item_workspaces returned an exhaustive 8-site writer set, none in scripts/cocoindex_pipeline/**. The in-repo precedent that does it correctly is lib/intelligence/pipeline.ts (production), plus the manual association surface app/api/items/[id]/workspaces/route.ts:145,170.

In-repo precedent (RESEARCH §4.4). storeAsContentItem (lib/intelligence/pipeline.ts:577-734) documents the contract verbatim (L589-593): “one content_items row per unique source_url, linked to many workspaces via content_item_workspaces. If the URL is already in the KB, attach the existing row to this workspace and skip creating a duplicate.” ensureWorkspaceLink (L549-569) pre-checks the junction and inserts the (workspace_id, content_item_id) pair idempotently on the composite PK (content_item_id, workspace_id) (pipeline.ts:543-544). This is the exact behaviour the canonical ingest is missing.

Use-case anchors

• Phew Q&A pairs (live). Phew, the first client, has authored their own Q&A pairs — one live use case on the canonical layer today.
• Phew product guides (next). Both are derived from the underlying canonical data, not separately ingested dumps. They depend on the canonical records being correctly associated to the Phew workspace(s).

---

Goals / Non-goals

Goals

Close the canonical-association ingest gap and establish a curated-corpus discipline so the Wikipedia model is real end-to-end: one canonical record associable to many workspaces, idempotently, via an explicit and forward-compatible junction-write contract; and the corpus stays clean (one record, in AI-ready form, never a wholesale dump).

Success looks like: a curated source folder ingests into content_items with correct content_item_workspaces associations; re-ingesting the same corpus re-stamps the same associations without duplication; ID-45/T7 can run on top of this model with workspace association working; and the junction-write contract is identical whether v1 (operator-side curation) or v1.1 (ingest-side declarative manifest) drives it — so v1.1 adds a writer without reworking the contract.

Non-goals (explicit)

• Building any client-facing onboarding UI (“connect your content folder”, CX.28). v1 ships a contract/checklist, not a UI. → v1.1 (C-7).
• Ingest-side include/exclude filtering or skip-not-raise. → v1.1 (OQ-69-4).
• Database-source ingest or broad SMB data-restructuring breadth. → v1.1 (C-7).
• Cross-workspace dedup / UC8 Candidate C. → v1.1 (V11-10).
• Cross-workspace ops / scope-tag taxonomy. → v1.1 (C-6 / UC9).
• Coupling to the upload route source binding (app/api/upload/route.ts). → v1.1 (C-2).
• Writing source_documents.workspace_id, or any workspace-on-content_items shortcut. → never (architecturally wrong, see Scope boundary / BI-2).
• Ontology / controlled-vocabulary hardening (content_type / ingest_source / entity_types value-sets). → separate data-quality concern, out of scope.
• Changing the mandatory-manifest abort semantics for the Path-B form-write. Whether the canonical-workspace mapping rides the same manifest or a separate one, and any change to mandatory-manifest semantics, is a TECH design choice bounded by BI-4’s contract-stability requirement.

---

Scope boundary

v1 vs v1.1 boundary (OQ-69-2, ratified — load-bearing)

The most important scope statement in this spec. The v1→v1.1 gap is deliberately negligible: nothing in v1 may be specified in a way that v1.1 would have to rework. v1.1 covers deferral-register items V11-2, V11-3, V11-4, V11-7, V11-9, V11-10 and C-1, C-2, C-4, C-5, C-6, C-7 (docs/themes/canonical-pipeline/reference/deferral-register.md).

Concern	v1 (this Task — theme 3 core + theme 5 slice)	v1.1+ (deferred)
Cross-workspace association	content_item_workspaces written by an idempotent upsert keyed on (content_item_id, workspace_id). v1 driver = operator-side post-ingest curation via app/api/items/[id]/workspaces (OQ-69-1 option c).	Ingest-side declarative writer (extended manifest workspace_ids: [...] per prefix — OQ-69-1 option a), reusing the identical junction contract.
Source kinds	File sources only (the fixed binary/text set the pipeline already accepts).	Database sources (the “or databases” half of C-7); broad SMB data-restructuring.
Curation / include-exclude (OQ-69-4)	Operator-side: a clean, staged folder is the contract. No ingest filter. The curated corpus is an operator responsibility.	Ingest-side allowlist/manifest so unsupported files are skipped, not raised (today adapters.py:74-77 raises).
Cross-workspace dedup	Association is dedup-agnostic — v1 makes no cross-workspace dedup decision (watch V11-10 / UC8 Candidate C).	UC8 Candidate C cross-workspace dedup upgrade.
Client onboarding UX	None. No client-facing onboarding UI in v1 (watch C-7). v1’s readiness model is a checklist/contract document, not a UI.	”Connect your content folder” onboarding flow (CX.28) + cross-workspace ops / scope-tag (C-6 / UC9).
Upload-route source binding	Not depended upon (watch C-2). v1 must not couple to app/api/upload/route.ts.	P7 upload-route disposition binds at cocoindex source-binding architecture.

Forward-compatibility watch items (must hold in v1 to avoid v1.1 rework): C-7 (no client-facing onboarding UX in v1); V11-10 (keep v1 association idempotent and dedup-agnostic); C-6 (no cross-workspace ops / scope-tag surface in v1); C-2 (v1 must not depend on the upload-route source binding).

Provenance / ontology boundary (investigated S291)

• source_documents is the provenance/lineage layer (FK’d from content_items.source_document_id), correctly written by cocoindex; its workspace_id column stays NULL in canonical ingest. Workspace association rides content_item_workspaces (M2M), NOT source_documents.workspace_id. (Stated explicitly so no downstream spec writes workspace_id to source_documents — architecturally wrong, it would break one-record-many-workspaces.)
• The ontology (controlled-vocabulary value-sets for content_type / ingest_source / entity_types) is a read-only validation boundary, core-seeded, never written by ingest → out of scope for ID-69. Ontology hardening is a separate data-quality concern; not specified here.

---

Behavior

Each invariant is numbered and testable. The acceptance criterion states how a Checker verifies it.

BI-1 — Canonical record has no intrinsic workspace; association is M2M only

A content_items record carries no workspace column. Its association to any workspace exists only as one or more rows in content_item_workspaces. The ingest path writes content_items independently of any workspace, and a record with zero content_item_workspaces rows is a valid (un-associated) canonical record.

• Acceptance: content_items has no workspace_id column (schema check). A canonical record can be queried back after ingest with zero junction rows and is still a complete record (has content_text, embedding, source_document_id). No code path infers a workspace from content_items itself.

BI-2 — Workspace association is written ONLY to content_item_workspaces, never to source_documents

Workspace association for a canonical record is written exclusively to content_item_workspaces. The canonical ingest does not set source_documents.workspace_id; that column remains NULL for canonically-ingested provenance rows.

• Acceptance: After ingest of a workspace-associated record, source_documents.workspace_id IS NULL for the corresponding provenance row, while content_item_workspaces holds the (content_item_id, workspace_id) pair. No writer sets source_documents.workspace_id on the canonical path (verifiable by ast-dataflow column-writes --table source_documents --column workspace_id showing no canonical-path writer).

BI-3 — Junction-write contract: idempotent upsert keyed on the composite (content_item_id, workspace_id)

Associating a record to a workspace is an idempotent upsert keyed on the composite PK (content_item_id, workspace_id). Associating the same pair twice produces exactly one junction row, not two; the operation never raises on a pre-existing pair and never creates a duplicate. This mirrors the ensureWorkspaceLink pre-check/upsert idiom (lib/intelligence/pipeline.ts:549-569).

• Acceptance: Calling the association twice for the same (content_item_id, workspace_id) yields a single junction row (count = 1). A re-association of an already-linked pair succeeds (no error) and leaves the row count unchanged. A test asserts row-count invariance across a repeated association.

BI-4 — Junction-write contract is identical across v1 (operator-side) and v1.1 (ingest-side) drivers

The junction-write contract defined in BI-3 is driver-agnostic: the same idempotent-upsert-on-composite-key contract is what the v1 operator-side curation surface (app/api/items/[id]/workspaces) uses and what a v1.1 ingest-side declarative writer (extended manifest workspace_ids: [...]) will use. Adding the v1.1 ingest-side writer must not require any change to the junction contract, schema, or idempotency semantics.

• Acceptance: The junction-write contract is documented once and referenced by both drivers. A reviewer can confirm the v1 operator-side write and the (documented, not-yet-built) v1.1 ingest-side write target the identical composite key with identical upsert semantics. No v1-only assumption (surrogate- key insert, non-idempotent blind insert, workspace-on-record shortcut) is baked in that v1.1 would have to unwind.

BI-5 — A canonical record can be associated to MANY workspaces

The model supports one canonical record associated to multiple workspaces (the Wikipedia one-record-many-workspaces shape). Associating a record to a second workspace does not disturb its first association; both junction rows coexist.

• Acceptance: A single content_item_id can hold ≥2 content_item_workspaces rows (distinct workspace_ids) simultaneously. Querying the record’s workspaces returns all associated workspaces. Adding a second association leaves the first intact.

BI-6 — Re-ingest re-stamps the same canonical identity and preserves associations idempotently

Re-ingesting a file re-stamps the same canonical identity (deterministic uuid5(namespace, "ci:" + rel_path), flow.py:1234-1240) — a changed-bytes re-ingest is an UPSERT of the content_items row, not a duplicate. Existing content_item_workspaces associations for that identity are preserved across re-ingest (the re-ingest does not orphan, drop, or duplicate them).

• Acceptance: Ingest a file, associate it to a workspace, re-ingest the same file (changed bytes). The content_item_id is unchanged, the content_items row is updated (not duplicated), and the prior content_item_workspaces row(s) still resolve the record to the same workspace(s) with the same row count.

BI-7 — Base-path stability is part of the canonical contract; base-path drift forks identity

Because canonical identity is rel_path-derived, the source base path (COCOINDEX_SOURCE_PATH) is a load-bearing part of the canonical contract. Moving or re-rooting the corpus changes every rel_path and therefore forks every record into a new identity. The readiness/curation model must state base-path stability as an explicit operator obligation (the staged layout is part of the canonical contract).

• Acceptance: The readiness/curation contract names base-path stability as a required operator obligation and explains the identity-fork consequence of drift. A reviewer can point to the stated obligation. (No code assertion — this is a documented contract invariant; the infra that mounts a stable corpus volume is ID-66’s concern, referenced not owned.)

BI-8 — Workspace association is explicit, never inferred from folder layout or classification

Workspace association is an explicit decision (operator-side curation in v1). It is not inferred from the curated folder layout, nor from the LLM-derived content_type / primary_domain classification (flow.py:1278-1279, :1350). The folder layout encodes neither taxonomy nor workspace; both are derived/declared separately.

• Acceptance: No code path maps folder path → workspace, or content_type/primary_domain → workspace, for canonical-record association. Association rows exist only because an operator (v1) or a declarative manifest (v1.1) explicitly asserted them. A test/review confirms no implicit folder-or-classification-to-workspace mapping is introduced.

BI-9 — Curated-corpus discipline: a clean staged folder is the v1 contract (operator-side, no ingest filter)

In v1 the corpus is kept clean by operator-side discipline: only AI-ready, supported-format files are staged into the source root. v1 adds no ingest-side include/exclude filter (that is v1.1, OQ-69-4). The readiness contract states that staging an unsupported file is an operator error, and documents that the pipeline today raises on unsupported formats (adapters.py:74-77) — so a clean staged folder is a precondition, not an ingest-time tolerance.

• Acceptance: The readiness/curation contract states the clean-staged-folder operator obligation and that v1 adds no ingest filter. No ingest-side allowlist/skip code is introduced in v1 (a reviewer confirms the raise-on-unsupported behaviour is unchanged for v1). The contract notes the v1.1 upgrade path (skip-not-raise) without implementing it.

BI-10 — AI-ready canonical-form readiness model is a client-readable contract (not a UI, not the internal ingest contract)

v1 produces a concrete, client-readable readiness contract/checklist: what enters the corpus, in what shape, and what a client must do to a plain-file source before it is staged. This is distinct from the internal ingest contract and contains no client-facing UI (watch C-7). It scopes to file sources in v1; database sources are named as v1.1.

• Acceptance: The readiness model exists as a documented, client-readable checklist for file sources, distinct from internal ingest mechanics, with no UI deliverable. It explicitly defers database sources and any onboarding UX to v1.1. A reviewer can confirm the checklist is consumable by a non-developer client admin and contains no implementation/UI scope.

BI-11 — Association is dedup-agnostic in v1

v1 makes no cross-workspace dedup decision. The association contract neither merges nor de-duplicates records across workspaces; it only links existing canonical identities to workspaces. (Watch V11-10 / UC8 Candidate C → v1.1.)

• Acceptance: No v1 code path merges two content_items rows or removes a record on the basis of cross-workspace duplication during association. The association is purely additive/idempotent on the junction. A reviewer confirms no dedup behaviour is introduced that a v1.1 dedup feature would have to re-architect.

BI-12 — Operator-side association is admin/editor-gated (RLS-coupled)

The operator-side association write (v1 driver) is permitted only for admin / editor roles, matching the existing ciw_insert RLS policy on content_item_workspaces (RESEARCH §4.1). Reviewer / viewer roles cannot create associations. Reads of a record’s workspaces are governed by ciw_select (USING (true)).

• Acceptance: An admin or editor can create an association via the operator surface; a reviewer/viewer attempt is rejected at the auth/RLS boundary. The operator API uses getAuthorisedClient() ({ success }) and authFailureResponse(auth) for failure routing. A test asserts the role gate.

BI-13 — ID-69 gates ID-45/T7 (sequencing invariant)

ID-69’s canonical-association + readiness model is a prerequisite for ID-45/T7 (“Phew full-corpus reingest via cocoindex”). T7 must not run a full-corpus reingest that produces canonical records with no workspace association; it consumes ID-69’s model so association works (OQ-69-5, ratified). This PRODUCT states the dependency; the Orchestrator adds ID-69 to T7’s dependencies.

• Acceptance: This spec records the ID-69 → ID-45/T7 gating relationship explicitly and surfaces it to the Orchestrator. (The actual dependencies edit on the ledger is the Orchestrator’s action; the Checker verifies the invariant is stated here, not that the ledger is already edited.)

---

KH quality bars (apply to any downstream implementation)

Surfaced so Executors inherit them via TECH/PLAN:

• No silent Supabase failures. Any TS-side junction read/write uses sb() / tryQuery() from @/lib/supabase/safe; composite responses via warningsEnvelope().
• Auth (see BI-12). getAuthorisedClient() returns { success } (not { authorised }); authFailureResponse(auth) routes failures; content_item_workspaces insert is admin/editor-gated.
• No barrel re-exports — direct file imports only.
• TanStack Query for any data fetching on the association surface.
• bun run test (not bun test); Python tests via python3 -m pytest scripts/tests/.
• Test philosophy — tests verify real behaviour (idempotency, association preservation across re-ingest, role gate), not implementation. Reference docs/reference/test-philosophy.md for behaviour-change Subtasks.
• UK English, DD/MM/YYYY dates, semantic tokens only for any UI (none in v1).

---

Open questions deferred to {69.3} TECH

Mechanism choices the PRODUCT deliberately leaves to TECH, bounded by the invariants above (especially BI-3 / BI-4):

• Manifest carrier (OQ-69-1 long-term). Whether the v1.1 ingest-side multi-workspace mapping rides an extended Path-B manifest (workspace_ids: [...] per prefix) or a separate canonical-association manifest. v1 does not build it, but TECH should pick the shape so the v1 junction contract (BI-4) is provably reusable.
• v1 operator surface exactness. Whether the v1 operator-side path uses app/api/items/[id]/workspaces as-is or needs a thin batch/idempotency hardening to satisfy BI-3. TECH confirms against the current route (route.ts:145,170).
• Re-ingest association preservation mechanism (BI-6). How association preservation is guaranteed across re-ingest given path-derived identity — TECH states whether this is automatic (associations key off the stable uuid5 identity) or needs an explicit guard.

---

Provenance pointers

• Predecessor: docs/specs/ID-69-client-corpus-onboarding/RESEARCH.md (S290, ratified).
• Ratified OQ answers OQ-69-1..5: Liam, S291.
• In-repo precedent: lib/intelligence/pipeline.ts:540-734 (storeAsContentItem / ensureWorkspaceLink).
• Gap evidence: scripts/cocoindex_pipeline/flow.py:1106-1561, workspace_resolver.py:144-186, adapters.py:74-77; content_item_workspaces schema (supabase/migrations/20260416102457_pre_squash_reconciliation.sql:3511).
• Manual association surface: app/api/items/[id]/workspaces/route.ts:145,170.
• Forward-compat / deferral homes: docs/themes/canonical-pipeline/reference/deferral-register.md (V11-2,3,4,7,9,10
  • C-1,2,4,5,6,7); decision-graph Q2.1 / CX.28 / CX.29.
• Intersections: ID-45/T7 (gated by ID-69), ID-66 (corpus volume / base path), ID-68 (IP separation — external-folder-canonical rationale).

Changelog

Date	Session	Note
31/05/2026	S291	{69.2} PRODUCT authored. Encoded ratified OQ-69-1..5: v1 = operator-side curation driving a driver-agnostic idempotent junction-write contract (BI-3/BI-4) reusable unchanged by the v1.1 ingest-side writer; file-source readiness; operator-side curation discipline; dedup-agnostic, no client UX, no upload-route coupling. 13 numbered Behavior invariants. Explicit v1 vs v1.1 boundary + source_documents/ontology out-of-scope boundary. States ID-69 gates ID-45/T7 (BI-13).