ID-110 — Re-home url_import onto reference_items: PRODUCT (behaviour spec)

ID-110 — Re-home url_import onto reference_items: PRODUCT

{110.2} PRODUCT artefact. Behaviour spec for re-pointing the manual single-URL ingest route (/api/ingest/url) so that pasting one URL lands one reference_items row + one source_documents provenance row per normalised URL (evidence), never a content_items row (knowledge). Written from the perspective of the three consumers of this surface: the admin/editor pasting a URL on the item page, the UI caller (components/create-content/url-ingest-form.tsx → IngestionSuccessCard) consuming the route’s JSON response, and the reference layer (downstream search/MCP) that reads the landed row.

• Task: ID-110 “Re-home manual-URL url_import to reference_items” (bl-298, from ID-107 {107.4}).
• Predecessor: {110.1} RESEARCH (ratified S348); this spec commits the decisions Liam ratified in the {110.1} journal — not the RESEARCH §7 OQ defaults, which are cosmetically still “pending” in RESEARCH.md.
• Binding ratification (from the {110.1} ledger journal, S348 — the load-bearing answers):
  1. Fresh ID-110 contract that EXTENDS ID-75 — no reopen of the closed ID-75 Task.
  2. Option C — synchronous app route + an owner-gated reference_ingest SECURITY DEFINER RPC that writes the sd+ri pair (resolving the pipeline-only-writer RLS posture without a new app-write policy on the table).
  3. Manual-URL paste = evidence → reference_items (not an adoption act; closes RESEARCH Option D).
  4. The ~55 legacy content_items.url_import rows stay read-only; they re-land at ID-45/T7. ID-110 changes only the forward write path.
  5. No workspace column on reference_items (honour ID-75 BI-7); originating user/workspace is provenance-only.
  6. (Only if Option B had been chosen — it was not) a dedicated ingest ledger. Moot under Option C.
• Authored: 13/06/2026, fresh Planner dispatch (Q-PLANNER-2: a DIFFERENT instance from the {110.1} RESEARCH author).
• Language: UK English; dates DD/MM/YYYY. No client-specific literals.

This spec defines behaviour + numbered invariants. The RPC field contract, migration plan, the body-producer decision (PullMD-in-request vs reuse extractFromUrl — flagged RESEARCH §9 / surfaced here as Open Question OQ-B), RLS grant mechanics, and test planning are {110.3} TECH concerns.

---

Problem

/api/ingest/url (POST, the manual single-URL “add content” option on the item page) TODAY inserts a content_items row stamped ingestion_source: 'url_import' (app/api/ingest/url/route.ts:191, .from('content_items').insert(...) at :207), then classifies it, summarises it, scores it, runs inferLayer, and returns layer/topic/guide suggestions to the caller (response shape route.ts:472-492). Under the ratified ID-75 O4/D4 reframe this is definitionally wrong: content_items is client-adopted knowledge; a raw external URL the user pasted is evidence, and “external gets one reference per URL” (ID-75 PRODUCT DP-2). The URL belongs on reference_items — the global, workspace-less reference layer — exactly as the RSS-feed route already lands there via cocoindex.

Code-intelligence orientation (cited verbatim, not paraphrased). gitnexus_query({query: "reference_items url_import ingestion reference ingest", repo: "knowledge-hub"}) returned no execution-flow processes (processes: [], process_symbols: []) — confirming there is no indexed execution flow wiring the manual-URL route to the reference layer today (the re-point does not exist yet). The standalone definitions it surfaced were the live URL write-path tests (scripts/tests/test_cocoindex_url_write_path.py:TestUrlLandingDeclaresEvidencePair :224-345; scripts/tests/test_cocoindex_flow_failure_mode.py:TestUrlPerItemFailureIsolation :1868-2166; scripts/tests/test_url_source.py:TestCrossWorkspaceCollapse), the UI caller (components/create-content/url-ingest-form.tsx:UrlIngestForm :67-394, handleSubmit :116-196), and the E2E spec (e2e/tests/content-ingestion-url.spec.ts).

gitnexus_context({name: "UrlIngestForm", file_path: "components/create-content/url-ingest-form.tsx"}) — incoming: NewItemTabs (app/item/new/new-item-tabs.tsx, 1 direct caller); outgoing: IngestionSuccessCard (components/create-content/ingestion-success-card.tsx), IngestionProgress, DedupWarning (components/shared/dedup-warning.tsx), ClaudePromptButton. So the re-point’s UI blast radius is exactly one form (UrlIngestForm) and its success surface (IngestionSuccessCard, props IngestionSuccessCardProps :34-49: itemId, title, contentType, domain?, subtopic?, warnings?, dedupMatches?, suggestedLayer?).

gitnexus_context({name: "POST", file_path: "app/api/ingest/url/route.ts"}) — resolves the route POST handler (:26); the body insert + response shape were read directly (:180-215, :472-492). Python is outside ast-dataflow’s TS corpus; the RI/SD field contract the RPC must mirror was read directly from scripts/cocoindex_pipeline/flow.py:_ingest_url_body :2709-2761 (the declare_row for both rows), and the ingestion_source route-scoping comment from scripts/cocoindex_pipeline/url_source.py:55-100 (INGESTION_SOURCE_RSS_FEED = "rss_feed" hardcode with the verbatim “url_import joins when a manual-URL route ships (post-v1)” comment, :57-60). Schema + RPC grant pattern read directly from supabase/migrations/20260606121451_id75_reference_items_layer.sql (:6-23 columns + CHECK + UNIQUE; :44 SELECT-only RLS) and supabase/migrations/20260606130224_id75_reference_search_rpcs.sql (:1-71 SET search_path = public, extensions; SECURITY DEFINER; per-function REVOKE-from-anon grant pattern).

The reframe in one line: the manual-URL route MUST write one reference_items row + one source_documents provenance row per normalised URL, via an owner-gated reference_ingest RPC, and MUST stop writing content_items.

Goals

• Re-point the forward manual-URL write path to the reference layer (evidence), faithful to the ratified ID-75 contract (BI-9/BI-11/BI-23, TECH §6.3).
• Preserve the synchronous accept UX: the user still gets an immediate response with the landed reference’s title/summary and a navigable link to it.
• Reconcile the RLS conflict the KH-idiomatic way (owner-gated RPC), with no new app write policy on reference_items and no change to the ID-75 pipeline-only-writer posture for the feed path.

Non-goals

• Retro-converting the ~55 legacy content_items.url_import rows. They stay read-only legacy lineage and re-land at ID-45/T7 (ID-75 BI-10). ID-110 touches only forward writes.
• Adding a workspace column / junction to reference_items (breaches ID-75 BI-7, RATIFIED-DO-NOT-BUILD).
• Reopening ID-75 or modifying its ratified spec/schema.
• Changing the RSS-feed route (url_source.py stays rss_feed; no async enumeration widening — that was the rejected Option B path).
• bl-17 (RSS routes auth-design review) — see Behaviour §6 disposition; out of scope.
• bl-119 (Phase-5 UI test coverage) — see Behaviour §6 disposition; the behaviour this spec mandates is testable, but authoring the UI test suite is bl-119’s own scope.
• bl-251 (feed_articles(passed=true) partial index) — see Behaviour §6 disposition; the feed-source enumeration is untouched by Option C, so this stays separate tech-debt.

Figma

Figma: none provided. This is a backend re-point whose only UI surface is the existing UrlIngestForm success/error states; no new visual design. The one user-visible reduction (dropping layer/topic/guide suggestions — §3) is a content removal from an existing card, not a new screen. Warm Meridian semantic-token discipline applies to any copy/affordance change (§3.4, §4.3).

---

Behavior

1. Forward write target — the core re-point

1.1. When an admin or editor submits a valid, SSRF-passing, non-duplicate URL to /api/ingest/url, the route MUST land exactly one reference_items row and exactly one source_documents row for that normalised URL, and MUST land zero content_items rows. Validation: after a successful ingest of URL u, reference_items has one row with source_url = normalise(u), source_documents has one row with source_url = normalise(u), and content_items has no row whose source_url = normalise(u) created by this request.

1.2. The landed reference_items row MUST carry ingestion_source = 'url_import' (the CHECK at reference_items_layer.sql:18-19 already admits this value — no migration is required to permit it). The RSS-feed route’s value ('rss_feed') is unchanged. Validation: the landed row’s ingestion_source column equals the string 'url_import'.

1.3. The reference row’s identity MUST be the ID-75 deterministic scheme: id = uuid5(KH_PIPELINE_DOC_NS, "ri:" + normalise(url)) and its provenance row’s id = uuid5(KH_PIPELINE_DOC_NS, "sd:" + normalise(url)), with reference_items.source_document_id pointing at that source_documents.id. A content_items/ci: seed MUST NEVER be minted from a manual URL. Validation: the landed ri.id/sd.id equal the uuid5 seeds for the normalised URL; the FK ri.source_document_id = sd.id holds; no ci:-seeded row exists for the URL.

1.4. The reference row MUST satisfy the full ID-75 field contract the async path produces (flow.py:_ingest_url_body :2732-2760): title (NOT NULL), body (NOT NULL, the canonical body of record), summary, source_url, published_at (nullable; original publication time when derivable, never ingest time), primary_domain, primary_subtopic, layer = 'research' (the v1 constant — validated by the validate_layer_key trigger), embedding, source_document_id, ingestion_source = 'url_import', op_id. workspace_ids/any workspace column MUST NOT be written (BI-7). Validation: the landed row has non-null title, body, layer = 'research'; the row’s column set matches the ID-75 contract; no workspace column is written.

1.5. The source_documents provenance row MUST carry storage_path = source_url = normalise(url), a URL-derived filename/mime_type/file_size, content_hash, op_id, and an extraction_method consistent with how the body was produced (mirroring flow.py:2714-2730). For a PDF URL the row reflects the Docling path; for an HTML URL the PullMD/extractor path. (Which body producer the synchronous route uses is OQ-B.) Validation: the source_documents row has storage_path = source_url = normalise(url) and a non-null extraction_method.

2. Idempotency, dedup, and the URL-already-exists contract

2.1. Re-submitting the same URL (after normalisation) MUST be idempotent at the reference layer: it MUST NOT create a second reference_items row, and MUST NOT error the caller with a hard failure. The UNIQUE(source_url) constraint (reference_items_source_url_key) plus the deterministic uuid5 PK make re-landing the same URL converge on the same row (update-in-place, ID-75 BI-2). Validation: ingesting URL u twice yields exactly one reference_items row for normalise(u); the second request returns a non-error response.

2.2. The route’s pre-write URL-already-exists check MUST be re-pointed from content_items.source_url (route.ts:74-90) to the reference layer (reference_items.source_url). When the URL already exists as a reference, the route returns the existing-reference signal the UI already understands (the url_already_exists branch — see §4.2), carrying the existing reference’s id + title. Validation: submitting a URL that already exists as a reference_items row returns url_already_exists: true with existing_item.id = the existing reference id (not a content_items id).

2.3. The route MUST preserve a dedup-by-similarity soft warning path equivalent to today’s (route.ts:153-172): when the incoming body is highly similar to existing reference material, the response MAY carry duplicate_matches and a dedup_status, but the reference still lands (soft block, not hard reject) — UNLESS §2.1 exact-URL idempotency already short-circuits. Validation: a near-duplicate (different URL, similar body) still lands a reference row and the response carries duplicate_matches/dedup_status when matches exist.

Open question (OQ-D): does reference-layer dedup-by-similarity carry the same product weight for evidence as it did for adopted knowledge, or is exact-URL idempotency (§2.1) sufficient and the similarity soft-warning dropped for references? RESEARCH O3 notes the feed path discards content_type; it does not specify a reference similarity-dedup UX. Recommended default: keep exact-URL idempotency (§2.1) as the hard contract; treat the similarity soft-warning as best-effort and resolve its retention in {110.3} TECH.

3. Synchronous response — what the caller sees back (the UX reduction)

3.1. The route MUST remain synchronous: a single POST returns the landed reference’s metadata in the response body (no async “processing, check back later” model). This is the ratified Option C contract — the synchronous accept UX is preserved. Validation: a successful POST returns HTTP 2xx with a JSON body containing the landed reference’s id and title in the same request (no polling required to obtain them).

3.2. The success response MUST carry, at minimum: the landed reference id, title, source_url, summary, and a dedup_status/warnings array equivalent to today’s. It SHOULD carry primary_domain/primary_subtopic (the classifier output the reference row already stores). Validation: the success-response JSON contains non-empty id, title, source_url; the id resolves to the landed reference_items row.

3.3. The response MUST drop the content_items-only affordances: suggested_layer / topic_suggestion / guide_section_suggestions MUST NOT be returned (references carry no per-row layer choice — layer is the constant 'research' — and no topic/guide-section inference). The content_type field, which references do not carry, MUST NOT be presented as a classification the user can act on. This is a deliberate, documented UX reduction. Validation: the success-response JSON contains no suggested_layer, topic_suggestion, or guide_section_suggestions keys.

3.4. The success surface (IngestionSuccessCard, today consuming IngestionSuccessCardProps: itemId, title, contentType, domain?, subtopic?, warnings?, dedupMatches?, suggestedLayer?) MUST be updated so it no longer renders the layer-suggestion control (suggestedLayer) or a content-type classification for a reference landing, and its “view item” navigation MUST resolve to the reference (its detail/search surface), not a content_items item page that no longer exists for this URL. Validation: after a URL ingest, the success card shows no layer-suggestion control, and its primary navigation resolves to the landed reference, not a /item/<content_items-id> route.

Open question (OQ-N): what is the canonical “view this reference” destination for a freshly-landed manual reference — a reference-detail view, the reference-search surface pre-filtered to it, or (interim) a copyable reference id with no dedicated page? ID-75 shipped reference_search / reference_get_verbatim RPCs but the RESEARCH does not cite a user-facing reference-detail page. Recommended default: if no reference-detail page exists yet, the success card surfaces the landed title/summary + a copyable id and omits a dead “view item” link rather than linking to a non-existent page; promote a proper destination as a follow-on if absent. Resolve in {110.3} TECH against the actual routes.

4. States, errors, and edge cases (preserved from the existing route)

4.1. Auth. The route MUST keep getAuthorisedClient(['admin','editor']) and route failures via authFailureResponse(auth) (route.ts:30-31). A viewer/reviewer or unauthenticated caller MUST be rejected before any write. The owner-gated reference_ingest RPC is the write seam; the route’s role gate is unchanged. Validation: a non-admin/editor caller receives the auth-failure response and no reference_items row is created.

4.2. URL already exists. When §2.2 matches, the route returns url_already_exists: true

• existing_item: { id, title }; the UI’s existing branch (handleSubmit url-ingest-form.tsx:152-157) resets to idle and surfaces the existing item without treating it as an error. Validation: the response carries url_already_exists: true and the UI shows the existing reference rather than an error state.

4.3. SSRF rejection. The route MUST keep validateUrl(url) (route.ts:68) and return a 400 with the validation error for a rejected URL — no fetch, no write. SSRF parity with the feed path (validate_url) is reused, not re-implemented. Defence-in-depth in the RPC/writer is a TECH concern. Validation: an SSRF-failing URL returns 400 and creates no reference_items/ source_documents row.

4.4. Empty/low-content body. The route MUST keep the quality gate: a body under the minimum threshold (today <100 chars ⇒ 422; <500 chars ⇒ soft warning, route.ts:98-110) returns the same 422/warning behaviour, because reference_items.body is NOT NULL and a near-empty body is not useful evidence. Validation: a URL yielding <100 chars of body returns 422 and lands no reference row.

4.5. Rate limit. The existing 10/min rate limit (route.ts:39) MUST be preserved. Validation: the 11th request inside a minute is rate-limited as today.

4.6. Partial-failure atomicity. A request MUST NOT leave a dangling source_documents row with no reference_items row, nor a reference_items row whose source_document_id FK is unsatisfiable (the FK is NOT NULL ... ON DELETE RESTRICT). Either both rows land or neither does (the owner-gated RPC is the natural transaction boundary). Validation: on an induced mid-write failure, neither a source_documents nor a reference_items row for the URL persists (no orphaned provenance row).

4.7. Caller error display. Network errors, 4xx, and 5xx MUST surface through the existing UrlIngestForm error states (handleSubmit :159-196): error copy via setErrorMessage, the step indicator transitions to error. WCAG 2.1 AA: error state MUST NOT rely on colour alone (existing pattern keeps an error message string + icon). Validation: a 5xx response drives the form to its error state with a visible error message (not colour-only).

5. RLS / writer posture (consumer-visible invariants)

5.1. The reference_items table MUST remain write-policy-free for app callers: ID-110 MUST NOT add an app-side INSERT/UPDATE RLS policy on reference_items. The synchronous app write path MUST go through the owner-gated reference_ingest SECURITY DEFINER RPC (the same posture as reference_search/reference_get_verbatim being the only authenticated reference surface). Validation: reference_items has no app-callable INSERT/UPDATE policy after ID-110; the only authenticated write surface is the reference_ingest RPC.

5.2. The reference_ingest RPC MUST follow the ID-75 grant pattern (reference_search_rpcs.sql:1-71): SET search_path = public, extensions, SECURITY DEFINER, per-function REVOKE EXECUTE FROM anon, GRANT EXECUTE TO authenticated/ service_role. The caller’s role gate (admin/editor) lives in the route (§4.1); the RPC is defence-in-depth, not the primary authorisation. Validation: anon cannot execute reference_ingest; authenticated can; the function is SECURITY DEFINER with a pinned search_path.

5.3. The feed path’s pipeline-only-writer posture (asyncpg owner connection, _ingest_url_body) MUST be unchanged. The new RPC is an additional gated write seam for the synchronous route, not a replacement for the async writer. Validation: url_source.py/flow.py feed write path is byte-unchanged by ID-110.

6. Backlog reconciliation (explicit dispositions)

6.1. bl-17 (RSS routes auth-design review) — OUT OF SCOPE. bl-17 concerns the RSS feed serving routes (app/api/feeds/[workspaceId]/rss/route.ts) being intentionally unauthenticated, and whether the blanket /api/** auth-exemption is sufficient. ID-110 touches the manual URL ingest route, which keeps its ['admin','editor'] gate (§4.1) and adds no new public route. No surface overlap with the feed-serving auth question. Disposition: note + leave to bl-17’s own spec phase.

6.2. bl-119 (Ingestion Phase-5 UI coverage — upload progress / dedup / URL-exists) — PARTIALLY FOLDED as behaviour, test-authoring stays bl-119. ID-110 changes the URL-exists behaviour (§2.2 now resolves against the reference layer) and the dedup-warning surface (§2.3, §4.2), so any bl-119 UI tests touching the URL path MUST assert the reference contract, not the old content_items one. The behaviour is fully specified here and testable; authoring the Phase-5 UI test suite remains bl-119’s scope and is not an ID-110 deliverable. {110.3} TECH MUST flag the bl-119 test surfaces that this re-point invalidates so they are re-pointed, not left asserting the dead contract. Disposition: fold the behavioural contract in (§2.2/§2.3/§4.2); leave suite authoring to bl-119; flag affected tests in TECH.

6.3. bl-251 (feed_articles(passed=true) partial index) — OUT OF SCOPE. bl-251 is a per-walk full-table-scan fix on the feed enumeration (url_source.py:93-96, SELECT ... WHERE passed = true). Option C does not widen feed enumeration or touch url_source.py (§5.3), so ID-110 neither helps nor worsens this scan and adds no feed_articles query. It remains independent tech-debt. Disposition: note + leave as separate tech-debt (relevant before ID-45/T7, not ID-110).

7. Dead-code cleanup (forward-write completeness)

7.1. Once the route stops calling inferLayer (references use the 'research' constant), the url_import special-case in lib/layer-inference.ts:197 and its inclusion in the ingestionSource union (:21) become dead for the live path. ID-110 SHOULD remove the url_import branch or document it as legacy-only. This is low-priority cleanup; the behavioural contract above does not depend on it. Validation: after ID-110 no live code path passes ingestionSource: 'url_import' to inferLayer.

7.2. The dead reference_items comment in lib/intelligence/pipeline.ts:438 (inside the retired ID-75 WP-E block) is not a writer and needs no behavioural change; noted so a future reader is not misled into thinking a TS reference writer already exists. Validation: grep -rln "reference_items" app/ lib/ --include=*.ts returns only the new reference_ingest call site (route/lib) plus that dead comment — no other TS writer.

---

Open questions (collected)

• OQ-B — body producer (→ TECH). Does the synchronous route call PullMD in-request (parity with the feed path’s body-of-record, but a network round-trip inside the ≤60s window) or reuse the route’s existing extractFromUrl Readability output (faster, but a potential body-of-record quality divergence from the feed path)? RESEARCH §9 flags this as an unresolved TECH decision. Recommended default: PullMD in-request for body-of-record parity, with Docling for PDF URLs, unless TECH measures the latency as unacceptable — then extractFromUrl with a documented parity caveat. Resolve in {110.3} TECH.
• OQ-D — reference similarity-dedup retention (→ TECH). See §2.3. Recommended default: exact-URL idempotency is the hard contract; similarity soft-warning is best-effort.
• OQ-N — “view this reference” destination (→ TECH). See §3.4. Recommended default: surface title/summary + copyable id; omit a dead link if no reference-detail page exists; promote a proper destination as follow-on if absent.

These are TECH-time resolutions; none blocks the behavioural contract (§1–§7), which is fully determined by the S348 ratification.

Changelog

Date	Session	Note
13/06/2026	S350	{110.2} PRODUCT authored (fresh Planner, Q-PLANNER-2). Grounded on the {110.1} ledger-journal S348 ratification (NOT the cosmetically-pending RESEARCH §7 OQs): fresh ID-110 extending ID-75, Option C (sync route + owner-gated reference_ingest SECURITY DEFINER RPC), manual-URL = evidence → reference_items, ~55 legacy rows read-only (re-land T7), no workspace column. 7 behaviour groups (§1 write target, §2 idempotency/dedup, §3 sync response + UX reduction, §4 states/errors, §5 RLS/writer posture, §6 backlog reconciliation, §7 dead-code). bl-17 out-of-scope, bl-119 behaviour-folded/suite-deferred, bl-251 out-of-scope. 3 TECH-time OQs (OQ-B body producer, OQ-D dedup retention, OQ-N view destination). Code-intel cited verbatim (gitnexus_query empty processes; UrlIngestForm/POST context; direct file:line reads of route, flow.py RI/SD contract, url_source.py, schema + RPC grant pattern).