ID-75 — PullMD remote URL/feed source: external reference layer + explicit adoption (O4)

ID-75 — PullMD remote URL/feed source: external reference layer + explicit adoption (O4)

{75.2} PRODUCT artefact. This is the product-behaviour phase of the spec-driven lifecycle: it states what the post-pivot external-content model must do, as numbered, testable Behavior invariants a Checker can verify against acceptance criteria. It does not decide implementation mechanism (that is {75.3} TECH) and does not decompose into Subtasks (that is {75.4} PLAN).

• Task: ID-75 “Wire PullMD into cocoindex correctly (remote URL/feed source)” (bl-217).
• Predecessor: {75.1} RESEARCH.md (04/06/2026) — Option-A cocoindex custom/remote source empirically VIABLE against cocoindex[postgres]==1.0.3 (RESEARCH §4); decided constraints 1–9 (RESEARCH §5) carried into this spec.
• Ratified frame (binding — do not re-litigate): O4 / D4 — Liam, S316 (05/06/2026), docs/research/s314-id75-reframe-options.md §7 (DP-1..DP-7). content_items = client-adopted knowledge. External URL/RSS material lands a reference/evidence layer and becomes canonical knowledge ONLY by explicit adoption/derivation with citation lineage.
• Working name: this spec uses reference_items for the new peer Layer-5 class throughout — RATIFIED as the canonical name (OQ-75-1, Liam, S316, 05/06/2026).
• Authored: S316 (05/06/2026), fresh Planner dispatch (Q-PLANNER-2).
• Status: DRAFT — Checker PASS_WITH_NOTES (note resolved); OQ-75-1..4 RATIFIED (Liam, S316): reference_items / conversion EXCLUDED v1 / human-or-AI-assisted+human- confirm / entities deferred. OQ-75-5 (CV plumbing) pending. Full ratification (incl. OQ-75-5) precedes {75.3} TECH.
• Language: UK English; dates DD/MM/YYYY.

---

Summary

External URL/RSS articles that pass the relevance gate land as evidence, not knowledge: one source_documents provenance row plus one global, workspace-less reference_items row per normalised URL — never a content_items row. External material becomes canonical knowledge only through an explicit, cited adoption/derivation act. ID-75 wires PullMD into the cocoindex pipeline as the single producer of that landing (Option A), retires the TypeScript legacy promotion path, and fixes the two-surface retrieval data contract (canonical knowledge vs reference/evidence) that ID-71 inherits.

---

Problem

Two defects converge on this Task:

1. The wiring defect. The cocoindex Stage-2 adapter hands PullMD — a remote-URL fetcher — a local container path (adapters.py:74-77), which can never resolve; PullMD’s contract is GET {PULLMD_SERVICE_URL}/api?url=<http(s) URL> (RESEARCH §1). The flow has exactly one source connector (localfs.walk_dir, flow.py:2307-2311) while URL/RSS feeds are inherently remote.
2. The definitional defect. The live promotion path predates the platform pivot: passed feed articles auto-promote into content_items (lib/intelligence/pipeline.ts:577-735, ingest_source: 'rss_feed' at :653), making external publications structural peers of the client’s own canonical knowledge. Liam’s reframe (docs/research/s314-id75-reframe-liam.md) holds that this is bid-era legacy, not normative: post-pivot, the platform’s centre is the client’s own canonical company knowledge (Wikipedia one-canonical-record practice), and the question is what separation is required between client-own content and externally-sourced content. The S316 O4 ratification answers it: external content is evidence that informs; canonical knowledge is what the client has authored or adopted.

Code-intelligence orientation (mandatory per .gitnexus/CLAUDE.md “Always Do”; cited verbatim, run 05/06/2026 against this HEAD):

• gitnexus_query({query: "feed article promotion content items pipeline", repo: "knowledge-hub"}) returned processes: [] (no indexed cross-module execution flow for the promotion path) with the relevant symbols as standalone definitions: Function:lib/intelligence/pipeline.ts:storeAsContentItem (:576-734), Function:lib/intelligence/pipeline.ts:processFeedSource (:276-536), Function:lib/intelligence/pipeline.ts:isDuplicate (:216-254), plus the consumer surfaces app/api/intelligence/workspaces/[id]/articles/route.ts:GET (:10-94) and app/api/feeds/[workspaceId]/rss/route.ts:GET (:23-106).
• gitnexus_context({name: "storeAsContentItem", repo: "knowledge-hub"}) — the legacy promotion symbol this spec retires (BI-11): incoming calls: exactly 1 (Function:lib/intelligence/pipeline.ts:processFeedSource); outgoing calls: classifyContent (lib/ai/classify.ts), normaliseUrl (lib/intelligence/content-extractor.ts), inferContentType, ensureWorkspaceLink (both lib/intelligence/pipeline.ts), sb (lib/supabase/safe.ts); accesses isDuplicate; processes: []. Verdict: LOW structural blast radius (1 direct caller, zero indexed execution-flow participation — gitnexus indexes no processes for this symbol; the runtime chain is the three poll triggers converging on runPipeline → processFeedSource → storeAsContentItem, grounded at lib/intelligence/pipeline.ts:771-905 per docs/research/s314-id75-feed-current-state.md §2.1-2.4), but HIGH product significance: it is the sole live writer of external content into content_items, i.e. the exact behaviour O4 retires.
• The Python pipeline (scripts/cocoindex_pipeline/flow.py) is outside ast-dataflow’s TypeScript corpus; its pins are cited from the S314-re-verified options paper (s314-id75-reframe-options.md §2 common substrate) rather than re-derived.

---

Definitional frame (ratified, binding)

From docs/research/s314-id75-reframe-options.md §7 — restated here because every invariant below is downstream of it:

DP	Decision binding on this spec
DP-1	O4 / D4 — content_items = client-adopted knowledge. External material lands a reference layer; O1/O2/O5 are dead; O3 survives as O4’s landing half.
DP-2	One-canonical-record practice applies to the client’s knowledge subjects only; “external gets one reference per URL”.
DP-3	Two separate retrieval surfaces — canonical knowledge (content_items + q_a_pairs) vs reference/evidence — with unambiguous citation semantics.
DP-4	External content is global and workspace-less (q_a_pairs precedent: no workspace FK; retrieval-time scoping). The platform must be extensible beyond today’s URL/RSS feeds. Semantics (Liam, verbatim intent): external content “informs”, provides “insight”, adds context, and may be subjective; client-specific data is normally factual/canonical and is “supported” by external data.
DP-5	External material becomes canonical knowledge only by explicit adoption/derivation with citation lineage — never automatically. The relevance gate admits evidence, not knowledge.
DP-6	Decide now — the knowledge-layer landing is decided; this PRODUCT specs it.
DP-7	Peer-class name, Layer-5 register entry, and the Q1a/Q1b storage-rule amendment are proposed here as OQs (OQ-75-1, OQ-75-5; BI-22).

Note on superseded conclusions: s314-id75-usecase-affordances.md §5 and s314-ontology-boundary.md §4 concluded (pre-reframe) that gate-passed articles should be content_items. Per the options paper’s method note, their facts (affordances A1–A15, use-cases U1–U17, boundary axes) are reused below; their conclusions are superseded by the O4 ratification.

---

Goals / Non-goals

Goals

• Make the ratified O4 landing model real end-to-end: gate-passed external articles land source_documents (provenance-hardened) + reference_items (global, embedded, classified, citable) via the cocoindex Option-A remote source — a single producer, with PullMD receiving a real http(s) URL.
• Retire the bid-era TS promotion path so no new external content lands content_items directly.
• Fix the two-surface retrieval data contract (DP-3) so ID-71’s tooling review designs against a decided substrate.
• Spec the adoption/derivation path as the destination (DP-5), sliced so the landing half ships first (S316 sequencing note).
• Amend the ontology’s storage-decision rule (Q1a/Q1b) and propose the Layer-5 register entry so future activities route external material correctly.

Non-goals (explicit)

• Rebuilding or moving the poller (s314-id75-feed-current-state.md §7 variant (b)) — discovery, scoring, triage and RSS-out stay exactly as they are (BI-12).
• Re-ingesting existing prod feed/URL rows — data movement is ID-45/T7’s (RESEARCH §7); legacy promoted content_items rows (url_import 55 + rss_feed 40) are not retro-converted by ID-75 (BI-10).
• Building or consolidating MCP tools / UI surfaces over the reference layer — ID-71 owns the tool surface; this spec fixes the data contract only (BI-16).
• Building the citations substrate — ID-58 owns it; BI-13/BI-15 spec the behaviour it must support.
• reference_chunks sibling table and entity extraction over references — upgrade paths named, not built (BI-17, OQ-75-4).
• Q&A extraction from references — no use-case requires it (s314-id75-usecase-affordances.md A10).
• Sentiment signals (U16) — no data home; backlog, not ID-75.
• Reference-row version history — out of scope v1 (BI-2 notes the ID-58 design input).
• The PullMD HTTP contract itself — already correct (adapters.py:144-218).

---

Scope boundary — the two slices (S316 sequencing note)

O4 is reachable FROM O3: this spec defines both halves but slices them so the landing half can ship ahead of the adoption half, with adoption as the specced destination.

Slice	Invariants	Ships
Slice 1 — landing (the O3 half)	BI-1..BI-12, BI-16..BI-21 (data contract + mechanics)	ID-75’s implementation scope: source_documents hardening, reference_items class + retrieval substrate, feed re-point, legacy-path retirement, Option-A wiring.
Slice 2 — adoption (the O4 half)	BI-13..BI-15	Specced here as the destination. Citation substrate = ID-58 (CV 24 citation_kind, planned citations table); adoption affordance + citation-resolution tooling = ID-71 design inputs. Nothing in slice 1 may be built in a way slice 2 would have to unwind.

Ontology and sequencing invariants (BI-22, BI-23) bind both slices.

---

Behavior

Each invariant is numbered and testable. The acceptance criterion states how a Checker verifies it. “Normalised URL” means the canonical URL-normalisation rule whose definition is {75.3} TECH’s (RESEARCH constraint 3); the invariants here state only its observable consequences. The TS normaliseUrl rule (lib/intelligence/content-extractor.ts:96-121) and the Python rule must agree, or the same article minted under two identities breaks BI-2/BI-8 (s314-id75-feed-current-state.md §7 seam 2).

A — Landing model (the O3 half)

BI-1 — A gate-passed external article lands the evidence pair, never content_items

When a feed article passes the relevance gate, the canonical-side landing is exactly two rows: one source_documents row (the fetched artefact + provenance, BI-4) and one reference_items row (BI-3). No content_items row is created for it — not by the pipeline, not by any fallback. The relevance gate admits evidence, not knowledge (DP-5).

• Acceptance: After a passed article lands, the source_documents and reference_items rows exist and are linked (reference_items.source_document_id FK); no content_items row exists whose identity derives from that URL. A test asserts zero content_items writes on the URL landing path.

BI-2 — Reference identity is deterministic, URL-keyed, and update-in-place

A reference row’s identity is uuid5(_KH_PIPELINE_DOC_NS, "ri:" + normalise(url)); the provenance row’s is uuid5(..., "sd:" + normalise(url)). The "ci:" prefix is never minted for URLs. The same URL — re-enumerated, re-fetched, or arriving via a second acquisition route — collapses to the same single row pair. A re-fetch whose content changed updates the existing reference row in place (one reference per URL = the latest fetched representation, with provenance refreshed); it never inserts a sibling. Reference-row version history is out of scope for v1; the citation-stability consequence for adopted records (a derived record cites a reference whose body later changes) is named as an ID-58 design input, not solved here.

• Acceptance: Landing the same URL twice yields row counts of exactly 1 in each table with unchanged PKs; a changed-content re-fetch updates the body under the same PK. A grep/test confirms no code path seeds "ci:" from a URL.

BI-3 — Reference-row content contract

Each reference_items row carries, at minimum: id (per BI-2), title, body (the PullMD-extracted markdown — the canonical body of record), summary, source_url, published_at, classification (primary_domain / primary_subtopic / layer per CVs 01/02/03), embedding vector(1024), source_document_id FK, op_id, and acquisition provenance (BI-9). published_at is the original publication timestamp where the source provides one (nullable when unavailable) and survives landing — it is never flattened into ingest time (affordance A5: recency drives the sales/marketing use-cases U5/U8/U13). feed_articles.raw_content demotes to a relevance-scoring scratchpad; where the two bodies differ, the PullMD body on the reference row is canonical (resolves tension T3, s314-id75-usecase-affordances.md §7).

• Acceptance: A landed reference row exposes every listed field; published_at round-trips the feed item’s original publication date; the body equals the PullMD extraction, not feed_articles.raw_content.

BI-4 — source_documents provenance hardening with two-way URL recoverability

URL-sourced provenance rows carry (RESEARCH constraints 1–4 — ratified, save that constraint 3’s idempotency seeds are superseded by BI-2: "ri:" replaces "ci:" for URL-sourced knowledge rows; the "sd:" half stands): a net-new source_documents.source_url TEXT column (not JSONB; nullable — file rows have no source URL); storage_path = the canonical normalised URL; deterministic URL-derived filename / mime_type / file_size (the columns are NOT NULL and no File exists); extraction_method (pullmd_<X-Source> or docling) and pullmd_share_id (the durable re-read handle). A downstream consumer can recover both directions: “this row came from URL X” (query source_url / storage_path) and “which rows hold URL X” (the deterministic PK and the join contract reference_items.source_url = source_documents.source_url = normalised feed_articles.external_url). This resolves RESEARCH OQ-3.

• Acceptance: For a landed URL: source_url and storage_path carry the normalised URL; filename/mime_type/file_size are populated deterministically; pullmd_share_id round-trips via GET <PULLMD>/s/<id>; both recovery directions are demonstrated by query.

BI-5 — The relevance gate is the evidence line, and only that

The gate (feed_articles.passed, or explicit human selection at the triage UI) is the boundary between the ingestion ledger and the reference layer. It is never a boundary into content_items — passing the gate confers “worth keeping as evidence”, not “part of our knowledge” (DP-5: “adoption is the line; gate = worth keeping as evidence”).

• Acceptance: The only automated consequence of passed = true on the canonical side is the BI-1 evidence pair. No code path conditions a content_items write on passed.

BI-6 — Filtered articles land nothing canonical-side

Articles that fail the gate (passed = false) remain feed_articles rows only — no source_documents row, no reference_items row. The 90-day filtered-row cleanup (cleanup_filtered_articles()) is unchanged and interacts safely with enumeration: it deletes only passed = false rows, which the BI-18 enumeration never selects, so cleanup never deletes an enumerated item. Rejected articles never pollute search, coverage, or any retrieval surface.

• Acceptance: A filtered article produces zero canonical-side rows; the cleanup cron’s delete predicate and the enumeration filter are disjoint by construction (reviewable).

B — The global reference layer

BI-7 — References are global and workspace-less; outside the ID-69 junction contract

reference_items carries no workspace FK and participates in no junction rows — ever. There is no reference_item_workspaces table (mirror of the q_a_pairs RATIFIED-DO-NOT-BUILD anti-pattern, docs/ontology/32-q-a-pair.md §6). A reference row with zero workspace attachment is not a degenerate state; it is the only state. Workspace/activity relevance is computed at retrieval time — in v1 via classification + recency + acquisition origin; the planned scope_tag[]/anti_scope_tag[] columns (CV 21, status planned) are the named extension when they land, mirroring Q-OQR1-06. The ID-69 BI-1..BI-13 junction contract governs content_items only: references sit outside it; adopted/derived records sit inside it (BI-14).

• Acceptance: Schema check — no workspace column, no junction table. No code path writes a workspace association for a reference row. Retrieval filters operate on row attributes, not attachment.

BI-8 — One reference per URL across workspaces

feed_articles deduplicates per workspace (UNIQUE (workspace_id, external_url)), so the same URL discovered in N workspaces yields N ledger rows — but exactly one reference_items row and one source_documents row, with all N ledger rows’ backlinks (BI-10) resolving to the same reference. This resolves tension T5 (s314-id75-usecase-affordances.md §7) without junction rows: cross-activity availability is intrinsic because the reference is global.

• Acceptance: Seed the same URL into two workspaces’ feeds; after landing, reference and provenance row counts are 1 each, and both feed_articles rows backlink the same reference_item_id.

BI-9 — Acquisition-route extensibility (DP-4)

The reference layer’s contract is acquisition-route-agnostic. v1’s route is the relevance-gated feed ledger (BI-18); the manual URL-ingest surface becomes a second route when {42.12} re-points it (BI-23); future connectors (e.g. research-workspace sources) are further routes. Every reference row records its acquisition route as queryable provenance (vocabulary per OQ-75-5). No route may bypass the reference layer and land external material directly in content_items — route extensibility never re-opens the DP-5 boundary.

• Acceptance: The reference-row shape carries an acquisition-provenance field populated by the v1 route; the contract documentation (register entry, BI-22) states route-agnosticism and the bypass prohibition; adding a hypothetical second route requires no schema change to the reference class.

C — Ingestion-boundary re-point + legacy retirement

BI-10 — feed_articles promotion FK re-points to the reference layer

feed_articles gains a new nullable reference_item_id FK. From cutover, promotions write reference_item_id and never content_item_id — the content_item_id backlink is retired for new rows. Transition behaviour:

• Existing content_item_id backlinks and their promoted content_items rows are retained read-only as bid-era legacy lineage — ID-75 performs no retro-conversion. The designated moment legacy external rows are re-landed per O4 is the ID-45/T7 fresh-corpus re-ingest into a fresh target (near-zero migration cost, options paper §2 common substrate).

• Lifecycle independence: a reference row’s existence does not depend on its feed_articles row(s), nor vice versa — deleting either side severs the backlink (the nullable-FK idiom mirrors the existing content_item_id FK ON DELETE SET NULL) without cascading into the other.

• Consumer surfaces keyed on content_item_id (item-detail FeedArticleFields join, Browse “SI” badge on metadata.source — s314-id75-feed-current-state.md §4) continue to work for legacy rows; for new rows the equivalent affordances ride the reference surface (BI-16). Re-pointing those UI/MCP consumers is named, ID-71-inherited follow-on work — flagged here so nothing goes dark silently.

• Acceptance: Post-cutover, new passed articles have reference_item_id set and content_item_id NULL; pre-cutover rows are untouched; deleting a feed row leaves the reference row intact (and vice versa nulls the backlink).

BI-11 — The TS legacy promotion path is retired/gated

storeAsContentItem (lib/intelligence/pipeline.ts:577-735) — including its classifyContent call and ensureWorkspaceLink junction write — must not produce new content_items or content_item_workspaces rows after cutover (options paper §4 constant 8: new bid-era promotions would contradict the adopted model). The poller’s write scope ends at feed_articles (+ its scoring/summary fields). Whether retirement is code removal or an explicit gate is TECH’s choice; the behaviour is absolute. The sibling manual-URL path (app/api/ingest/url/route.ts, ingest_source: 'url_import') carries the same obligation at its own re-point moment ({42.12}, BI-23).

• Acceptance: Post-cutover, zero new content_items rows with ingest_source = 'rss_feed' appear; a test (or DB assertion across a poll run) verifies the poll path writes no content_items / content_item_workspaces rows.

BI-12 — Poller scope untouched; feed_articles remains the workspace-scoped ledger

Settled constants 1–2 (options paper §4), restated as an invariant: discovery, polling cadence (15-minute cron), extraction-for-scoring, the relevance gate, flags/refinement loop, triage UI, metrics, and public RSS out are unchanged — live polling stays exactly as-is. feed_articles remains the workspace-scoped ingestion-boundary ledger: dedup UNIQUE (workspace_id, external_url), audit of passed AND filtered rows, 90-day filtered cleanup. ID-75 swaps the promotion step only.

• Acceptance: A reviewer can diff the poll path (runPipeline → processFeedSource) and confirm the only behavioural change is the promotion-step swap (BI-10/BI-11); all feed_*-table surfaces (triage tabs, flags, RSS out, metrics) behave as before.

D — Adoption / derivation: the only path to canonical knowledge (the O4 half)

BI-13 — Adoption is explicit, cited, and never automatic

External material enters content_items only through a deliberate adoption act at human or AI-assisted instigation (actor per OQ-75-3) — never through the relevance gate, a batch job, or any automatic rule. The primary flavour is derivation: a person/agent AUTHORS a new canonical record (their analysis, position, summary, or guide section) citing one or more reference rows; the reference never converts; lineage rides the planned citations substrate (ID-58; CV 24 citation_kind, docs/ontology/24-citation-kind.md) — structurally the established q_a_extractions → q_a_pairs promote-with-lineage pattern (docs/ontology/32-q-a-pair.md §1). Derived records mint fresh identities (they are new authored works, not URL-keyed; "ci:" + normalise(url) is never used — BI-2). The variant conversion flavour (promote the reference row itself into a content_items row) is per OQ-75-2 (default-lean: excluded from v1).

• Acceptance: No code path creates a content_items row from a reference row without an explicit, attributable adoption action. A derived record’s identity is not derivable from its cited URL(s).

BI-14 — Adopted/derived records are ordinary canonical records

A record created by adoption/derivation is a first-class content_items row: it satisfies the full canonical consumer contract (ID-56 C-1..C-30 — populated, source-attributed, embedded, classified, chunked) and sits inside the ID-69 junction contract (BI-1..BI-5: no intrinsic workspace; association via content_item_workspaces only; many-workspace capable). Semantics per DP-4: the derived record is the client’s authored, normally factual/canonical position, supported by cited external evidence which may be subjective — the citation lineage is what makes that support inspectable rather than implicit.

• Acceptance: An adopted/derived record passes the same contract checks as any client-corpus record (embedding present, chunks present, junction-associable, zero-junction valid); nothing distinguishes it structurally except its citation lineage.

BI-15 — Citation lineage resolves two-hop; the adoption act is role-gated

From any adopted/derived canonical record, a consumer can resolve the full provenance chain: canonical record → citation(s) → reference_items row → source_documents row → original URL. Every adopted/derived record carries at least one resolvable citation to its source reference(s) — no orphan adoption (a record claiming external derivation without resolvable lineage is invalid). The adoption act itself is admin/editor-gated (mirroring ID-69 BI-12); reviewer/viewer roles cannot adopt.

• Acceptance: Given a derived record’s id, the two-hop chain resolves to a concrete source_url by queries alone; an adoption attempt by a reviewer/viewer is rejected at the auth/RLS boundary (via getAuthorisedClient() { success } + authFailureResponse(auth) on any TS surface).

E — Two retrieval surfaces + the data contract ID-71 inherits

BI-16 — Two retrieval surfaces with unambiguous citation semantics (DP-3)

The platform exposes two structurally separate retrieval surfaces: the canonical surface (content_items + q_a_pairs — “what we know / our positions”) and the reference surface (reference_items — “what the outside world published”). A consumer (human, app, or MCP-driven AI) can always tell which surface a result came from; the two are never silently blended into one ranked pool by default. Citation semantics are unambiguous: citing a reference is citing external evidence (possibly subjective, per DP-4); citing a canonical record is citing company knowledge. The reference surface’s data contract — fixed by this spec; tool design and consolidation owned by ID-71 (docs/reference/tasks/ID-71.md remits (c)/(d)) — is: each result exposes title, summary, source_url, published_at, classification, acquisition origin (BI-9), and the provenance chain (reference → source_documents → URL). Retrieval follows the q_a_search two-step precedent (32-q-a-pair.md §5): a ranked preview step plus a verbatim fetch step, with the embedding excluded from the verbatim payload (AI-consumer-first payload discipline). Reference reads are available to all authenticated roles (corpus-level, as q_a_pairs); writes are pipeline-only.

• Acceptance: The reference search RPC + data shape exist and return every contract field; no default retrieval path interleaves reference rows into canonical-surface results; a reviewer confirms the contract documentation names ID-71 as the tool owner.

BI-17 — v1 retrieval substrate: whole-record embedding; upgrade paths named, not built

v1-minimal per the q_a_pairs precedent: the reference row carries its own embedding vector(1024) column searched by its own RPC — no chunk table for references in v1. The upgrade path is a sibling reference_chunks table (the CONTENT_CHUNKS_SCHEMA shape keyed on reference_item_id) — stated as the destination, not specced here. content_chunks is never widened with a polymorphic nullable parent (it would break the NOT-NULL writer contract, flow.py:1234, and muddy the canonical table). Entity mentions over references: per OQ-75-4 (default-lean: deferred — entity_mentions.content_item_id NOT NULL stands unmodified in v1; the U17 longitudinal competitor/entity record is the named driver for revisiting).

• Acceptance: Schema check — embedding column on the reference row, no reference chunk table, content_chunks/entity_mentions parent contracts unchanged. The upgrade path is documented in the register entry (BI-22).

F — Acquisition mechanics (Option A, settled constants)

BI-18 — The cocoindex source enumerates gate-passed feed_articles rows

Per Option A (empirically viable, RESEARCH §4 — settled constant 3): a cocoindex custom/remote source yields URL items enumerated from feed_articles rows with passed = true (plus the BI-9 acquisition routes as they land) — not from feed_sources roots; poll-and-discover and relevance scoring stay the poller’s job (resolves RESEARCH OQ-2; tension T2). Workspace context comes from the enumerated item (RESEARCH constraint 7) and is used for provenance only — it never produces a workspace attachment on the reference row (BI-7). The source/connector machinery is data-model-agnostic; whether the built-in PgTableSource is reused or the source is hand-rolled is TECH’s choice, as is the named-closure/_LoopRunner discipline (RESEARCH §6 risk note).

• Acceptance: The source’s enumeration predicate selects only gate-passed ledger rows; no poll/scoring logic exists in the cocoindex flow; landing a passed row requires no feed_sources read.

BI-19 — Continuous, unattended landing; per-item failure is recorded, isolated, retried

Landing is a standing, unattended behaviour: an article that passes the gate appears on the reference surface without operator action, on a bounded cadence (the feed-to-reference latency is two-stage: poller discovery every 15 minutes, then the enumeration cadence). Whether the source is a snapshot re-walk or a live watch() is TECH’s choice (RESEARCH OQ-1), bounded by this invariant — a manually-triggered one-shot batch does not satisfy it. Per-item failure semantics: a failed fetch/extraction (PullMD non-2xx, timeout) lands no partial rows, is structured-logged (the existing _pullmd_to_markdown log-then-raise pattern — no silent failures), is retried on a subsequent enumeration, and never halts the remainder of the enumeration.

• Acceptance: With the pipeline running and no operator action, a newly-passed article lands within the documented cadence. A simulated PullMD failure produces a structured log entry, zero canonical-side rows for that item, continued processing of sibling items, and a successful landing on retry.

BI-20 — PDF-via-URL routes to Docling, and the route is recorded

When an enumerated URL resolves to a PDF (HEAD/content-type sniff, or .pdf suffix), the content routes to Docling over fetched bytes — never PullMD (which returns binary garbage at X-Quality≈0.5 for PDFs; RESEARCH constraint 5). The route is recorded, not silent (Liam’s stated preference, options paper §4 constant 4): the provenance row carries extraction_method = 'docling' with source_url and the URL-keyed identity (BI-2/BI-4) intact, so a consumer can observe that a URL-acquired item took the PDF route. This resolves RESEARCH OQ-4.

• Acceptance: Landing a PDF URL yields rows whose extraction_method is docling, whose source_url/identity are URL-derived, and whose body is the Docling extraction; no PullMD call is made for it.

BI-21 — SSRF-rejected URLs are skipped, logged, and surfaced — never silent

SSRF validation lives in the URL-extraction branch (the one place doing the network call; RESEARCH constraint 6). A URL that fails validation is skipped (no fetch attempt, no source_documents row, no reference_items row), logged with a structured reason, and surfaced to an operator-visible surface (Liam’s stated preference: skip-and-log + surface to UI; the concrete surface is TECH’s choice — the invariant is that an operator can discover rejections without reading pipeline logs). This resolves RESEARCH OQ-5.

• Acceptance: A seeded private-range/invalid URL produces zero canonical-side rows, a structured log entry with the rejection reason, and an operator-visible record of the rejection; sibling items are unaffected.

G — Ontology rule amendment + sequencing

BI-22 — The 4-question storage rule amends: Q1 splits Q1a/Q1b; adoption is the promotion edge

The generalisable storage-decision rule (s314-ontology-boundary.md §6) amends as follows:

• Q1a — client-canonical / client-adopted knowledge → content_items (full canonical contract: ID-69 BI-1..BI-13, ID-56 C-1..C-30).
• Q1b — external evidence admitted by a curation gate → the reference layer (global, workspace-less, embedded, cited as external evidence).
• Q4 sidecars promote into Q1b, not Q1a: feed_articles.reference_item_id is the promotion FK (BI-10).
• The Q1b→Q1a edge is the adoption act (BI-13) — explicit, cited, never automatic; mirroring the extraction→pair promotion edge the rule already acknowledges.
• Corollary 1 holds: placement is never solved with a content_type value — no news/feed/intelligence content type; the enum stays closed (docs/ontology/04-content-type.md:62; options paper §4 constant 7).

A new Layer-5 register entry is proposed: docs/ontology/33-reference-item.md in the 32-q-a-pair.md shape (frontmatter, table shape, lifecycle, retrieval, anti-patterns, relationships), with anti-pattern rows including: no workspace FK / no junction table (RATIFIED-DO-NOT-BUILD), never auto-promote into content_items, "ci:" never minted for URLs, and “do not model a feed item as a file on disk” (S300 §2.2). The entry’s name follows OQ-75-1; the CV value plumbing (CV 13 / CV 29 / acquisition-route vocabulary) follows OQ-75-5; the docs/ontology/README.md “Where does new data live?” rule gains the Q1a/Q1b split.

• Acceptance: The register entry and README amendment exist with the stated content once ratified; the Checker verifies this PRODUCT states the amendment and the entry proposal (authoring the entry itself may land with TECH/implementation, but the rule amendment is ratified with this spec).

BI-23 — Sequencing: slice boundary, proof re-shape, and adjacent-task reconciliation

• Slice boundary: the landing half (slices per Scope boundary) ships ahead of the adoption half; nothing in slice 1 may preclude slice 2 (e.g. the citation substrate’s ability to address reference rows). The adoption half’s implementation seats are ID-58 (citations) and ID-71 (affordance + tooling); this spec is their behavioural contract.

• Proof re-shape ({42.10}/{62.10}): the end-to-end proof — owned by ID-75, coordinated with ID-42 close-out and ID-62 fixture-staging — exercises a real http(s) URL → PullMD → source_documents + reference_items (the sd: + ri: seeds), NOT a content_items landing. The /stage driver stages a URL item (e.g. seeds a gate-passed ledger row the source enumerates), not a local HTML fixture; the Inv-9 GET <PULLMD>/s/<share_id> round-trip references a genuine remote fetch.

• {42.11}/{42.12} reconcile-at-spec-time flag (S316 sequencing note): ID-42.11 re-points the TS extraction tiers at PullMD for scoring input only (the poller still needs text to score relevance before the gate); ID-42.12 re-points the manual URL-ingest surface, at which moment that surface inherits BI-9/BI-11 (lands a reference row, never a content_items row). Both are compatible with O4 but MUST be verified against this spec when they are specced.

• ID-45/T7: the fresh-corpus re-ingest consumes this landing model; legacy external content_items rows are re-landed per O4 there, not migrated here (BI-10).

• Acceptance: The re-shaped proof asserts the sd:/ri: pair and the absence of a content_items row; the reconcile flags are recorded on the respective Tasks by the Orchestrator (this spec states them; the ledger edit is the Orchestrator’s action).

---

KH quality bars (apply to any downstream implementation)

Surfaced so Executors inherit them via TECH/PLAN:

• No silent failures. Python pipeline: structured-log-then-raise per the existing _pullmd_to_markdown pattern. Any TS-side surface: sb() / tryQuery() from @/lib/supabase/safe; composite responses via warningsEnvelope().
• Auth (BI-15): getAuthorisedClient() returns { success } (not { authorised }); authFailureResponse(auth) routes failures; adoption is admin/editor-gated.
• DDL via CLI only (supabase migration new + db push), staging-first (turayklvaunphgbgscat); verify supabase/.temp/project-ref before any push. New PL/pgSQL functions (the reference search RPC) include SET search_path = public, extensions.
• No barrel re-exports — direct file imports only. TanStack Query for any app-side data fetching over the reference surface.
• bun run test (not bun test); Python tests via python3 -m pytest scripts/tests/.
• Test philosophy — tests verify real behaviour (idempotent landing, update-in-place, gate/adoption boundaries, failure isolation), not implementation (docs/reference/test-philosophy.md).
• UK English, DD/MM/YYYY dates; semantic tokens only for any UI (none in slice 1).
• Tool discipline (Inv 2/3): {75.4} PLAN briefs for code-touching Subtasks MUST carry gitnexus_impact-before-edit and gitnexus_detect_changes-before-commit, plus a grep sweep for Python (RESEARCH §8).

---

Open questions (OQ-75-N — for Liam ratification with this spec)

OQ-75-1 — Peer-class name: reference_items vs evidence_items

RATIFIED (Liam, S316, 05/06/2026): reference_items — the default-lean stands; the spec body’s usage is now canonical. Register entry: 33-reference-item.md.

Default-lean: reference_items (register entry 33-reference-item.md). Grounds: Liam’s own phrasing — “external gets one reference per URL” (DP-2); “reference” names the artefact’s citation role on both surfaces, whereas “evidence” names only its argumentative use and reads oddly for background/contextual material (e.g. a sector guide a client keeps for orientation). The spec body uses reference_items throughout; a different ratified name changes labels only.

OQ-75-2 — Conversion flavour of adoption: in scope, deferred, or excluded?

RATIFIED (Liam, S316, 05/06/2026): EXCLUDED from v1 — derivation-only Q1b→Q1a edge; the regulator-guidance counter-case revisits at ID-58 spec time.

Default-lean: EXCLUDED from v1 (derivation-only Q1b→Q1a edge). Grounds: conversion blurs D4’s stewardship logic — a snapshot of someone else’s text becomes “our record” without an authored act (options paper §2 O4 variant flavour); no identified use-case requires it now. The known counter-case (a regulator’s guidance the client operationalises verbatim) can be reconsidered as a deliberate decision at ID-58 spec time, where the promoted_to_content_item_id lineage shape would be designed if admitted.

OQ-75-3 — Who performs adoption in v1: human-only, or AI-assisted?

RATIFIED (Liam, S316, 05/06/2026): human, or AI-assisted with explicit human confirmation — never autonomous. Admin/editor-gated per BI-15.

Default-lean: human, or AI-assisted with explicit human confirmation — never autonomous AI adoption. The adoption act is admin/editor-gated either way (BI-15). AI may draft the derived record and propose citations; a human ratifies the act that makes it canonical knowledge. (ID-71 designs the affordance; this OQ fixes the trust posture it must implement.)

OQ-75-4 — Entity-mentions home for references

RATIFIED (Liam, S316, 05/06/2026): option (a) — defer entirely in v1. The entity_mentions.content_item_id NOT-NULL writer contract stays unmodified; U17 is the named revisit driver.

Options: (a) defer entity extraction for references entirely (default-lean), or (b) add a nullable reference_item_id to entity_mentions. Grounds for (a): matches the v1-minimal whole-record-embedding posture (BI-17); leaves the entity_mentions.content_item_id NOT-NULL writer contract (flow.py:1198) unmodified. The named driver for revisiting is U17 — the longitudinal competitor/entity record over external news — which becomes pressing when the competitor-research workspace is specced.

OQ-75-5 — CV plumbing for the reference layer

PART-RATIFIED (Liam, S316, 05/06/2026): items (1) and (3) ACCEPTED — CV 29 gains remote_url; the reference-row acquisition vocabulary reuses CV 13 semantics (no fifth provenance vocabulary). Item (2) HELD for {75.3} TECH with this steer: “as the platform isn’t live yet, migration items should be aligned to the new structure” — i.e. prefer clean alignment over preserving legacy-only values; the rss_feed/url_import disposition and the adopted_from_reference value land with TECH’s migration detail.

Three linked vocabulary decisions, proposed for ratification with the register entry:

1. CV 29 cocoindex_source_kind gains a remote value (proposal: remote_url) — all 7 current baseline values are file-surface adapters.
2. CV 13 ingest_source: rss_feed/url_import become legacy-only values for content_items (new external content never lands content_items directly — BI-11). Decide what adopted/derived records stamp (proposal: a new adopted_from_reference value, keeping the authored-derivation act queryable) — and decide it BEFORE the planned CHECK hardening freezes the set (13-ingest-source.md:93).
3. Acquisition-route vocabulary on the reference row (BI-9): whether it reuses CV 13 or gets its own small CV (proposal: reuse CV 13 semantics scoped to the reference table, avoiding a fifth provenance-ish vocabulary — the options paper’s O2 risk).

---

Verification (Q-EX2 pre-ratification check)

This PRODUCT cites no new external-library API surfaces. The Option-A cocoindex mechanics it relies on (BI-18/BI-19) were empirically verified in {75.1} RESEARCH §4.1 against the installed pin cocoindex[postgres]==1.0.3 on 04/06/2026 (mount_each, LiveMapView, LiveMapFeed, LiveMapSubscriber, PgTableSource — all PRESENT; a minimal custom URL source passes both runtime-checkable protocols; verdict: Option A VIABLE), and re-confirmed as a settled constant in the options paper §4.3 (S314, 05/06/2026). PullMD’s HTTP contract is in-repo and already correct (adapters.py:144-218). No further import-and-call checks were required for this behaviour-level spec; {75.3} TECH re-runs the check for any concrete API it cites.

---

Provenance pointers

• Ratified frame: docs/research/s314-id75-reframe-options.md §7 (Liam, S316, 05/06/2026); authoritative question docs/research/s314-id75-reframe-liam.md (S314).
• Predecessor: docs/specs/ID-75-pullmd-cocoindex/RESEARCH.md ({75.1}, 04/06/2026) — Option-A viability, constraints 1–9, OQs 1–5 (all five resolved or OQ’d above: OQ-1 → BI-19; OQ-2 → BI-18; OQ-3 → BI-4; OQ-4 → BI-20; OQ-5 → BI-21).
• Evidence inventories: docs/research/s314-id75-feed-current-state.md (legacy promotion path, seams 1–7); docs/research/s314-id75-usecase-affordances.md (U1–U17, A1–A15, T1–T8); docs/research/s314-ontology-boundary.md (4-question rule, q_a_pairs promotion precedent, register shape).
• Contract neighbours: docs/specs/ID-69-client-corpus-onboarding/PRODUCT.md (BI-1..BI-13 — adopted records inside, references outside); docs/ontology/32-q-a-pair.md + 21-scope-tag.md (corpus-level/no-workspace-FK precedent); docs/ontology/24-citation-kind.md (planned citation substrate, ID-58); docs/reference/tasks/ID-71.md (tooling remit inheriting the two-surface contract).
• Legacy promotion symbol: lib/intelligence/pipeline.ts:577-735 (storeAsContentItem; gitnexus context cited in Problem).
• Settled design input: docs/themes/canonical-pipeline/reference/pullmd-wiring-design-s300.md (re-opened for the content_items half only; mechanics retained).

Changelog

Date	Session	Note
05/06/2026	S316	{75.2} PRODUCT authored under the ratified O4/D4 frame (DP-1..DP-7). 23 numbered Behavior invariants across landing model, global reference layer, boundary re-point + legacy retirement, adoption/derivation, two-surface retrieval contract, acquisition mechanics, and ontology amendment + sequencing. Resolved RESEARCH OQs 1–5 at behaviour level. Five OQs (OQ-75-1..5) proposed for Liam ratification: peer-class name, conversion flavour, adoption actor, entity-mentions home, CV plumbing. DRAFT pending ratification.
05/06/2026	S316	Checker audit: PASS_WITH_NOTES — one nit (BI-4 “constraints 1–4, all ratified” obscured BI-2’s "ri:"-supersedes-"ci:" amendment of RESEARCH constraint 3) — fixed in BI-4. OQ-75-1..4 RATIFIED (Liam): reference_items canonical; conversion flavour EXCLUDED from v1 (derivation-only; ID-58 revisit); adoption = human or AI-assisted with explicit human confirmation, never autonomous; entity mentions over references deferred entirely in v1 (NOT-NULL contract untouched; U17 revisit driver). OQ-75-5 (CV plumbing) remains open — full ratification precedes {75.3} TECH.