Reference Item

Reference Item

§1 Mission

A reference_item is a global, workspace-less record of external evidence: a web page, article, or feed item admitted through a curation gate, stored verbatim as markdown, embedded, and retrievable by AI as cited external evidence — never as client-canonical knowledge. It is the Q1b box in the “Where does new data live?” rule (README; s314-ontology-boundary.md §6 as amended by ID-75 BI-22, and re-split by ID-133 Decision A): external evidence admitted by a relevance gate lands here, while client-canonical and client-adopted knowledge lands in q_a_pairs / source_documents (Q1a, three-way split post-ID-131 — see README “Where does new data live?”).

The class exists to hold the DP-5 line: the relevance gate admits evidence, not knowledge. External material becomes canonical knowledge only by an explicit, cited adoption or derivation act (BI-13) — never automatically. One row exists per normalised URL (UNIQUE (source_url), BI-2/BI-8); the corpus is shared across all workspaces, with workspace context carried as provenance on the staging ledger only (BI-7).

§2 Table shape

2.1 reference_items columns (landed M1 shape — 20260606121451_id75_reference_items_layer.sql)

Column	Type	Nullable	Notes
id	uuid	NO	PK; pipeline-minted uuid5("ri:" + normalised URL) — deliberately NO DEFAULT (BI-2)
title	text	NO	Ledger title (feed-declared); classifier suggested_title fallback if ever empty (D-10)
body	text	NO	PullMD/Docling markdown — the canonical body of record (BI-3)
summary	text	YES	v1: latest non-null ledger ai_summary (scoring-derived provenance accepted, D-10)
source_url	text	NO	Canonical normalised URL; UNIQUE — one reference per URL (BI-2/BI-8); the join contract (BI-4)
published_at	timestamptz	YES	Original publication time; never ingest time (BI-3)
primary_domain	text	YES	From the existing extract_classification pass (D-10)
primary_subtopic	text	YES	From the existing extract_classification pass (D-10)
layer	text	YES	v1 constant 'research'; trigger-validated against layer_vocabulary
embedding	vector(1024)	YES	Whole-record embedding (BI-17 — NO chunk table); HNSW cosine index
source_document_id	uuid	NO	FK → source_documents(id) ON DELETE RESTRICT — provenance chain integrity (BI-15)
ingestion_source	text	NO	CHECK ('rss_feed','url_import') — CV 13 semantics re-homed per TECH §6.3 (BI-9)
op_id	uuid	YES	Pipeline operation id
created_at	timestamptz	NO	now()
updated_at	timestamptz	NO	now(); BEFORE UPDATE trigger

No workspace FK and no junction table exist on this class — see §6.

2.2 ingestion_source CHECK extension rule (BI-9)

The acquisition-route vocabulary is extended by migration only: a new acquisition route adds a new value to the CHECK constraint and changes nothing else about the class. v1 values: 'rss_feed' (the relevance-gated feed ledger route, BI-18) and 'url_import' (the {42.12} manual URL re-point). Adding a hypothetical further route (e.g. a research-workspace connector) requires no schema change to reference_items beyond the CHECK value (BI-9 acceptance).

The column uses the canonical ingestion_source spelling (S236 Liam ruling, recorded in 13-ingest-source.md Notes; TECH D-11/OQ-T3) — a net-new table aligns to the canonical name rather than propagating the legacy content_items.ingest_source spelling (pre-ID-131; content_items itself is now DROPPED).

2.3 RLS / write surface

Corpus-level SELECT for all authenticated roles; no app-side write policies — writes are pipeline-only via the asyncpg owner connection (BI-16; q_a_pair_history precedent).

§3 Lifecycle

Rows land via the cocoindex URL source (TECH WP-C): the flow enumerates gate-passed ledger rows (feed_articles.passed = true), fetches via PullMD (Docling for PDFs, BI-20), and declares the source_documents + reference_items evidence pair.

• Land: uuid5("ri:" + normalised URL) mints the stable PK; the same URL seen from N ledger rows (across workspaces) lands exactly one item (BI-8).
• Update-in-place (BI-2): changed content re-fetches under the same PK (the D-4 content-epoch memo token drives re-fetch); the row is overwritten, not versioned. No version history exists in v1 — q_a_pair_history-style snapshots are NOT built here. ID-58 (citations/adoption) is the named design input for whether shipped citations require version-on-cite snapshots over references; until ID-58 decides, citation consumers must treat a reference body as mutable.
• No publication workflow: admission is the relevance gate itself (DP-5 — the gate admits evidence, not knowledge). There is no publication_status column and no review state machine on this class.

§4 Acquisition-route contract (BI-9)

The class is acquisition-route-agnostic. Every row records its route in ingestion_source as queryable provenance. The binding prohibition: no route may bypass the reference layer and land external material directly in q_a_pairs or source_documents (Q1a) — route extensibility never re-opens the DP-5 boundary. New routes inherit BI-9/BI-11 at the moment they land (a route lands a reference row, never a canonical row).

§5 Retrieval

Two-step retrieval mirroring q_a_search / q_a_get_verbatim exactly (BI-16; 32-q-a-pair.md §5):

1. reference_search(p_query text, p_query_embedding vector(1024), p_limit integer DEFAULT 20) — Step 1 ranked preview. Returns every BI-16 contract field (classification, acquisition origin, provenance chain head) with embedding_score numeric(5,4) + fulltext_score numeric(5,4) as separate columns (N9 precedent — no blended score exposed). Filters embedding IS NOT NULL. Internal ORDER BY uses embedding*0.6 + fulltext*0.4.
2. reference_get_verbatim(p_reference_id uuid) — Step 2 verbatim fetch: the full row excluding embedding (AI-consumer-first payload discipline).

ID-71 is the named MCP tool owner over these RPCs (BI-16 acceptance) — ID-75 fixes the data contract only; tool design (naming, params, response shaping) is ID-71’s. No default retrieval path interleaves reference rows into canonical-surface results: neither RPC touches source_documents/q_a_pairs, and no canonical RPC was modified — the two-surface separation is structural.

§6 Anti-patterns

The following are explicitly RATIFIED-DO-NOT-BUILD per the ID-75 spec chain:

Anti-pattern	Status	Rationale
reference_items.workspace_id FK / reference_item_workspaces junction	RATIFIED-DO-NOT-BUILD (BI-7)	The reference corpus is global and workspace-less; workspace context is provenance on the staging ledger (feed_articles.workspace_id) only. Mirrors the 32-q-a-pair.md §6 workspace-FK retirement. Future maintainers MUST NOT add either.
Auto-promotion into q_a_pairs / source_documents (Q1a)	RATIFIED-DO-NOT-BUILD (BI-13 / DP-5)	The Q1b→Q1a edge is the explicit, cited adoption act — human or AI-assisted with explicit human confirmation, never autonomous. No code path converts a reference row into a canonical row.
"ci:" uuid5 minted from a URL	RATIFIED-DO-NOT-BUILD (BI-2)	URL identity mints "ri:" (reference) and "sd:" (source document) seeds only. No URL-derived content_items PK exists (pre-ID-131 framing; content_items is now DROPPED) — adopted/derived records mint fresh ids regardless.
Modelling a feed item as a file on disk	RATIFIED-DO-NOT-BUILD (S300 §2.2)	A feed/URL item is URL-keyed remote content; forcing it through a file-shaped landing (path, bytes-on-disk, file metadata as identity) breaks the URL join contract (BI-4).
Widening content_chunks / entity_mentions with a reference parent	RATIFIED-DO-NOT-BUILD (BI-17 / OQ-75-4)	Both NOT-NULL writer contracts (content_item_id) stand unmodified in v1. References carry a whole-record embedding, not chunks; entity mentions over references are deferred entirely (U17 is the revisit driver — see §7).

§7 Upgrade paths (named, not built)

Documented so future work extends rather than re-litigates the v1 shape:

• reference_chunks sibling table — if long-document chunked retrieval is needed, a sibling table is the path (BI-17); never widen content_chunks with a reference parent.
• Entity mentions over references — the U17 longitudinal competitor/entity record is the named revisit driver (OQ-75-4 ratified deferral); a future design adds a reference-side mention surface without touching the entity_mentions.content_item_id NOT-NULL contract.
• scope_tag[] / anti_scope_tag[] columns — when CV 21 lands, query-time workspace relevance can be computed over references the same way as q_a_pairs; until then the corpus is unscoped.
• Fresh PullMD-body summarisation pass — v1 summary is the scoring-derived ledger ai_summary (D-10); a dedicated summarisation pass over the PullMD body is the named non-blocking upgrade.
• GIN expression fulltext index — v1 reference_search computes fulltext over title || ' ' || COALESCE(summary,'') || ' ' || body inline (corpus is small); a GIN expression index is the named scale upgrade.

Relationships

• reference_items.source_document_id FK → source_documents(id) ON DELETE RESTRICT — provenance chain integrity (BI-15). With source_documents.source_url (added in the same migration), the chain reference → source document → URL is resolvable by query in both directions (BI-4).
• feed_articles.reference_item_id FK → reference_items(id) ON DELETE SET NULL — the promotion backlink (BI-10): the Q4 staging/audit sidecar promotes into Q1b, mirroring the content_item_id idiom. Legacy content_item_id rows are untouched read-only lineage.
• Adjacent to CV 13 ingest_source — reference_items.ingestion_source reuses CV 13 semantics; rss_feed/url_import re-home to this class per TECH §6.3 (see 13-ingest-source.md Notes).
• Adjacent to CV 29 cocoindex_source_kind — the remote_url baseline value records the external surface for URL-acquired items (column still planned).
• Adjacent to CV 16 extraction_method — the linked source_documents row carries pullmd_* or docling (BI-20).
• Sibling of 32-q-a-pair.md — the other corpus-level, workspace-less Layer-5 retrieval class; the two-step retrieval shape and the anti-pattern discipline are deliberate mirrors.
• Layer 5 (Knowledge Graph): reference_item is the canonical Layer-5 “external evidence” entity (O4/D4).

Consumers

• Cocoindex URL source (scripts/kb_pipeline/ — TECH WP-C) — sole writer, via the asyncpg owner connection (BI-16).
• reference_search / reference_get_verbatim RPCs (M2) — the read surface.
• ID-71 MCP tool (named owner) — the future AI retrieval consumer over the two RPCs.
• ID-58 citations + adoption flow (slice 2) — the future Q1b→Q1a adoption act and citation substrate; consumes reference_items ids as citable addresses.
• {62.10} end-to-end proof — asserts the evidence-pair landing set (sd + ri, zero content_items rows, pre-ID-131 framing; content_items is now DROPPED so the zero-rows assertion is moot by construction) per TECH §5.

Notes

• Status: M1 MERGED + LIVE on staging (turayklvaunphgbgscat, S318, 06/06/2026) via 20260606121451_id75_reference_items_layer.sql. The M2 retrieval RPCs land with WP-B.
• References carry no content_type — the classifier’s content_type output is discarded (D-10); placement is never solved with a content_type value (BI-22 corollary 1; the enum stays closed per 04-content-type.md).
• Legacy lineage (pre-ID-131): the 55 url_import + 40 rss_feed legacy content_items rows stayed read-only legacy lineage (no retro-conversion, BI-10); content_items itself was DROPPED at ID-131 M6, so this lineage is now historical record only — re-landing as references happens at the ID-45/T7 fresh-corpus re-ingest (TECH §6.2).
• Table COMMENT (landed, quoted verbatim from the migration): “Global, workspace-less external reference/evidence layer (ID-75, O4/D4). One row per normalised URL. Never auto-promotes into content_items.” The content_items reference is pre-ID-131 phrasing at time of authorship — the current Q1a target is q_a_pairs / source_documents.