ID-75 — PullMD remote URL/feed source: TECH spec (O4 landing + reference layer)

ID-75 — PullMD remote URL/feed source: TECH spec (O4 landing + reference layer)

{75.3} TECH artefact. Translates the ratified {75.2} PRODUCT behaviour invariants (BI-1..BI-23) into an implementation plan against the current codebase. Authored by a fresh Planner dispatch (Q-PLANNER-2); PRODUCT.md read in full as input.

• Task: ID-75 “Wire PullMD into cocoindex correctly (remote URL/feed source)” (bl-217).
• Predecessors: {75.1} RESEARCH.md (04/06/2026, Option-A VIABLE); {75.2} PRODUCT.md (05/06/2026, ratified-for-TECH; OQ-75-1..4 ratified, OQ-75-5 items 1+3 ratified, item 2 decided HERE per Liam’s steer — see §6.3).
• Ratified frame (binding): O4/D4 per docs/research/s314-id75-reframe-options.md §7. content_items = client-adopted knowledge; external URL/RSS material lands source_documents + reference_items (OQ-75-1 ratified name); adoption/derivation is the only path to canonical knowledge.
• Scope: Slice 1 (landing, BI-1..BI-12, BI-16..BI-21) is the implementation scope. Slice 2 (adoption, BI-13..BI-15) is specced as contract only — implementation seats are ID-58 (citations) and ID-71 (affordance + tooling). BI-22/BI-23 bind both.
• Status: DRAFT — pending Checker audit + Liam ratification of OQ-T1..OQ-T3 (§10).
• Language: UK English; dates DD/MM/YYYY.

---

1. Context

1.1 Code-intelligence orientation (mandatory per .gitnexus/CLAUDE.md; cited verbatim, run 06/06/2026 against this HEAD)

• gitnexus_query({query: "pullmd remote source ingestion", repo: "knowledge-hub"}) returned processes: [] (no indexed cross-module execution flow) with the relevant symbols as standalone definitions: Function:scripts/cocoindex_pipeline/adapters.py:convert_binary_to_markdown (:55-90), Function:scripts/cocoindex_pipeline/adapters.py:_pullmd_to_markdown (:144-217), Function:scripts/cocoindex_pipeline/adapters.py:extract_source_provenance (:262-307), Function:scripts/cocoindex_pipeline/flow.py:bound_ingest_file (:2616-2639), plus the test surfaces TestSourceDocumentProvenanceWritePath (scripts/tests/test_cocoindex_flow_write_path.py:1052-1180).
• gitnexus_query({query: "content_items write path cocoindex", repo: "knowledge-hub"}) also returned processes: []; standalone definitions: Function:scripts/cocoindex_pipeline/flow.py:_ingest_file_body (:1683-1822), Function:scripts/cocoindex_pipeline/flow.py:_ingest_content_branch (:1826-2111), Function:scripts/cocoindex_pipeline/server.py:_stage_handler (:198-298), Function:scripts/cocoindex_pipeline/server.py:start_cocoindex_thread (:495-568), TestPerItemFailureIsolation (scripts/tests/test_cocoindex_flow_failure_mode.py:1403-1661).
• gitnexus_context per symbol this spec mandates modifying (verdicts: gitnexus indexes zero processes for all of them — the runtime chains run through the cocoindex engine and the Vercel cron, edges gitnexus does not trace — so structural blast radius is LOW throughout; product significance is stated per symbol):
  • convert_binary_to_markdown — incoming calls: exactly 1 (_ingest_content_branch); outgoing: _docling_to_markdown, _pullmd_to_markdown, _passthrough_markdown. LOW / change locus of the WP-D HTML-branch retirement.
  • extract_source_provenance — incoming calls: exactly 1 (_ingest_content_branch); outgoing: _pullmd_to_markdown. LOW / same retirement.
  • app_main (flow.py:2384-2834) — incoming calls: 0 (engine-invoked via KH_PIPELINE_APP); outgoing: 18 resolved callees (stage-error helpers, Stage-5, manifest loader, counters). LOW structural / HIGH product significance: it is the single Stage-1 binding site where the second source lands (WP-C).
  • processFeedSource (lib/intelligence/pipeline.ts:276-536) — incoming calls: exactly 1 (runPipeline); outgoing includes storeAsContentItem, isDuplicate, normaliseUrl, pollFeed, scoreRelevance. LOW structural / HIGH product significance: its passed-branch (:495-519) is the promotion step BI-10/BI-11 swap.
  • normaliseUrl (lib/intelligence/content-extractor.ts:95-120) — incoming calls: 3 (isDuplicate, processFeedSource, storeAsContentItem, all in lib/intelligence/pipeline.ts). LOW / the TS half of the BI-2/BI-8 parity contract.
• The Python pipeline is outside ast-dataflow’s TypeScript corpus; Python grounding below used gitnexus + targeted grep sweeps (file:line citations throughout), per .ast-dataflow/CLAUDE.md.

1.2 Current state (re-grounded at this HEAD — line numbers have shifted since PRODUCT was authored)

The defect. adapters.py:74-77: the HTML suffix branch hands str(file.file_path.path) (a local container path) to _pullmd_to_markdown, whose contract is GET {PULLMD_SERVICE_URL}/api?url=<http(s) URL> (adapters.py:144-217 — contract correct, input categorically wrong). Repeated at adapters.py:283 (extract_source_provenance HTML branch).

The flow shape. One Stage-1 source: localfs.walk_dir(source_path, live=True, recursive=True) (flow.py:2560-2564) → coco.mount_each(coco.component_subpath( "ingest_file"), bound_ingest_file, source.items(), …7 targets) (flow.py:2645-2656). bound_ingest_file is a NAMED closure carrying run context across the _LoopRunner daemon-thread boundary ({66.19}; functools.partial is incompatible — engine reads __name__/__qualname__). Per-item containment at the mount boundary (ID-80.9): unexpected escapes are tallied + logged, never abort the batch. Targets are mounted managed_by=ManagedBy.USER (rows only, no DDL) via mount_table_target(DB_CTX, …) (flow.py:2506-2555); raw pool access for non-target SQL uses pool = coco.use_context(DB_CTX) (Stage-5 precedent, flow.py:2371, stage_5.py throughout).

Operational posture (post-dates RESEARCH — load-bearing for BI-19). bl-221/ID-83 retired the continuous watcher: /stage only lands bytes; ingestion fires on the bearer-gated POST /walk running a one-shot KH_PIPELINE_APP.update_blocking(live=False, full_reprocess=…) behind a single-flight lock (server.py:18-20, :330-360, :397-410; bearer = CRON_SECRET). full_reprocess=True is the cache-invalidating full re-walk (server.py:381). There is therefore no standing cadence today — BI-19 requires this spec to add one (§2 D-3).

Identity + write sites. _KH_PIPELINE_DOC_NS = uuid.UUID("fbfaf1ff-…") (flow.py:1479); content branch seeds uuid5(NS, "sd:"+rel_path) / "ci:"+rel_path (flow.py:1875-1876) and declares sd_target rows with storage_path=rel_path, content_hash, NOT-NULL filename/mime_type/file_size, extraction_method, pullmd_share_id (flow.py:1902-1921; SOURCE_DOCUMENTS_SCHEMA flow.py:1163-1188). CONTENT_ITEMS_SCHEMA (flow.py:1111-1141) carries no source_url/ingest_source (the parity seam — irrelevant here: URLs never land content_items under O4).

The TS legacy promotion path (BI-11 retirement target). lib/intelligence/pipeline.ts: gate at :495 (if (passed)) → storeAsContentItem (:577-735) — dedup-by-source_url, content_items INSERT with ingest_source: 'rss_feed' (:653), feed_articles.content_item_id backlink (:674-678), classifyContent, ensureWorkspaceLink junction write (:541-569, called :615/:734). normaliseUrl (content-extractor.ts:95-120): lowercase hostname, strip 7 tracking params (utm_source/utm_medium/utm_campaign/utm_term/ utm_content/ref/source), strip trailing slash (non-root). feed_articles.external_url is stored already normalised by this function at insert time.

Schema facts. feed_articles (20260416102457_pre_squash_reconciliation.sql:3690): external_url text NOT NULL, passed boolean NOT NULL DEFAULT false, content_item_id uuid (nullable, ON DELETE SET NULL idiom), published_at, ai_summary, title NOT NULL, dedup UNIQUE (workspace_id, external_url). source_documents has no source_url column (net-new, RESEARCH §2.4); its extraction_method CHECK already admits pullmd_* + docling (20260526074944_id42_pullmd_provenance.sql). cleanup_filtered_articles() (pre-squash :392) deletes passed=false rows only. ingestion_quality_log (pre-squash, flag_type CHECK of 6 values, content_item_id nullable, details jsonb) is the existing operator-visible quality surface (BI-21 home, §2 D-9). content_items.layer is trigger-validated against layer_vocabulary (validate_layer_key(), pre-squash :3299); research is a valid layer key (scripts/kb_pipeline/layer_inference.py:23). HNSW embedding-index precedent: USING hnsw (embedding vector_cosine_ops) WITH (m=16, ef_construction=64) (pre-squash :4683).

Precedents reused. q_a_pairs corpus-level RLS + grants (20260520225456_t6_q_a_pairs_full_schema.sql: grant_standard_public_table_access, SELECT-only policies, REVOKE EXECUTE FROM anon, SET search_path = public, extensions); two-step retrieval RPCs (20260520231524_t6_q_a_search_rpcs.sql: separate embedding_score/fulltext_score NUMERIC(5,4), 0.6/0.4 internal blend, verbatim step excludes the embedding). SSRF logic to port: lib/extraction/url-validation.ts (protocol allowlist http/https; loopback + 10/8, 172.16/12, 192.168/16, 169.254/16 rejection). update_updated_at_column() trigger fn exists (pre-squash :3285).

Reference PRODUCT.md for all behaviour; this spec restates none of it.

---

2. Design decisions (with rejected alternatives — ledger discipline ID-34)

D-1 — Hand-rolled URL source, not PgTableSource. The source is a small class in a new scripts/cocoindex_pipeline/url_source.py yielding (normalised_url, UrlItem) pairs from one SELECT over feed_articles WHERE passed = true. REJECTED: cocoindex.connectors.postgres._source.PgTableSource — its verified signature (§9) takes table_name + optional columns, i.e. whole-table row enumeration; it cannot express the filtered predicate, nor the cross-workspace URL-dedup grouping BI-8 requires (N ledger rows → 1 item), without post-filtering that re-implements the hand-rolled source anyway. The localfs DirWalker is the structural reference implementation (RESEARCH §4.1).

D-2 — Snapshot source; watch() feeds the snapshot via update_all() then mark_ready(). (CORRECTED S319 — {75.16}.) The source’s items() returns a LiveMapView-conforming object. The original D-2 claim (“watch(subscriber) calls subscriber.mark_ready() and returns — the engine consumes the snapshot per walk via __aiter__”) is false on the real engine: cocoindex 1.0.3 mount_each routes ANY watch-bearing source through _MountEachLiveComponent, and _mount_live_component (api.py:326) always starts process_live() regardless of live mode — items flow EXCLUSIVELY through items.watch(subscriber); __aiter__ is reached only via subscriber.update_all(). A mark_ready-only watch() therefore enumerates NOTHING (S319 live finding: walks completed with url tally 0 and no sd/ri rows). The correct one-shot contract is await subscriber.update_all() then await subscriber.mark_ready() — in catch-up mode mark_ready() terminates the watch, giving exactly one snapshot feed per walk (bl-221 posture preserved; the localfs _LiveDirItems twin minus its watchfiles loop). Executable contract record: scripts/tests/test_url_source_engine_consumption.py (four-case probe matrix). REJECTED (unchanged): live watch() over Postgres LISTEN/NOTIFY — contradicts the ratified one-shot walk posture, adds daemon-thread change-signal risk for no v1 behaviour gain.

D-3 — BI-19 standing cadence = poller-fired walk nudge (+ scheduled fallback). The TS poller, at the end of a runPipeline run in which ≥1 article newly passed, fires a fire-and-forget POST {COCOINDEX_WORKER_URL}/walk with Authorization: Bearer {CRON_SECRET} (the /walk route already validates exactly this bearer, server.py:397-410). Failure-tolerant by construction: the enumeration is snapshot-based, so a missed nudge only delays landing until the next nudge — nothing is lost. A Coolify scheduled task POSTing /walk hourly is the belt-and-braces fallback. Documented cadence (BI-19 acceptance): poller discovery ≤15 min + nudge-triggered walk ≈ minutes; worst-case (nudge lost) ≤1 h via the fallback. This is part of the promotion-step swap (BI-12: the promotion step is the ONLY poll-path change). REJECTED: cron-only walks every 15 min — walks the whole localfs corpus on a timer for no event; REJECTED: manual walks only — fails BI-19 (“a manually-triggered one-shot batch does not satisfy it”). OQ-T2 (§10) ratifies this cross-service call.

D-4 — BI-2 update-in-place via a content_epoch memo token. The per-item component and the PullMD fetch are memoised on (normalised_url, content_epoch), NOT on the URL alone (a URL-only memo would return stale markdown forever, silently breaking BI-2’s changed-content re-fetch). v1 epoch = the ledger row’s ingested_at (max across the URL’s rows) — i.e. one fetch per article in steady state; the recorded changed-content re-fetch route is POST /walk {"full_reprocess": true} (cache-invalidating, server.py:381), which re-fetches and UPSERTs under the same PK. A later slice may bump the epoch on poller re-discovery. The existing _pullmd_to_markdown(url) (memo key = url) is retired with the localfs HTML branch (WP-D); its HTTP body is extracted to a plain non-memo helper wrapped by the new epoch-keyed @coco.fn.

D-5 — BI-11 retirement = code removal, not a gate. Remove the if (passed) promotion block (pipeline.ts:495-519), storeAsContentItem (:577-735), ensureWorkspaceLink (:541-569), and inferContentType (PLAN verifies it has no other callers; isDuplicate STAYS — it guards ledger inserts). Grounds: the platform is pre-live; Liam’s OQ-75-5(2) steer is to align to the new structure, not preserve legacy; a dead gate rots and invites re-enablement. The legacy lineage lives in DATA (retained content_item_id backlinks + promoted rows, BI-10), not code.

D-6 — ingest_url lives in flow.py beside ingest_file; leaf logic in new modules. New leaf modules: url_normalise.py (D-8), url_validation.py (SSRF port), url_source.py (D-1). The per-item component (ingest_url + _ingest_url_body) and REFERENCE_ITEMS_SCHEMA live in flow.py, mirroring ingest_file exactly (named closure, component subpath pinning, containment). REJECTED: a separate flow_url.py for the component — circular-import risk against the schemas/extractors/counters flow.py owns.

D-7 — Backlink write via raw pool UPDATE (Stage-5 precedent). After declaring the sd:/ri: pair, the URL branch runs UPDATE feed_articles SET reference_item_id = $1 WHERE external_url = ANY($2) AND passed over pool = coco.use_context(DB_CTX), with $2 = the raw ledger URLs captured at enumeration (precise even if normalisation rules ever drift from stored values). REJECTED: mounting feed_articles as a declare_row target — declare_row is a PK-keyed UPSERT; we are updating existing rows whose PKs the item does not carry.

D-8 — URL-normalisation parity via a shared fixture. Port normaliseUrl (content-extractor.ts:95-120) rule-for-rule to scripts/cocoindex_pipeline/url_normalise.py. A shared JSON fixture (scripts/tests/fixtures/url_normalisation_parity.json, input→expected pairs including tracking-param, trailing-slash, case, and idempotency cases) is consumed by BOTH a pytest and a Vitest guard (extending the existing pipeline-parity guard-test pattern). Drift on either side breaks tests on both sides (BI-2/BI-8 seam 2). Python re-applies normalise() to enumerated URLs defensively (idempotent on the already-normalised ledger values).

D-9 — BI-21 operator surface = ingestion_quality_log. SSRF rejections write a row: flag_type='ssrf_rejected' (CHECK extended by M1), severity='error', content_item_id=NULL, details={source_url, reason, op_id, feed_article_ids} — plus the structured log. DB-queryable via the existing quality-log surface; the dedicated UI filter is named ID-71-adjacent follow-on (the invariant requires operator discoverability without log access, which a quality-log row provides). REJECTED: a rejection column on feed_articles (pollutes the ledger); run-level webhook error fields (rejections are per-item, not run failures).

D-10 — v1 classification fields on the reference row. primary_domain / primary_subtopic from the existing extract_classification pass (its content_type output is discarded — references carry no content_type, preserving the closed enum, BI-22 corollary 1); layer = 'research' constant (matches the existing url_import → research inference fallback, layer_inference.py:184-189; trigger-validated against layer_vocabulary); title = ledger title (NOT NULL, feed-declared) with classifier suggested_title as fallback if ever empty; summary = latest non-null ledger ai_summary (nullable; scoring-derived provenance accepted v1 — a fresh PullMD-body summarisation pass is a named non-blocking upgrade).

D-11 — reference_items.ingestion_source spelling. The acquisition-route column (BI-9, OQ-75-5(3): reuse CV 13 semantics) uses the canonical ingestion_source spelling per the S236 Liam ruling (CV 13 doc Notes) — a net-new table aligns to the canonical name rather than propagating the legacy ingest_source spelling. Flagged as OQ-T3 (nit) since it diverges from the content_items column spelling until the Q-OQR1-16 rename lands.

D-12 — PDF route detection. .pdf path-suffix check first (cheap, deterministic); otherwise an httpx HEAD content-type sniff with application/pdf match; HEAD failure ⇒ assume HTML and let PullMD try (its failure is contained per BI-19). PDF bytes are fetched with the same SSRF-validated URL and routed to the existing _docling_to_markdown(content_bytes, filename) (memoised on content hash); extraction_method='docling', mime_type='application/pdf' (BI-20).

---

3. Proposed changes (work packages)

WP-A — Migration M1: the reference layer (id75_reference_items_layer)

Via supabase migration new id75_reference_items_layer (CLI-only DDL; staging-first turayklvaunphgbgscat; verify cat supabase/.temp/project-ref before push). Content:

SET search_path = public, extensions;

-- 1. reference_items — peer Layer-5 class (O4/D4; PRODUCT BI-3/BI-7).
--    id is PIPELINE-MINTED uuid5('ri:'+normalised URL) — deliberately NO DEFAULT.
CREATE TABLE public.reference_items (
  id uuid PRIMARY KEY,
  title text NOT NULL,
  body text NOT NULL,                -- PullMD/Docling markdown: the canonical body of record
  summary text NULL,
  source_url text NOT NULL,          -- canonical normalised URL (join contract, BI-4)
  published_at timestamptz NULL,     -- original publication time; never ingest time (BI-3)
  primary_domain text NULL,
  primary_subtopic text NULL,
  layer text NULL,                   -- v1 constant 'research'; validated below
  embedding vector(1024) NULL,       -- whole-record embedding, BI-17 (no chunk table)
  source_document_id uuid NOT NULL
    REFERENCES public.source_documents(id) ON DELETE RESTRICT,  -- provenance chain integrity (BI-15)
  ingestion_source text NOT NULL
    CHECK (ingestion_source IN ('rss_feed','url_import')),      -- CV 13 semantics, BI-9 / §6.3
  op_id uuid NULL,
  created_at timestamptz NOT NULL DEFAULT now(),
  updated_at timestamptz NOT NULL DEFAULT now(),
  CONSTRAINT reference_items_source_url_key UNIQUE (source_url) -- one reference per URL (BI-2/BI-8)
);
-- NO workspace FK. NO junction table. RATIFIED-DO-NOT-BUILD (BI-7; 32-q-a-pair.md §6 mirror).

CREATE INDEX idx_reference_items_embedding ON public.reference_items
  USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64);
CREATE INDEX idx_reference_items_published_at ON public.reference_items (published_at DESC);
CREATE INDEX idx_reference_items_source_document_id ON public.reference_items (source_document_id);

CREATE TRIGGER set_reference_items_updated_at BEFORE UPDATE ON public.reference_items
  FOR EACH ROW EXECUTE FUNCTION public.update_updated_at_column();
CREATE TRIGGER trg_validate_reference_items_layer
  BEFORE INSERT OR UPDATE OF layer ON public.reference_items
  FOR EACH ROW EXECUTE FUNCTION public.validate_layer_key();

-- RLS: corpus-level read for all authenticated roles; NO app-side write policies —
-- writes are pipeline-only via the asyncpg owner connection (BI-16).
ALTER TABLE public.reference_items ENABLE ROW LEVEL SECURITY;
SELECT public.grant_standard_public_table_access('public.reference_items'::regclass);
-- anon SELECT and authenticated CRUD grants above are overridden by RLS; the policy
-- below is the effective gate (q_a_pair_history precedent).
CREATE POLICY reference_items_select ON public.reference_items
  FOR SELECT TO authenticated USING (true);
-- (no INSERT/UPDATE/DELETE policies — q_a_pair_history precedent)

-- 2. feed_articles promotion FK re-point (BI-10) — mirrors the content_item_id idiom.
ALTER TABLE public.feed_articles ADD COLUMN IF NOT EXISTS reference_item_id uuid NULL
  REFERENCES public.reference_items(id) ON DELETE SET NULL;
CREATE INDEX IF NOT EXISTS idx_feed_articles_reference_item_id
  ON public.feed_articles (reference_item_id) WHERE reference_item_id IS NOT NULL;

-- 3. source_documents provenance hardening (BI-4; RESEARCH constraint 1 ratified).
ALTER TABLE public.source_documents ADD COLUMN IF NOT EXISTS source_url text NULL;
CREATE INDEX IF NOT EXISTS idx_source_documents_source_url
  ON public.source_documents (source_url) WHERE source_url IS NOT NULL;

-- 4. BI-21 operator surface: extend the quality-log flag_type enum (D-9).
ALTER TABLE public.ingestion_quality_log DROP CONSTRAINT IF EXISTS ingestion_quality_log_flag_type_check;
ALTER TABLE public.ingestion_quality_log ADD CONSTRAINT ingestion_quality_log_flag_type_check
  CHECK (flag_type = ANY (ARRAY['duplicate','low_quality','missing_field','review_needed',
                                'stale','conflicting','ssrf_rejected']));

COMMENT ON TABLE public.reference_items IS
  'Global, workspace-less external reference/evidence layer (ID-75, O4/D4). One row per normalised URL. Never auto-promotes into content_items.';

No PL/pgSQL functions in M1 ⇒ no REVOKE-anon needed here. Post-apply: regenerate database.types.ts via the documented supabase gen types command; consume row shapes via Tables<'reference_items'>.

WP-B — Migration M2: two-step retrieval RPCs (id75_reference_search_rpcs)

Mirrors q_a_search/q_a_get_verbatim exactly (BI-16; 32-q-a-pair.md §5):

• public.reference_search(p_query text, p_query_embedding vector(1024), p_limit integer DEFAULT 20) — Step 1 ranked preview. Returns (reference_id, title, summary_preview, body_preview, embedding_score numeric(5,4), fulltext_score numeric(5,4), source_url, published_at, primary_domain, primary_subtopic, layer, ingestion_source, source_document_id) — every BI-16 contract field (classification, acquisition origin, provenance chain head). Filter embedding IS NOT NULL. Separate scores (N9 precedent); internal ORDER BY embedding*0.6 + fulltext*0.4. Fulltext over title || ' ' || COALESCE(summary,'') || ' ' || body (corpus is small v1; a GIN expression index is the named scale upgrade).
• public.reference_get_verbatim(p_reference_id uuid) — Step 2 verbatim fetch: the full row excluding embedding (AI-consumer-first payload discipline, BI-16).

Both: LANGUAGE plpgsql STABLE SECURITY DEFINER SET search_path = public, extensions; REVOKE EXECUTE … FROM anon; + GRANT EXECUTE … TO authenticated, service_role; per RLS-PATTERN P-4 (vector params use bare vector in GRANT/REVOKE signatures, per the T6 note). No default retrieval path interleaves reference rows into canonical-surface results — neither RPC touches content_items/q_a_pairs, and no existing canonical RPC is modified (BI-16 two-surface separation is structural). MCP tool design over these RPCs is ID-71’s (the spec fixes the data contract only).

WP-C — Python: URL substrate + flow wiring

New leaf modules:

• scripts/cocoindex_pipeline/url_normalise.py — normalise_url(url: str) -> str, the exact TS rule (D-8): lowercase hostname; delete query params utm_source, utm_medium, utm_campaign, utm_term, utm_content, ref, source; strip trailing slash from non-root path; return input unchanged on parse failure.
• scripts/cocoindex_pipeline/url_validation.py — validate_url(url) -> tuple[bool, str | None], porting lib/extraction/url-validation.ts verbatim: http/https only; reject loopback hostnames (localhost, 127.0.0.1, 0.0.0.0, ::1, [::1]) and private/reserved IPv4 ranges (10/8, 172.16/12, 192.168/16, 169.254/16) (BI-21; RESEARCH constraint 6).
• scripts/cocoindex_pipeline/url_source.py — UrlItem frozen dataclass: url (normalised), title, summary, published_at, ingestion_source ('rss_feed' v1), content_epoch (max ingested_at ISO string — D-4), ledger_urls: tuple[str, ...] (raw stored values, for the D-7 backlink predicate), workspace_ids: tuple[str, ...] (provenance/attribution only — never written to reference_items; workspace provenance is recoverable via the feed_articles backlink join, BI-7/BI-18). FeedUrlSource(pool) with .items() returning a LiveMapView-conforming snapshot iterator (D-2): SELECT external_url, title, ai_summary, published_at, ingested_at, workspace_id FROM feed_articles WHERE passed = true — no feed_sources read, no scoring logic (BI-18); rows grouped Python-side by normalise_url(external_url) into one item per URL (BI-8); watch(subscriber) = await subscriber.update_all() then await subscriber.mark_ready() (corrected D-2 contract, {75.16} — a mark_ready-only no-op never enumerates on the real engine).

adapters.py:

• Extract the PullMD HTTP body of _pullmd_to_markdown into a plain (non-memo) _pullmd_http_get(url) -> PullmdResult (unchanged contract, log-then-raise pattern intact); add @coco.fn(memo=True) async def _pullmd_fetch(url: str, content_epoch: str) -> PullmdResult delegating to it (D-4 memo key).
• Retire the localfs HTML branches (WP-D rationale): convert_binary_to_markdown’s _HTML_EXTENSIONS branch (:74-82) and extract_source_provenance’s (:282-305) now raise/return a LOUD named error: HTML content lands via the URL source (ID-75); the file corpus does not route HTML to PullMD. _pullmd_to_markdown is deleted once both callers are re-pointed. The false comment at :75 goes with it.

flow.py:

• REFERENCE_ITEMS_SCHEMA = TableSchema(…) mirroring §WP-A columns the pipeline writes (id, title, body, summary, source_url, published_at, primary_domain, primary_subtopic, layer, embedding (encoder=_encode_pgvector), source_document_id, ingestion_source, op_id; created_at/updated_at OMITTED per the PG-default convention).
• Add "source_url": ColumnDef(type="text", nullable=True) to SOURCE_DOCUMENTS_SCHEMA; the localfs branch’s sd_target.declare_row dict adds "source_url": None explicitly.
• New per-item component ingest_url(item, ri_target, sd_target, *, op_id, stage_counter, …) + _ingest_url_body, mirroring ingest_file/_ingest_file_body:
  1. validate_url — on rejection: structured log (event: cocoindex.url_ssrf_rejected), ingestion_quality_log row (D-9), return with ZERO rows; siblings unaffected (BI-21).
  2. PDF sniff (D-12): PDF ⇒ SSRF-validated httpx GET bytes → _docling_to_markdown(bytes, filename); extraction_method='docling', mime_type='application/pdf', file_size=len(bytes), content_hash=sha256(bytes) (BI-20). HTML ⇒ _pullmd_fetch(item.url, item.content_epoch); extraction_method='pullmd_<x_source>' via the existing _PULLMD_X_SOURCE_METHODS mapping, pullmd_share_id from X-Share-Id, mime_type='text/html', file_size=len(markdown.encode()), content_hash=sha256(markdown.encode()) (RESEARCH constraints 4/8).
  3. extract_classification(markdown) (content_type output discarded — D-10) + embed_content_text(markdown).
  4. Mint sd_id = uuid5(NS, "sd:"+url) / ri_id = uuid5(NS, "ri:"+url); "ci:" is never seeded from a URL (BI-1/BI-2/BI-13).
  5. sd_target.declare_row: storage_path = source_url = normalised URL (RESEARCH constraint 2), URL-derived filename (last path segment, else hostname), provenance per step 2 (BI-4).
  6. ri_target.declare_row: the full BI-3 contract; layer='research'; ingestion_source=item.ingestion_source; op_id.
  7. Backlink (D-7): raw-pool UPDATE feed_articles SET reference_item_id=$1 WHERE external_url = ANY($2) AND passed (BI-10; all N workspace rows backlink the one reference, BI-8).
  8. _bump stage counters (source_walk, binary_conversion, llm_extraction, embedding, postgres_upsert ×2) per the Inv-17 semantics.
• app_main wiring (the genuine change locus): mount ri_target = await mount_table_target(DB_CTX, "reference_items", REFERENCE_ITEMS_SCHEMA, managed_by=ManagedBy.USER); build url_source = FeedUrlSource(pool=coco.use_context(DB_CTX)); bind via a NAMED closure bound_ingest_url (containment + per-branch failure tally 'url', exactly the bound_ingest_file pattern — functools.partial PROHIBITED) and await coco.mount_each(coco.component_subpath("ingest_url"), bound_ingest_url, url_source.items(), ri_target, sd_target); await handle.ready(). Per-item failures log + skip + retry on a later enumeration (memo miss on unchanged epoch only if no rows were declared — a failed item declared nothing, so the next walk re-runs it) (BI-19).
• EXECUTOR-VERIFY-1: memoisation of a @coco.fn(memo=True) component over a frozen dataclass arg (UrlItem) must be verified empirically at implementation time (the engine memo-keys on serialised args; FileLike has bespoke handling). Fallback if unsupported: pass scalar fields positionally. This is an engine-behaviour check, not an API-presence check (§9 covers presence).

WP-D — Localfs HTML branch retirement

Covered in WP-C adapters.py changes. Consequence: a .html/.htm file staged into the corpus now fails LOUDLY per-file (contained at the mount boundary) instead of silently handing PullMD an unreachable path. Post-pivot client corpora are md/docx/pdf/xlsx; the {42.10} local-HTML proof shape is superseded by the BI-23 URL proof. _HTML_EXTENSIONS stays in _SOURCE_MIME_FALLBACK (mime resolution) but exits the conversion routing. Update test_cocoindex_adapters.py / test_cocoindex_flow_write_path.py HTML fixtures accordingly.

WP-E — TS legacy promotion retirement (BI-11) + walk nudge (D-3)

lib/intelligence/pipeline.ts:

• Delete the if (passed) { … storeAsContentItem … } block (:495-519) — the passed-branch keeps result.articlesPassed++ only. Delete storeAsContentItem, ensureWorkspaceLink, inferContentType (PLAN confirms no other callers via ast-dataflow callers); keep isDuplicate, scoring, summary, ledger writes, RSS out untouched (BI-12).
• Add the nudge: after a run with articlesPassed > 0, fire-and-forget fetch(${process.env.COCOINDEX_WORKER_URL}/walk, { method: 'POST', headers: { Authorization: Bearer ${process.env.CRON_SECRET} } }) with a short timeout and catch-and-log (a failed nudge is a delay, not a loss — D-3). Env: COCOINDEX_WORKER_URL net-new on Vercel (staging + prod); CRON_SECRET already exists both sides.
• Tool discipline (Inv 3): gitnexus_impact on processFeedSource/storeAsContentItem before edit; gitnexus_detect_changes before commit; ast-dataflow callers sweep for the deletions.

The sibling manual-URL path (app/api/ingest/url/route.ts, 'url_import') is NOT touched here — it inherits the same obligation at its own {42.12} re-point moment (BI-23 reconcile flag).

WP-F — Ontology + documentation edits (BI-22; lands with implementation)

• docs/ontology/33-reference-item.md — new Layer-5 register entry in the 32-q-a-pair.md shape (frontmatter, table shape, lifecycle, retrieval = the two RPCs, relationships). Anti-pattern rows (all RATIFIED-DO-NOT-BUILD): workspace FK / reference_item_workspaces junction; auto-promotion into content_items; "ci:" minted from a URL; modelling a feed item as a file on disk (S300 §2.2); widening content_chunks/entity_mentions with a reference parent (BI-17, OQ-75-4). Upgrade paths documented: reference_chunks sibling table; entity mentions (U17 driver); scope_tag[]/anti_scope_tag[] columns when CV 21 lands.
• docs/ontology/13-ingest-source.md — record the §6.3 re-homing decision.
• docs/ontology/29-cocoindex-source-kind.md — add remote_url baseline value (OQ-75-5(1) ratified; doc-only — the column is still planned).
• docs/ontology/README.md — “Where does new data live?” gains the Q1a/Q1b split + the adoption promotion edge (BI-22).

WP-G — {62.10} proof re-shape support

No new server surface needed (§5). server.py is unchanged except documentation: the _stage_handler docstring note that URL items are staged by seeding a gate-passed ledger row, not by staging bytes.

---

4. BI → change mapping (one-to-one; the Checker’s verification index)

BI	Implemented by	Slice
BI-1	WP-C ingest_url declares sd+ri only; no ci_target in its signature (structural impossibility of a content_items write); WP-E removes the TS writer	1
BI-2	WP-C step 4 ("ri:"/"sd:" uuid5 seeds; "ci:" never); UNIQUE(source_url); declare_row UPSERT under stable PK; D-4 epoch for changed-content re-fetch	1
BI-3	WP-A reference_items columns; WP-C step 6 (D-10 field sources; published_at from the ledger’s original value, never ingest time; PullMD body canonical)	1
BI-4	WP-A §3 (source_documents.source_url + index); WP-C step 5 (storage_path = normalised URL; URL-derived filename/mime/size; both recovery directions by query)	1
BI-5	WP-C enumeration predicate passed = true is the ONLY automated consequence wired; WP-E removes the passed-conditioned content_items write	1
BI-6	Enumeration selects passed = true only; cleanup_filtered_articles() deletes passed = false only — disjoint by construction (no change to the cron)	1
BI-7	WP-A: no workspace column, no junction table (comment + register anti-pattern); WP-C: UrlItem.workspace_ids never written to the row	1
BI-8	WP-C Python-side grouping by normalised URL (N ledger rows → 1 item); UNIQUE(source_url); D-7 backlinks all N rows	1
BI-9	WP-A ingestion_source column (CV 13 semantics, §6.3); contract documented in 33-reference-item.md incl. the bypass prohibition; new routes = new CHECK value only	1
BI-10	WP-A §2 (reference_item_id FK, ON DELETE SET NULL both directions); WP-C step 7 writes it; legacy content_item_id rows untouched; re-landing = ID-45/T7	1
BI-11	WP-E code removal (D-5); regression test asserts zero content_items/content_item_workspaces writes on the poll path	1
BI-12	WP-E diff scope = promotion block + nudge only; poller discovery/scoring/triage/RSS untouched (reviewable by diff)	1
BI-13	Contract only (slice 2): schema supports it — derived records mint fresh ids (no URL-derived content_items PK exists); no code path converts a reference row; ID-58 citations + ID-71 affordance build it	2
BI-14	Contract only: adopted records are ordinary content_items rows inside the ID-69 junction contract; nothing in WP-A/WP-C constrains them; adopted_from_reference ingest value reserved (§6.3)	2
BI-15	Substrate shipped: ri.source_document_id (RESTRICT) + sd.source_url make reference→sd→URL resolvable by query; citation hop + admin/editor gate land with ID-58/ID-71 (getAuthorisedClient() { success } + authFailureResponse(auth))	2
BI-16	WP-B two RPCs expose every contract field; no canonical RPC modified; no default blending (structural); ID-71 named tool owner in the register entry	1
BI-17	WP-A: embedding column, NO chunk table; content_chunks/entity_mentions schemas + NOT-NULL writer contracts untouched (OQ-75-4 ratified); upgrade paths in register entry	1
BI-18	WP-C FeedUrlSource enumerates feed_articles.passed = true only; no feed_sources read; no scoring; hand-rolled per D-1; workspace context provenance-only	1
BI-19	D-2 snapshot source + D-3 standing nudge/fallback cadence; per-item containment + structured log + retry-on-next-walk; no partial rows on failure (declare happens after fetch/extract succeed)	1
BI-20	WP-C step 2 / D-12: PDF → Docling over fetched bytes, never PullMD; extraction_method='docling' recorded with URL identity intact	1
BI-21	WP-C step 1 + D-9: skip (zero rows) + structured log + ingestion_quality_log row (ssrf_rejected); siblings unaffected	1
BI-22	WP-F register entry + README Q1a/Q1b amendment + CV doc updates	both
BI-23	§5 landing-set contract ({62.10}); WP-E leaves the {42.11}/{42.12} surfaces untouched with reconcile flags restated; ID-45/T7 consumes the landing model	both

---

5. Landing-set contract for {62.10} (the proof re-shape — BI-23)

The end-to-end proof exercises a real http(s) URL → PullMD → the evidence pair — explicitly NOT a content_items landing:

1. Stage: seed one gate-passed ledger row directly into staging (turayklvaunphgbgscat) via service-role insert: feed_articles (workspace_id, feed_source_id, external_url = normaliseUrl(<real public URL>), title, passed = true, published_at). No /stage byte-staging, no local HTML fixture — the /stage route remains file-fixture-only.
2. Trigger: POST {worker}/walk with Authorization: Bearer ${CRON_SECRET} (the existing bl-221 route; no new server surface).
3. Assert (the landing set):
   • source_documents row at id = uuid5(fbfaf1ff-…, "sd:" + normalisedUrl) with source_url = storage_path = normalisedUrl, populated filename/mime_type/file_size, extraction_method LIKE 'pullmd_%', non-null pullmd_share_id.
   • reference_items row at id = uuid5(fbfaf1ff-…, "ri:" + normalisedUrl) with body = PullMD markdown, embedding IS NOT NULL, source_document_id = the sd id, ingestion_source = 'rss_feed', published_at round-tripping the seeded value.
   • Zero content_items rows whose id = uuid5(NS, "ci:" + normalisedUrl) or whose source_url = the URL.
   • feed_articles.reference_item_id = the ri id; content_item_id IS NULL.
   • Inv-9 round-trip: GET {PULLMD}/s/{pullmd_share_id} returns 2xx (a genuine remote fetch backs the share id).
   • Idempotency leg: second /walk ⇒ row counts unchanged, PKs unchanged.

The proof URL must be a stable public page (politeness + determinism — {62.10} picks it; https://example.com/-class stability recommended).

---

6. Migration plan

6.1 Sequencing (slice 1; nothing here precludes slice 2)

1. M1 + M2 to staging (db push; verify project-ref first) → regenerate types → commit.
2. WP-C Python (leaf modules + flow wiring + adapters refactor) behind the existing deploy path (onprem-deploy.yml); Python tests green (python3 -m pytest scripts/tests/).
3. {62.10} proof against staging (§5) — the gate for step 4.
4. WP-E TS retirement + nudge (single PR; bun run test full regression). Ordering matters: the Python landing path must be proven before the TS promotion is removed, or passed articles land nowhere in the gap.
5. WP-F docs/ontology in the same window as 4 (BI-22 acceptance).
6. Prod push of M1/M2 + env (COCOINDEX_WORKER_URL) after Liam ratification — schema-parity workflow confirms prod ↔ staging.

6.2 Data disposition

No retro-conversion (BI-10): the 55 url_import + 40 rss_feed legacy content_items rows and their backlinks stay read-only legacy lineage; re-landing per O4 happens at the ID-45/T7 fresh-corpus re-ingest. No backfill of source_documents.source_url (no URL rows exist there yet — RESEARCH §2.4).

6.3 OQ-75-5 item 2 — DECIDED HERE (Liam’s pre-live steer: align to the new structure; no legacy-only preservation)

• rss_feed and url_import RE-HOME from content_items CV 13 to reference_items.ingestion_source (the M1 CHECK). They are not preserved as legacy-only content_items values: post-T7 (fresh corpus, legacy external rows re-landed as references) no content_items row will carry them.
• The forward content_items CV 13 set (recorded in 13-ingest-source.md now; enforced when the planned Build-Phase-1 CHECK lands — ID-75 does NOT ship that CHECK): drops rss_feed/url_import, adds adopted_from_reference — the value adopted/derived records stamp (BI-13/BI-14), decided now so the CHECK freezes correctly (13-ingest-source.md hardening note).
• v1 reference_items writer stamps 'rss_feed'; the {42.12} manual-URL re-point stamps 'url_import'; future routes extend the CHECK by migration (BI-9, no schema change to the class).

---

7. Testing and validation

Tests verify real behaviour per docs/reference/test-philosophy.md. bun run test / python3 -m pytest scripts/tests/.

Python (new: test_url_normalise.py, test_url_validation.py, test_url_source.py, URL cases in test_cocoindex_flow_write_path.py + test_cocoindex_flow_failure_mode.py):

• Parity fixture: every url_normalisation_parity.json case passes in pytest (D-8) — BI-2/BI-8 seam.
• SSRF: each blocked class rejected with reason; public URLs pass — BI-21.
• Enumeration: predicate selects only passed=true; same URL in 2 workspaces yields 1 item with both ledger_urls; no feed_sources query issued — BI-8/BI-18.
• Write path (faithful-mount harness precedent): landing declares exactly the sd+ri pair with the BI-3/BI-4 field contract; zero ci_target interactions — BI-1. Landing twice: same PKs, counts 1 — BI-2. Changed epoch: body updated under same PK — BI-2. PDF URL: docling route, no PullMD call — BI-20. Simulated PullMD 5xx: zero rows for the item, structured log, siblings land — BI-19. SSRF item: zero rows + quality-log row — BI-21. Backlink UPDATE hits all ledger rows — BI-10.
• Grep/test guard: no code path seeds "ci:" from a URL (assert on the module source — BI-2 acceptance).

TypeScript:

• Vitest parity guard consuming the same fixture against normaliseUrl (D-8).
• pipeline.ts regression: a full mocked poll run performs zero content_items/content_item_workspaces inserts (BI-11 acceptance) and fires the nudge exactly when articlesPassed > 0 (D-3); ledger/triage/RSS behaviour unchanged (BI-12 — existing suites keep passing un-edited except promotion expectations).
• Guard tests (mcp-fixture-sync / doc-freshness / pipeline-parity): update fixtures for the new doc paths + fixture file.

Integration/proof: §5 is the BI-23 acceptance run; the schema checks for BI-7/BI-17 are reviewable from M1 (no workspace column/junction; no reference chunk table; parent contracts untouched).

Slice-2 invariants (BI-13/14/15): verified at ID-58/ID-71 implementation; this spec’s checkable obligations are the substrate facts in §4 (fresh-id discipline, RESTRICT chain, reserved CV value).

---

8. Risks and mitigations

• Memo over UrlItem (EXECUTOR-VERIFY-1, §WP-C). If the engine cannot memo-key a frozen dataclass, fall back to scalar args. Verified empirically before the write-path slice lands.
• Idle-mode gate covers both sources. app_main returns early when COCOINDEX_SOURCE_PATH is unset/missing (flow.py:2396-2413), which would also skip URL enumeration. Deployed workers always mount the corpus, so v1 accepts this; named follow-up: split the gate if a URL-only deployment ever exists.
• Retirement gap. Removing the TS promotion before the Python path is proven would strand passed articles — mitigated by the §6.1 ordering (proof gates retirement).
• Nudge reachability. Vercel → on-prem /walk failure tolerance is by design (D-3); the hourly fallback bounds latency. Monitor via the existing pipeline_runs webhook (each walk emits run rollups).
• PullMD politeness/cost on full_reprocess. A full_reprocess=true walk re-fetches every passed URL. Acceptable at v1 corpus scale (≤ hundreds); named constraint for ID-45/T7 batch planning.
• RLS posture relies on the pipeline’s owner connection. App-side writes are blocked by policy absence; the asyncpg connection (table owner) bypasses RLS — same posture as every existing pipeline target table. No service_role key in Python; unchanged.
• ingestion_quality_log CHECK widening is additive; existing consumers SELECT by flag_type and are unaffected.
• Proxy publicRoutes: no new Next.js routes ship in slice 1 — N/A (RPCs are Postgres functions; tools are ID-71’s).

---

9. Verification (Q-EX2 pre-ratification empirical check)

Run 06/06/2026 against the installed pins (requirements.txt:49 cocoindex[postgres]==1.0.3; requirements.txt:67 httpx==0.28.1):

Symbol / surface	Result
cocoindex.__version__	1.0.3 (runtime-confirmed)
cocoindex._internal.api.mount_each	PRESENT
cocoindex._internal.live_component.LiveMapView / LiveMapFeed / LiveMapSubscriber	PRESENT; minimal custom URL source (__aiter__ + watch) passes isinstance for BOTH protocols — structurally accepted, re-confirming RESEARCH §4.1
cocoindex.connectors.postgres._source.PgTableSource.__init__	PRESENT — (self, pool: asyncpg.Pool, *, table_name: str, columns: Sequence[str] | None = None, pg_schema_name: str | None = None, row_factory…, row_type…); table-enumeration only ⇒ grounds for D-1 rejection
cocoindex.connectors.postgres._source.RowFetcher.items	PRESENT — (self, key: Callable[[RowT], StableKey]) -> AsyncIterator[tuple[StableKey, RowT]]
coco.fn (decorator), coco.component_subpath	PRESENT
coco.use_context(DB_CTX) raw-pool access; update_blocking(live=False, full_reprocess=…)	In-repo production precedent (flow.py:2371; server.py:350 — bl-221 empirically verified)

No ABSENT / SIGNATURE_DRIFT / BEHAVIOUR_DRIFT findings. PullMD’s HTTP contract is in-repo and already correct (adapters.py:144-217). The single named engine-behaviour uncertainty is EXECUTOR-VERIFY-1 (memo over a custom dataclass), flagged with a fallback in §WP-C.

---

10. Open questions for Liam (ratify with this spec)

OQ-T1 — OQ-75-5(2) disposition as decided in §6.3

Re-home rss_feed/url_import to reference_items.ingestion_source; forward content_items CV 13 set drops them and adds adopted_from_reference (frozen at the Build-Phase-1 CHECK, not shipped by ID-75). Default-lean: accept — it is the direct application of the “align to the new structure” steer.

OQ-T2 — BI-19 standing cadence mechanism (D-3)

Poller-fired fire-and-forget POST /walk nudge (reusing CRON_SECRET; new COCOINDEX_WORKER_URL env on Vercel) + hourly Coolify scheduled-walk fallback. Default-lean: accept — event-driven, bounded, no idle re-walks; alternatives rejected in D-3. Needs ratification because it adds a Vercel → on-prem call path.

OQ-T3 (nit) — reference_items.ingestion_source spelling

Canonical ingestion_source (S236 ruling) vs ingest_source (matching the legacy content_items column until Q-OQR1-16 renames it). Default-lean: ingestion_source (D-11) — net-new surfaces adopt the canonical name.

---

11. Provenance pointers

• Behaviour contract: docs/specs/ID-75-pullmd-cocoindex/PRODUCT.md (BI-1..BI-23).
• Option-A mechanics + constraints 1–9: docs/specs/ID-75-pullmd-cocoindex/RESEARCH.md.
• Ratified frame + DP-1..DP-7: docs/research/s314-id75-reframe-options.md §7.
• {75.2} ratification trail: task-list Task 75 / subtask 2 journal (S316 entries).
• Precedents: 20260520225456_t6_q_a_pairs_full_schema.sql (RLS/grants), 20260520231524_t6_q_a_search_rpcs.sql (two-step RPCs), 20260526074944_id42_pullmd_provenance.sql (provenance columns), docs/ontology/32-q-a-pair.md (register shape), bl-221 one-shot walk (scripts/cocoindex_pipeline/server.py:18-31).
• Tool-discipline propagation: {75.4} PLAN briefs MUST carry gitnexus_impact-before-edit + gitnexus_detect_changes-before-commit + Python grep sweeps (RESEARCH §8; .ast-dataflow/CLAUDE.md Inv 2/3).

Changelog

Date	Session	Note
06/06/2026	S318	{75.3} TECH authored under the ratified O4/D4 frame by a fresh Planner (Q-PLANNER-2). One-to-one BI→change mapping (§4); reference-layer schema (M1) + two-step RPCs (M2); hand-rolled snapshot URL source (D-1/D-2); epoch-keyed memo for update-in-place (D-4); TS promotion code removal + walk nudge (D-5/D-3); OQ-75-5(2) decided per the pre-live alignment steer (§6.3); {62.10} landing-set contract fixed (§5); Q-EX2 verification re-run against cocoindex==1.0.3 (§9) — all PRESENT. OQ-T1..OQ-T3 raised for ratification.
06/06/2026	S318	TECH RATIFIED (Liam). Checker PASS_WITH_NOTES; three line-ref nits + M1 grant/RLS comment fixed inline pre-commit. OQ-T1 ratified (CV 13 re-homing per §6.3 accepted); OQ-T2 ratified (poller-fired fire-and-forget POST /walk nudge + hourly scheduled fallback; CRON_SECRET reuse + net-new COCOINDEX_WORKER_URL Vercel env approved as a new Vercel→on-prem call path); OQ-T3 ratified (ingestion_source, canonical S236 spelling). Spec is the implementation basis for {75.4} PLAN decomposition.