ID-75 RESEARCH — Wire PullMD into cocoindex correctly (remote URL/feed source)

ID-75 RESEARCH — Wire PullMD into cocoindex correctly (remote URL/feed source)

Artefact: {75.1} RESEARCH — the first spec-chain artefact for Task ID-75. Status: Research input for {75.2} PRODUCT. Not a behaviour spec; no code, migrations, or ledger writes were made authoring it. Date: 04/06/2026. Author: Task Planner (Opus 4.8, 1M context) — fresh dispatch, read-only. Backlog item: bl-217 — “URL/RSS re-ingest — cocoindex remote source (not localfs pointer)”. Ratified design input (READ FIRST): docs/themes/canonical-pipeline/reference/pullmd-wiring-design-s300.md (Liam, S300).

This RESEARCH consolidates the S300 ratified design, then adds the one thing the design explicitly deferred to spec time and made load-bearing: the empirical cocoindex 1.0.3 custom/remote-source viability verdict. That verdict (§4) decides Option A vs the B2 fallback. It is A viable — verified against the installed cocoindex[postgres]==1.0.3 package source, not assumed.

---

1. Problem statement

The cocoindex Stage-2 adapter hands PullMD — a remote-URL fetcher — a local container path, which can never work. PullMD’s contract is GET {PULLMD_SERVICE_URL}/api?url=<http(s) URL> → text/markdown body + X-Source/X-Quality/X-Share-Id headers; it has no filesystem access to the cocoindex container and no file:// support (verified against the v2.x README, S299 §3). Handing it /cocoindex-state/corpus/test/x.html cannot resolve.

The root tension: the cocoindex 6-stage flow has exactly one source connector — a local filesystem walk (localfs.walk_dir) — but URL/RSS feeds are inherently remote. Feeds belong to the intelligence workspace (and a future research workspace) and would never naturally live on local disk. The adapter papered over the tension by passing a path string into a url: parameter that merely type-checked.

This Task wires PullMD so it receives a real http(s) source URL of URL/RSS-sourced content, and lands the result in content_items + source_documents with correct provenance, inside (or alongside) the cocoindex pipeline. It is NOT a re-ingest mechanism (see §7 OUT OF SCOPE).

1.1 Code-intelligence orientation (per .gitnexus/CLAUDE.md “Always Do”)

gitnexus_query({query: "cocoindex source connector localfs walk_dir mount_each ingest_file", repo: "knowledge-hub"}) surfaced the canonical ingest flow symbols verbatim:

• Function:scripts/cocoindex_pipeline/flow.py:ingest_file (flow.py:1389-1557) — the per-source-item component mounted by mount_each.
• Function:scripts/cocoindex_pipeline/flow.py:_to_source_relative (flow.py:1366-1385) — derives the rel_path that seeds the deterministic per-document uuid5 PKs.
• Function:scripts/cocoindex_pipeline/server.py:_stage_handler (server.py:196-290) — the {62.10} /stage fixture driver (writes fixture BYTES into the localfs corpus dir).

gitnexus_context({name: "ingest_file", file_path: "scripts/cocoindex_pipeline/flow.py"}) recorded the call-graph context:

• Verdict / blast radius: the only incoming caller is bound_ingest_file (the named closure inside app_main, flow.py:2358); the only outgoing call is _ingest_file_body (flow.py:1561). ingest_file participates in no indexed cross-module process (it is invoked by the cocoindex engine via mount_each, an edge gitnexus does not trace through the framework). Direct-caller count = 1; the eventual implementation will add a SECOND source binding and (under Option A) a non-suffix Stage-2 branch rather than rewrite ingest_file’s body — so the per-symbol blast radius for the existing localfs path is LOW, but the Stage-1 app_main binding site (flow.py:2307-2387) is the genuine change locus and MUST carry full tool-discipline (see §8).

ast-dataflow does not cover Python; per .ast-dataflow/CLAUDE.md the Python pipeline was grounded with gitnexus + a targeted grep sweep (file:line citations throughout §2).

---

2. Current-state evidence (file:line, gitnexus-grounded)

2.1 The adapter defect

scripts/cocoindex_pipeline/adapters.py, convert_binary_to_markdown (Stage-2 outer adapter), HTML branch:

# adapters.py:74-77 (the defect)
if suffix in _HTML_EXTENSIONS:
    # Pullmd service resolves local paths and remote URLs transparently.   # ← FALSE comment (:75)
    url = str(file.file_path.path)                                          # ← local container path (:76)
    result = await _pullmd_to_markdown(url)                                 # ← GET /api?url=<local path> (:77)

The same wrong input is repeated in the Stage-6 provenance fan-out extract_source_provenance (adapters.py:283): url = str(file.file_path.path).

The HTTP contract in _pullmd_to_markdown (adapters.py:144-218) is correct (the {42.1} POST/endpoint bug is fixed; it issues GET {PULLMD_SERVICE_URL}/api?url=<enc> with Authorization: Bearer, parses X-Source/X-Quality/X-Share-Id). It is the input value that is categorically wrong. This is a genuine defect, not a regression — the local-path assumption was never correct against PullMD’s contract.

2.2 The single-source flow shape

The flow binds exactly one source at Stage 1 (flow.py:2307-2311):

source = localfs.walk_dir(source_path, live=True, recursive=True)   # files on disk only

source.items() yields (relative_path_str, File) pairs into coco.mount_each (flow.py:2370-2382), which fans each item into the bound_ingest_file closure (flow.py:2358) → ingest_file (flow.py:1389). Every downstream row is keyed on the source-relative path rel_path via deterministic uuid5(_KH_PIPELINE_DOC_NS, "sd:"+rel_path) / "ci:"+rel_path PKs (flow.py:1357 namespace; the seed is rel_path). Every adapter except HTML consumes file bytes; the HTML adapter alone needs a network identity (a URL).

2.3 The Stage-6 source_documents write site

sd_target.declare_row(...) (flow.py:1667-1684) writes:

• storage_path = rel_path (flow.py:1670) — currently the localfs-relative path.
• content_hash = content_fingerprint (flow.py:1673; renamed from content_fingerprint by ID-64.10/64.11, S296).
• filename = file.file_path.path.name, mime_type = _resolve_source_mime(...), file_size = await file.size() (flow.py:1677-1679) — all three NOT NULL in prod (SOURCE_DOCUMENTS_SCHEMA, flow.py:1062-1064), all three derived from a File.
• extraction_method = provenance.extraction_method (pullmd_<X-Source> or docling), pullmd_share_id = provenance.pullmd_share_id (flow.py:1683-1684).

The migration 20260526074944_id42_pullmd_provenance.sql added the two pullmd-provenance columns (pullmd_share_id, extraction_method, both text NULL).

2.4 The live source_documents table — source_url is genuinely net-new

The prod table (20260416102457_pre_squash_reconciliation.sql:4028) carries filename, original_filename, mime_type, file_size, content_hash, storage_path, extraction_metadata jsonb, workspace_id, plus the two ID-42 provenance columns. It has no source_url column. The only source_url columns in the schema are on content_items (a different table) and ingestion_quality_log — confirmed by migration grep. So OQ-3’s source_documents.source_url is net-new, and the JSONB alternative (extraction_metadata) physically exists but is rejected per the S300 ratification (avoid JSONB platform-wide).

2.5 The feed_articles data model (where the real URL lives)

feed_articles (pre_squash_reconciliation.sql:3696) carries external_url text NOT NULL — the original source URL PullMD actually needs — and dedups on UNIQUE (workspace_id, external_url) (idx_feed_articles_dedup, migration :4839; plus a plain index idx_feed_articles_external_url, :4847). feed_sources is the workspace- scoped feed registry (source_type ∈ {rss, web, api}). The si-feed-onboarding strategy (docs/operations/si-feed-onboarding-strategy.md) onboards ~20 real remote Phew RSS/Atom feeds (gov.uk, Ofsted, Schools Week, Google News alerts). Feeds are remote, workspace- scoped, and already modelled — they would never naturally be files on disk.

2.6 The {62.10} /stage fixture driver (why the fixture shape must change)

_stage_handler (server.py:196-290) accepts a multipart/form-data body with a file part (raw BYTES) + destPath and writes the bytes into the localfs COCOINDEX_SOURCE_PATH corpus dir for the co-located walk_dir(live=True) watcher to pick up. It stages a file — exactly the local-HTML-fixture shape that {42.10} depends on and that PullMD cannot read. This is the front-door of the fixture-shape problem in §6.

---

3. The option space (A vs B2 vs .url-fallback)

Three options, per S300 §3 (full trade-off analysis there). Summarised:

Option	Shape	How PullMD gets a real URL	Verdict
A — cocoindex remote/custom source	A custom source connector yields workspace-scoped UrlItems instead of files; a second Stage-1 binding sits beside localfs.walk_dir; a non-suffix Stage-2 branch calls _pullmd_to_markdown(item.url).	Directly — the source value IS the URL.	PRIMARY (ratified). Matches the data model; single write path; cleanest idempotency. Empirically verified viable (§4).
B2 — fetch worker writes canonical tables	A separate fetch worker calls PullMD and writes content_items/source_documents behind the same declare_row schema, bypassing cocoindex Stages 2-6 for URLs.	The fetch worker calls GET /api?url=<real URL> itself.	FALLBACK only — iff A proves non-functional. Duplicates the canonical write path (drift risk: two producers of content_items).
C — .url pointer file on localfs	Stage a one-line .url pointer per URL into the corpus; a .url-suffix branch reads the URL from the file body.	From the pointer file’s contents.	Bounded-batch fallback ONLY (re-ingest, §7). Models a live remote feed as a file on disk — the abstraction mismatch S300 §4.0 forbids; forces folder-less feeds through a folder→workspace prefix manifest. NOT for live feeds.

The S300 recommendation: Option A, folding in Option B’s strength — put the PDF pre-route + SSRF validation in the URL-extraction branch (the one place doing the network call), reusing the retired lib/extraction/url-validation.ts / lib/intelligence/url-validation.ts logic. The whole recommendation is gated on the §4 empirical finding.

---

4. Empirical cocoindex 1.0.3 custom/remote-source viability (the load-bearing finding)

Why this gate exists (Q-EX2 / S252 cocoindex precedent): cocoindex 1.0.3 APIs that “look” present have historically been non-functional placeholders (the bind_target / flow["op_id"] gap). The S300 design made the Option-A recommendation explicitly conditional on empirically verifying that cocoindex 1.0.3 accepts a usable custom/remote source object. This section is that verification — run against the installed pin, not a survey doc.

4.1 Verification block

• Date: 04/06/2026.
• Pinned version: cocoindex[postgres]==1.0.3 (requirements.txt:49); installed at ~/Library/Python/3.14/lib/python/site-packages/cocoindex, cocoindex.__version__ == "1.0.3" (confirmed at runtime).
• Symbols / surfaces checked:

Symbol / surface	Source of truth	Result
cocoindex._internal.api.mount_each	_internal/api.py:427-520	PRESENT — imports and is a real coroutine accepting (fn, items, *args) where items is _ItemsType[T] (a keyed (key, value) iterable OR a LiveMapFeed/LiveMapView).
cocoindex._internal.live_component.LiveMapView	live_component.py:188-201	PRESENT — @runtime_checkable Protocol[_K,_V]: __aiter__() -> AsyncIterator[tuple[_K,_V]] (scannable) + watch(subscriber) (watchable).
cocoindex._internal.live_component.LiveMapFeed	live_component.py:172-184	PRESENT — @runtime_checkable Protocol[_K,_V]: watch(subscriber) -> None (watch-only, e.g. Kafka).
cocoindex._internal.live_component.LiveMapSubscriber	live_component.py:215-263	PRESENT — update_all / update(key,value) / delete(key) / mark_ready; wraps LiveComponentOperator.
Built-in remote source connectors	cocoindex/connectors/{amazon_s3,google_drive,kafka,oci_object_storage,postgres,...}	PRESENT — 14 connectors ship; 7+ are remote/cloud. None localfs-locked.
cocoindex.connectors.postgres._source.PgTableSource / RowFetcher.items(key=...)	connectors/postgres/_source.py:62-230	PRESENT — a query-driven source: RowFetcher.items(key: Callable[[Row], StableKey]) -> AsyncIterator[tuple[StableKey, Row]]. Enumerates Postgres rows into (key, row) — almost exactly the feed_articles.external_url enumeration Option A needs, off the shelf.

• Import-and-call check (run against the pin):

from cocoindex._internal.api import mount_each
from cocoindex._internal.live_component import LiveMapView, LiveMapFeed
class _UrlItem:
    def __init__(self, url): self.url = url
class _UrlSource:                       # a minimal custom source
    def __aiter__(self): return self._gen()
    async def _gen(self):
        yield ('k1', _UrlItem('https://example.com'))
    async def watch(self, subscriber): ...
src = _UrlSource()
assert isinstance(src, LiveMapView)     # → True
assert isinstance(src, LiveMapFeed)     # → True

Result: PRESENT + STRUCTURALLY ACCEPTED. Both protocols are runtime_checkable, and a minimal hand-rolled URL source object passes isinstance(src, LiveMapView) / isinstance(src, LiveMapFeed) — so the cocoindex engine accepts a custom source structurally, with no engine-side registration, no bind_* placeholder, and no Rust-side gap. The KH localfs source (connectors/localfs/_source.py:68-263) is itself just a plain DirWalker class exposing items() (returning a LiveMapView via _LiveDirItems) + a watch(subscriber: LiveMapSubscriber) method — i.e. the reference implementation a custom URL source would mirror.

4.2 Verdict — OPTION A IS VIABLE

cocoindex 1.0.3’s custom/remote-source surface is real, public, protocol-based, and functional — not a bind_target-class placeholder. A custom URL source is an ordinary Python class implementing __aiter__() (yield (key, UrlItem) from a scannable snapshot, e.g. SELECT external_url, workspace_id FROM feed_articles …) and optionally watch(subscriber) (for live RSS polling). It plugs into the existing mount_each call with no engine modification. The built-in PgTableSource further means much of the “enumerate feed_articles into keyed items” work may be reusable rather than hand-rolled.

Therefore: PRODUCT and TECH should proceed on Option A. The B2 fallback condition (S300: “iff cocoindex 1.0.3’s custom-source surface proves non-functional”) is NOT met — B2 is not triggered. No escalation; the spec chain continues on A. (Residual risk that belongs to {75.3} TECH, not here: the live-watch watch() change-signal semantics under the on-prem _LoopRunner daemon-thread loop — see §6 risk note — and whether a snapshot-only __aiter__ source without watch() is sufficient for the first slice.)

---

5. Decided constraints (carry into {75.2} PRODUCT)

1. OQ-3 (RATIFIED) — original-URL provenance is a NEW source_documents.source_url TEXT COLUMN, not JSONB. The extraction_metadata jsonb alternative is rejected (avoid JSONB platform-wide). Column is net-new (§2.4), nullable (localfs rows have no source URL), and queryable. A migration lands it (CLI-only DDL per CLAUDE.md; staging-first; SET search_path not needed for a plain ADD COLUMN).
2. storage_path for URL rows = the canonical normalised URL (the human-meaningful origin), with pullmd_share_id retained as the durable re-read handle (GET /s/<id> round-trips). Do NOT overload storage_path with a pullmd:// scheme unless a re-read use-case demands it.
3. Deterministic PK reseeds on a NORMALISED URL, not the file/pointer path: uuid5(_KH_PIPELINE_DOC_NS, "sd:"+normalise(url)) / "ci:"+normalise(url), so the same URL collapses to one row across re-enumeration. (PRODUCT states the idempotency invariant; TECH owns the normalisation rule.)
4. NOT-NULL filename/mime_type/file_size get deterministic URL-derived values for URL rows (no File exists): e.g. filename = last URL path segment or host; mime_type = text/html for PullMD-extracted; file_size = len(markdown) bytes. This is net-new and MUST be specced — current derivations assume a File.
5. PDF-via-URL pre-route is mandatory wherever a URL can be a PDF: HEAD/.pdf content-type sniff → Docling-over-fetched-bytes, NOT PullMD (PullMD returns binary garbage at X-Quality≈0.5 for PDFs).
6. SSRF validation lives in the URL-extraction branch (the one place doing the network call), reusing the retired Surface-A/B url-validation logic ported to Python. The DB- enumerated feed corpus is low-risk, but the manual URL-ingest path still needs it.
7. Workspace scoping comes from the item (UrlItem.workspace_id), NOT the .kh-workspace-map.json folder→workspace prefix manifest (a localfs-path concept). This also sidesteps the ID-219 workspace_resolution ERROR spam for non-prefix content.
8. extraction_method mapping is unchanged — pullmd_<X-Source> via the existing _PULLMD_X_SOURCE_METHODS frozenset (adapters.py:242-244); it keys off the URL item, not a suffix.
9. KH quality bars (inherited by every downstream subtask): UK English; no silent Supabase failures (Python pipeline uses structured-log-then-raise per the existing _pullmd_to_markdown pattern); CLI-only DDL with staging-first db push; tests verify real behaviour per docs/reference/test-philosophy.md; bun run test (TS guards) / python3 -m pytest scripts/tests/ (Python).

---

6. Dependency map (the 42.10 / 62.10 fixture-shape change)

ID-75 unblocks ID-42.10 and requires a 42.10 + 62.10 fixture-shape change (S300 §5.3, holds regardless of A/B/C):

• ID-42.10 (blocked; deps [7, 9]; “End-to-end HTML ingest proof against the deployed pullmd Service”) currently stages one local HTML source and asserts a pullmd-extracted content_items body + pullmd_* extraction_method + round-tripping pullmd_share_id. It is unsatisfiable as written — PullMD cannot read the staged local file. ID-75’s wiring changes what 42.10 proves: from “local HTML file → PullMD” to “remote URL → PullMD via the chosen wiring”. So 42.10 must be re-shaped (owned by ID-75, coordinated with ID-42’s close-out) to exercise a real http(s) URL.
• ID-62.10 / the /stage driver (server.py:196-290, §2.6): under Option A the /stage driver must stage a URL item (seed a feed_articles/feed_sources row the remote source enumerates, OR yield a URL through the source), not a local HTML fixture. The Inv-9 GET <PULLMD>/s/<share_id> round-trip then works because the share-id references a genuine remote fetch. Coordinated with ID-62 (fixture-staging infra) and ID-66 (on-prem co-location, where PullMD is a localhost sibling).
• Risk note for TECH (carry forward, do NOT resolve here): the per-item component runs on cocoindex 1.0.3’s _LoopRunner daemon thread (a separate event loop that does NOT copy the binder task’s ContextVar snapshot — the {66.19} lesson, flow.py:2318-2356). A custom URL source’s watch()/__aiter__ and any context binding must respect that boundary; functools.partial is incompatible with mount_each (needs real __name__/__qualname__). The second source binding must use a NAMED closure exactly as bound_ingest_file does.

Adjacent tracked items (reconcile in PLAN, not here): ID-45 (T7 full-corpus reingest; ID-75 is its prerequisite; the §7 batch note is ID-45’s data-movement step); ID-64 (pre-re-ingest readiness; owns the cutover the §7 “export + re-apply” option references); ID-219 (workspace_resolution ERROR spam — Option A sidesteps the prefix manifest for URL items).

---

7. Out of scope

• Existing prod feed re-ingest is OUT OF SCOPE (trivial data movement, not architecture). At cutover, either pull feed_articles.external_url straight from prod (SELECT external_url, workspace_id FROM feed_articles WHERE …) and re-ingest those URLs through whatever wiring ID-75 lands, OR export + re-apply after the ID-64/ID-66 cutover. Both are a one-shot URL list handed to the live wiring; neither warrants a bespoke mechanism. Owned by ID-45.
• Designing Option B2 or Option C in detail — A is viable (§4), so B2 is not built and C is reserved strictly for the bounded-batch re-ingest above (and even there a flat URL list beats per-URL .url pointer files).
• The PullMD HTTP contract itself — already correct (adapters.py:144-218, ID-42).

---

8. Tool-discipline propagation note for downstream subtasks (Inv 2/3)

Per .ast-dataflow/CLAUDE.md propagation discipline: the eventual {75.4} PLAN implementation-subtask briefs MUST carry the code-intelligence tool-discipline pattern, because the implementation touches symbols (the Stage-1 app_main source binding, the Stage-2 adapter branch, the Stage-6 declare_row). Specifically, every code-touching subtask brief must instruct the Executor to:

• run gitnexus_impact({target: <symbol>, direction: "upstream"}) before editing any symbol, and report the blast radius;
• run gitnexus_detect_changes() before committing, to confirm only expected symbols changed;
• for the Python pipeline, supplement with a grep sweep (ast-dataflow does not cover Python).

This is a brief-composition requirement (Inv 2 = Planner embeds it; Inv 3 = Executor applies it). A subtask dispatched without it on a code-touching brief is a brief defect — the PLAN author must not omit it.

---

9. Open questions to resolve in {75.2} PRODUCT

1. First-slice source mode — snapshot-only __aiter__, or full live watch()? §4 confirmed both LiveMapView (scannable + watchable) and LiveMapFeed (watch-only) are accepted. Is the first behavioural slice a snapshot enumeration of feed_articles.external_url (re-walk on each flow run, lean on PullMD’s memo + the normalised-URL PK for idempotency), deferring continuous RSS watch()-polling to a later slice — or must live polling be in-scope from invariant one? This shapes how many PRODUCT invariants there are and whether the source defines its own change-signal now.

2. Source enumeration target — feed_articles rows, feed_sources registry, or both? feed_articles.external_url holds already-discovered article URLs (one-per-article, the dedup unit); feed_sources.url holds feed roots needing a poll-and-discover step. Which does the source enumerate for the first invariant — and is the built-in PgTableSource (§4.1) reused, or a hand-rolled source? (Decision-shaped: it sets the source’s contract.)

3. storage_path content + the original-URL surfacing contract. Confirm storage_path = canonical normalised URL (constraint 2) AND that a URL row’s identity in source_documents is observably the URL (via source_url column AND/OR storage_path) — what exactly must a downstream consumer be able to query to recover “this row came from URL X”? (PRODUCT must make this a testable invariant; it drives the 42.10 re-shape.)

4. PDF-via-URL behaviour — silent route, or recorded? When a feed URL resolves to a PDF (HEAD sniff), the content routes to Docling not PullMD (constraint 5). Is the PDF-route a silent internal branch, or must the row record that it was PDF-routed (e.g. extraction_method = "docling" with the URL still in source_url)? This is a testable provenance invariant for PRODUCT.

5. SSRF rejection behaviour — what does the pipeline DO with a rejected URL? Skip-and- log, fail the item, or quarantine? The DB-enumerated corpus is trusted, but the invariant must state the observable outcome when a URL fails validation (no silent drop). PRODUCT defines the behaviour; TECH ports the validation logic.

---

End {75.1} RESEARCH. Empirical verdict: Option A is VIABLE against cocoindex==1.0.3 (custom/remote source is a real, runtime-checkable, protocol-based surface — not a bind_target-class placeholder; the built-in PgTableSource is a ready analogue for feed_articles enumeration). The spec chain proceeds on Option A; the B2 fallback condition is not met. No code, migrations, or ledger writes were made.