PullMD ↔ cocoindex wiring design (bl-217) — S300

PullMD ↔ cocoindex wiring design (bl-217) — S300

DRAFT — design input, not ratified. Read-only research; no code, no migrations, no ledger writes. This document scopes how a remote URL / RSS feed item reaches PullMD inside a localfs-only cocoindex pipeline, and recommends one wiring option. It does NOT design a re-ingest mechanism (that is the trivial data-movement note in §4) and it does NOT author or mutate any spec/ledger artefact. Ratification is Liam-owned.

Date: 2026-06-02 (S300). Author: Design sub-agent (Opus 4.8, 1M context) — READ-ONLY. Backlog item: bl-217 — “URL/RSS re-ingest — cocoindex remote source (not localfs pointer)” (needs_research). Supersedes lean of: pullmd-cocoindex-intent-s299.md §4.1 (the .url pointer-file “baseline”). Honours: pullmd-cocoindex-intent-s299.md §4.0 Liam design steer; §3 verified PullMD v2.x contract; §4.4 open questions.

RATIFIED (S300, Liam): Option A (cocoindex remote/custom source) confirmed as primary, with B2 as the fallback if cocoindex 1.0.3’s custom-source surface proves non-functional (empirically verify FIRST — bind_target-class risk). Land as a new Task (bl-217 stays needs_research with this doc attached until the spec chain is authored). OQ-3 decided: the original-URL provenance is a new source_documents.source_url COLUMN — NOT JSONB (avoid JSONB platform-wide); so §3.4’s “or extraction_metadata JSONB” alternative is rejected. Client grounding for the eventual spec (note the platform has moved on since these were written): kh-client-content-archive/docs/client-briefs/{Sector-Intelligence-Brief-Liam-Final, sector-intelligence-analysis, kb-hub-gap-analysis-response-s188, kb-hub-gap-analysis-liam}.md.

---

0. The remit in one line (do NOT drift)

bl-217 is NOT about designing a re-ingest mechanism. It is about wiring PullMD into the cocoindex pipeline correctly so PullMD receives a real http(s) URL instead of a local container path. Two needs are kept strictly separate throughout:

1. PullMD wiring correctness — the architecture question (§1–§3, §5). THE focus.
2. Re-ingest of existing prod feed_articles — trivial data movement (§4). One paragraph, no mechanism.

---

1. Problem

1.1 The adapter defect (file:line)

scripts/cocoindex_pipeline/adapters.py, convert_binary_to_markdown (Stage-2 outer adapter):

# adapters.py:62
suffix = file.file_path.path.suffix.lower()
...
# adapters.py:68-71  (the defect)
if suffix in _HTML_EXTENSIONS:
    # Pullmd service resolves local paths and remote URLs transparently.   # ← FALSE comment (:69)
    url = str(file.file_path.path)                                          # ← local container path (:70)
    result = await _pullmd_to_markdown(url)                                 # ← GET /api?url=<local path> (:71)

The same wrong input is repeated in the Stage-6 provenance fan-out extract_source_provenance (adapters.py:262): url = str(file.file_path.path).

PullMD is a remote-URL fetcher (GET /api?url=<http(s) URL> → text/markdown body + X-Source/X-Quality/X-Share-Id headers). It has no filesystem access to the cocoindex container and no file:// support (verified against the v2.x README, S299 §3). Handing it /cocoindex-state/corpus/test/x.html cannot work — it 4xx/5xxs or returns degraded garbage. The HTTP contract in _pullmd_to_markdown (adapters.py:123-197) is correct (the {42.1} POST /extract bug is fixed); the input value is categorically wrong. This is a genuine defect, not a regression — the local-path assumption was never correct against PullMD’s contract.

1.2 The core tension: localfs-only source vs a remote-URL extractor

The cocoindex 6-stage flow has exactly one source connector, bound at Stage 1 (flow.py:2113):

source = localfs.walk_dir(source_path, live=True, recursive=True)   # files on disk only

walk_dir().items() yields (rel_path, File) tuples; mount_each fans each File into ingest_file, which keys every downstream row on the source-relative path rel_path (flow.py:1436) via deterministic uuid5(_KH_PIPELINE_DOC_NS, "sd:"+rel_path) / "ci:"+rel_path PKs (flow.py:1470-1471; namespace _KH_PIPELINE_DOC_NS at flow.py:1187). Every adapter except HTML consumes file bytes. The HTML adapter is the only one that needs a network identity (a URL). That tension — “cocoindex walks files; PullMD needs URLs” — was never resolved, and the adapter papered over it by passing the path string into a url: parameter that happened to type-check.

1.3 The explicit two-needs split

S299 §4.0 (Liam’s steer) separates two genuinely different things the §4.1 pointer-file lean conflated:

Need	Shape	Where it lives today	This doc
(1) Ongoing URL/RSS feed ingestion	continuous, remote	the intelligence workspace SI pipeline (feed_sources → feed_articles → content_items), retired by ID-42	§3 — the wiring architecture (THE focus)
(2) Re-ingest of existing prod feed_articles	bounded, known, one-time	the prod feed_articles rows already extracted	§4 — trivial data movement, no mechanism

URL/RSS feeds are inherently remote — they belong to the intelligence workspace (and a future research workspace). They would never naturally live on localfs. So a .url-pointer-file-on-localfs is the wrong abstraction for need (1); it is considered only as a bounded fallback for need (2), if at all (§3.3, §4).

---

2. The intelligence-workspace grounding

The design must be grounded in how feeds/URLs are actually modelled, because that is where a fetched-and-extracted feed item is supposed to land.

2.1 Feeds are remote, workspace-scoped, and already have a data model

The SI feed model (verified against migrations + lib/intelligence/ + the intelligence-workspaces specs + docs/operations/si-feed-onboarding-strategy.md):

• feed_sources — the feed registry. Workspace-scoped (workspace_id FK → workspaces, ON DELETE CASCADE). source_type ∈ {rss, web, api} (CHECK at migration :1428). Carries url, polling_interval_minutes, consecutive_failures, is_active, etag, last_modified. This is the canonical home of “which remote URLs/feeds this workspace monitors”. The si-feed-onboarding-strategy adds 20 real Phew RSS/Atom feeds (gov.uk, Ofsted, Schools Week, Google News alerts) — all remote URLs.
• feed_articles — one row per polled article. Workspace-scoped. Dedup is UNIQUE (workspace_id, external_url) (idx_feed_articles_dedup, migration :4839). Carries external_url (the original source URL — the thing PullMD actually needs), external_id (GUID), raw_content/content, extraction_method (CHECK currently {rss_content, fetch, jina_reader, firecrawl, summary_fallback}, migration :3713), passed, ingested_at.
• content_items — passed articles are promoted here (pipeline.ts::storeAsContentItem). This is the same table the cocoindex pipeline writes (flow.py Stage 6). So the SI pipeline and the canonical pipeline converge on content_items — they are two producers of the same destination.
• intelligence_workspaces satellite (intelligence-workspaces PRODUCT/TECH, S243/S244): three typed columns company_profile_id, guide_id, relevance_threshold. The workspace is the scoping unit for feeds. intelligence is one of the 6 baseline core-provenance application types; a research workspace is the named future sibling (S299 §4.0).

The retiring TS surfaces (ID-42 Surfaces A + B) are all remote-URL fetchers, never local-file readers:

• Surface A — SI cascade lib/intelligence/content-extractor.ts::extractContent(item) fetches item.url; pollWebSource() (feed-poller.ts) Firecrawl-scrapes a source.url. Driven by the pollers that hold the source URL. The p0-web-si-web-feed-handler spec shows the web-source poll path: feed_sources.source_type='web' → scrape → synthesise ParsedFeedItem → dedup by URL → extract → score → promote.
• Surface B — URL-ingest lib/extraction/url.ts::extractFromUrl(url): SSRF-validated fetch(url) → Readability (HTML) or unpdf (PDF). Sole caller app/api/ingest/url/route.ts:94 (the manual “ingest a URL” API).

2.2 Where a fetched-and-extracted feed item is supposed to land

A feed item’s natural lifecycle: a remote source_url (held in feed_sources/feed_articles) → fetch+extract to markdown → land in content_items (workspace-scoped, embedded, classified) + source_documents (provenance: storage_path, content_hash, extraction_method, pullmd_share_id). The canonical pipeline already writes exactly that pair of tables (flow.py Stage 6). The only missing piece is the front door: how the remote URL enters a flow whose sole source connector reads local files. The feed item should never be modelled as a file on disk — its identity is a workspace-scoped URL, and feed_articles.external_url is its durable home.

---

3. Options for correct PullMD wiring

Objective: feed PullMD the original http(s) source URL of URL/RSS-sourced content, and land the result in content_items + source_documents with correct provenance, inside (or alongside) the cocoindex pipeline. Three options.

For all three, recall the PullMD v2.x contract (S299 §3): GET {PULLMD_SERVICE_URL}/api?url=<enc>, Authorization: Bearer <PULLMD_API_TOKEN>, raw text/markdown body, headers X-Source/X-Quality/X-Share-Id; GET /s/<share_id> round-trips the cached markdown (auto-refresh >1h; last-good snapshot if the source dies). PullMD does not parse PDFs (binary garbage at X-Quality≈0.5) — a PDF pre-route is mandatory wherever a URL can be a PDF.

Option A — a proper cocoindex remote / custom source connector

Shape. Replace (or sit beside) localfs.walk_dir with a custom cocoindex source connector that yields URLs rather than files. cocoindex is a pipeline framework, not localfs-locked: a custom source is an object exposing the same (key, value).items() contract mount_each expects. The source enumerates the workspace’s feed corpus — e.g. queries feed_sources (for live RSS/web feeds) and/or feed_articles.external_url — and yields (natural_key, UrlItem) where UrlItem carries { url, workspace_id, external_id, source_type, first_seen_at }. A new Stage-2 branch extracts via PullMD from item.url.

How PullMD gets a real URL. Directly — the source connector’s value is the URL (or an object carrying it). _pullmd_to_markdown(item.url) receives a genuine http(s) URL.

Changes in adapters.py / flow.py.

• flow.py Stage 1: add a second source binding (a RemoteUrlSource / FeedSource connector) alongside localfs.walk_dir, OR a parallel mount_each over the URL source. The localfs path stays for genuinely-local content (markdown/PDF/DOCX); the URL source handles feeds.
• adapters.py: the new branch keys on the item type (a UrlItem), not a file suffix; it calls _pullmd_to_markdown(item.url) directly. convert_binary_to_markdown’s suffix routing is bypassed for URL items (they have no file/suffix). The PDF pre-route (HEAD content-type sniff) lives in this branch: PDFs route to Docling-over-fetched-bytes, not PullMD.
• flow.py Stage 6 declare_row: storage_path becomes the URL identity (see provenance below); the rel_path-derived PK seed (uuid5(ns,"sd:"+rel_path)) is reseeded on a normalised URL (e.g. uuid5(ns,"sd:"+normalise(url))) so the same URL collapses to one row regardless of source-connector churn. The workspace is resolved from the UrlItem.workspace_id directly (no .kh-workspace-map.json prefix lookup — that manifest is a localfs-path concept).

Idempotency / memo / deterministic PK. Idempotency is cleaner than localfs here: _pullmd_to_markdown is already @coco.fn(memo=True) keyed on url (adapters.py:124), so re-enumerating an unchanged URL issues no second HTTP call. Deterministic PK on normalised URL makes re-ingest idempotent across re-enumeration. The one subtlety: cocoindex’s content-hash/file-mtime memo for the source item does not apply to a URL source — the source has to define its own change signal (e.g. feed_articles.ingested_at, or PullMD’s X-Quality/etag), or accept re-fetch-and-memo-dedup on every walk.

Provenance. storage_path = the canonical URL or pullmd://<share_id> (see §3.4); extraction_method = pullmd_<X-Source> via the existing mapping (works unchanged — it just keys off the URL item not a suffix); pullmd_share_id = X-Share-Id; original URL = the UrlItem.url, recorded durably (a source_url column on source_documents, or in extraction_metadata JSONB — net-new, §3.4).

PDF-via-URL + SSRF (§4.4). PDF pre-route mandatory (HEAD/.pdf sniff → Docling). SSRF: the source connector enumerates URLs from trusted DB rows (feed_sources/feed_articles), so the attack surface is low — but the manual URL-ingest path (Surface B) still needs SSRF validation; fold the retired lib/extraction/url-validation.ts logic into the connector or a Python equivalent.

Trade-offs. Most faithful to the data model (feeds are remote → the source is remote). Highest build cost: a net-new connector, a second source binding, a non-suffix Stage-2 branch, and a source-side change-signal. Diverges most from the current single-source flow shape — but that divergence is correct, because the feed domain genuinely is a second source.

Option B — fetch-then-handoff (a thin fetch layer feeds the existing flow)

Shape. Keep the cocoindex flow as-is (localfs-only) but separate the “get the markdown” step from the pipeline. A small fetch layer (a Python step, a TS route, or a worker) takes a URL, calls PullMD, and hands the extracted markdown to the pipeline — either by (b1) writing the markdown as a .md file into COCOINDEX_SOURCE_PATH (so the existing passthrough adapter ingests it as plain markdown), or (b2) calling a direct content_items/source_documents write with the PullMD provenance attached.

How PullMD gets a real URL. The fetch layer calls GET /api?url=<the real URL> itself, outside the suffix-routing adapter. PullMD never sees a path.

Changes in adapters.py / flow.py.

• (b1) Minimal: no adapter change for the URL path — the handoff writes a .md file and the existing _passthrough_markdown branch handles it. BUT provenance is lost unless the handoff also stamps extraction_method/pullmd_share_id — passthrough records neither (adapters.py:286). So (b1) needs a sidecar that carries provenance from the fetch layer to the Stage-6 write (a metadata file alongside the .md, parsed by a new provenance branch). This re-introduces a localfs artefact for a remote item — the abstraction mismatch S299 §4.0 warns against, just moved one hop.
• (b2) Bypasses cocoindex Stage 2-6 entirely for URLs — the fetch layer owns the embed/classify/write. That duplicates Stages 3-6 (extraction, embedding, chunking, declare_row) outside the flow, which is a large correctness liability (two write paths to content_items, drift risk).

Idempotency / memo / deterministic PK. (b1) inherits the localfs memo/PK story keyed on the written file’s rel_path — so the PK seed is the file path, not the URL, and two handoffs of the same URL produce two rows unless the fetch layer dedups first. (b2) must re-implement idempotency by hand. PullMD’s own 1h cache + X-Share-Id help but don’t substitute for the flow’s memo.

Provenance. (b1) needs a provenance sidecar; (b2) the fetch layer writes provenance directly. Original-URL recording is net-new either way.

PDF-via-URL + SSRF (§4.4). The fetch layer is the natural home for both the PDF pre-route and SSRF validation (it is the thing doing the network call) — a genuine advantage of B: the security

• content-type logic lives in one explicit place, reusing the retired Surface-A/B logic.

Trade-offs. (b2) is the cleanest conceptually (a remote fetch worker that writes the canonical tables) but duplicates the cocoindex write path — high drift risk, and it sidelines the very pipeline ID-28/31 built. (b1) keeps the pipeline but smuggles a remote item back onto localfs via a .md+sidecar, collapsing toward Option C’s mismatch. B’s real value is isolating fetch/SSRF/PDF concerns; its cost is either pipeline duplication (b2) or a localfs round-trip (b1).

Option C — .url pointer-file on localfs (Liam’s caveat: likely WRONG for live feeds)

Shape (the S299 §4.1 lean — now demoted). Stage a tiny .url pointer file per URL into COCOINDEX_SOURCE_PATH; the adapter, on a .url suffix, reads the URL from the file body and passes THAT to PullMD: target_url = (await file.read_text()).strip(); await _pullmd_to_markdown(target_url).

How PullMD gets a real URL. From the pointer file’s contents (not its path) — the one-line conceptual fix to the §1.1 defect.

Changes in adapters.py / flow.py. Smallest of all three for the adapter: add a .url branch to _URL_POINTER_EXTENSIONS, read body, call PullMD; add the same branch to extract_source_provenance. No Stage 1/3/4/5/6 changes. But the PK seed (uuid5(ns,"sd:"+rel_path)) keys on the pointer-file path, so two pointers for one URL → two rows unless reseeded on normalised URL (§4.4 Q4). Workspace still resolves via the .kh-workspace-map.json prefix manifest — which means every feed URL needs a folder placement, an awkward fit for a workspace-scoped, folder-less feed.

Idempotency / memo / deterministic PK. Memo on the URL inside _pullmd_to_markdown is fine; the file-tier memo and PK are pointer-path-keyed (wrong granularity for a URL). Re-seed PK on normalised URL to fix.

Provenance. As §3.4; original URL is in the pointer body (so trivially recordable) — the one ergonomic upside.

PDF-via-URL + SSRF (§4.4). PDF pre-route still required in the .url branch. SSRF is worse here: a pointer file’s contents are attacker-influenceable if any non-trusted path writes them, so URL validation must move into pointer generation or the adapter.

Trade-offs. Lowest adapter cost, but models a live remote feed as a file staged on disk — exactly the mismatch S299 §4.0 calls out. It forces feeds (workspace-scoped, folder-less) through a folder→workspace prefix manifest. Acceptable only as a bounded batch fallback for need (2) (§4), and even there a flat URL list is simpler than per-URL pointer files. Not recommended for the live-feed wiring (need 1).

3.4 Provenance — what to record (applies to whichever option lands)

Today storage_path = rel_path (flow.py:1499) and the row carries content_hash, filename, mime_type, file_size, extraction_method, pullmd_share_id (flow.py:1496-1513). For a URL-sourced row:

• storage_path — two candidates: (a) the canonical normalised URL, or (b) pullmd://<X-Share-Id> (a real URI: GET /s/<id> round-trips the bytes). Leaning: store the URL in storage_path (it is the human-meaningful origin) and keep pullmd_share_id as the durable re-read handle (already exists, indexed, asserted by 42.10 Inv-9). Avoid overloading storage_path with a pullmd:// scheme unless UC re-reads need it.
• extraction_method — pullmd_<X-Source> via the existing _PULLMD_X_SOURCE_METHODS mapping (adapters.py:221-223); unchanged across options.
• pullmd_share_id — X-Share-Id; unchanged.
• filename / mime_type / file_size — NOT NULL in prod (flow.py:1506-1508). For a URL there is no file; derive deterministically (filename = last URL path segment or the host; mime_type = text/html for PullMD-extracted; file_size = len(markdown) bytes). This is net-new and MUST be specced — the current NOT-NULL derivations assume a File.
• Original source URL — record durably even when storage_path already holds it, for query ergonomics: a source_url column on source_documents (net-new, queryable) OR inside extraction_metadata JSONB. Net-new; spec alongside the chosen option.

3.5 Recommendation — **Option A (cocoindex remote/custom source), with Option B’s fetch

concerns folded in**

Recommend Option A. Rationale, tied to the workspace model:

1. It matches the data model. Feeds are remote and workspace-scoped (feed_sources / feed_articles / intelligence_workspaces). A remote source connector makes the pipeline’s front door match the domain’s actual shape, instead of disguising a URL as a file (C) or splitting the write path (B2). S299 §4.0 explicitly steers here for the live-feed need.
2. Idempotency is cleaner, not harder. _pullmd_to_markdown is already memoised on url; reseeding the deterministic PK on a normalised URL gives true per-URL idempotency without the pointer-path/file-path granularity bug A/C/B1 all share.
3. No localfs round-trip for remote content. B1 and C both smuggle a remote item back onto disk (a .md+sidecar, or a .url file) and then through a folder→workspace prefix manifest — awkward for folder-less feeds and a maintenance burden. A binds the workspace directly from the item.
4. It does not duplicate the canonical write path. Unlike B2, A keeps Stages 3-6 (extraction, embedding, chunking, declare_row) as the single producer of content_items / source_documents — the convergence the platform already relies on.

Fold in Option B’s strength: put the PDF pre-route + SSRF validation in the URL-extraction branch (the one place doing the network call), reusing the logic from the retiring lib/extraction/url-validation.ts / lib/intelligence/url-validation.ts. That is the part B gets right; A should adopt it rather than scatter it.

Caveat / cost to flag for Liam: Option A is the largest build (a net-new cocoindex source connector + a second Stage-1 binding + a non-suffix Stage-2 branch + a source-side change signal), and it depends on cocoindex 1.0.3 actually accepting a custom/remote source object that satisfies the walk_dir().items() (key, value) contract mount_each consumes — this must be empirically verified before committing (the project’s hard-won lesson: cocoindex 1.0.3 APIs that “look” present are sometimes non-functional placeholders, e.g. the historical bind_target/flow["op_id"] gap). If the custom-source surface proves unusable in 1.0.3, Option B2 (a fetch worker that writes the canonical tables behind the same declare_row schema) is the fallback — accepting the write-path-duplication cost — and Option C is reserved strictly for bounded batch re-ingest (§4), never live feeds.

---

4. The trivial re-ingest note (existing prod feed_articles) — NO mechanism

The existing prod feed_articles that need re-ingesting are a bounded, known, one-time set; this is data movement, not architecture. Two plain options, pick at cutover time (Liam’s explicit steer — do not build a mechanism around either): (1) pull the relevant URLs straight from the prod feed_articles table — SELECT external_url, workspace_id FROM feed_articles WHERE … (the exact predicate is the spike’s source_url IS NOT NULL shape) — and re-ingest those URLs through whatever wiring §3 lands; or (2) export the rows and re-apply them after the preview→prod cutover (the ID-64 / ID-66 cutover sequence). Both are a one-shot list of URLs handed to the live wiring; neither warrants a bespoke component. (If a localfs path is ever wanted for this bounded case, a flat URL list is simpler than per-URL .url pointer files — Option C demoted to here, if at all.)

---

5. Landing recommendation

5.1 Fold into ID-42, or a new Task?

Recommend a NEW Task, with ID-42 closing on its actual ratified scope. Reasons:

• ID-42’s ratified scope is local-HTML→PullMD, not remote-URL ingestion (S299 §1.2; ID-42 PRODUCT §Out-of-scope explicitly punts full-corpus/feed_articles reingest to T7). Every ID-42 artefact frames “Surface C” as the cocoindex .html/.htm→PullMD path and never says where the URL comes from. The remote-source wiring is a different capability than the one ID-42 specced.
• 42.10 is blocked and unsatisfiable as written (it stages a local HTML fixture for PullMD, which PullMD cannot read). bl-217 is the design that unblocks 42.10 by changing what 42.10 proves — from “local HTML file → PullMD” to “remote URL → PullMD via the chosen wiring”. So bl-217’s output feeds ID-42’s close-out rather than living inside it.
• ID-42’s specs are stale on infrastructure. They are written against Cloud Run (manifests, cloud-run-deploy.yml, Secret Manager, sidecar topology α). Per CLAUDE.md, Cloud Run is fully retired (S298) — the pipeline now deploys to an IONOS VPS via Coolify (ID-66, .github/workflows/onprem-deploy.yml). A new Task can spec the wiring against the current on-prem reality instead of inheriting Cloud Run framing.
• The schema has already moved past ID-42 TECH. ID-42 TECH describes source_documents columns (content_fingerprint, extraction_method/pullmd_share_id as net-new) and a non-functional Stage-6 write path. Since then: ID-28.20 made Stage-6 declare_row functional; S296 (ID-64.10/64.11) renamed columns (content→ from content_text, content_hash→ from content_fingerprint) and added NOT-NULL filename/mime_type/file_size. A fresh spec chain reflects this; amending ID-42 in place would fight stale text.

Concretely: author bl-217 as a new Task ID-N (the spec chain {N.1 RESEARCH} → {N.2 PRODUCT} → {N.3 TECH} → {N.4 PLAN} per the project convention), with ID-42 retaining ownership of the PullMD deploy + adapter HTTP contract + provenance schema (done: 42.5/42.6/42.7/42.9) and ID-42’s 42.10

• Surface-A/B retirement (42.11) gated on the new Task’s wiring landing. Cross-link bl-217 ↔ ID-42 ↔ ID-45.

5.2 What the spec chain would cover

• RESEARCH — already largely done (pullmd-cocoindex-intent-s299.md + this doc). The net-new research is the empirical cocoindex-1.0.3 custom/remote source verification (does a custom source object satisfying walk_dir().items()’s (key,value) contract actually run under mount_each on the daemon-thread loop?). This gates the whole Option-A recommendation.
• PRODUCT — invariants: PullMD receives a real http(s) URL; URL-sourced rows land in content_items + source_documents with correct provenance (extraction_method=pullmd_*, pullmd_share_id round-trips, original URL recorded); PDF-via-URL routes to Docling not PullMD; workspace scoping is correct; idempotent across re-enumeration.
• TECH — the source-connector (or fetch-layer) implementation; the Stage-1/Stage-2/Stage-6 changes; the NOT-NULL filename/mime_type/file_size derivations for URL rows; the normalised-URL PK seed; the source_url/extraction_metadata original-URL column; SSRF + PDF pre-route placement; migration (if a source_documents.source_url column lands).
• PLAN — decomposition; explicitly sequence: wiring → re-prove 42.10 against a remote URL → unblock 42.11 retirement → hand to ID-45 for the bounded batch re-ingest (§4).

5.3 Relationship to the 42.10 / 62.10 fixture-shape change

S299 §4.3 already flagged this and it holds regardless of A/B/C: 42.10’s and 62.10’s fixtures must change from “stage a local HTML file and expect PullMD to extract it” to “exercise a real http(s) URL through the chosen wiring”. Under Option A, 62.10’s /stage driver stages a URL item (or seeds a feed_articles/feed_sources row the remote source enumerates), not a local HTML fixture; the Inv-9 GET <PULLMD>/s/<share_id> round-trip then works because the share id references a genuine remote fetch. This is a required adjustment to 42.10 + 62.10, owned by the new Task and coordinated with ID-62 (fixture-staging infra) and ID-66 (on-prem co-location, where PullMD is a localhost sibling).

5.4 Adjacent tracked items to reconcile

• ID-45 (T7 — Phew full-corpus reingest, spec_needed) names “URL” as a content type to reingest but has no mechanism. bl-217’s wiring is its prerequisite; the §4 batch note is ID-45’s data-movement step.
• ID-64 (pre-re-ingest readiness, in_progress) owns the staging/preview → verify → prod cutover that the §4 “export + re-apply after cutover” option references.
• ID-219 (ready) — during re-ingest the flow spams workspace_resolution ERRORs for content not under a form-mapped prefix. Option A sidesteps the prefix manifest for URL items entirely (workspace comes from the item), which is additional motivation to bind workspace from the source rather than from a localfs path.
• Deferral register — the PullMD/URL wiring gap is still not recorded in docs/themes/canonical-pipeline/reference/deferral-register.md (verified — no PullMD/URL row). The new Task should add it (or bl-217’s existence + this doc may suffice as the canonical home).

---

6. Open questions for Liam to ratify (decision-shaped)

1. Wiring option — confirm Option A (cocoindex remote/custom source) as the live-feed path? It best matches the workspace data model and keeps a single write path, but it is the largest build and depends on cocoindex 1.0.3 actually supporting a usable custom/remote source — to be empirically verified first. If that verification fails, fall back to Option B2 (a fetch worker writing the canonical tables) and reserve Option C strictly for the bounded batch re-ingest. Ratify the primary + the fallback ordering.

2. Landing — new Task ID-N (recommended) vs fold into ID-42? A new spec chain lets the wiring be specced against the current on-prem / Coolify reality and the post-S296 schema, with ID-42 closing on its actual deploy+adapter+provenance scope and 42.10/42.11 gated on the new Task. Confirm this split (and that bl-217 → new Task, not an ID-42 amendment).

3. Provenance for URL rows — storage_path = canonical URL (lean) with pullmd_share_id as the re-read handle, plus a net-new source_url column (or extraction_metadata JSONB) for the original URL? And confirm the deterministic PK reseeds on a normalised URL (not file/pointer path) so re-ingest is idempotent, and that the NOT-NULL filename/mime_type/file_size get deterministic URL-derived values. (Secondary, but blocks TECH.)

---

End. DRAFT design input — not ratified. No code, migrations, or ledger changes were made. Single recommendation: wire via a cocoindex remote/custom source connector (Option A), folding in a fetch-layer PDF-pre-route + SSRF check; land it as a NEW Task (not an ID-42 amendment), with ID-42 closing on its real scope and 42.10/42.11 gated on the new wiring. Empirically verify the cocoindex-1.0.3 custom-source surface before committing to A; fall back to B2 if it is non-functional.