Skip to content
Canonical

PullMD + cocoindex URL/HTML extraction — original intent vs current drift (S299)

PullMD + cocoindex URL/HTML extraction — original intent vs current drift (S299)

Section titled “PullMD + cocoindex URL/HTML extraction — original intent vs current drift (S299)”

Date: 2026-06-02 (S299). Author: Research sub-agent (Opus 4.8, 1M context) — READ-ONLY. No code or ledger changes. Purpose: Clarify what PullMD + cocoindex were ALWAYS intended to do for URL/HTML extraction, so the next session can correctly design the URL re-ingest mechanism (the .url pointer-file branch, or a better alternative). This is design-input only; nothing here is ratified.

Verdict (one line): remote-URL ingest = OUT OF ORIGINAL SCOPE for the cocoindex canonical pipeline, AND the adapters.py:69 local-path→PullMD behaviour is a genuine defect (it can never work — PullMD is a remote-URL-only fetcher and cannot read a local container path). The two facts compound: ID-42’s end-to-end proof (42.10) was specced to pass a LOCAL HTML file through PullMD, which is impossible by PullMD’s contract; and the real prod need — re-fetching the original source URLs of URL/RSS-sourced content — was never wired into the localfs-only pipeline at all.

0. TL;DR for the next-session designer

Section titled “0. TL;DR for the next-session designer”
  1. The canonical cocoindex pipeline only ever walks LOCAL files (localfs.walk_dir, flow.py:4, :2113; ID-28 TECH §P-2). It has no remote-URL source by design. This is original intent, not drift — ID-28/ID-31 deliberately built a local-filesystem substrate (Docling for binary, passthrough for markdown, PullMD for HTML).
  2. PullMD is a remote-URL fetcher (GET /api?url=<http(s)-url> → markdown). It does not read local files / file:// paths (confirmed against the v2.x README, §3 below). So handing it a local container path cannot work — it is not “drift that regressed”, it is a contract mismatch that was never correct.
  3. ID-42 mis-modelled HTML ingest as “local .html file → PullMD”. Every ID-42 artefact (RESEARCH/PRODUCT/TECH/PLAN) frames “Surface C” as the cocoindex .html/.htm→PullMD path, and 42.10 proves it by staging a local HTML fixture. None of the four specs ever states where the URL PullMD needs comes from. The adapter’s url = str(file.file_path.path) (adapters.py:70) plus the false comment “Pullmd service resolves local paths and remote URLs transparently” (adapters.py:69) is the crystallised defect.
  4. The real prod need is remote-URL re-ingestion — re-fetching the original source URLs of the existing URL-import + RSS/sector-intelligence corpus through PullMD. That is what the retired TS cascade did (extractFromUrl(url), extractContent(item.url)), and it is what the cocoindex path must absorb. This need is real and tracked (ID-45 full-corpus reingest), but the mechanism to feed a URL into a localfs-only flow was never designed. That is the gap the next session must close.

1. Original intent — was remote-URL re-ingestion always in scope?

Section titled “1. Original intent — was remote-URL re-ingestion always in scope?”

1.1 The canonical pipeline was specced as LOCAL-FILESYSTEM-ONLY

Section titled “1.1 The canonical pipeline was specced as LOCAL-FILESYSTEM-ONLY”

The cocoindex 6-stage flow is, by original design, a local-file pipeline:

  • flow.py:4 (Stage 1): source walk -> connectors.localfs.walk_dir(live=True, recursive=True).
  • flow.py:2113: the live source binding is localfs.walk_dir(...).
  • ID-28 TECH §P-2 (docs/specs/id-28-cocoindex-flow-scaffolding/TECH.md:100): “opens the source feed via localfs.walk_dir(source_path, live=True, recursive=True).items()”. The whole reactive App / mount_each / declare_row shape is built around walk_dir().items() yielding (relative_path, File) tuples — i.e. files on disk.
  • ID-28 TECH :344: source = localfs.walk_dir(source_path, live=True, recursive=True).
  • flow.py docstring: “Source-binding folder: env var COCOINDEX_SOURCE_PATH (T8 ships EMPTY default; T7 stages files post-T8 stable)”. The ingestion model is: stage files into a watched folder, cocoindex walks them.

There is no remote-URL source connector anywhere in the flow, and none was specced. The Stage-2 adapter table (flow.py:5-12, ID-28 TECH §P-3) is purely a per-MIME local file content converter: Docling for PDF/DOCX/XLSX bytes, passthrough for markdown/txt, and “PullMD HTTP client for HTML”. The HTML branch was always the odd one out — it is the only adapter that needs a network identity (a URL) rather than file bytes, and that tension was never resolved in the spec.

1.2 What ID-42 actually scoped — local-HTML→markdown, NOT remote-URL re-fetch

Section titled “1.2 What ID-42 actually scoped — local-HTML→markdown, NOT remote-URL re-fetch”

ID-42 (“Deploy pullmd + retire HTML extraction cascade”) is the Task that owns PullMD. Its scope, across all four artefacts, is consistently “make the cocoindex local .html/.htm → PullMD path live”:

  • RESEARCH §1 “Surface C” (docs/specs/id-42-pullmd-deploy/RESEARCH.md:115-128): “cocoindex HTML→pullmd adapter (CODE EXISTS, RUNTIME-DEAD) … the intended future HTML path”. It describes convert_binary_to_markdown routing .html/.htm → _pullmd_to_markdown(url) and flow.py wiring content_text = source.transform(...). It never asks where url comes from — the implicit assumption throughout is that the thing being ingested is a local HTML file in the cocoindex folder.
  • PRODUCT §Scope + per-surface table (PRODUCT.md:61-69): Surface C is “the path made live (deploy + adapter contract fix). It is the target of the swap.” Inv-7 (PRODUCT.md:129-137): “stage a known HTML source through the cocoindex flow against the deployed pullmd Service” — i.e. a file staged into the folder.
  • TECH §Context (TECH.md:54-72): “Make the cocoindex .html/.htm→pullmd extraction path live”. The adapter rewrite (WP-A) fixes the HTTP contract (POST /extract → GET /api?url=) but keeps passing str(file.file_path.path) as the url — the local path is never questioned.
  • PLAN 42.10 (task-list.json Task 42 / subtask 10): the end-to-end proof “stages ONE real HTML source through the cocoindex .html/.htm→pullmd path”. The testStrategy says “stages one real HTML source” — and given the pipeline is localfs-only, that “source” is a local file.

Explicitly OUT of ID-42 scope (PRODUCT §Out-of-scope, PRODUCT.md:51-59): “Full-corpus / feed_articles reingest through pullmd — belongs to T7. ID-42 proves one HTML source ingests; it does not backfill the existing corpus.” So ID-42 never claimed to handle the existing URL-sourced corpus — that was punted to the reingest Task.

1.3 Where remote-URL re-ingestion DID get acknowledged — ID-45 (full-corpus reingest)

Section titled “1.3 Where remote-URL re-ingestion DID get acknowledged — ID-45 (full-corpus reingest)”

The only place the original planning acknowledges URL-sourced content flowing into the canonical pipeline is the full-corpus reingest scope:

  • ID-31 PLAN line 305 (docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md:305): “S262 CLARIFICATION: Scope updated to all Phew content types … Full-corpus reingest is the intent: Q&A pairs, capability/methodology, sector intelligence, procurement content (markdown/PDF/DOCX/URL). The detailed decomposition … are tracked in ID-45.”

So the intent that URL-sourced content must end up in the canonical pipeline is on record (ID-45). But the mechanism — how a URL becomes something localfs.walk_dir can hand to the PullMD adapter — was never designed in ID-42, ID-31, or ID-28. ID-45 names “URL” as a content type to reingest without saying how the localfs-only flow ingests a URL. That missing mechanism is exactly the gap.

1.4 The PullMD spike always proved REMOTE-URL extraction

Section titled “1.4 The PullMD spike always proved REMOTE-URL extraction”

Critically, the spike that justified adopting PullMD only ever tested it as a remote-URL fetcher — never as a local-file reader:

  • 0.9-spike-S4 bake-off (docs/themes/canonical-pipeline/spike-findings/0.9-spike-S4-pullmd-bakeoff.md): the corpus is “source_url” values pulled from prod content_items (WHERE source_url IS NOT NULL, §2.1); the harness calls GET http://localhost:3000/api?url=<encoded> with real remote URLs (§2.2 Path A); the X-Source distribution (§3.3) shows readability/trafilatura/playwright/reddit — all network fetchers. The spike’s entire value proposition is “PullMD fetches and extracts REMOTE URLs better than KH’s cascade”. Local files never entered the spike.

This is the smoking gun: the proven, intended use of PullMD is remote-URL fetching, and the spike corpus was the existing URL-sourced prod corpus — the very content that needs re-ingesting. The drift is that this proven remote-URL usage got wired into a local-file-only pipeline by passing a local path where a URL belongs.

2. The gap — current code vs intent

Section titled “2. The gap — current code vs intent”

2.1 What the code does today (the defect)

Section titled “2.1 What the code does today (the defect)”

scripts/cocoindex_pipeline/adapters.py:

  • convert_binary_to_markdown (:56) routes .html/.htm files to PullMD.
  • :69 (the false comment): # Pullmd service resolves local paths and remote URLs transparently.
  • :70 (the defect): url = str(file.file_path.path) — passes the local container path (e.g. /cocoindex-state/corpus/test/x.html, per flow.py:1423 BUG-A note) as the url= query param.
  • :71: result = await _pullmd_to_markdown(url) → issues GET {PULLMD_SERVICE_URL}/api?url=/cocoindex-state/corpus/test/x.html.

The HTTP contract is now correct (the {42.1} POST /extract bug is fixed — _pullmd_to_markdown at :123-197 correctly does GET /api?url=, Bearer auth, raw-text body, captures X-Source/X-Quality/X-Share-Id). The Stage-6 write path is also now functional (ID-28.20 landed): flow.py:1492-1511 sd_target.declare_row writes storage_path, extraction_method (:1508) and pullmd_share_id (:1509) via the extract_source_provenance fan-out (adapters.py:242, called at flow.py:1446).

But the input is wrong. PullMD receives a filesystem path, not an HTTP URL. PullMD has no filesystem access to the cocoindex container and no file:// support (§3). The call either 4xx/5xx errors (PullMD can’t resolve the “URL”) or, worse, returns degraded garbage — it cannot extract content that lives on another container’s disk.

2.2 Is this a defect against the spec, or never-specced?

Section titled “2.2 Is this a defect against the spec, or never-specced?”

Both — and that distinction matters for the fix:

  • Defect against PullMD’s contract (genuine drift): The :69 comment asserting PullMD “resolves local paths … transparently” is factually false against the v2.x README (§3). This is the same drift-class the {42.1} RESEARCH §2.3 flagged for the (now-fixed) POST /extract bug: “a spec/code artefact cited an external API shape that was never empirically exercised against the running service.” The local-path assumption was authored from the ID-28 spec sketch (“inner-tier takes str (pullmd URL …)”, ID-28 TECH :424), where the parameter was named url: str but was only ever fed a local path string. The type annotation url: str made the wrong value type-check.
  • Never-specced (scope gap): No ID-42 artefact ever specified the remote-URL ingestion case. ID-42 was scoped to local-HTML→markdown (§1.2), so “how does a URL-sourced document get re-fetched through PullMD?” was simply out of frame. The cocoindex pipeline’s localfs-only source (§1.1) means there was never a code path that could deliver a real URL to the adapter. So 42.10 (the local-HTML proof) is unsatisfiable as written (PullMD can’t read the local file), and the actual prod requirement (re-fetch original URLs) has no design at all.

In short: the adapter behaviour is a defect (it can’t work), and the capability the prod corpus actually needs (remote-URL re-ingest) was never specced into the canonical pipeline.

2.3 What URL-handling behaviour the cocoindex path must absorb (from the retiring TS cascade)

Section titled “2.3 What URL-handling behaviour the cocoindex path must absorb (from the retiring TS cascade)”

ID-42 retires two live TS surfaces that BOTH take a remote URL and fetch it:

  • Surface A — sector-intelligence cascade (lib/intelligence/content-extractor.ts:170 extractContent(item)): fetches item.url over the network through tiers rss_content → direct-fetch → Jina (r.jina.ai/${url}) → Firecrawl → summary. Driven by RSS/web pollers (feed-poller.ts) that already hold the source URL. Writes feed_articles.extraction_method.
  • Surface B — URL-ingest (lib/extraction/url.ts:47 extractFromUrl(url)): SSRF-validated fetch(url) → Readability (HTML) or unpdf (PDF) branch. Sole caller app/api/ingest/url/route.ts:94 — the manual “ingest a URL” API.

The behaviour the cocoindex+PullMD path must absorb is therefore “given an original http(s) URL, re-fetch and extract it to markdown” — exactly PullMD’s native contract. The retired surfaces never dealt in local files; they dealt in URLs. The cocoindex path, being local-file-only, currently has nowhere to put a URL.

3. Correct PullMD usage (confirmed from spike + v2.x README)

Section titled “3. Correct PullMD usage (confirmed from spike + v2.x README)”

Re-verified 2026-06-02 against https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/README.md (v2.x) and cross-checked against the spike (0.9-spike-S4) and {42.1} RESEARCH §2.1.

PullMD is a remote-URL fetcher. It does not read local files.

  • Convert: GET /api?url=<url-encoded http(s) URL>. The url param is required and the documented input is web URLs (http/https; Reddit auto-detected). No local-file / file:// support is documented — all examples are remote URLs. (README §“URL Input Scope”: “accepts remote URLs only … no mention of local file paths or file:// URIs.”)
  • Query params: url (required), format (md|text|json, default md), nocache (bypass 1h cache), render (force|skip Playwright), frontmatter, plus Reddit params (comments, comment_depth 1-10, comment_limit, lang).
  • Response: raw text/markdown body (default format=md); headers X-Source ∈ {reddit, cloudflare, readability, readability-fallback, trafilatura, playwright}, X-Quality (0.0-1.0), X-Share-Id (8-hex permalink).
  • Share round-trip: GET /s/:id re-serves the cached markdown by share id; auto-refreshes if the cached row is > 1h old; serves the last-good snapshot if the source URL dies — “share links keep working even when the original URL dies.” This is the durable re-read handle.
  • Auth: PULLMD_AUTH_MODE ∈ {disabled (default), single-admin, multi-user}; token Authorization: Bearer pmd_<32-char-base62>. ID-42 ratified single-admin + a PULLMD_API_TOKEN secret (TECH §RATIFIED-DECISIONS 2).
  • Version: v2.x; pin :2 / :2.0.0 (:latest still tracks v1.x for back-compat).
  • PDF caveat (from spike §5.4): PullMD does NOT parse PDFs — returns binary garbage at X-Quality≈0.5. A PDF pre-route (HEAD content-type / .pdf sniff) is required before any PullMD call. In the cocoindex flow this is intrinsic (PDF→Docling, only .html/.htm→PullMD).

Conclusion: the correct way to use PullMD is to hand it the original http(s) source URL of the content. The current str(file.file_path.path) is categorically the wrong input.

Section titled “4. Recommended design for next session (design-level only)”

The objective: feed PullMD the original source URL of URL/RSS-sourced content, through a pipeline whose only source connector is localfs.walk_dir. The challenge is bridging “cocoindex walks files” with “PullMD needs URLs”.

4.0 S299 LIAM DESIGN STEER — reconsider remote-source vs localfs pointer (supersedes the §4.1 “baseline” lean)

Section titled “4.0 S299 LIAM DESIGN STEER — reconsider remote-source vs localfs pointer (supersedes the §4.1 “baseline” lean)”

Liam’s steer at S299 (load-bearing — do NOT default to §4.1): URL/RSS feeds are inherently remote and belong to the intelligence workspace (+ the future “research” workspace) — they would never naturally live on localfs. So a .url-pointer-file-on-localfs is likely the wrong abstraction for the live feed case. Separate two genuinely different needs the §4.1 recommendation conflates:

  1. Batch re-ingest of existing prod content’s original URLs — a bounded, known, one-time set (the ID-45 concern). A pointer-file / URL-list could plausibly serve this.
  2. Ongoing URL/RSS feed ingestion — continuous, remote, the data source for the intelligence workspace (and future research workspace); this is exactly what the retiring feed-poller.ts / lib/intelligence/content-extractor.ts / lib/extraction/url.ts cascade does today (§2.3). Modelling a live remote feed as files staged on disk is the mismatch.

cocoindex is a pipeline framework, not localfs-locked — it supports custom source connectors. So the next-session spec must evaluate a proper remote URL/feed source (a cocoindex remote source, or a fetch-then-handoff layer) for need (2), tied to the intelligence/research workspace data model — rather than treating the §4.1 pointer-file as the default. The pointer-file (§4.1) should be considered only for the bounded batch-reingest case (1), if at all. This decision is OPEN and Liam-owned; §4.1 below is documented as one option, not the baseline.

4.1 The .url pointer-file pattern (one option for the batch case — see §4.0)

Section titled “4.1 The .url pointer-file pattern (one option for the batch case — see §4.0)”

Stage a tiny pointer file per URL-sourced document into the cocoindex-watched corpus folder, instead of (or alongside) the extracted content. The adapter, on seeing a .url suffix, reads the URL from the file body and passes THAT to PullMD.

Concretely:

  1. New suffix branch in adapters.py. Add .url to a new _URL_POINTER_EXTENSIONS set. In convert_binary_to_markdown, for .url files: target_url = (await file.read_text()).strip() then result = await _pullmd_to_markdown(target_url). This is the ONE-line conceptual fix to the §2.1 defect: the URL comes from the file contents, not the file path. (The existing .html/.htm local-file branch can either be removed, or kept only for genuinely-local HTML files — but note PullMD still can’t read those, so local .html files would need a different extractor, e.g. a local Readability/Docling HTML path. See §4.4.)
  2. Pointer file format. Simplest: a one-line file containing the bare URL. Richer (recommended): a small structured file (e.g. .url.json or front-matter) carrying { "url": "...", "source_document_id": "...", "workspace_id": "...", "ingest_source": "url_import|rss_feed", "first_seen_at": "..." } so provenance + identity survive the round-trip and the deterministic PK seed (flow.py:1466 uuid5(ns, "sd:"+rel_path)) can be made stable per-URL rather than per-pointer-path.
  3. Pointer generation = the reingest mechanism (ID-45). A reingest step enumerates the existing URL-sourced corpus (content_items.source_url WHERE ingest_source IN ('url_import','rss_feed') — exactly the spike’s query, §2.1) and writes one pointer file per URL into COCOINDEX_SOURCE_PATH. cocoindex’s walk_dir(live=True) picks them up and the adapter re-fetches each through PullMD. For ongoing (non-backfill) URL ingest, the manual app/api/ingest/url route and the RSS/web pollers would write a pointer file instead of running the retired TS cascade.

Why a pointer file (vs a custom cocoindex source connector): it preserves the entire localfs-only substrate (idempotency via content-hash memo, deterministic uuid5 PKs, mount_each/declare_row write path) with zero changes to Stages 1/3/4/5/6 — only the Stage-2 adapter gains a branch. It also keeps the memo key sane: memoise _pullmd_to_markdown on the URL (it already does, adapters.py:124), so re-walking an unchanged pointer file issues no second HTTP call.

4.2 storage_path + provenance — what to record

Section titled “4.2 storage_path + provenance — what to record”

Today storage_path = rel_path (the local pointer-file path, flow.py:1495). For URL-sourced rows this is the wrong durable identity (it points at an ephemeral pointer file, not the content origin). Recommended:

  • storage_path should record the PullMD share-permalink URI: pullmd://<X-Share-Id> (the {42.1} RESEARCH §2.2 design — “what makes source_documents.storage_path = 'pullmd://<share_id>' a real URI rather than a magic string”). GET /s/<share_id> then round-trips to the exact extracted bytes (auto-refresh > 1h; last-good snapshot if source dies). This makes the row re-readable independent of the pointer file. Decision for the spec: whether to put the PullMD URI in storage_path or keep rel_path there and rely on the existing pullmd_share_id column (flow.py:1509) for the round-trip. Leaning: keep rel_path semantics consistent across all source types and use pullmd_share_id as the canonical re-read handle (it already exists, is indexed, and 42.10 Inv-9 already asserts it).
  • extraction_method already maps X-Source → pullmd_<source> via extract_source_provenance (adapters.py:242-287); this works unchanged for the .url branch (it routes by suffix — add .url to the HTML-equivalent branch so it awaits _pullmd_to_markdown(target_url) and maps the header).
  • Original URL provenance: record the original source URL somewhere durable (candidate: a source_url column on source_documents, OR inside the existing extraction_metadata JSONB). The pointer-file body has it; the row should keep it so the content can be re-fetched again later without the pointer file. This is net-new and should be specced alongside the .url branch.

4.3 How this satisfies 42.10 Inv-7/8/9

Section titled “4.3 How this satisfies 42.10 Inv-7/8/9”

42.10’s three invariants (testStrategy in task-list.json Task 42 subtask 10) become satisfiable for the first time, because PullMD finally receives a real URL:

  • Inv-7 (content_items.content_text non-empty markdown): stage a .url pointer containing a known-good http(s) URL → adapter reads the URL → PullMD fetches + extracts → content_text is real markdown. (Today, with a local path, PullMD returns nothing usable → Inv-7 fails by construction.)
  • Inv-8 (source_documents.extraction_method = pullmd_* matching live X-Source): the existing extract_source_provenance mapping applies to the .url branch unchanged.
  • Inv-9 (source_documents.pullmd_share_id = X-Share-Id, GET /s/<id> → 200): PullMD only mints a real X-Share-Id for a real fetched URL; the round-trip works because the share id references a genuine remote fetch, not a failed local-path lookup.

Knock-on for 62.10 (the fixture-staging harness that “wires {42.10}”, task-list.json Task 62 subtask 10): its driver currently “stages the HTML” as a local fixture file and expects PullMD to extract it. Under this design, 62.10 must stage a .url pointer fixture (a file whose body is a stable known-good test URL), not a local HTML file. The Inv-9 GET http://localhost:3000/s/<share_id> round-trip (62.10 already specs PullMD as a localhost sibling via ID-66) then works because the share id is real. This is a required adjustment to 62.10’s fixture shape, not just 42.10.

4.4 Open design questions for the spec author to resolve

Section titled “4.4 Open design questions for the spec author to resolve”
  1. Local .html files — keep, drop, or re-route? PullMD can’t read them. If the corpus ever contains genuinely-local HTML (not URL-sourced), it needs a different extractor (local Readability, or Docling’s HTML support). Decision: does v1 need local-HTML at all, or is ALL HTML URL-sourced (→ .url pointers)? The spike corpus suggests all HTML is URL-sourced. Simplest: drop the .html/.htm→PullMD local-file branch entirely and make .url the sole PullMD entry point; if local HTML appears later, route it to a local HTML extractor. This also retires the false adapters.py:69 comment cleanly.
  2. Pointer format: bare-URL line vs structured .url.json/front-matter (§4.1.2). The structured form is recommended to carry identity + original-URL provenance, but adds a parse step. Pick one and pin it.
  3. storage_path semantics for URL rows (§4.2): pullmd://<share_id> URI vs keep rel_path + rely on pullmd_share_id. Affects how UC re-reads resolve content.
  4. Deterministic PK seed for URL rows: today uuid5(ns, "sd:" + rel_path) (flow.py:1466). If two pointer files (e.g. backfill + live) reference the same URL, they should collapse to one row. Recommend seeding the PK on a normalised URL, not the pointer-file path, so re-ingest is idempotent across pointer churn. (Mirrors the existing dedup intent at flow.py:1592, :1662.)
  5. PDF-via-URL: URL-sourced PDFs must NOT hit PullMD (binary garbage, spike §5.4). The .url branch needs a content-type pre-route: HEAD the URL (or sniff .pdf), and for PDFs fetch + hand bytes to Docling instead of PullMD. This is the cocoindex equivalent of the retired Surface B PDF branch (url.ts → unpdf). Spec this into the .url adapter.
  6. SSRF / URL validation: the retired extractFromUrl had SSRF protection (lib/extraction/url-validation.ts). PullMD does its own fetching, but the pointer-file contents are attacker-influenceable if any non-trusted path writes them — decide whether URL validation moves into pointer generation or the adapter.
  7. Where the gap is recorded: this gap is NOT yet in docs/themes/canonical-pipeline/reference/deferral-register.md (verified — no PullMD/URL entry). The next session should add it, and reconcile with ID-45 (which names “URL” as a reingest content type but lacks the mechanism) and the 42.10/62.10 fixture-shape change.

5. Citations (file:line / doc section)

Section titled “5. Citations (file:line / doc section)”

Code (current state):

  • scripts/cocoindex_pipeline/adapters.py:56,69,70,71 — .html/.htm → PullMD with local path as url; the false “resolves local paths and remote URLs” comment.
  • scripts/cocoindex_pipeline/adapters.py:123-197 — _pullmd_to_markdown: correct v2.x HTTP contract (GET /api?url=, Bearer, raw-text, header capture). Contract bug from {42.1} is FIXED.
  • scripts/cocoindex_pipeline/adapters.py:242-287 — extract_source_provenance: X-Source → pullmd_<source> mapping, suffix-routed.
  • scripts/cocoindex_pipeline/flow.py:4,2113 — Stage-1 source = localfs.walk_dir (local files only).
  • scripts/cocoindex_pipeline/flow.py:1432,1466,1492-1511 — rel_path storage_path, uuid5 PK seed, Stage-6 declare_row (now functional; writes extraction_method, pullmd_share_id).
  • lib/extraction/url.ts:47 (extractFromUrl(url)), lib/intelligence/content-extractor.ts:170 (extractContent(item.url)) — retiring TS surfaces that take REMOTE URLs.

Specs (original intent):

  • docs/specs/id-42-pullmd-deploy/RESEARCH.md:115-128 (Surface C), §2.1-2.3 (contract + drift), §5 (schema).
  • docs/specs/id-42-pullmd-deploy/PRODUCT.md:51-69 (scope / per-surface), :129-157 (Inv-7/8/9).
  • docs/specs/id-42-pullmd-deploy/TECH.md:54-72 (context), §RATIFIED-DECISIONS, §WP-A/WP-E.
  • docs/specs/id-42-pullmd-deploy/PLAN.md — 42.5-42.13 decomposition; 42.10 the proof.
  • docs/specs/id-28-cocoindex-flow-scaffolding/TECH.md:100,344,421,424 — localfs.walk_dir reactive shape; inner-tier str (pullmd URL …) naming.
  • docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md:305 — ID-45 full-corpus reingest names “URL” as a content type (intent on record; mechanism absent).
  • task-list.json Task 42 subtask 10 (42.10 proof, BLOCKED on ID-28.20 — now landed) + Task 62 subtask 10 (62.10 fixture harness “wires {42.10}”).

Spike + external:

  • docs/themes/canonical-pipeline/spike-findings/0.9-spike-S4-pullmd-bakeoff.md §2.1-2.3, §3.3, §5.4 — PullMD proven as REMOTE-URL fetcher; corpus = prod source_urls; PDF caveat.
  • https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/README.md (v2.x, fetched 2026-06-02) — remote-URL-only; GET /api?url=; /s/:id; single-admin Bearer; no local-file support.

End. Design-level only — the next session owns specifying the .url pointer branch (or alternative), the pointer format, storage_path/provenance for URL rows, the PDF-via-URL pre-route, and the 42.10/62.10 fixture-shape change, then ratifying with Liam.