pullmd evaluation — URL/RSS extraction + sourcedocuments lineage fit

pullmd evaluation — URL/RSS extraction + source_documents lineage fit

Date: 2026-05-08 Branch: content-items-investigation (worktree) Author: Claude (Opus 4.7, 1M context) Subject: AeternaLabsHQ/pullmd — Self-hosted URL-to-Markdown service (PWA + REST + MCP + Claude Code skill) Status: Evaluation only — re-use / integrate / hybrid framing per parent feedback in graphify-evaluation-feedback.md and 07-synthesis-feedback.md. No production integration changes proposed without explicit decision.

1. Executive summary

pullmd is a Node.js + Python self-hosted service whose entire job is “URL → cached, refreshable, share-able markdown.” It cascades through Cloudflare’s native markdown short-circuit → Mozilla Readability + Trafilatura on static HTML → Playwright headless Chromium for JS-heavy pages. Each conversion gets an 8-hex share id; /s/:id is a live endpoint that auto-refreshes when the cached row is older than 1h, and falls back to the last good snapshot if the source dies. Reddit threads are first-class citizens with comment-tree extraction. PWA + REST + MCP server (Streamable-HTTP, 3 tools: read_url, get_share, list_recent) + a downloadable Claude Code skill. AGPL v3, 80.1 MB Docker image (multi-arch amd64/arm64), version 2.0 stable.

The fit with KH is strong but bounded. pullmd cleanly dominates the URL-extraction half of three KH paths (P4 TS URL ingest, P9 RSS Tier 2-3, the Tier 2-3 fallback inside lib/intelligence/content-extractor.ts). Its share-link contract directly enables three things KH has been wrestling with in 0.7.4: (a) stable URL identity across re-fetches so the version-chain on source_documents can hang off it, (b) the parent_id re-upload chain extending naturally to URL re-fetches (pullmd’s “snapshot at this share id” is exactly what 0.7.4-Q7 asks for), and (c) a deterministic content-hash (the share-id contract demands it) that makes URL detect_reupload semantics work where today’s URL ingest has none. None of which require changing how content_items is structured.

The fit is bounded because pullmd does NOT do PDFs (KH P4’s unpdf path stays), does NOT do binary uploads (P7 stays), does NOT do RSS discovery / relevance scoring / 4-tier orchestration (P9’s higher-level work is orthogonal), and does NOT do the entity classification / dedup / chunking / embedding side of ingest (Layer 4 of the canonical pipeline stays exactly as-is). So pullmd is a shape-adapter replacement for the URL extractor inside Layer 3 of the canonical pipeline, not a pipeline-wide replacement.

The 07-synthesis-feedback §3.2 question — “Do we need to store binaries?” — pullmd answers half of it. For URL inputs, pullmd’s share-id IS the binary-equivalent provenance: the 8-hex id is stable, the markdown is cached for 90 days post-write, and the URL+timestamp+content-hash chain gives the audit trail. So URL source_documents rows in v1 can be metadata-only with no bucket (the field user asked for in §3.2): storage_path becomes a sentinel pullmd://${share_id} with the content live in pullmd’s SQLite. That handles the “users already store docs in SharePoint, don’t want a second repository” concern for URLs entirely. It does NOT handle the same concern for binary uploads (PDFs, DOCX) where pullmd has no role — see §6 for the binary story.

Recommendation: HYBRID. Re-use pullmd as the URL→markdown shape adapter for P4 and pullmd as Tier 2-3 of P9, defer the MCP-from-MCP question, defer Reddit support sequencing decision, keep all Layer 4 work (dedup/classify/embed/chunk) on KH side. Run pullmd as a single shared instance per-deployment (one Vercel KH ↔ one pullmd container), not per-client. Effort: ~1-2 weeks for P4 swap + ~1 week for P9 Tier-2/2.5 swap + ~half-week for source_documents pullmd-share-id schema bridge.

Overall confidence: 84%. The 16% drag is: (a) pullmd’s “Cloudflare native markdown” short-circuit is undocumented in detail (could be a paid Cloudflare AI feature that adds ops cost), (b) AGPL v3 means a self-hosted deployment is fine but if KH ever forks pullmd code into the platform proper, KH must also be AGPL-licensed downstream — this needs Liam call, (c) pullmd’s quality scoring is described but not evaluated against KH’s existing extractor on a real corpus — would need a 50-URL bake-off before commit, (d) batch ingestion has no native API (single-URL only) so high-volume re-ingest may need throttling, (e) pullmd is at v2.0 (active but young), single-vendor risk.

2. What pullmd is — single-paragraph

pullmd is a self-hosted Node.js Express service (~80 MB Docker image, AGPL v3) whose core artefact is a GET /api?url=… endpoint that returns the URL’s content as clean markdown, an X-Share-Id 8-hex permalink in the response header, an X-Source header indicating which extractor won (reddit | cloudflare | readability | readability-fallback | trafilatura | playwright), and an X-Quality confidence score (0.0-1.0). The service is composed of three Docker containers via compose: the Node app (Express + better-sqlite3 cache), a Trafilatura Python sidecar (FastAPI), and an optional Playwright Python sidecar (FastAPI + Chromium adds ~3.7 GB). The same conversions are accessible via PWA (vanilla JS, dark/paper themes, service worker), REST API, MCP Streamable-HTTP server (3 tools: read_url, get_share, list_recent), and a downloadable Claude Code skill. Cache rows persist 90 days post-last-write; /s/:id is a live endpoint that re-fetches if the row is older than 1h and serves the last good snapshot if the source becomes unreachable. Three auth modes (disabled / single-admin / multi-user); session cookies + API keys + bearer-token compat; share links remain unauthenticated by design. Reddit threads receive special handling (full comment trees, configurable depth 1-10, optional Reddit OAuth). No PDF support. No batch endpoint. Single-URL only.

3. KH’s current URL extraction infrastructure — what pullmd would replace or augment

3.1 Inventory

KH’s URL extraction lives in three places, with significant duplication:

Path	Entry point	Extractor stack	Lines	Hits prod?
P4 TS URL ingest (Layer 3 URL adapter)	`app/api/ingest/url/route.ts` calls `extractFromUrl()` from `lib/extraction/url.ts`	(1) SSRF validation `lib/extraction/url-validation.ts`; (2) fetch w/ 15s timeout, 20MB limit, redirect-follow + re-validate; (3) PDF branch → unpdf via `lib/extraction/pdf.ts`; (4) HTML branch → JSDOM + Mozilla Readability + Turndown via `lib/extraction/html.ts` + `lib/extraction/og-metadata.ts` regex	~300 LOC across 8 files	YES — 55 prod rows
P9 RSS Tier 2-3 extractor	`lib/intelligence/content-extractor.ts:extractContent()`	4-tier cascade: (1) RSS `content:encoded` via Turndown; (2) direct fetch + `extractMainContentHtml()` regex + Turndown; (2.5) Jina Reader (`https://r.jina.ai/${url}`); (3) Firecrawl (`@mendable/firecrawl-js` w/ FIRECRAWL_API_KEY); (4) summary fallback. Fail-fast in production if FIRECRAWL_API_KEY missing	~335 LOC	YES — 28 prod rows on `ingest_source='rss_feed'`
lib/ai/extract-content.ts	Per-item structured-data extraction via Claude (different concern)	NOT URL extraction — operates on already-extracted `content_items.content`. Sends to Claude with a JSON schema, returns structured JSON	~165 LOC	n/a — different feature

Important distinction the brief glosses: lib/ai/extract-content.ts is not URL extraction. It’s Claude-driven structured data extraction from already-stored content_items.content. It has no overlap with pullmd. The actual URL extraction infrastructure is lib/extraction/url.ts + dependencies (P4) and lib/intelligence/content-extractor.ts (P9 Tier 2-3).

3.2 Side-by-side extractor capabilities

Capability	KH `lib/extraction/url.ts` (P4)	KH `lib/intelligence/content-extractor.ts` (P9)	pullmd
Static HTML extraction	JSDOM + Readability + Turndown	regex `<article>` / `<main>` extract + Turndown	Readability + Trafilatura (parallel, quality-scored)
JS-heavy pages	None — Readability fails on hydrated SPAs	Falls through to Jina Reader → Firecrawl	Playwright headless Chromium (sidecar)
Cloudflare-protected pages	Blocked unless cached	Firecrawl handles via headless browser	Cloudflare native short-circuit (undocumented mechanism)
PDF extraction	unpdf with `mergePages: false` + `\n\n---\n\n` join	Not handled (RSS items aren’t typically PDFs)	Not supported
OG metadata	Hand-rolled regex (`lib/extraction/og-metadata.ts`)	Firecrawl returns `metadata.ogImage`	Not explicitly exposed in headers
SSRF protection	`validateUrl()` blocks 10/8, 172.16/12, 192.168/16, 169.254/16, localhost, non-http(s)	None — relies on Vercel egress + Firecrawl	Not documented (likely none — self-hosted in trusted env)
Redirect following + re-validation	Yes — re-runs `validateUrl()` on `response.url` after fetch	Yes — `redirect: 'follow'`; Firecrawl resolves Google News URLs via `metadata.sourceURL`	Yes — normalised URLs
Reddit thread support	None — generic page extraction, comments lost	None	First-class — full comment trees, depth 1-10, optional Reddit OAuth
Quality scoring	None — Readability succeeds or extraction throws	None — tier success/failure via word-count threshold	`X-Quality` 0.0-1.0 + `X-Source` header indicating winning extractor
Stable share id	None — no identity across re-fetches	None	8-hex share id, 90-day TTL post-write
Cache	None	In-memory `embeddingCache` (1h TTL) — but only for embeddings, not extraction	SQLite (`better-sqlite3`), 1h freshness window, 90-day retention
Last-good-snapshot fallback	None — fails on dead URL	Falls through tiers to summary fallback	Returns last cached if source unreachable
Rate limiting	None	`getGlobalRateLimiter().waitForDomain()` per-domain throttle	Not documented
Timeout	15s fetch	`EXTRACTION_TIMEOUT_MS` (varies, ~20-30s)	Not documented (Playwright default)
Size limit	20 MB content-length check	None explicit	Not documented

3.3 The capability delta

pullmd ADDS to KH: Playwright fallback (large win — KH currently has no JS-rendered fallback in P4; P9 hands off to Firecrawl which is paid), Reddit support (large win — KH ingests forum content but currently mangles Reddit threads), stable share-id with refresh contract (large win — directly enables source_documents lineage for URLs per 0.7.4-Q7), 90-day cached fallback (medium win — eliminates the “URL is gone, content is gone” failure mode), quality scoring (medium — gives ingest pipeline a signal for whether to trust the extraction).

pullmd LACKS vs KH: PDF extraction (KH must keep unpdf path for binary URLs — pullmd can’t replace this), SSRF protection (KH must keep validateUrl() in front of any pullmd call — pullmd assumes trusted callers), structured OG metadata in response (KH currently uses og:image for thumbnails, og:author for author_name; pullmd doesn’t expose these as separate fields, only via the markdown body — would need to either parse from markdown or re-fetch the page for metadata), per-domain rate limiting (KH’s global rate limiter would still need to gate calls to pullmd’s /api?url= since pullmd doesn’t gate per-source-domain itself).

pullmd REPLACES in KH: the Readability + JSDOM dependency chain in lib/extraction/html.ts (P4); the multi-tier cascade in lib/intelligence/content-extractor.ts Tier 2 / Tier 2.5 / Tier 3 (P9) — pullmd is a richer single-tier replacement that subsumes all three.

pullmd OVERLAPS with KH: Turndown (both use it; KH could stop bundling it if pullmd is the only HTML→MD path), Firecrawl (pullmd’s Playwright + Cloudflare path obviates Firecrawl for JS-heavy pages — could drop the FIRECRAWL_API_KEY ops dependency entirely if pullmd handles all the cases Firecrawl currently does, save ~$X/month).

4. Question 1 — Does pullmd replace P4 (URL ingest) entirely?

Verdict: Yes for the EXTRACTION step, no for the rest of P4.

P4 is 18-step pipeline (per 0.1-ts-url-ingest.md). pullmd would replace step 6 (extractFromUrl()) and the SSRF validation in step 4 stays as a pre-filter. Everything else — auth (1), rate limit (2), validation (3), existing-URL guard (5), quality gate (7), content_type detection (8), source_domain parse (9), embedding (10), dedup (11), content_items INSERT (12), chunks (13a), date extraction (13b), classify (14), summarise (15), quality-score (15b), layer (16), topic suggestion (17), guide-section (17b), re-fetch (18) — is KH’s responsibility and stays put.

4.1 The replacement contract

Today:

1
const validation = validateUrl(url);  // SSRF
2
if (!validation.valid) throw new Error(validation.error);
3
const extracted = await extractFromUrl(url);  // returns ExtractedContent
4
// ExtractedContent: { title, content, author, excerpt, ogImage, ogDescription, ogDate, extractionMethod, pageCount?, contentLength }

With pullmd as URL adapter:

1
const validation = validateUrl(url);  // SSRF — still ours
2
if (!validation.valid) throw new Error(validation.error);
3

4
// PDF special-case stays — pullmd doesn't do PDFs
5
if (isPdfUrl(url)) {  // sniff via HEAD or extension
6
  return extractPdfText(await fetchPdf(url));  // existing unpdf path
7
}
8

9
const response = await fetch(`${PULLMD_URL}/api?url=${encodeURIComponent(url)}`, {
10
  headers: { Authorization: `Bearer ${PULLMD_API_KEY}` },
11
  signal: AbortSignal.timeout(30_000),
12
});
13

14
const markdown = await response.text();
15
const shareId = response.headers.get('X-Share-Id');         // 8-hex
16
const source = response.headers.get('X-Source');             // 'readability' | 'playwright' | 'cloudflare' | ...
17
const quality = parseFloat(response.headers.get('X-Quality') ?? '0');
18

19
// pullmd returns markdown only — no separate title/author/og fields.
20
// Recover them from markdown body (first H1) + a separate og-metadata fetch
21
// OR accept the trade-off: title comes from URL extraction (always-present),
22
// author/og:image become best-effort on top of pullmd's call.

4.2 Required changes to P4

Step in P4	Change	Effort
4 SSRF	No change — keep `validateUrl()`	0
5 Existing-URL guard	No change	0
6 `extractFromUrl()`	Swap implementation — wrap `fetch(${PULLMD_URL}/api?url=)`	~3-4h
6a (NEW) PDF detection	Add HEAD-request sniff + branch to `extractPdfText()` since pullmd doesn’t do PDFs	~2h
6b (NEW) OG metadata	Either parse from pullmd markdown (lossy) OR keep a parallel og-only fetch (`extractOgMetadata()` from raw HTML) — recommend parallel fetch since og:image is the thumbnail UX	~3-4h
6c (NEW) Capture share_id	Persist `X-Share-Id` to `source_documents.metadata.pullmd_share_id` (or as a new typed column — see §5.4)	~1h
7 Quality gate	Augment with `X-Quality` threshold check — reject extractions below 0.3 quality OR wrap as `ingestion_quality_log` warning	~1h
8-18	No change	0

Total P4 swap: ~10-13h, single file mostly + a small ExtractedContent adapter to keep the contract. Tests covering the existing P4 flow stay; new tests around the pullmd response shape, PDF branch detection, and share-id capture.

4.3 Side-by-side extraction quality

I cannot empirically run pullmd against KH’s 55 production URLs in this evaluation (no live pullmd instance). But the architectural prior is strong: pullmd’s Readability + Trafilatura + Playwright stack strictly dominates KH’s Readability-only stack on JS-heavy pages, ties on static HTML (both use Readability), and adds Reddit comments which KH currently loses entirely. The only quality regression risk is on Cloudflare-protected pages where KH today silently 403s and pullmd’s “Cloudflare native markdown” short-circuit may or may not work — that’s why a 50-URL bake-off pre-commit is recommended (see §10 Open Questions).

4.4 Verdict

P4 swap is RECOMMENDED for v1, conditional on a 50-URL extraction-quality bake-off. Effort is bounded (~10-13h), the contract change is local (one file in lib/extraction/), and pullmd’s capabilities strictly dominate the current Readability-only path except on PDFs (kept) and OG metadata (small adapter). The biggest single architectural unlock is the share-id, which feeds directly into §5.

This is the single most interesting integration angle, because it directly resolves three open questions in 0.7-synthesis / 0.7.4:

0.7.4-Q7 (URLs become source_documents in v2 canonical reframe? — Liam ratified v1): pullmd’s share-id IS the source_document identity for URL inputs. The version chain hangs off it.
§3.1 Re-upload UPDATEs existing content_items (Liam ratified v1): for URL re-fetches, pullmd’s “/s/:id refreshes if older than 1h” gives a deterministic re-fetch trigger. The new content goes via content_history v2 against the same content_items row, while pullmd cache shows the diff naturally.
§3.2 Markdown bucket — bucket-only-for-binaries v1 (Liam: do we even need binaries?): for URL inputs, no bucket needed in v1 OR v2 — pullmd IS the cache, KH stores pullmd_share_id as the durable handle.

For URL inputs:

1
-- source_documents row written when ingest_source='url_import' or 'rss_feed'
2
INSERT INTO source_documents (
3
  id,                       -- KH-side uuid (always present)
4
  filename,                 -- from URL: 'extracted-from-{hostname}-{path-slug}.md' OR title-derived
5
  uploaded_by,              -- user.id (P4) or PIPELINE_SYSTEM_USER_ID (P9)
6
  mime_type,                -- 'text/markdown' (post-pullmd conversion)
7
  file_size,                -- byte length of pullmd-returned markdown
8
  content_hash,             -- MD5 of normalised markdown content (matches detect_reupload semantics)
9
  storage_path,             -- 'pullmd://{share_id}'  ← sentinel form, no bucket write
10
  extracted_text,           -- the pullmd markdown verbatim (source-of-truth for diffs)
11
  extraction_metadata,      -- JSONB: { source: 'pullmd', share_id, x_source: 'readability', x_quality: 0.85, fetched_at, original_url }
12
  version,                  -- 1 on first fetch; +1 on re-fetch detected
13
  parent_id,                -- NULL on first fetch; chains on re-fetch
14
  pipeline_run_id,          -- if cron-triggered
15
  workspace_id,             -- if RSS path
16
  status                    -- 'processed' once content lands in content_items.content
17
);

Key insight: storage_path = 'pullmd://${share_id}' is the sentinel form 0.7.4 §4.4 Q5 asked about. It’s not magic-string-hacky here — it’s a real URI scheme pointing at a real (pullmd) endpoint. A user can take that share id, append it to the pullmd public URL, and get the exact bytes back. Audit trail intact.

5.2 Re-fetch / re-upload behaviour

Today (P4): the existing-URL guard at step 5 of P4 silently early-returns the existing content_items row. No re-fetch ever happens. No version chain. If the page changed, KH never knows.

With pullmd share-id:

1
1. User triggers ingest (or cron) on URL X.
2
2. Existing-URL guard: SELECT * FROM source_documents WHERE
3
     extraction_metadata->>'original_url' = X
4
     AND archived_at IS NULL
5
     ORDER BY version DESC LIMIT 1;
6
3a. If miss: fresh ingest. POST to pullmd, capture share_id, INSERT source_documents v1, INSERT content_items, classify+chunk+embed (Layer 4).
7
3b. If hit (existing URL):
8
    - GET ${PULLMD_URL}/s/${existing_share_id}  ← live endpoint, auto-refresh if >1h old
9
    - Compare returned content_hash vs source_documents.content_hash
10
    - If identical: short-circuit, return existing content_item (today's behaviour)
11
    - If different:
12
        - INSERT new source_documents v2 with parent_id = v1.id
13
        - Compute diff (existing source_document_diffs infrastructure)
14
        - UPDATE existing content_items.content (Liam-ratified §3.1 behaviour)
15
        - This triggers content_history v2 (existing trigger)
16
        - Send diff-review notification (existing infra)

This is the biggest single UX unlock in the whole 0.7 synthesis — exactly the friction point Liam called out in 07-synthesis-feedback. pullmd makes it cheap. Without pullmd, KH would need to: (a) build its own URL-content cache (operationally heavy), (b) build its own re-fetch trigger logic (~5h), (c) build its own snapshot-on-failure (annoying — needs storage), (d) handle the “URL changed but content didn’t” case (content_hash already does this). With pullmd: (a) is free (SQLite cache), (b) is one HTTP call, (c) is free (last-good-snapshot built in), (d) works.

5.3 What about the 1h freshness window?

pullmd refreshes on /s/:id GET if the row is older than 1h. KH’s re-fetch cycle is rarer — daily-ish for RSS, on-demand for P4. Two options:

(a) Honour pullmd’s 1h freshness: every KH /s/:id hit may or may not re-fetch depending on cache age. Nondeterministic from KH’s perspective but lower load.
(b) Force-refresh: add a ?refresh=true parameter (if pullmd supports — needs check) OR call /api?url= directly (always fresh) and ignore the existing share. Wastes the cache benefit but deterministic.

Recommend (a). KH’s cron-driven re-fetch can be a no-op if pullmd cache is fresh — that’s actually a feature, not a bug.

5.4 Schema implications

Option A — typed column (recommended for v1):

1
ALTER TABLE source_documents ADD COLUMN pullmd_share_id text;
2
-- NULL for binary uploads (P7), set for URL-extracted (P4, P9)
3
CREATE INDEX idx_source_documents_pullmd_share_id ON source_documents(pullmd_share_id) WHERE pullmd_share_id IS NOT NULL;

Pros: queryable, indexable, drift-resistant. Cons: another column.

Option B — JSONB-only:

1
-- already in scope: extraction_metadata JSONB
2
-- write: { ..., pullmd: { share_id, x_source, x_quality, fetched_at } }

Pros: zero schema change. Cons: not indexable without a partial index on the JSONB key.

Recommend Option A. The share_id is identity-shaped data, deserves a column.

5.5 Verdict

This is the single highest-ROI pullmd integration. It costs a bounded migration + schema change + ~4h of share-id capture wiring, and unlocks the §3.1 + 0.7.4-Q7 dynamic the synthesis explicitly identifies as “the biggest single UX unlock.” If we adopt pullmd at all, this is the piece to commit to first.

6. Question 3 — NEW5/NEW7 collapse: do we need binary storage at all?

6.1 The user’s question (07-synthesis-feedback §3.2, verbatim)

Do we need to store binaries? If all content is extracted to markdown anyway, and we have the file provenance (for audit purposes), could all content be saved as markdown in a DB column/table? If users already store documents in, for example, SharePoint and other systems, they likely won’t want to have an additional location where a document is stored, and we need to consider that when a user edits the content, it’s no longer the original file that they uploaded. In future, we could look to sync with forms like SharePoint or Notion. But for the time being, our aim remains being the mechanism by which users can utilise AI to traverse and benefit from the content that they ingest to the platform, rather than us serving as ‘another’ document repository.

This is a product-level question, not just a technical one. Let me try to answer cleanly given pullmd’s capabilities.

6.2 Three input shapes, three answers

For URL inputs (P4, P9):

No bucket needed in v1 or v2. The original lives at the URL forever (or doesn’t, but pullmd’s last-good-snapshot covers that). The markdown lives in pullmd’s SQLite cache (90-day) and content_items.content (permanent). The audit trail is pullmd_share_id + original_url + fetched_at + content_hash — that’s enough to prove what KH read at ingest time. pullmd resolves this question entirely for URLs.

For markdown inputs (P8 EP2):

No bucket needed in v1. As 0.7.4 §4.4 already concluded: content lives in content_items.content and source_documents.extracted_text. The “original markdown” IS the canonical content. No binary to preserve.
Optional in v2: if “download as markdown.docx” UX wants byte-perfect re-download, add a bucket. But this is download-side, not upload-side.
pullmd plays no role here — markdown is already canonical, no extraction needed.

For binary inputs (P7 PDF/DOCX/etc.):

This is where the user’s question really lands. Today, KH’s documents bucket holds the binary, source_documents.extracted_text holds the post-extraction markdown, content_items.content holds the same markdown.
Per the user’s intuition, a binary is “another document repository” that competes with SharePoint/Notion. Strictly speaking, KH could:
- Drop the binary after extraction
- Keep only source_documents.extracted_text + content_items.content
- Capture provenance metadata (filename, mime_type, file_size, content_hash, uploader) without the bytes
What’s lost if we drop binaries:
- Re-extraction with future improved extractors (mammoth/unpdf bug fixes, new chunking strategies)
- Diff against original-original (the markdown extract becomes the new ground truth)
- Compliance: if a client’s regulator asks “show me the actual document the AI was reasoning over,” KH can only show the markdown extract, not the original PDF
- User UX: the “re-download as original PDF” link breaks (currently aspirational — 0.7.4 noted no UI uses this today)
What’s gained if we drop binaries:
- Storage cost reduction (pennies for SMBs but real for high-volume sectors)
- Honesty: KH stops being “yet another file store”
- Simpler RLS (no bucket policies)
- Faster onboarding (“upload a file” becomes “extract a file” mentally)

Verdict on §3.2: pullmd doesn’t directly answer the binary question — it only handles URLs. But it does set a precedent for “external service holds the source-of-truth, KH holds the markdown extract + provenance metadata.” That precedent could be extended to binaries via:

Option A: keep documents bucket as-is in v1, defer the question.
Option B: drop bucket, keep source_documents.extracted_text only, add original_uri typed column for “where the user said the original lives” (sharepoint URL, drive URL, etc.) — KH never holds the bytes.
Option C: hybrid — bucket optional. v1 default is bucket-on (legal compliance posture), with a per-client setting retain_binaries = true | false.

Option C is closest to the user’s intuition and matches the “one Supabase project per client” principle. But this is a Liam call, not pullmd-driven.

6.3 NEW5 and NEW7 collapse

NEW5 (D2 markdown storage_path) — Liam ratified inline-only: pullmd reinforces this. URL paths use pullmd://${share_id} sentinel. Markdown paths use inline://${id} sentinel or NULL. Binary paths use real bucket path. Three sentinels → schema can either accept a NULL storage_path OR a CHECK constraint allowing the three forms. Recommend NULL — simplest, matches “data quality is paramount.”

NEW7 (re-upload detection in D2 v1) — Liam ratified v1: pullmd’s content_hash + share_id contract makes URL re-upload detection trivial. Markdown re-upload detection still needs MD5-of-normalised-content (existing detect_reupload extends easily — same hash, same filename + uploader semantics). pullmd accelerates NEW7 for URLs but doesn’t directly help markdown. Markdown still needs the NEW7 work.

6.4 Verdict

pullmd resolves the binary question for URL inputs entirely (no storage needed). For binary uploads, the question stays open and is a Liam decision separate from pullmd. The strongest opinion I’ll commit to: for v1, keep the documents bucket for binary uploads (compliance posture, low cost, reversible), but adopt the “pullmd holds the source-of-truth” precedent for URL-shaped inputs. This is option (b) without committing to the full binary-drop yet.

7. Question 2 — RSS pipeline (P9) component: Tier 2-3 fit

7.1 What pullmd replaces in P9

P9’s lib/intelligence/content-extractor.ts:extractContent() 4-tier cascade:

Tier	Today	With pullmd
1: RSS `content:encoded`	Turndown on inline content	No change — pullmd doesn’t see the RSS payload, only URLs. Tier 1 stays as-is.
2: direct fetch + `<article>` regex + Turndown	regex extraction (lossy)	Replace with pullmd `/api?url=` call (richer extraction)
2.5: Jina Reader	`https://r.jina.ai/${url}` (free, public service)	Replace with pullmd OR keep both — pullmd is richer but Jina is free; pullmd saves the round-trip if it works
3: Firecrawl	paid API, Cloudflare-bypass + Google News redirect resolution	Mostly replaceable with pullmd — pullmd’s Playwright + Cloudflare native short-circuit covers most Firecrawl use cases. Google News redirect resolution would need verification (does pullmd follow the chain?)
4: summary fallback	use `item.summary ?? item.title` (very low quality)	No change — pullmd’s “last good snapshot” partially mitigates this but doesn’t fully replace — if pullmd has never seen the URL, no snapshot exists

7.2 What pullmd does NOT replace in P9

P9 is much more than URL extraction. It includes:

getDueFeedSources() — workspace-scoped feed source discovery
pollFeed / pollWebSource — RSS-specific polling with feed_sources row updates
parseFeedItems — rss-parser library + Atom <category term> regex
isDuplicate (workspace-scoped via feed_articles table)
The 4-tier discovery cascade (Tier 1 RSS → Tier 2-3 URL → Tier 4 fallback) — pullmd is one path inside this; the cascade orchestration stays
Embedding pre-filter via OpenAI text-embedding-3-large + relevance threshold
LLM relevance scoring via Claude getModelForTier('quality')
Article summary via Claude haiku-4-5
feed_articles INSERT (always — even for filtered)
storeAsContentItem (only for passed) → calls Layer 4 ingest
Concurrency guard via si_processing_queue
Per-domain rate limiting via getGlobalRateLimiter()

The relevance filter + 4-tier discovery cascade is orthogonal to pullmd. pullmd is just a better Tier 2-3. The relevance filter happens AFTER extraction (against the markdown content) — pullmd doesn’t see relevance and shouldn’t.

7.3 Effort estimate for P9 swap

Step	Effort
Replace Tier 2 (direct fetch + regex) with pullmd `/api?url=`	~3h
Decide Tier 2.5 fate: keep Jina parallel (cheap fallback if pullmd fails) OR drop	~1h decision + 0-1h impl
Replace Tier 3 (Firecrawl) with pullmd’s Playwright path; preserve Google News redirect resolution OR verify pullmd handles it	~4-5h (verification + impl)
Capture `X-Source` and `X-Quality` to `feed_articles.extraction_method` (existing column) — `'pullmd_readability' \| 'pullmd_playwright' \| 'pullmd_cloudflare' \| 'pullmd_reddit'` extends the existing CHECK constraint	~1h migration + ~1h impl
Update `extraction_method` taxonomy in classification telemetry	~1h

Total P9 swap: ~10-12h. About the same as P4. Could be done in parallel.

7.4 The Reddit support angle

Per brief Q4: “KH’s clients ingest forum-style content for sector intelligence. Reddit support is a unique pullmd feature.”

Today’s Reddit story in KH: the 4-tier extractor would currently treat a Reddit URL like any other — direct fetch returns the SPA shell, falls through to Jina/Firecrawl, gets a degraded extraction with comments mostly lost.

With pullmd: Reddit becomes first-class. Comment trees (depth 1-10), nested structure preserved, OAuth-optional (better rate limits with credentials, public JSON otherwise).

Sector intelligence value: depends entirely on whether clients’ RSS feeds include Reddit threads. The RSS feed model is built around news/blog feeds; Reddit subreddits do offer RSS (https://reddit.com/r/subreddit/.rss) but the items in those feeds are URLs to threads, not the threads themselves. So the value chain is: RSS feed for r/SECTOR → thread URL → pullmd extracts thread + comments → KH classifies the multi-comment markdown as one content_item.

This works mechanically but raises a content-modeling question: a Reddit thread with 50 comments is one content_items row with 50 perspectives in it. Classification will probably struggle (not a “policy” or “case_study” — what is it? “discussion”?). Q&A autosplit might be the right shape if questions/answers exist in the thread. But this is all post-pullmd, KH-side work.

Verdict on Q4: pullmd delivers Reddit extraction cleanly. Whether KH wants to ingest Reddit threads is a product decision, not a pullmd one. Recommend: capability adopted (pullmd path supports Reddit), product decision deferred (do clients actually want this?). At minimum, Reddit URLs would no longer silently fail in P4/P9 — that’s a quality win even if no client asks for it.

8. Question 6 — Self-hosted vs SaaS for “one Supabase project per client”

8.1 The deployment topology question

KH’s principle: “One Supabase project per client — simple isolation, not multi-tenant RLS.” (CLAUDE.md). pullmd is self-hosted only (no SaaS). So the question is: one pullmd per client, or one pullmd shared across all clients?

8.2 Per-client pullmd vs shared pullmd

Dimension	Per-client	Shared
Cost	~$X × N clients (Cloud Run / Fly / VPS)	$X total
Cache utility	Per-client misses (low cache hit rate per pullmd)	Cross-client cache hits (e.g. two clients ingest the same gov.uk URL → one extraction)
Privacy	Strong — client A never sees client B’s pullmd cache	Weaker — shared SQLite holds both clients’ URLs (URL is metadata only, content is the public web, but the URLS themselves leak ingest patterns)
Compliance	Per-client auth = per-client config	Shared auth (could use API keys per client)
Operational	N services to monitor	1 service
Failure mode	One client’s pullmd crash → only that client affected	Shared crash → all clients affected

8.3 The honest take

For URL extraction, the cache content is public web content — there’s no private data leakage if client A and client B both extract https://www.gov.uk/cabinet-office/some-policy. The URLs themselves are metadata; if two clients are tracking the same gov.uk feed, that’s not a secret either.

Recommend: shared pullmd instance per deployment region. One container, shared cache, API-key-auth per client. This matches the “one record, many views” principle (KH’s mental model of duplication-as-bad applied at the cache layer too).

Exception: if a client has a confidentiality posture that requires private URL ingestion (e.g. ingesting URLs from their own intranet — though SSRF would block this), they get a private pullmd. This would be rare.

8.4 Operational concerns

The Playwright sidecar adds ~3.7 GB. This is non-trivial for Cloud Run cold starts. Options:

(a) Run pullmd in a long-running container (Fly.io, ECS Fargate, etc.) — 3.7 GB image is fine when warm
(b) Skip Playwright sidecar — fall back to Trafilatura-only extraction. Loses Playwright fallback for JS-heavy pages, gains lighter image (~80 MB)
(c) Run two pullmd instances: one with Playwright (cold-start ok), one without (Vercel-friendly cron worker hits the lighter one for fast-path cases)

Recommend (a) for v1. KH’s deploy posture is already mixed (Vercel for the app, Cloud Run for Python pipelines) — adding one more long-running service is incremental, not categorical.

8.5 Verdict

Single shared pullmd container per deployment region. Long-running (not Cloud Run cold-start). Auth via API keys per client. ~$X/month operational cost.

9. Question 7 — pullmd’s MCP server vs KH’s MCP

9.1 pullmd’s 3 MCP tools

read_url — fetch and convert a URL
get_share — retrieve by share id
list_recent — browse recent conversions

9.2 KH’s MCP tool surface (post-pruning per ongoing work)

KH’s MCP tools live in lib/mcp/ and are documented in docs/generated/mcp-inventory.md. The full list is long (~30 tools). Tools that overlap with pullmd’s surface:

KH has no read_url direct tool — its URL extraction is wrapped inside create_content_item and ingest endpoints, both of which trigger Layer 4 work. read_url is shape-different (returns markdown, doesn’t ingest).
KH has no get_share — closest is get_content_item which retrieves by KH uuid not pullmd share id.
KH has no list_recent for raw extractions — closest is content search.

9.3 Could KH consume pullmd’s MCP rather than re-implement?

Two integration shapes:

(a) KH server-side calls pullmd’s REST API (chosen pattern in §4-5 above). MCP-over-MCP is not invoked.
(b) KH proxies pullmd’s MCP through to Claude Desktop/CoWork so users in Claude can directly invoke read_url via KH’s MCP server.

(a) is the right pattern for ingest pipelines (server-to-server, deterministic).

(b) is interesting because KH’s product strategy (per the AI strategy doc quoted in graphify-evaluation-feedback) is that “Claude is the primary AI interface — where most AI-powered interaction happens.” If a user in Claude Desktop wants to “summarise this URL into the KB,” that’s exactly what read_url (pullmd) → create_content_item (KH) chained would do.

Recommend defer (b). It’s a feature, not a pipeline necessity. Adding pullmd’s MCP to KH’s MCP exposure is ~1 day of adapter work + a security review (forwarding MCP calls). Worth a backlog item; not v1.

9.4 Verdict

Use pullmd via REST from KH’s pipeline (pattern a). Defer MCP-from-MCP exposure (pattern b) to a backlog item. This matches the broader pattern in 07-synthesis-feedback: “P10 MCP create — this is a priority area because it’s where the client is already utilising Claude Desktop and Claude CoWork to create content.” Adding pullmd’s MCP into KH’s MCP surface aligns with that priority but isn’t blocking.

10. Re-use vs integrate vs reinvent vs hybrid — final verdict

10.1 Decision matrix (mirrors graphify-evaluation §6.4 format)

ID	Decision	Recommendation	Confidence
PM1	Adopt pullmd as URL→markdown shape adapter for P4?	YES — replaces `lib/extraction/url.ts` + `lib/extraction/html.ts` HTML branch; PDF stays unpdf; SSRF stays KH-side	85% — conditional on 50-URL bake-off
PM2	Adopt pullmd for P9 Tier 2-3 (URL extraction in RSS pipeline)?	YES — replaces direct-fetch regex + Jina + Firecrawl on URL paths; Tier 1 (RSS `content:encoded`) and Tier 4 (summary fallback) stay	80% — Google News redirect resolution needs verification
PM3	Use pullmd `share_id` as the source_documents identity for URL inputs?	YES — directly resolves 0.7.4-Q7 + §3.1 + §3.2; new typed column `pullmd_share_id` on `source_documents`; sentinel `pullmd://${id}` in `storage_path`	90% — biggest single architectural unlock
PM4	Drop binary bucket entirely (per 07-synthesis-feedback §3.2)?	NO for v1, decision deferred to Liam separately. pullmd doesn’t help with binaries; URL bucket-drop is what pullmd enables	60% — Liam call
PM5	Drop Firecrawl from KH’s stack once pullmd is wired?	YES if pullmd Playwright covers Cloudflare + Google News chain successfully — saves ops cost	70% — needs verification
PM6	Single shared pullmd instance per deployment region (vs per-client)?	YES — public web content, cross-client cache hits valuable, simpler ops	80%
PM7	Adopt Reddit support?	YES capability-wise — pullmd makes it free; product decision (do clients want it?) deferred	75% — Reddit ingest may surface content-modeling questions (multi-comment threads)
PM8	Mount pullmd’s MCP alongside KH’s MCP for Claude Desktop/CoWork?	DEFER to backlog. Server-side REST integration is the v1 pattern. MCP exposure is a feature, not infrastructure	70%
PM9	Drop existing `lib/extraction/html.ts` + `lib/extraction/turndown.ts` from KH?	YES once both P4 and P9 are pullmd-backed; Turndown stays only for P9 Tier 1 (RSS `content:encoded`)	75%
PM10	AGPL v3 self-hosted is licence-compatible with KH’s posture?	LIKELY YES (self-hosted, no fork into KH), but Liam should sign off given AGPL’s “network service” clause	80% — needs Liam confirmation

10.2 Re-use vs integrate vs reinvent vs hybrid

Re-use (run pullmd unchanged, call its REST API): the recommended pattern for v1. Bounded effort, clean contract, reversible.

Integrate (consume pullmd’s MCP, fold its share-id into KH schema): §5 + §9 — the pieces of pullmd that align with KH’s primitives (share-id → source_documents, MCP → KH MCP) get folded in cleanly. Doesn’t require forking pullmd source.

Reinvent (build URL+Reddit+Playwright extraction in KH): NOT recommended. This is exactly the trap graphify-evaluation-feedback warned against: “re-use of battle-tested infra,” “what should we be doing not what’s lowest disruption.” pullmd’s Playwright + Reddit + Cloudflare path is meaningful engineering work that’s already done. KH should not duplicate.

Hybrid (pullmd for URL/Reddit, KH-native for PDF + binary + markdown): this is the actual recommendation. pullmd handles the URL shape; KH handles PDFs (unpdf), binaries (mammoth/unpdf + bucket), markdown (P8 inline). Layer 4 (dedup/classify/embed/chunk) stays KH-side for all shapes.

10.3 The phased plan

Phase A — Foundation (v1, pre-launch, ~1.5 weeks):

Stand up shared pullmd Docker container (Trafilatura sidecar + Playwright sidecar + Node app). Auth: single API key. Cloud Run / Fly / wherever. ~4-6h ops.
Add pullmd_share_id typed column to source_documents + index. ~1h migration.
Add original_url typed column to source_documents (currently only in extraction_metadata JSONB). ~30 min migration.
Build the extractFromPullmd(url) adapter in lib/extraction/pullmd.ts returning the existing ExtractedContent interface. PDF special-case stays as a pre-branch. OG metadata via parallel fetch (small extractOgMetadata call to recover og:image etc.). ~6-8h.
Bake-off test: 50 prod URLs through pullmd vs current extraction, score quality manually. ~3-4h.
Decision gate: if bake-off positive, proceed. If not, abort cleanly.

Phase B — P4 swap (v1, ~1 week):

Replace extractFromUrl() in lib/extraction/url.ts with pullmd-backed implementation. Keep PDF branch and SSRF as-is.
Wire share-id capture into P4 ingest flow. INSERT source_documents row with storage_path='pullmd://${share_id}' for URL inputs.
Update existing-URL guard at step 5 of P4 to use the new re-fetch flow per §5.2.
Update P4 tests; new tests around pullmd response shape, PDF branch detection, share-id capture, re-fetch behaviour.

Phase C — P9 swap (v1, ~1 week, parallel with B):

Replace Tier 2 + Tier 2.5 + Tier 3 in content-extractor.ts with pullmd. Keep Tier 1 (RSS content:encoded) and Tier 4 (summary fallback).
Wire share-id capture into RSS storeAsContentItem. Same source_documents flow as P4.
Migrate feed_articles.extraction_method CHECK constraint to add pullmd_* values.
Update P9 tests.

Phase D — Cleanup (v1, ~half-week):

If pullmd Playwright covers Firecrawl use cases: drop @mendable/firecrawl-js, remove FIRECRAWL_API_KEY from env. Update health endpoint. Update fail-fast policy in checkFirecrawlApiKey() (now checkPullmdAvailable()).
Drop lib/extraction/html.ts (Readability + JSDOM) — only used by P4, now superseded.
Update extraction_method taxonomy in classification telemetry, reference docs.

Phase E — Defer to backlog:

Reddit ingest as product feature (BL-PM-1)
pullmd MCP as KH MCP exposure (BL-PM-2)
Drop binary bucket question (BL-PM-3 — Liam call, separate from pullmd)

Total v1 effort: ~3-3.5 weeks for Phases A-D. Compare with the canonical-pipeline foundation (~2 weeks) and per-path refactors (~6 weeks) — pullmd integration is a sub-component of the per-path refactor for P4 and P9, not orthogonal work.

10.4 Sequencing with re-ingest

pullmd swap should happen BEFORE re-ingest, not after. Re-ingesting 617 prod rows through pullmd produces a clean baseline (every URL row gets a pullmd_share_id, every URL extraction goes through the new path). Re-ingesting through old lib/extraction/url.ts and then swapping wastes the re-ingest opportunity.

This means the canonical-pipeline plan in 0.7-synthesis §5.2 needs amendment:

Phase A foundation adds pullmd container + pullmd_share_id column work
Phases B/C/D/E proceed unchanged but use pullmd instead of legacy extractors for URL paths
The “re-ingest 617 prod rows” gate (Stream 1 finale) happens AFTER pullmd swap

10.5 Verdict on the four framings

Re-use: YES (run pullmd unchanged, REST API)
Integrate: YES (share-id → source_documents, eventually MCP→MCP)
Reinvent: NO (would duplicate Playwright + Reddit + Cloudflare work that’s done)
Hybrid: YES (pullmd for URL+Reddit, KH-native for PDF+binary+markdown+Layer-4)

The hybrid framing wins. Adopt pullmd as the URL shape adapter; keep everything else KH-native.

11. Open questions for parent session

ID	Question	Why it matters	Recommendation
PM-Q1	50-URL bake-off vs current extractor before commit?	Prevents committing to a regression on edge cases (Cloudflare, JS-heavy, rare CDN setups)	YES — half-day spend, cheap insurance
PM-Q2	AGPL v3 acceptable for self-hosted use?	If KH never forks pullmd source into the platform, AGPL “network service” clause likely doesn’t trigger. But Liam should confirm	Liam call — likely yes
PM-Q3	Drop Firecrawl entirely if pullmd’s Playwright covers it?	Saves ops cost ($X/month + maintenance). Risk: pullmd’s Cloudflare path may not match Firecrawl’s JS-rendering coverage on long-tail	Defer until pullmd is live; verify on real corpus
PM-Q4	Adopt Reddit ingest as product feature?	pullmd makes the capability free; product decision is “do clients want forum content?”	Defer to product backlog; capability adopted, product gate later
PM-Q5	Per-client vs shared pullmd instance?	Public web URL extraction has limited per-client privacy concern; shared cache valuable	Shared in v1; per-client only if a client requests it
PM-Q6	Add `pullmd_share_id` typed column or JSONB-only?	Identity-shaped data deserves a column; JSONB-only saves migration but loses indexability	Typed column (Option A in §5.4)
PM-Q7	Drop `lib/extraction/html.ts` + Readability + JSDOM dependency once pullmd is wired?	~3 dependencies removed (`@mozilla/readability`, `jsdom`, possibly Turndown if RSS Tier-1 also moves to pullmd later)	Yes — clean up post-Phase D
PM-Q8	URL re-fetch policy: respect pullmd 1h freshness vs force-refresh per ingest?	Affects re-ingest cost (force-refresh = pullmd hits source every time; 1h = mostly cached)	Respect pullmd 1h (Option (a) in §5.3)
PM-Q9	source_documents.storage_path NULL vs sentinel `pullmd://${id}` vs `inline://${id}`?	Schema clarity vs sentinel-as-magic-string trade-off	Use sentinels — they ARE valid URIs (pullmd://), audit-trail intact
PM-Q10	Mount pullmd MCP alongside KH MCP for Claude Desktop/CoWork?	Aligns with “Claude is primary interface” but ~1 day work + security review	Defer to backlog (BL-PM-2)
PM-Q11	If pullmd MCP is exposed, do we share the share-id with users in Claude?	UX question — the share-id is a public refresh handle	Defer with Q10
PM-Q12	Can pullmd handle Google News redirects?	Today’s Firecrawl path handles `metadata.sourceURL` resolution. pullmd’s docs are silent	Verify via test before dropping Firecrawl
PM-Q13	Does pullmd respect robots.txt?	Compliance / ethical scraping posture	Verify — likely yes (Trafilatura does); if not, add check upstream
PM-Q14	Rate limiting at the pullmd ↔ source-domain interface?	KH’s `getGlobalRateLimiter()` sits on KH side; pullmd has no documented rate limiting. So source domains may see double-rate when KH calls pullmd which calls them. Could trip Cloudflare protection	Add per-domain rate limiting at the pullmd-call site OR run pullmd through a known-good egress IP

11.1 The most important questions

PM-Q1 (bake-off) and PM-Q2 (AGPL) are the two GOs/NO-GOs. Without those answered, Phases A-D can’t start.

PM-Q3 (Firecrawl) and PM-Q12 (Google News) bound the Phase D cleanup scope.

PM-Q9 (storage_path sentinels) is the single schema decision that ties pullmd into the canonical-pipeline source_documents work — needs to be locked early in Phase A.

12. What pullmd does NOT solve (for completeness)

So that the recommendation isn’t oversold:

PDF extraction. KH keeps unpdf. pullmd has no PDF handling.
Binary uploads (DOCX, XLSX, etc.). KH keeps mammoth + custom DOCX handling. pullmd is URL-only.
Layer 4 of the canonical pipeline — dedup, classify (Pass 1 + Pass 2), embed, chunk. pullmd has no role.
The 4-tier discovery cascade orchestration in P9. pullmd is one tier inside it; the orchestration stays.
Embedding pre-filter + relevance scoring in P9. Different concern (post-extraction filtering).
The 10-path → canonical-pipeline collapse. pullmd accelerates the URL adapter inside Layer 3 but doesn’t change the architecture overall.
The Q&A docx pipeline (P3, P6). pullmd has no role.
Manual content creation (P5) + MCP create (P10). Different shape — these are text-input paths, not URL-input.
detectQAPairs() orphan and similar build-not-wired findings. Orthogonal to extraction.
The “drop binary bucket” question. pullmd handles URLs only; binary policy is independent.
Test infrastructure audits. Orthogonal.
Cloud Run scheduler decision (P1 retirement). Production-readiness track owns this.

pullmd is a focused URL→markdown service. The recommendation’s confidence comes from that focus, not from pretending it’s a wider solution.

13. Confidence assessment

Section	Confidence	Reason
§2 What pullmd is	95%	Sourced from README + Docker Hub + WebFetch verification
§3 KH URL infrastructure inventory	95%	Direct file-level audit + 0.1-ts-url-ingest + 0.1-rss-intelligence-pipeline grounding
§4 P4 replacement	88%	Architectural prior strong; 50-URL bake-off would tighten to 95%
§5 source_documents + share-id	90%	Mechanism is clear; Liam’s §3.1 + §3.2 ratifications align directly with pullmd’s contract
§6 NEW5/NEW7 + binary bucket	75%	Half answered (URLs), half deferred (binaries) — Liam call
§7 P9 RSS Tier 2-3	82%	Mechanism clear; Google News redirect handling is the unverified piece
§8 Self-hosted topology	80%	Standard ops trade-offs; no surprises
§9 MCP overlap	85%	Server-side REST is the right pattern; MCP-from-MCP is feature work
§10 Decision matrix + verdict	85%	Hybrid framing is honest; bake-off gates the commit
§11 Open questions	88%	Per-question confidence attached
§12 What pullmd does NOT solve	95%	Defensive scope-bounding
Overall evaluation	84%

Items below 90%:

§6 binary bucket question (75%) — this is a Liam-product-decision, not pullmd-driven. pullmd evaluation alone doesn’t move the bar.
§7 P9 Tier 3 (82%) — Firecrawl has known-good Google News redirect resolution; pullmd’s coverage is unverified.
§4 P4 swap (88%) — strong architectural prior but unverified empirically.
§10 verdict (85%) — bake-off gates the commit; until then the verdict is “lean YES with verification.”
§8 ops topology (80%) — depends on actual cost/latency on a real Cloud Run / Fly deployment; unverified.

14. References

pullmd

AeternaLabsHQ/pullmd — main repo, README inspected
Docker Hub: aeternalabshq/pullmd 80.1 MB amd64+arm64, version 2.0.0 stable, :latest still on v1.x
License: GNU AGPL v3
Architecture: Node.js Express + better-sqlite3 + Trafilatura Python sidecar (FastAPI) + Playwright Python sidecar (FastAPI + Chromium ~3.7 GB optional)
MCP Streamable-HTTP transport, 3 tools: read_url, get_share, list_recent

KH URL extraction infrastructure (audited)

lib/extraction/url.ts — main URL fetch orchestrator (P4 entry from lib/ai/extract-content.ts is unrelated structured-data extraction, NOT URL extraction)
lib/extraction/html.ts — JSDOM + Readability + Turndown
lib/extraction/pdf.ts — unpdf
lib/extraction/url-validation.ts — SSRF protection
lib/extraction/og-metadata.ts — regex OG metadata
lib/extraction/turndown.ts — Turndown configuration with GFM
lib/extraction/markdown-front-matter.ts — YAML/TOML front-matter parser (unused by URL path)
lib/intelligence/content-extractor.ts — P9 4-tier cascade
app/api/ingest/url/route.ts — P4 entry point
app/api/upload/route.ts — P7 binary upload (NOT pullmd-applicable)

Phase 0 audits referenced

docs/plans/phase-0-investigation/0.1-ts-url-ingest.md — P4 detailed audit
docs/plans/phase-0-investigation/0.1-rss-intelligence-pipeline.md — P9 detailed audit
docs/plans/phase-0-investigation/0.7-synthesis.md — canonical pipeline architecture
docs/plans/phase-0-investigation/0.7.4-source-documents-history-relationship.md — source_documents architecture
docs/plans/phase-0-investigation/07-synthesis-feedback.md — Liam feedback, especially §3.1 (re-upload UPDATEs), §3.2 (binary storage question), and “Remaining third-party tools to evaluate”
docs/plans/phase-0-investigation/graphify-evaluation.md + feedback — structural template for this evaluation
docs/plans/phase-0-investigation/trpc-evaluation-feedback.md — re-use bias framing

End of pullmd evaluation. Overall confidence 84%. 14 open questions surfaced for parent ratification (PM-Q1 through PM-Q14). Verdict: HYBRID — adopt pullmd as URL shape adapter for P4 and P9 Tier 2-3, capture share_id as source_documents identity for URL inputs, defer Reddit ingest as product feature, defer MCP-from-MCP exposure to backlog, keep all binary handling and Layer 4 work KH-side. ~3-3.5 weeks effort across Phases A-D, gated by 50-URL bake-off (PM-Q1) and AGPL v3 confirmation (PM-Q2).