RESEARCH — {42.1} Deploy pullmd Cloud Run Service + retire HTML extraction cascade

RESEARCH — {42.1} Deploy pullmd Cloud Run Service + retire HTML extraction cascade

Task: ID-42 — Deploy pullmd Cloud Run Service + retire current HTML extraction cascade. Spec-chain artefact: {42.1} RESEARCH (first of {42.1}→{42.2}→{42.3}→{42.4}). Date: 25/05/2026 (S262 / ID-42 research wave). Author: Task Planner (Opus 4.7, 1M context) — read-only research + empirical verification. Branch: worktree-agent-a4c33af067201ce94 (transient worktree off main@f7767c92). Status: Research complete. Returns to Orchestrator for ratification before {42.2} PRODUCT.

UK English throughout. Per OQ-3 (Q-EX2 forcing function) every external-API and schema citation in this doc is backed by an empirical import/inspect/fetch check against the live pin — see §6 Verification.

---

0. Executive summary

ID-42 has three coupled scope elements, and the verified facts re-order their risk:

1. Deploy the pullmd Cloud Run Service (the D4 blocker). pullmd was never deployed. PULLMD_SERVICE_URL is an S258 placeholder (https://pullmd-not-yet-deployed-staging.example.com) mounted at .github/workflows/cloud-run-deploy.yml:345. There is NO pullmd manifest, NO cloudbuild config for it, and NO image in Artifact Registry.

2. Fix the cocoindex HTML adapter so it can actually talk to pullmd. This is a NEW finding not in the ID-42 brief. scripts/cocoindex_pipeline/adapters.py::_pullmd_to_markdown() is contract-wrong on two counts against the empirically-verified pullmd v2.x HTTP surface: it POSTs to /extract (that endpoint does not exist — the real one is GET /api?url=) and it reads response.json()["markdown"] (the real response is a raw text/markdown body, not JSON). Deploying pullmd alone does NOT make HTML ingest work; the adapter must be rewritten. See §2.

3. Retire the live HTML extraction cascade — but ONLY after (1)+(2) make HTML ingest work end-to-end through deployed pullmd. The cascade (lib/intelligence/content-extractor.ts::extractContent()) is the live production sector-intelligence ingestion path today. There are in fact three distinct live HTML-extraction surfaces (§1), not one — the ID-42 brief named only the cascade.

The single most important scope correction: the gap-analysis intel and the ID-42 acceptance criteria assume pullmd_share_id + extraction_method ∈ {pullmd_readability | pullmd_playwright | pullmd_cloudflare} live on source_documents. Neither column exists on source_documents (verified §6). extraction_method exists ONLY on feed_articles (sector-intelligence), with the legacy enum {rss_content, fetch, jina_reader, firecrawl, summary_fallback}. The cocoindex pipeline (which writes source_documents / content_items / q_a_extractions) does not write any extraction_method at all today. So the schema-column work is real net-new DDL, and which table it lands on is a load-bearing ratification question for {42.2}/{42.3} (§5).

T7/T14 disposition (the gating question for Liam, §3): the canonical PLAN gates T14 (cascade retirement) on “T8 stable in prod + T7 (Phew first-ingest) complete.” Neither T7 nor a stable HTML ingest path is a tracked Task. ID-42 is the missing “stable HTML path.” Recommendation: scope ID-42 to make HTML ingest work + retire the web/HTML cascade independently of T7 (T7 is a Q&A-markdown path that does not exercise pullmd at all), and surface a crisp split-vs-fold question to Liam rather than deciding unilaterally.

---

1. Acceptance item 1 — full live HTML-surface inventory

Enumerated via grep across lib/, app/, scripts/ (excluding node_modules, .next, tests) + ID-42 task-list dispatch brief + cross-check against PLAN §4.14 T14 deletion list. Three distinct live surfaces produce HTML→markdown today; the ID-42 brief named only Surface A.

Surface A — P9 sector-intelligence 4-tier cascade (LIVE in production)

The path the ID-42 brief calls “the OLD 4-tier HTML cascade”. Confirmed live and the default RSS/web ingestion path.

Symbol / site	file:line	Role
extractContent(item)	lib/intelligence/content-extractor.ts:170	The 4-tier cascade. Tier 1 rss_content (Turndown on content:encoded, line 180); Tier 2 fetch (direct fetch + extractMainContentHtml() regex + Turndown, lines 196-240); Tier 2.5 jina_reader (https://r.jina.ai/${url}, lines 242-272); Tier 3 firecrawl (@mendable/firecrawl-js, lines 274-315); Tier 4 summary_fallback (lines 317-333).
extractMainContentHtml(html)	lib/intelligence/content-extractor.ts:37	<article>/<main> regex extractor used by Tier 2. ID-42 acceptance grep targets this symbol name.
checkFirecrawlApiKey()	lib/intelligence/content-extractor.ts:136	Pipeline-startup fail-fast guard; throws in production if FIRECRAWL_API_KEY missing. Called once from runPipeline (pipeline.ts:777).
isFirecrawlConfigured()	lib/intelligence/content-extractor.ts:27	Read by health/status surfaces.
Sole production caller of extractContent	lib/intelligence/pipeline.ts:372	const extraction = await extractContent(item); inside processFeedSource.
pollWebSource(source)	lib/intelligence/feed-poller.ts:336	Web-source poller. Scrapes with @mendable/firecrawl-js (lines 430-435) → emits raw HTML in contentEncoded → extractContent does the Turndown. This is a SECOND live Firecrawl call site distinct from the cascade Tier 3.
pollFeed(source)	lib/intelligence/feed-poller.ts:169	RSS poller (no Firecrawl); feeds extractContent via contentEncoded / direct-fetch tiers.

Cascade entrypoints (cron + manual + MCP), the things that must keep working:

Entrypoint	file:line	Notes
Cron poll	app/api/cron/intelligence-poll/route.ts:18	await runPipeline(supabase) — the scheduled production ingest.
Manual trigger	app/api/intelligence/trigger-poll/route.ts:14	await runPipeline(supabase).
MCP tool	lib/mcp/tools/intelligence.ts:120	dynamic import of runPipeline.
Source-test endpoint	app/api/intelligence/workspaces/[id]/sources/[sourceId]/test/route.ts:51,80	Admin “test this source” — calls pollWebSource(…, {dryRun:true}) / pollFeed. Must keep working post-retirement.

runPipeline → processFeedSource → (pollFeed|pollWebSource) → extractContent writes feed_articles.extraction_method (pipeline.ts:395, 480) and, for passed articles, content_items via storeAsContentItem (pipeline.ts:577). Retiring Surface A means re-pointing this whole chain at pullmd — it is NOT a delete-only operation; the RSS Tier 1 rss_content path (inline content:encoded, no URL fetch) has no pullmd equivalent and must be preserved or explicitly retired (decision for {42.2}).

Surface B — P4 URL-ingest (LIVE, separate path, NOT in the ID-42 brief)

A completely separate URL→markdown path that the ID-42 brief does not mention but PLAN §4.14 T14 explicitly targets for deletion (lib/extraction/html.ts).

Symbol / site	file:line	Role
extractFromUrl(url)	lib/extraction/url.ts:47	SSRF-validated fetch → PDF branch (unpdf via extractPdfText) OR HTML branch (extractFromHtml = JSDOM + Mozilla Readability + Turndown) + extractOgMetadata. Returns ExtractedContent.
extractFromHtml(html, url)	lib/extraction/html.ts:28	JSDOM + @mozilla/readability + Turndown.
Sole caller	app/api/ingest/url/route.ts:94	dynamic import of extractFromUrl. The manual “ingest a URL” API.

PLAN §4.14 deletes lib/extraction/html.ts + removes @mozilla/readability + jsdom + unpdf. Decision for {42.2}/{42.3}: is Surface B in ID-42 scope (re-point app/api/ingest/url at pullmd, keep the PDF branch on Docling/unpdf), or is it deferred to a later Task? The PLAN treats both A and B as T14 retirement scope. The cleanest framing: ID-42 covers the HTML branches of A and B; PDF/binary stays on Docling (cocoindex) / unpdf (Surface B) per the Docling-owns-binary ratification (03-tech-stack.md §6-§7).

Surface C — cocoindex HTML→pullmd adapter (CODE EXISTS, RUNTIME-DEAD)

The intended future HTML path, wired but non-functional.

Symbol / site	file:line	Role
convert_binary_to_markdown(file)	scripts/cocoindex_pipeline/adapters.py:42	Outer-tier @coco.fn(memo=True). Routes .html/.htm → _pullmd_to_markdown(url).
_pullmd_to_markdown(url)	scripts/cocoindex_pipeline/adapters.py:88	Inner @coco.fn(memo=True). Raises RuntimeError when PULLMD_SERVICE_URL unset (lines 90-95). When set, POST {url}/extract {"url":url} → response.json()["markdown"]. Contract-wrong — see §2.
Flow wiring	scripts/cocoindex_pipeline/flow.py:669	content_text = source.transform(convert_binary_to_markdown). So the cocoindex flow DOES route HTML through pullmd — but the adapter is dead (placeholder URL + wrong contract).

Surface C writes neither feed_articles.extraction_method nor any source_documents extraction-method (verified — flow.py:536 only writes extractor_kind for Q&A). So the ID-42 acceptance criterion “extraction_method reflecting the pullmd source header” has no write site today — it is net-new wiring (§5).

Inventory verdict

• Three live HTML surfaces (A live-prod, B live-prod, C code-dead).
• Cascade retirement (PLAN T14) targets A (content-extractor.ts) AND B (lib/extraction/html.ts) AND the 4 npm deps (@mozilla/readability, jsdom, unpdf, @mendable/firecrawl-js).
• ID-42 acceptance grep grep -r 'firecrawl|FIRECRAWL|jina.ai|extractMainContentHtml' will hit: content-extractor.ts (Tiers 2/2.5/3 + extractMainContentHtml + Firecrawl guards), feed-poller.ts (pollWebSource Firecrawl), the cloud-run-deploy.yml:324 FIRECRAWL_API_KEY secret mount, and the ontology CV docs/ontology/16-extraction-method.md. All must be retired/updated for the gate to pass.

---

2. Acceptance item 2 — empirical pullmd HTTP-contract findings

Verified 25/05/2026 by WebFetch against https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/README.md (pullmd v2.x), and cross-checked against the two phase-0 docs that stood up a live container (0.8.4-pullmd-evaluation.md §2-§4 and 0.9-spike-S4-pullmd-bakeoff.md §2.3, which both ran docker compose up and hit the live endpoint).

2.1 The real contract (pullmd v2.x)

Aspect	Verified value
Convert endpoint	GET /api?url=<encoded> (NOT POST /extract).
Query params	url (required), format (md|text|json, default md), nocache (bypass 1h cache), render (force|skip Playwright), frontmatter, plus Reddit params (comments, comment_depth 1-10, comment_limit, lang).
Response body	Raw Markdown with Content-Type: text/markdown by default. ?format=json returns structured JSON. There is NO POST /extract and NO top-level {"markdown": …} on the default path.
Response headers	X-Source ∈ {reddit, cloudflare, readability, trafilatura, playwright}; X-Quality (0.0-1.0 confidence); X-Share-Id (8-hex permalink).
Share-link	GET /s/:id — live endpoint; re-fetches from source if cached row > 1h; serves last-good snapshot if source dead. Same headers as /api.
Auth	PULLMD_AUTH_MODE ∈ {disabled (default), single-admin, multi-user}. When ≠ disabled, /api is protected. Token via Authorization: Bearer pmd_<32-char-base62> (SHA-256 hashed server-side) OR pullmd_session cookie (7-day, HttpOnly, SameSite=Lax).
Health endpoint	NOT documented. No /health in the README. The phase-0 spike used GET / (root) returning 200 as the readiness smoke-check (0.9-spike-S4 §2.3).
Other endpoints	GET /api/stream?url= (SSE), POST /mcp (3 tools), GET /web-reader.zip, GET /help.
Deploy env vars	PLAYWRIGHT_URL (sidecar, ~3.7 GB, optional), TRAFILATURA_URL (sidecar fallback), PULLMD_ADMIN_EMAIL/PULLMD_ADMIN_PASSWORD (bootstrap if auth on), PUBLIC_URL, REDDIT_CLIENT_ID/SECRET/USER_AGENT (optional), DISABLE_PUBLIC_HISTORY.

2.2 Share-id semantics (for the pullmd_share_id provenance contract)

• The 8-hex X-Share-Id is a stable handle, 90-day TTL post-last-write, surfaced on every /api and /s/:id response.
• GET /s/:id is the durable re-read: KH can store pullmd_share_id and later fetch the exact bytes (auto-refresh if >1h, last-good-snapshot if source dead). This is what makes source_documents.storage_path = 'pullmd://<share_id>' a real URI (03-tech-stack.md §7.1) rather than a magic string.
• Quality signal: X-Quality < 0.6 correlates with degraded extraction (the bake-off found all 5 PDF-garbage rows scored 0.5; a partial Google-News extraction scored 0.45). Candidate ingest gate: reject/flag below 0.6. Note pullmd does NOT parse PDFs — it returns binary garbage at X-Quality≈0.5 (bake-off §5.4), so a PDF pre-route (HEAD content-type / .pdf sniff) is required before any pullmd call. (For ID-42 the cocoindex flow already routes PDF→Docling and only .html/.htm→pullmd, so the PDF pre-route is intrinsic to Surface C; it matters for Surface A/B retirement.)

2.3 LOAD-BEARING DRIFT — the current adapter cannot work even with pullmd deployed

scripts/cocoindex_pipeline/adapters.py:96-102 (verbatim):

response = httpx.post(
    f"{pullmd_url}/extract",
    json={"url": url},
    timeout=60.0,
)
response.raise_for_status()
return response.json()["markdown"]

Against the verified contract this is SIGNATURE_DRIFT + BEHAVIOUR_DRIFT:

1. Wrong endpoint/method. POST /extract does not exist. Correct: GET /api?url=<encoded>.
2. Wrong response parse. .json()["markdown"] fails — the default response is a raw text/markdown body (response.text), not JSON. (Either switch to response.text, or pass ?format=json and parse the documented JSON shape — TBD in {42.3}; raw-text is simpler and matches what the bake-off harness consumed.)
3. No auth header. If pullmd is deployed with PULLMD_AUTH_MODE != disabled, the call needs Authorization: Bearer pmd_…. Decision: run pullmd single-admin with a bearer token mounted via Secret Manager, OR disabled behind Cloud Run internal-ingress network isolation (the cocoindex Service is ingress: internal per T8 §P-1).
4. X-Source/X-Quality/X-Share-Id are dropped. httpx.post(...).json()["markdown"] discards the headers ID-42 needs for extraction_method + pullmd_share_id provenance. The rewrite must capture them.

Provenance origin to record (ID-42 acceptance bullet 5): this drift is the same class of failure as the Q-EX2 ExtractByLlm incident (docs/research/cocoindex-1.0.3-extractbyllm-spec-reality-investigation.md): a spec/code artefact cited an external API shape that was never empirically exercised against the running service. The _pullmd_to_markdown POST /extract shape was authored from the T8 spec sketch, not from the pullmd README — and the README says GET /api?url=. Caught here at {42.1} per OQ-3 rather than at Executor time.

2.4 Implication for the cocoindex Inv-6 health probe

T8 TECH Inv-6 asserts GET https://<service-url>/health → 200. pullmd has no documented /health. For ID-42, the pullmd Service readiness check should be GET / (root, 200) — NOT /health. {42.3} must specify the smoke-verify command accordingly (mirrors the existing cloud-run-deploy.yml “Smoke verify” gcloud run … describe pattern, but for a Service not a Job).

---

3. Acceptance item 3 — T7/T14 disposition recommendation

3.1 What the canonical PLAN says (verified)

• PLAN §4.14 T14 (Cocoindex absorption cleanup): scope-in deletes lib/intelligence/content-extractor.ts (Tier 2/2.5/3), lib/extraction/, lib/extraction/html.ts, and removes @mozilla/readability + jsdom + unpdf + @mendable/firecrawl-js. Dependencies (verbatim, PLAN line 493): “T8 stable (cocoindex flow ingesting in production) + T7 complete (Phew first-ingest validated).”
• PLAN §4.7 T7 (Phew Q&A first-ingest): stages Phew’s Q&A markdown source files into the cocoindex-tracked localfs folder → cocoindex emits q_a_extractions → promote-all to q_a_pairs. T7 depends on T8 (PLAN line 324). T7 is a markdown-passthrough path (_passthrough_markdown) — it does NOT exercise pullmd / the HTML adapter at all.
• Neither T7 nor “a stable HTML ingest path” is a tracked Task in task-list.json today (33 top-level tasks; ID-42 is spec_needed, 0 subtasks).

3.2 The actual dependency structure

T14’s gate couples two independent things under one “stable” umbrella:

• (a) HTML cascade retirement depends on the HTML→pullmd path working in prod (Surfaces A/B → pullmd). This is exactly ID-42’s scope. It has nothing to do with T7 — T7 ingests Q&A markdown (passthrough), never touching pullmd or the cascade.
• (b) Q&A-markdown first-ingest (T7) depends on the cocoindex flow + the Q&A extraction contract, NOT on HTML extraction.

The PLAN folded both into “T14 after T7” because T14 is a bulk cleanup that also deletes lib/bid-library-ingest/extract-qa-pairs.ts (the Pattern A/B Q&A parser) — and THAT deletion genuinely depends on T7 (you can’t delete the legacy Q&A parser until Phew Q&A is re-ingested via cocoindex). But the HTML-cascade subset of T14 does not.

3.3 Recommendation (for Liam to ratify — NOT decided here)

Recommend: scope ID-42 to the HTML-extraction subset only, independent of T7. Concretely:

• ID-42 makes HTML ingest work end-to-end through deployed pullmd (deploy + adapter fix + schema + wiring) and retires the HTML surfaces: content-extractor.ts Tier 2/2.5/3
  • extractMainContentHtml + Firecrawl/Jina + (decision) lib/extraction/html.ts HTML branch + the 3 HTML-only npm deps (@mozilla/readability, jsdom, @mendable/firecrawl-js).
• Explicitly out of ID-42: deletion of lib/bid-library-ingest/extract-qa-pairs.ts (the Q&A Pattern A/B parser) and scripts/kb_pipeline/ — those stay gated on T7 (Phew Q&A re-ingest) per the PLAN. unpdf removal stays gated on Surface-B PDF disposition.
• ID-42’s “stable HTML path in prod” then becomes the satisfied half of T14’s gate; when T7 lands later, the residual Q&A-parser cleanup completes T14.

This avoids ID-42 either (i) waiting on an untracked T7 it does not actually depend on, or (ii) deleting the Q&A parser before Phew Q&A is re-ingested (data-loss risk).

Scoping note (do NOT resolve here): a parallel investigation is assessing the broader “full Phew corpus reingest” strategy. This T7 analysis is scoped strictly to ID-42’s own dependency on a stable HTML path. Flagged, not resolved: whether the existing sector-intelligence feed_articles corpus + the P4-ingested content_items should be re-ingested through pullmd after the swap (the bake-off §5.5 noted Google-News rows where Firecrawl returned 0 words that pullmd recovers) is a corpus-reingest question that belongs to that parallel investigation, not ID-42.

3.4 Crisp question for Liam

ID-42 / T7-T14 scope split. The canonical PLAN gates T14 cascade-retirement on “T8 stable + T7 (Phew Q&A first-ingest) complete,” but T7 is a Q&A-markdown path that never exercises pullmd or the HTML cascade — the cascade’s only real dependency is a working HTML→pullmd path, which IS ID-42. Do you want ID-42 to retire the HTML extraction surfaces (content-extractor.ts Tier 2/2.5/3, Firecrawl, Jina, extractMainContentHtml, and the HTML branch of lib/extraction/) independently of T7 — leaving the Q&A-parser deletion (lib/bid-library-ingest/extract-qa-pairs.ts) and scripts/kb_pipeline/ removal gated on T7 as the PLAN specifies? (Recommended: yes — split. Alternative: fold the whole HTML+Q&A retirement into one post-T7 Task, which blocks ID-42’s cascade-retirement element on an untracked T7.)

---

4. Acceptance item 4 — Cloud Run deploy-mechanics findings

4.1 How the existing pipeline deploys (the pattern ID-42 extends)

.github/workflows/cloud-run-deploy.yml (verified):

• Triggers: push on main → kh-prod-494815 (Production env); push on production-readiness → kh-staging-494815 (Staging env); workflow_dispatch → manual. Path-trigger allowlist: scripts/**/*.py, requirements.txt, cloudrun/**, .gcloudignore, the workflow file itself.
• Auth: WIF (google-github-actions/auth@v2) using vars.GCP_WIF_PROVIDER + vars.GCP_DEPLOY_SA_EMAIL (stored as Environment variables, not secrets — non-sensitive resource IDs). No long-lived JSON keys.
• Build: gcloud builds submit --config=cloudrun/cloudbuild.yaml (main pipeline image) and --config=cloudrun/cloudbuild-cocoindex.yaml (cocoindex image — pullmd EXCLUDED, see §4.2). Both pass --service-account (post-April-2024 GCP has no Cloud Build default SA).
• Deploy: gcloud run jobs replace cloudrun/jobs/{env}-{tenant}.yaml (Jobs) + gcloud run services replace cloudrun/services/{env}-{tenant}-cocoindex.yaml (cocoindex Services). Per-tenant loop for TENANT in phew kpf.
• Secret mount (the key step, lines 292-355): gcloud run jobs update --set-secrets for the 11 base secrets on Jobs; gcloud run services update --set-secrets for the cocoindex Services with COCOINDEX_SECRETS = <11 base>,PIPELINE_RUN_WEBHOOK_URL=…, PULLMD_SERVICE_URL=PULLMD_SERVICE_URL:latest (line 345). --set-secrets is declarative — it REPLACES the whole mount set each run. Runtime SA ({phew,kpf}-pipeline-sa) needs roles/secretmanager.secretAccessor.

4.2 The pullmd-deploy gap (verified)

• pullmd is deliberately NOT in the cocoindex image. T8 §P-1 + Inv-9 assert zero pullmd binaries + zero Playwright deps in kh-cocoindex-pipeline:latest (the AGPL network-service boundary O-Q3). cloudbuild-cocoindex.yaml comment (workflow line 197): “the AGPL boundary (pullmd excluded)”. So pullmd MUST be its own Service/image — it is not bundled anywhere today.
• There is no pullmd manifest, image, or build. No cloudrun/services/*pullmd*.yaml, no cloudbuild-pullmd.yaml, no pullmd entry in Artifact Registry. The ONLY pullmd artefact in the deploy chain is the PULLMD_SERVICE_URL secret reference at line 345 — which resolves to the S258 placeholder.
• PULLMD_SERVICE_URL is a placeholder secret. Value https://pullmd-not-yet-deployed-staging.example.com (S258, created solely to let the cocoindex Service deploy succeed — gcloud run services update --set-secrets fails if a referenced secret version does not exist). adapters.py:_pullmd_to_markdown raises RuntimeError when unset; when set to the placeholder, every HTML extraction HTTP call fails (unreachable host → on top of the §2.3 contract bug).

4.3 What ID-42 deploy work entails (findings, not the plan)

pullmd ships as Docker images (aeternalabshq/pullmd ~80 MB base) + a Playwright sidecar (~3.7 GB) + a Trafilatura sidecar — a three-container compose topology (0.8.4-pullmd-evaluation.md §2). Cloud Run runs one container per Service, so the deploy shape is a {42.3}-level decision among:

• (α) Single pullmd Service with Playwright + Trafilatura as additional containers in the same Service (Cloud Run multi-container/sidecar support) — closest to upstream compose, but bumps memory/cold-start (the 3.7 GB Playwright image needs min_instances=1 to avoid cold-start pain per 0.8.4 §8.4).
• (β) pullmd Service that calls hosted Playwright/Trafilatura via PLAYWRIGHT_URL / TRAFILATURA_URL (separate Services) — more Services, finer scaling.
• (γ) pullmd-only Service with Playwright disabled (Trafilatura/Readability only) — lightest (~80 MB), but loses JS-render + the Reddit/Cloudflare wins the bake-off showed. Likely too lossy for v1.

Deploy-mechanics facts ID-42 must honour (all verified against the existing workflow):

• New cloudrun/services/{prod,staging}-pullmd.yaml manifest(s). The cocoindex deploy is per-tenant (phew/kpf); pullmd is a shared Service per deployment region (0.8.4 §8 — public-web content, cross-tenant cache hits valuable, simpler ops), so ID-42 likely deploys one shared pullmd Service per env, not per-tenant. This diverges from the per-tenant cocoindex pattern and should be ratified in {42.2}/{42.3}.
• New cloudbuild-pullmd.yaml OR direct gcloud run deploy --image aeternalabshq/pullmd:<pinned-tag> from the public image (pin a version, not :latest — the eval noted :latest lagged behind 2.0.0).
• Path-trigger allowlist must add cloudrun/services/*pullmd* (and cloudbuild-pullmd.yaml if added) so pushes touching pullmd manifests trigger the workflow.
• PULLMD_SERVICE_URL secret value must be replaced (not added) in both kh-staging-494815 and kh-prod-494815 Secret Manager with the real Cloud Run Service URL after first deploy. If pullmd uses auth, a PULLMD_API_TOKEN-style secret is added to the COCOINDEX_SECRETS mount loop (line 345).
• Network isolation: cocoindex Services are ingress: internal. If pullmd is also ingress: internal + same VPC, PULLMD_AUTH_MODE=disabled is defensible (no public exposure). If pullmd needs public ingress (e.g. for the share-link UX), it needs auth. Decision for {42.2}/{42.3}.
• Smoke-verify: gcloud run services describe <pullmd-service> + a GET / 200 check (NOT /health — §2.4).

---

5. Acceptance item 5 — schema-column plan

5.1 Empirical schema state (verified §6)

• source_documents columns (live, database.types.ts:2877-2940): archived_at, archived_by, content_hash, created_at, extracted_text, extraction_metadata (Json), file_size, filename, id, mime_type, op_id, original_filename, parent_id, pipeline_run_id, status, storage_path, uploaded_by, version, workspace_id. No pullmd_share_id. No extraction_method.
• feed_articles.extraction_method EXISTS (database.types.ts:1228, string | null), CHECK enum {rss_content, fetch, jina_reader, firecrawl, summary_fallback} per docs/ontology/16-extraction-method.md. This is the sector-intelligence column; it is NOT on source_documents and NOT written by the cocoindex flow.
• No migration anywhere references pullmd_share_id / pullmd_readability / pullmd_playwright / pullmd_cloudflare (verified — zero grep hits in supabase/migrations/).

5.2 The two-table reality (load-bearing for {42.2}/{42.3})

The ID-42 acceptance criteria conflate two different extraction-method concepts:

Concept	Table	Status
Sector-intelligence article extraction route	feed_articles.extraction_method	EXISTS; enum needs pullmd_* values added IF Surface A is re-pointed at pullmd.
Canonical-pipeline source provenance	source_documents (extraction_metadata Json today)	NO typed extraction_method column; NO pullmd_share_id. Both net-new.

Both surfaces need attention, and they are different DDL. The ID-42 acceptance bullet “extraction_method reflecting the pullmd source header” must be pinned to a specific table in {42.2}. Recommendation to carry into {42.2}/{42.3}:

1. source_documents.pullmd_share_id text NULL + partial index WHERE pullmd_share_id IS NOT NULL — per 0.8.4-pullmd-evaluation.md §5.4 Option A (identity-shaped data deserves a column; queryable, drift-resistant). This is the provenance handle backing storage_path = 'pullmd://<share_id>'.
2. extraction_method on source_documents — recommend a typed column added with a CHECK enum that extends the existing 5 values with pullmd_readability, pullmd_playwright, pullmd_cloudflare, pullmd_reddit, pullmd_trafilatura, docling (mapping pullmd’s X-Source header values + Docling for binary). Alternative: keep extraction provenance in the existing extraction_metadata JSONB ({pullmd:{share_id,x_source,x_quality,fetched_at}}) — zero DDL but not indexable. {42.3} ratifies typed-column-vs-JSONB; the gap-analysis intel’s wording (“columns on source_documents”) leans typed-column.
3. feed_articles.extraction_method enum extension — IF Surface A (sector intel) is re-pointed at pullmd, the CHECK constraint feed_articles_extraction_method_check must add pullmd_* values, and CV docs/ontology/16-extraction-method.md baseline_values updated (the CV already records firecrawl retired-S236 + the Docling/pullmd direction in §Notes — so this is a sanctioned change, not a surprise).

5.3 Confirm-in-scope verdict

pullmd_share_id + source_documents.extraction_method are legitimately in ID-42 scope: cascade retirement strands the provenance contract otherwise (today feed_articles.extraction_method records which extractor ran; if Surface A retires to pullmd and writes to source_documents/content_items via the cocoindex path, the provenance must land somewhere typed). Migration discipline (CLAUDE.md): DDL via supabase migration new + db push only (never MCP); cat supabase/.temp/project-ref before push; any new PL/pgSQL fn needs SET search_path = public, extensions + the REVOKE-anon pattern. Schema-parity (schema-parity.yml) must show prod↔staging match after the migration. The op_id migration precedent (source_documents.op_id added T8 §P-4) shows the ADD COLUMN IF NOT EXISTS + partial-index pattern this should mirror.

---

6. Verification (OQ-3 — empirical import/inspect/fetch checks)

All checks run 25/05/2026 in the worktree (main@f7767c92).

#	Subject	Method	Pin / source	Result
V1	pullmd v2.x HTTP contract	WebFetch of AeternaLabsHQ/pullmd README (main)	pullmd v2.x	GET /api?url= returns raw text/markdown; headers X-Source/X-Quality/X-Share-Id; GET /s/:id share-link; auth via Bearer pmd_…; no /health. Verified — adapter’s POST /extract+.json()["markdown"] is SIGNATURE_DRIFT+BEHAVIOUR_DRIFT.
V2	cocoindex pin	python3 -c "import cocoindex; print(cocoindex.__version__)"	requirements.txt: cocoindex[postgres]==1.0.3	1.0.3 — PRESENT, pin matches.
V3	httpx (adapter import)	python3 -c "import httpx; print(httpx.__version__)"	transitive (not pinned in requirements.txt)	0.28.1 — PRESENT. Note: httpx is imported by adapters.py but NOT explicitly pinned in requirements.txt; {42.3} should pin it (it ships transitively via cocoindex today, fragile).
V4	source_documents.pullmd_share_id	grep database.types.ts lines 2877-2940 + supabase/migrations/	live generated types	ABSENT — column does not exist; never migrated.
V5	source_documents.extraction_method	grep database.types.ts 2877-2940	live generated types	ABSENT on source_documents.
V6	feed_articles.extraction_method	grep database.types.ts:1228 + docs/ontology/16-extraction-method.md	live generated types	PRESENT (string|null), enum {rss_content,fetch,jina_reader,firecrawl,summary_fallback}.
V7	pullmd_* extraction-method values	grep supabase/migrations/	live migrations	ABSENT — zero hits for pullmd_readability/pullmd_playwright/pullmd_cloudflare/pullmd_share_id.
V8	PULLMD_SERVICE_URL placeholder + mount site	read cloud-run-deploy.yml:345 + adapters.py:90-95	workflow + adapter	Mounted at line 345 in COCOINDEX_SECRETS; adapter raises RuntimeError when unset. Placeholder value per ID-42 brief (S258). PRESENT as reference, value is placeholder.
V9	extractContent sole prod caller	grep lib/,app/,scripts/ (non-test)	source	pipeline.ts:372 only (other hits are tests/comments). PRESENT — single live caller.
V10	npm deps to retire	grep package.json	source	@mendable/firecrawl-js ^4.23.0, @mozilla/readability ^0.6.0, jsdom ^28.1.0, unpdf ^1.6.2 — all PRESENT.
V11	cocoindex ExtractByLlm precedent	cross-ref docs/research/cocoindex-1.0.3-extractbyllm-spec-reality-investigation.md	research doc	Confirms the same drift-class; pullmd-adapter drift caught here BEFORE Executor (the desired OQ-3 outcome).

Escalation check: the pullmd contract drift (V1) is SIGNATURE_DRIFT/BEHAVIOUR_DRIFT, NOT ABSENT-blocking for THIS research artefact — pullmd exists and the correct contract is now documented. The fix is a spec/code correction owned by {42.3}/Executor, not a version-pin upgrade or a research blocker. Recorded inline (§2.3) per OQ-3 BEHAVIOUR_DRIFT handling; no STOP required for {42.1}.

---

7. Recommendations for {42.2} PRODUCT (fresh Planner)

1. State the three surfaces explicitly (A live-prod cascade, B live-prod URL-ingest, C code-dead cocoindex adapter) and decide per-surface in/out of ID-42 scope. Recommend: A + B HTML branches IN; PDF/binary OUT (stays Docling/unpdf); Q&A-parser deletion OUT (gated on T7).
2. Behaviour invariants must cover (testable by Checker): (a) pullmd Service resolves at stable HTTPS in both envs; (b) PULLMD_SERVICE_URL reflects the real URL in both; (c) one HTML source ingests end-to-end → non-empty content_items.content with the pullmd-sourced extraction_method; (d) pullmd_share_id present on the produced source_documents row; (e) post-retirement grep firecrawl|FIRECRAWL|jina.ai|extractMainContentHtml returns zero non-test hits; (f) sector-intelligence cron still ingests post-swap (regression).
3. Sequencing invariant: retirement (element 3) MUST be gated on elements 1+2 being green in prod — encode “no deletion before HTML ingest verified end-to-end through deployed pullmd” as an explicit invariant.
4. Carry the two open ratifications to {42.2}: (i) shared-vs-per-tenant pullmd Service; (ii) extraction_method typed-column-vs-JSONB on source_documents.

8. Recommendations for {42.3} TECH

1. Rewrite _pullmd_to_markdown to GET {PULLMD_SERVICE_URL}/api?url=<encoded>, read response.text (or ?format=json), capture X-Source/X-Quality/X-Share-Id, add Authorization: Bearer if auth-on. Return shape must carry the headers up to the write site (today it returns bare str).
2. Pin httpx in requirements.txt (V3 — currently transitive-only).
3. pullmd Service deploy mechanics — manifest(s), build/image-pin, path-trigger allowlist, Secret Manager replace, GET / smoke-check (NOT /health), ingress/auth posture.
4. Migration — pullmd_share_id + (decision) extraction_method on source_documents via ADD COLUMN IF NOT EXISTS + partial index (mirror T8 §P-4 op_id migration); feed_articles.extraction_method CHECK extension if Surface A re-points; CV 16-extraction-method.md update.
5. Proposed-changes mapping must be one-to-one against {42.2} invariants, with the KH quality bars baked in (semantic tokens N/A here; UK English; no silent Supabase failures via sb()/tryQuery(); bun run test; tests verify real behaviour per docs/reference/test-philosophy.md).

---

Appendix A — files read (all absolute)

• /Users/liamj/Documents/development/knowledge-hub/lib/intelligence/content-extractor.ts
• /Users/liamj/Documents/development/knowledge-hub/lib/intelligence/pipeline.ts
• /Users/liamj/Documents/development/knowledge-hub/lib/intelligence/feed-poller.ts
• /Users/liamj/Documents/development/knowledge-hub/lib/extraction/url.ts
• /Users/liamj/Documents/development/knowledge-hub/scripts/cocoindex_pipeline/adapters.py
• /Users/liamj/Documents/development/knowledge-hub/scripts/cocoindex_pipeline/flow.py (grep)
• /Users/liamj/Documents/development/knowledge-hub/.github/workflows/cloud-run-deploy.yml
• /Users/liamj/Documents/development/knowledge-hub/docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md (§4.7, §4.8, §4.14)
• /Users/liamj/Documents/development/knowledge-hub/docs/specs/id-28-cocoindex-flow-scaffolding/TECH.md (§P-1, §P-3, §P-4, §P-6, Inv-6/Inv-9)
• /Users/liamj/Documents/development/knowledge-hub/docs/plans/phase-0-investigation/0.8.4-pullmd-evaluation.md
• /Users/liamj/Documents/development/knowledge-hub/docs/plans/phase-0-investigation/0.9-spike-S4-pullmd-bakeoff.md
• /Users/liamj/Documents/development/knowledge-hub/docs/plans/phase-0-investigation/architecture/03-tech-stack.md (§3, §4, §6, §7)
• /Users/liamj/Documents/development/knowledge-hub/docs/ontology/16-extraction-method.md
• /Users/liamj/Documents/development/knowledge-hub/docs/research/cocoindex-1.0.3-extractbyllm-spec-reality-investigation.md
• /Users/liamj/Documents/development/knowledge-hub/supabase/types/database.types.ts (source_documents, feed_articles)
• /Users/liamj/Documents/development/knowledge-hub/docs/reference/task-list.json (ID-42 record)

Appendix B — external sources fetched

• https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/README.md — pullmd v2.x HTTP contract (25/05/2026; V1).

---

End of {42.1} RESEARCH. Returns to Orchestrator. Two ratifications surfaced for Liam (T7/T14 split §3.4; shared-vs-per-tenant Service + typed-column-vs-JSONB §7.4). One load-bearing drift recorded (pullmd adapter contract §2.3) — caught at spec time per OQ-3, not at Executor time.