PRODUCT — {42.2} Deploy pullmd Cloud Run Service + retire current HTML extraction cascade

PRODUCT — {42.2} Deploy pullmd Cloud Run Service + retire current HTML extraction cascade

Task: ID-42 — Deploy pullmd Cloud Run Service + retire current HTML extraction cascade. Spec-chain artefact: {42.2} PRODUCT (second of {42.1}→{42.2}→{42.3}→{42.4}). Date: 25/05/2026 (S262 / ID-42 spec wave). Author: Task Planner (Opus 4.7, 1M context) — fresh instance per Q-PLANNER-2; NOT the {42.1} RESEARCH author. This is a fresh review pass over the ratified RESEARCH. Predecessor: docs/specs/id-42-pullmd-deploy/RESEARCH.md (Checker-PASS, main@04dac485). Status: PRODUCT draft. Returns to Orchestrator for ratification before {42.3} TECH.

UK English throughout. Behaviour invariants are numbered and individually testable so a Checker can acceptance-verify each against the deployed Service, a real HTML ingest, and the retirement grep gate.

---

Summary

Deploy the pullmd HTML→Markdown extraction Service to Cloud Run in both kh-staging-494815 and kh-prod-494815, point the cocoindex pipeline at it via a real PULLMD_SERVICE_URL, prove one HTML source ingests end-to-end through deployed pullmd (non-empty body + pullmd provenance recorded), and only THEN retire the legacy HTML extraction surfaces (the live 4-tier sector-intelligence cascade and the live URL-ingest HTML branch). PDF/binary extraction stays on Docling/unpdf; the Q&A-parser deletion and scripts/kb_pipeline/ removal are out of scope (separately tracked T7/T14 Tasks).

Problem

PULLMD_SERVICE_URL is an S258 placeholder (pullmd-not-yet-deployed-…); pullmd was never deployed, so the cocoindex HTML path is runtime-dead. Meanwhile the live HTML extraction path is a 4-tier cascade (rss_content → direct-fetch regex → Jina → Firecrawl → summary-fallback) plus a separate Readability/JSDOM URL-ingest branch. Liam ratified (S262) that ID-42 retires the HTML extraction surfaces independently of T7 — the T7/T14 split is approved, so T7 (full-corpus reingest) and T14 (collapse-list cleanup) are now separately-tracked Tasks. ID-42’s job is the HTML surfaces only.

Scope (ratified S262)

In scope

• Deploy pullmd as a Cloud Run Service in staging + prod.
• Replace the PULLMD_SERVICE_URL placeholder with the real Service URL in both envs.
• Make HTML ingest work end-to-end through deployed pullmd (the cocoindex .html/.htm→pullmd path), capturing pullmd provenance (pullmd_share_id + pullmd-sourced extraction method).
• Net-new schema for the provenance contract (pullmd_share_id on source_documents; the extraction-method column placement is OQ-2 below — do NOT pre-decide it here).
• Retire the HTML branches of the legacy surfaces (see per-surface table) — gated on the end-to-end ingest proof.

Out of scope (explicitly)

• PDF / binary extraction — stays on Docling (cocoindex) / unpdf (URL-ingest). pullmd does not parse PDFs (returns binary garbage at quality ≈ 0.5; RESEARCH §2.2).
• Q&A-parser deletion (lib/bid-library-ingest/extract-qa-pairs.ts) and scripts/kb_pipeline/ removal — gated on T7 (Phew Q&A re-ingest). Not ID-42.
• Full-corpus / feed_articles reingest through pullmd — belongs to T7. ID-42 proves one HTML source ingests; it does not backfill the existing corpus.
• unpdf removal — gated on Surface-B PDF disposition (a later Task).

Per-surface disposition (the three live HTML surfaces, RESEARCH §1)

Surface	What it is	file (entrypoint)	HTML branch in ID-42?
A	P9 sector-intelligence 4-tier cascade (LIVE in prod) — rss_content / fetch (extractMainContentHtml regex) / jina_reader / firecrawl / summary_fallback	lib/intelligence/content-extractor.ts::extractContent()	IN — retire Tier 2 fetch+extractMainContentHtml, Tier 2.5 jina_reader, Tier 3 firecrawl (decision: see Inv-10/Inv-11 on Tier 1 rss_content + summary_fallback preservation)
B	P4 URL-ingest HTML branch (LIVE in prod) — JSDOM + Mozilla Readability + Turndown	lib/extraction/url.ts::extractFromUrl() → lib/extraction/html.ts::extractFromHtml()	IN (HTML branch only) — re-point at pullmd; keep the PDF branch (unpdf/extractPdfText)
C	cocoindex HTML→pullmd adapter (CODE EXISTS, runtime-dead)	scripts/cocoindex_pipeline/adapters.py::_pullmd_to_markdown()	IN — this is the path made live (deploy + adapter contract fix). It is the target of the swap, not a retirement target.

Surfaces A + B are retired (HTML branches) once Surface C is proven live end-to-end.

Behaviour

Group 1 — pullmd Service deployment

1. (Service resolves — staging.) A pullmd Cloud Run Service exists in kh-staging-494815 and resolves at a stable HTTPS URL (https://…run.app or a mapped custom domain). The URL is stable across redeploys — a subsequent revision deploy does not change the resolvable base URL the cocoindex Service consumes.

   • Acceptance: gcloud run services describe <pullmd-service> --project kh-staging-494815 --format='value(status.url)' returns one HTTPS URL; a readiness probe GET <url>/ returns HTTP 200 (NOT /health — pullmd has no documented /health endpoint; RESEARCH §2.4). Two consecutive describes (before/after a no-op redeploy) return the same URL.

2. (Service resolves — prod.) The same holds in kh-prod-494815: a pullmd Service resolves at a stable HTTPS URL and GET <url>/ returns 200.

   • Acceptance: the Inv-1 commands against --project kh-prod-494815.

3. (Pinned image, not :latest.) The deployed pullmd Service runs a pinned image tag, not a floating :latest — the phase-0 eval found :latest lagged behind 2.0.0 (RESEARCH §4.3).

   • Acceptance: gcloud run services describe … --format='value(spec.template.spec.containers[0].image)' shows a concrete version tag (e.g. …/pullmd:2.x.y), not …:latest.

4. (Deploy is reproducible from source control.) pullmd deploy is wired into the existing deploy mechanism (.github/workflows/cloud-run-deploy.yml + a cloudrun/services/*pullmd* manifest and/or build config), and the workflow’s path-trigger allowlist fires on changes to the pullmd manifest. A redeploy from a clean checkout reproduces the same Service (no manual console-only steps).

   • Acceptance: a cloudrun/services/{staging,prod}-pullmd*.yaml (or equivalent) exists under source control; the workflow path-trigger list includes the pullmd manifest glob; re-running the deploy job produces a healthy Service.

Group 2 — wiring (PULLMD_SERVICE_URL reflects reality)

5. (PULLMD_SERVICE_URL reflects the real URL — both envs.) The PULLMD_SERVICE_URL Secret Manager value in kh-staging-494815 equals the real staging pullmd Service URL, and the value in kh-prod-494815 equals the real prod pullmd Service URL. Neither retains the S258 placeholder (https://pullmd-not-yet-deployed-…example.com).

   • Acceptance: gcloud secrets versions access latest --secret=PULLMD_SERVICE_URL --project <env> returns a URL that (a) matches the Inv-1/Inv-2 Service URL for that env and (b) contains no not-yet-deployed / example.com substring. The cocoindex Service revision that consumes the mount (cloud-run-deploy.yml:345, COCOINDEX_SECRETS=…,PULLMD_SERVICE_URL=PULLMD_SERVICE_URL:latest) has been re-deployed so the running revision sees the new value.

6. (cocoindex can reach pullmd.) From the cocoindex Service’s network position, an HTTP call to PULLMD_SERVICE_URL succeeds (no unreachable-host / DNS / auth-rejection error). If pullmd runs with auth enabled, the cocoindex Service presents valid credentials and is accepted; if pullmd runs auth-disabled behind internal ingress, the call is reachable without credentials.

   • Acceptance: a smoke call (GET {PULLMD_SERVICE_URL}/api?url=<a-known-good-html-url>) issued from the cocoindex Service context returns HTTP 200 with a non-empty text/markdown body. (Auth posture is OQ-1 / a {42.3} decision; this invariant only asserts the call is accepted, however auth is configured.)

Group 3 — HTML ingest works end-to-end through deployed pullmd

7. (One HTML source ingests → non-empty body.) Ingesting one real HTML source through the cocoindex .html/.htm→pullmd path produces a content_items row whose body (content_items.content / content_text) is non-empty and contains Markdown-shaped text extracted from that source (not raw HTML, not an empty string, not a placeholder).

   • Acceptance: stage a known HTML source through the cocoindex flow against the deployed pullmd Service; query the produced content_items row and assert length(content) > 0 and that the body is recognisably the source’s main content rendered as Markdown. No mocking of pullmd — the call hits the deployed Service.

8. (Pullmd-sourced extraction method recorded.) The produced row records an extraction-method value that identifies pullmd as the extractor — derived from pullmd’s X-Source response header (one of readability / trafilatura / playwright / cloudflare / reddit), surfaced as a pullmd_* extraction-method value (e.g. pullmd_readability).

   • Acceptance: the produced row’s extraction-method field equals a pullmd_* value consistent with the live X-Source header for that fetch. The target table/column for this value is OQ-2 (below) — this invariant asserts the value is recorded and pullmd-attributed wherever OQ-2 lands it; it does NOT pre-decide the column.

9. (pullmd_share_id present on the produced source_documents row.) The source_documents row produced by the HTML ingest carries a non-null pullmd_share_id equal to pullmd’s X-Share-Id response header (the 8-hex permalink handle), and that handle round-trips: GET {PULLMD_SERVICE_URL}/s/<share_id> re-serves the extracted Markdown.

   • Acceptance: the produced source_documents row has pullmd_share_id matching the X-Share-Id returned by the live fetch; a follow-up GET …/s/<share_id> returns 200 with non-empty Markdown. (source_documents.pullmd_share_id is net-new DDL — RESEARCH §5.1 / verified ABSENT today; this invariant presumes the migration has landed.)

Group 4 — sequencing gate (LOAD-BEARING)

10. (No deletion before verified end-to-end ingest.) No legacy extraction surface (Surface A Tier 2/2.5/3, Surface B HTML branch, any HTML-extraction npm dependency, any Firecrawl/Jina call site) is deleted or disabled in main until Inv-7, Inv-8 and Inv-9 are demonstrably green in prod. Retirement commits MUST be ordered strictly after the end-to-end-ingest proof; a single commit that both deletes a legacy surface AND first introduces the pullmd path is a violation (the two must be separable and ordered).
    • Acceptance: the commit/PR history shows the pullmd-ingest-proof landing before any surface-deletion commit; at no point on main does a state exist where the legacy HTML cascade is removed but pullmd HTML ingest is unproven. The Checker can verify by bisecting: at the deletion commit’s parent, Inv-7/8/9 already pass.

Group 5 — retirement (gated on Group 4)

11. (Surface A HTML tiers retired.) After the gate, the sector-intelligence cascade no longer performs HTML extraction via direct-fetch regex (extractMainContentHtml), Jina (r.jina.ai), or Firecrawl (@mendable/firecrawl-js). The HTML→Markdown work for web/RSS sources flows through pullmd.

    • Acceptance: extractMainContentHtml, the r.jina.ai fetch, and the @mendable/firecrawl-js cascade-Tier-3 call are gone from lib/intelligence/content-extractor.ts; the Firecrawl scrape in lib/intelligence/feed-poller.ts::pollWebSource() is gone or re-pointed at pullmd.

12. (Tier 1 rss_content + summary_fallback disposition is explicit.) The cascade’s Tier 1 rss_content path (inline content:encoded, no URL fetch — has no pullmd equivalent) and Tier 4 summary_fallback are either preserved or explicitly retired by deliberate decision, not deleted as collateral of the HTML-tier removal. (RESEARCH §1 flags rss_content as having no pullmd equivalent.)

    • Acceptance: {42.3}/PLAN states the decision for each; the implemented behaviour matches. The Checker verifies an RSS source with inline content:encoded still ingests if rss_content is preserved, OR that its retirement is documented and the replacement path works.

13. (Surface B HTML branch retired; PDF branch intact.) After the gate, app/api/ingest/url HTML extraction flows through pullmd, and the JSDOM/Mozilla-Readability HTML branch (lib/extraction/html.ts::extractFromHtml) is retired. The PDF branch of extractFromUrl (unpdf/extractPdfText) still works.

    • Acceptance: ingesting an HTML URL via app/api/ingest/url produces a non-empty extraction via pullmd; ingesting a PDF URL still produces a non-empty extraction via the PDF branch; extractFromHtml (JSDOM/Readability) is removed or no longer reached for HTML.

14. (npm HTML-extraction dependencies removed.) The HTML-only extraction npm dependencies are removed from package.json: @mendable/firecrawl-js, @mozilla/readability, jsdom. unpdf is NOT removed by ID-42 (PDF branch / OQ out of scope).

    • Acceptance: package.json no longer lists @mendable/firecrawl-js, @mozilla/readability, or jsdom; unpdf remains. bun install succeeds and the app builds.

15. (Retirement grep gate — scoped.) After retirement, a grep for firecrawl|FIRECRAWL|jina.ai|extractMainContentHtml over the live HTML-extraction code paths returns zero hits. Because a fresh-pass grep across lib/,app/,scripts/ finds eight non-test files today, the gate is scoped to the in-ID-42 surfaces and the legitimately-out-of-scope matches are enumerated as explicit exclusions so the gate is unambiguous:

    MUST be zero after retirement (in-scope live HTML surfaces):

    • lib/intelligence/content-extractor.ts (Tier 2/2.5/3 + extractMainContentHtml + Firecrawl guards checkFirecrawlApiKey/isFirecrawlConfigured)
    • lib/intelligence/feed-poller.ts (pollWebSource Firecrawl scrape)
    • lib/intelligence/types.ts (the 'firecrawl' extraction-method union member — removed alongside the cascade)
    • app/api/intelligence/workspaces/[id]/sources/[sourceId]/test/route.ts (only if it carries a Firecrawl/Jina reference after the cascade swap — verify post-retirement)
    • the cloud-run-deploy.yml FIRECRAWL_API_KEY secret mount

    Explicitly EXCLUDED from the gate (legitimately out of ID-42 scope — must NOT block the gate, and must NOT be deleted by ID-42):

    • scripts/kb_pipeline/extract.py (the r.jina.ai line) — scripts/kb_pipeline/ removal is T7-gated, not ID-42.

    • scripts/backfill-source-url-firecrawl.ts — a standalone one-shot migration/backfill script; its disposition is a separate clean-up, not part of the live cascade. (If Liam wants it removed in ID-42, that is an OQ — see note below.)

    • lib/env-server.ts (FIRECRAWL_API_KEY Zod env schema entry) and lib/intelligence/url-validation.ts (a code comment referencing firecrawl) — removed only if the corresponding runtime usage is gone; the comment/schema entry is cosmetic and may trail the runtime removal into {42.3}/PLAN clean-up.

    • any __tests__/** and *.test.ts hits (the gate is non-test by construction).

    • documentation (docs/ontology/16-extraction-method.md, RESEARCH.md, this PRODUCT.md) — docs reference the retired terms by design.

    • Acceptance: the Checker runs the gate as grep -rIn -E 'firecrawl|FIRECRAWL|jina\.ai|extractMainContentHtml' lib/ app/ --include='*.ts' | grep -vE '__tests__|\.test\.' and asserts zero hits in the in-scope set above; any residual hit must fall in the explicit-exclusion list with a recorded rationale, or the gate fails. (Note: scripts/ is intentionally NOT in the gate’s path set because the only scripts/ hits are the T7-gated kb_pipeline extractor and the standalone backfill script.)

Group 6 — regression guard (cascade keeps working through the swap)

16. (Sector-intelligence cron still ingests post-swap.) After the pullmd swap and Surface-A retirement, the scheduled sector-intelligence ingest still runs and still produces feed_articles + (for passed articles) content_items rows. RSS sources, web sources, the manual trigger, the MCP runPipeline tool, and the admin “test this source” endpoint all still ingest.

    • Acceptance: a cron poll (app/api/cron/intelligence-poll) and a manual trigger (app/api/intelligence/trigger-poll) each produce new feed_articles rows for a live source; the admin source-test endpoint (app/api/intelligence/workspaces/[id]/sources/[sourceId]/test) returns a successful dry-run extraction; no entrypoint 500s due to a removed-symbol reference. The full regression suite (bun run test) passes.

17. (Existing feed_articles corpus is not corrupted.) Retirement does not rewrite or invalidate the existing feed_articles rows (including the 27 prod rows recorded with extraction_method='firecrawl', RESEARCH §5 / CV 16-extraction-method.md). Their historical extraction_method values remain readable; the CHECK-enum and CV are only extended (not destructively pruned) by ID-42. Pruning firecrawl from the enum/CV is gated on the corpus-reingest Task (T7), per the Ratified-S237 lockstep rule — ID-42 does not prune.

    • Acceptance: the pre-existing feed_articles rows with extraction_method='firecrawl' are still present and readable after ID-42; the feed_articles_extraction_method_check constraint still admits the legacy values; docs/ontology/16-extraction-method.md still lists the legacy values (extension only, no destructive prune in ID-42).

18. (Tests verify real behaviour.) New/changed tests for the pullmd path assert real extraction behaviour (non-empty body, correct provenance, gate ordering), not implementation shape, per docs/reference/test-philosophy.md. The end-to-end ingest proof (Inv-7/8/9) exercises the deployed Service, not a mock.

Open questions (carry to {42.3} TECH — for Liam to ratify, NOT resolved here)

• OQ-1 — pullmd Service topology + auth posture (shared vs per-tenant). The existing cocoindex deploy is per-tenant (phew/kpf loop). pullmd extracts public-web content where cross-tenant cache hits are valuable and ops are simpler as a single shared Service per env (RESEARCH §4.3 / phase-0 eval §8). This diverges from the per-tenant pattern. Question for Liam: deploy one shared pullmd Service per env (recommended), or one per tenant? Coupled sub-question: auth posture — PULLMD_AUTH_MODE=disabled behind ingress: internal + same VPC as cocoindex (no public exposure, no token), or single-admin/multi-user with a PULLMD_API_TOKEN-style secret added to the COCOINDEX_SECRETS mount? (Inv-6 holds either way; this decides how the call is accepted.)

• OQ-2 — extraction-method storage: typed column vs JSONB on source_documents; and whether feed_articles.extraction_method enum extends. source_documents has no extraction_method column today (verified ABSENT, RESEARCH §5.1); provenance lives in extraction_metadata JSONB. Question for Liam (two parts):

  1. For the canonical pipeline (source_documents/content_items), record the pullmd extraction method as a new typed column source_documents.extraction_method with a CHECK enum (extending the legacy 5 values with pullmd_readability, pullmd_playwright, pullmd_cloudflare, pullmd_reddit, pullmd_trafilatura, docling) — queryable, drift-resistant, indexable — OR keep it in the existing extraction_metadata JSONB ({pullmd:{share_id,x_source,x_quality,fetched_at}}) — zero DDL but not indexable? (The gap-analysis wording “columns on source_documents” leans typed-column; the typed-column choice also makes Inv-8 directly queryable.)
  2. IF Surface A (sector-intelligence) re-points at pullmd and continues writing feed_articles.extraction_method, the feed_articles_extraction_method_check constraint must extend to admit pullmd_* values (the live constraint today is exactly {rss_content, fetch, jina_reader, firecrawl, summary_fallback}), and CV docs/ontology/16-extraction-method.md baseline_values must add them. Per the Ratified-S237 lockstep rule, firecrawl is not pruned from the enum/CV until the corpus-reingest Task (T7) drops it — ID-42 extends, never destructively prunes. Confirm this enum-extension-without-prune is the intended ID-42 boundary.

  (Both OQs are deferred to {42.3} TECH for resolution with Liam; the invariants above are written to hold regardless of which way each OQ lands — Inv-8 asserts the value is recorded “wherever OQ-2 lands it”, and Inv-6 asserts the call is “accepted, however auth is configured”.)

---

End of {42.2} PRODUCT. Returns to Orchestrator for ratification. A fresh Planner authors {42.3} TECH against this PRODUCT (Q-PLANNER-2). Two ratifications carried as OQ-1/OQ-2 for Liam; the load-bearing sequencing gate is Inv-10 (no deletion before verified end-to-end ingest).