TECH — {42.3} Deploy pullmd Cloud Run Service + retire HTML extraction cascade

TECH — {42.3} Deploy pullmd Cloud Run Service + retire HTML extraction cascade

Task: ID-42 — Deploy pullmd Cloud Run Service + retire current HTML extraction cascade. Spec-chain artefact: {42.3} TECH (third of {42.1}→{42.2}→{42.3}→{42.4}). Date: 25/05/2026 (S265 / ID-42 spec wave). Author: Task Planner (Opus 4.7, 1M context) — fresh instance per Q-PLANNER-2; NOT the {42.1} RESEARCH or {42.2} PRODUCT author. Fresh review pass over both ratified predecessors. Predecessors: docs/specs/id-42-pullmd-deploy/PRODUCT.md (18 numbered invariants); docs/specs/id-42-pullmd-deploy/RESEARCH.md (empirical pullmd v2.x contract + schema state). Status: TECH draft. Returns to Orchestrator for ratification. Carries ONE OQ-FOR-PARENT (the cocoindex Stage-6 write-path capability gate for Inv-7/8/9) plus one installed-reality divergence handled in-spec.

UK English throughout. Proposed changes map one-to-one against the 18 PRODUCT invariants (§Proposed changes per invariant). Every proposed change names a concrete, runnable verification. KH quality bars baked in (DDL-via-CLI-only, SET search_path, REVOKE-anon, getAuthorisedClient() discriminated union, sb()/tryQuery() no-silent-failure, no barrel re-exports, bun run test, tests verify real behaviour per docs/reference/test-philosophy.md).

---

RATIFIED DECISIONS (S265 — baked in, NOT re-opened)

These resolve PRODUCT’s OQ-1 and OQ-2. The TECH spec treats them as decided.

1. Topology = per-tenant pattern, PHEW-ONLY. kpf is decommissioned (was dogfooding; no longer exists). Deploy ONE pullmd Cloud Run Service per env (kh-staging-494815 + kh-prod-494815) for the phew tenant only. NO kpf pullmd Service. The pullmd deploy step does NOT iterate the existing for TENANT in phew kpf loop — pullmd is a single phew Service per env. kpf-decommissioned was recorded as a platform fact; the *-kpf-cocoindex.yaml manifests were removed in S274 (Liam ratification) along with their live references in cloudrun/services/README.md and docs/runbooks/cocoindex-deploy.md. The phew kpf loops in cloud-run-deploy.yml remain a separate, tracked clean-up.
2. Auth = single-admin Bearer token. Deploy pullmd with PULLMD_AUTH_MODE=single-admin. Mint Authorization: Bearer pmd_<32-char-base62> (SHA-256 hashed server-side per RESEARCH §2.1), store as a NEW Secret Manager secret PULLMD_API_TOKEN, add it to the COCOINDEX_SECRETS mount loop. Bootstrap admin creds via PULLMD_ADMIN_EMAIL / PULLMD_ADMIN_PASSWORD env on the pullmd Service.
3. Storage = typed column + CHECK enum. Add source_documents.extraction_method (NET-NEW typed column, CHECK enum extending the legacy 5 with the pullmd_* + docling values), plus source_documents.pullmd_share_id text NULL + partial index. Both mirror the T8 §P-4 op_id ADD COLUMN IF NOT EXISTS + partial-index precedent (supabase/migrations/20260521203414_t8_op_id_propagation.sql). Extend feed_articles_extraction_method_check (add pullmd_*, no prune of firecrawl per Ratified-S237 lockstep — that’s T7-gated). Update CV docs/ontology/16-extraction-method.md (extension only).

---

Context

What is being built

Make the cocoindex .html/.htm→pullmd extraction path live by (a) deploying pullmd as a phew-only Cloud Run Service in both envs, (b) rewriting the contract-wrong adapter to the real pullmd v2.x HTTP surface, (c) landing net-new provenance schema on source_documents, (d) wiring the captured pullmd headers to a write site, then — gated strictly behind the end-to-end ingest proof (Inv-10) — retiring the live HTML extraction surfaces (Surface A Tiers 2/2.5/3 + Surface B HTML branch + the 3 HTML-only npm deps).

How the current system works (validated against installed code, 25/05/2026)

Surface C — cocoindex adapter (the swap target). scripts/cocoindex_pipeline/adapters.py:88 _pullmd_to_markdown(url: str) -> str is contract-wrong on two counts (verified verbatim, lines 96–102): it POSTs /extract with json={"url": url} and reads response.json()["markdown"]. The real pullmd contract (re-verified 25/05/2026 against the AeternaLabsHQ/pullmd README, matching RESEARCH §2.1) is GET /api?url=<encoded> returning a raw text/markdown body with headers X-Source/X-Quality/X-Share-Id. The adapter returns a bare str, discarding the headers Inv-8/Inv-9 require. It is routed from adapters.py:55-58 (convert_binary_to_markdown outer-tier, .html/.htm only) and consumed at scripts/cocoindex_pipeline/flow.py:684 content_text = source.transform(convert_binary_to_markdown).

Surface C write path — NON-FUNCTIONAL in cocoindex 1.0.3 (load-bearing finding, see Risks R1 + OQ-FOR-PARENT). flow.py:744-765 mounts source_documents via mount_table_target(... managed_by=ManagedBy.USER) then calls source.bind_target(sd_target, key_fields=("id",), op_id=flow["op_id"]). Both bind_target and flow["op_id"] carry # type: ignore[attr-defined] and the flow docstring (lines 30-39) + the comment at 753-756 state plainly that these are spec-sketch placeholders that cocoindex 1.0.3 does not expose — “real wiring lands when the flow-scope op_id API is finalised” / “mount_table_target exposes NO per-row UPSERT completion callback … real invocation waits on a public callback surface (TECH.md §P-5 amendment queue)”. The SOURCE_DOCUMENTS_SCHEMA (flow.py:560-568) declares only id, storage_path, content_fingerprint, op_id — no extraction_method, no pullmd_share_id, and there is no functional Stage-6 row UPSERT today. Empirically: from cocoindex.connectors.postgres import mount_table_target, ColumnDef, TableSchema and from cocoindex.connectorkits.target import ManagedBy all resolve in the installed cocoindex==1.0.3; the placeholder is specifically the .bind_target(...) / flow["op_id"] flow-graph write API, not the imports.

Surface A — sector-intelligence cascade (live prod, functional TypeScript path). lib/intelligence/content-extractor.ts:170 extractContent() returns ExtractionResult with a method field; Tier 1 rss_content (line 180, inline content:encoded, no URL fetch), Tier 2 fetch+extractMainContentHtml (line 37 / 220), Tier 2.5 jina_reader (https://r.jina.ai/${url}, line 244), Tier 3 firecrawl (@mendable/firecrawl-js, lines 276-278), Tier 4 summary_fallback (line 331). Guards isFirecrawlConfigured() (line 27) + checkFirecrawlApiKey() (line 136). Sole prod caller lib/intelligence/pipeline.ts:372; pipeline.ts:395 + pipeline.ts:480 both write feed_articles.extraction_method = extraction.method; passed articles flow to storeAsContentItem (pipeline.ts:577) which writes content_items via sb() (line 594 — real, functional, no-silent-failure path). A SECOND Firecrawl call site is feed-poller.ts:336 pollWebSource() (import + scrape at lines ~430-435), surfaced through the admin source-test route (app/api/intelligence/workspaces/[id]/sources/[sourceId]/test/route.ts:51) which returns firecrawlCalled / firecrawlCreditsExpected in its response (lines 57-75).

Surface B — URL-ingest (live prod). lib/extraction/url.ts:47 extractFromUrl() branches PDF (extractPdfText/unpdf, line 97) vs HTML (extractFromHtml, line 115). lib/extraction/html.ts:28 extractFromHtml() uses dynamic import('jsdom') + import('@mozilla/readability') (lines 33-34) + Turndown. Sole caller app/api/ingest/url/route.ts:94.

Deploy mechanics (validated against .github/workflows/cloud-run-deploy.yml). Triggers: push main → kh-prod-494815 (Production), push production-readiness → kh-staging-494815 (Staging) — there is no staging-branch deploy path; staging is deploy-only mirror sync per CLAUDE.md. Path-trigger allowlist (lines 39-45): scripts/**/*.py, requirements.txt, cloudrun/**, .gcloudignore, the workflow file. WIF auth via vars.GCP_WIF_PROVIDER + vars.GCP_DEPLOY_SA_EMAIL. cocoindex Services deployed via gcloud run services replace cloudrun/services/{env}-{tenant}-cocoindex.yaml in a for TENANT in phew kpf loop (lines 261-268); secrets mounted declaratively via gcloud run services update --set-secrets (lines 347-355) with COCOINDEX_SECRETS including PULLMD_SERVICE_URL=PULLMD_SERVICE_URL:latest (line 345). The cocoindex image build (cloudbuild-cocoindex.yaml) actively asserts (Inv-9) zero pullmd binaries in the image (AGPL boundary O-Q3, lines 113-158) — confirming pullmd MUST be its own Service. Existing Service manifest precedent: cloudrun/services/{prod,staging}-phew-cocoindex.yaml (ingress: internal, minScale/maxScale=1, gen2, europe-west2, per-tenant SA reuse).

Schema state (validated against supabase/types/database.types.ts + migrations). source_documents (lines 2877-2940) has NO extraction_method, NO pullmd_share_id. feed_articles.extraction_method exists (line 1228, string | null); the live CHECK is feed_articles_extraction_method_check = {rss_content, fetch, jina_reader, firecrawl, summary_fallback} (supabase/migrations/20260416102457_pre_squash_reconciliation.sql:3713, verified verbatim). The op_id migration (20260521203414_t8_op_id_propagation.sql) is the exact precedent: ALTER TABLE … ADD COLUMN IF NOT EXISTS … ; CREATE INDEX IF NOT EXISTS … WHERE … IS NOT NULL; COMMENT ON COLUMN …; with a file-level SET search_path = public, extensions; and NO PL/pgSQL functions (so no REVOKE needed — the REVOKE-guard lints CREATE FUNCTION public.* only, per scripts/check-revoke-guard.ts).

Most-relevant files (absolute)

• /Users/liamj/.../scripts/cocoindex_pipeline/adapters.py:88 — adapter to rewrite.
• /Users/liamj/.../scripts/cocoindex_pipeline/flow.py:560,684,744-765 — schema decl + transform + Stage-6 write (placeholder).
• /Users/liamj/.../scripts/tests/test_cocoindex_adapters.py:398-458 — adapter tests asserting the OLD wrong contract (must be rewritten).
• /Users/liamj/.../requirements.txt — httpx pin to add (transitive-only today).
• /Users/liamj/.../.github/workflows/cloud-run-deploy.yml:39-45,261-268,316-355 — deploy wiring.
• /Users/liamj/.../cloudrun/services/staging-phew-cocoindex.yaml — manifest precedent.
• /Users/liamj/.../supabase/migrations/20260521203414_t8_op_id_propagation.sql — migration precedent.
• /Users/liamj/.../lib/intelligence/content-extractor.ts, feed-poller.ts, types.ts, pipeline.ts — Surface A retirement.
• /Users/liamj/.../lib/extraction/html.ts, url.ts — Surface B retirement.
• /Users/liamj/.../docs/ontology/16-extraction-method.md — CV extension.

---

Proposed changes

Organised by work-package (WP-A … WP-G). The per-invariant mapping is in §Proposed changes per invariant; this section gives the implementation detail.

WP-A — Adapter rewrite (adapters.py::_pullmd_to_markdown)

Contract. Replace the wrong POST /extract + .json()["markdown"] with the verified v2.x contract. The return type must change from str to a structured result carrying the provenance headers so the write site (WP-E) can persist them.

• Endpoint: GET {PULLMD_SERVICE_URL}/api?url=<urlencoded>. URL-encode the target via urllib.parse.quote(url, safe="") and pass as the url query param (use httpx’s params={"url": url} so encoding is handled correctly).
• Auth: send Authorization: Bearer <PULLMD_API_TOKEN> (single-admin, ratified). Read the token from os.environ["PULLMD_API_TOKEN"]; raise a clear RuntimeError if unset (mirrors the existing PULLMD_SERVICE_URL fail-fast at lines 90-95 — no silent failure).
• Async client: use httpx.AsyncClient (empirically present in httpx==0.28.1) inside the async def, NOT the current synchronous httpx.post(...) (which blocks the event loop inside an @coco.fn async function). Pattern: async with httpx.AsyncClient(timeout=httpx.Timeout(60.0)) as client: resp = await client.get(f"{pullmd_url}/api", params={"url": url}, headers=headers).
• Response: resp.raise_for_status(); body is resp.text (raw markdown — NOT JSON). Capture resp.headers.get("X-Source"), resp.headers.get("X-Quality"), resp.headers.get("X-Share-Id") (httpx.Headers is case-insensitive — empirically verified).
• Return shape: a typed result, e.g. a frozen dataclass PullmdResult(markdown: str, x_source: str | None, x_quality: float | None, share_id: str | None) (or a TypedDict). Memo-key caveat: the inner @coco.fn(memo=True) is memoised on url; the richer return value is fine (memo key is the input arg, not the output). The OUTER convert_binary_to_markdown currently returns str and feeds content_text — see WP-E for how the headers surface without breaking the content_text text flow.
• Error handling: no silent failures. raise_for_status() propagates HTTP errors; httpx.RequestError (connect/timeout) propagates to the flow’s Stage-error rollup (flow.py:771 _classify_stage_exception). Add a structured log on failure consistent with the existing _logger usage.
• PDF pre-route note: no change needed in the adapter — convert_binary_to_markdown (adapters.py:51-58) already routes .pdf/.docx/.xlsx→Docling and only .html/.htm→pullmd, so pullmd is never handed a PDF on Surface C. The PDF pre-route matters only for Surface A/B retirement (Surface B keeps its unpdf PDF branch).

Tests (rewrite test_cocoindex_adapters.py:398-458). The three existing tests assert the OLD contract (POST /extract, .json()["markdown"]) and WILL break — they are rewritten to assert: GET /api with params={"url": …}; Authorization: Bearer header present; raw resp.text returned as markdown; X-Source/X-Quality/X-Share-Id captured onto the result; RuntimeError when PULLMD_SERVICE_URL OR PULLMD_API_TOKEN unset; HTTP errors propagate via raise_for_status(). Mock httpx.AsyncClient (not httpx.post). These remain unit-level (mocked transport) — the NON-mocked end-to-end proof is Inv-7/8/9 (WP-F).

WP-B — Pin httpx (requirements.txt)

Add an explicit pin under “Cocoindex pipeline”: httpx==0.28.1 # explicit pin — adapters.py uses httpx.AsyncClient; transitive via cocoindex today (fragile). Empirically verified installed version is 0.28.1 (OQ-3 check). This guards against a cocoindex minor bump silently moving the httpx major.

WP-C — pullmd Service deploy (manifests + workflow wiring + secrets)

Manifests (NEW): cloudrun/services/{staging,prod}-pullmd.yaml (phew-only — NO kpf manifest). Mirror the cocoindex Service manifest shape with pullmd-specific deltas:

• metadata.name: kh-pullmd-phew; labels tenant: phew, component: pullmd-service.
• Sidecar topology decision (RESEARCH §4.3 α/β/γ): recommend α — single Service, multi-container (pullmd + Playwright + Trafilatura as additional containers). Rationale: closest to upstream docker compose, keeps PLAYWRIGHT_URL/TRAFILATURA_URL as in-pod localhost calls (no extra Service ingress/auth surface), and the bake-off found Playwright/Cloudflare/Reddit wins are worth the cost. The 3.7 GB Playwright container mandates autoscaling.knative.dev/minScale: "1" (avoid cold-start; matches the cocoindex minScale:1 precedent) and a higher memory limit (≥ 4Gi on the pullmd container; size the Playwright container per upstream guidance). maxScale: "1" keeps a single warm instance (public-web extraction, cross-request cache reuse). γ (Playwright-disabled) is rejected as too lossy for v1; β (separate sidecar Services) is rejected as more ops surface for no v1 benefit.
• PINNED image tag — NOT :latest. Use aeternalabshq/pullmd:<pinned-2.x.y> (the eval found :latest lagged 2.0.0; RESEARCH §4.3). Pin the concrete tag in the manifest. (If the public image cannot be referenced cross-registry by Cloud Run, mirror it into Artifact Registry via a one-line gcloud builds submit / docker pull && push step — a cloudbuild-pullmd.yaml is optional; the direct public-image reference is preferred if it resolves.)
• run.googleapis.com/ingress: internal (same posture as cocoindex; the cocoindex Service is the only consumer and is itself ingress: internal in the same project). Combined with single-admin Bearer auth this is defence-in-depth: network-internal AND token-gated.
• Env on the pullmd container: PULLMD_AUTH_MODE=single-admin, PULLMD_ADMIN_EMAIL + PULLMD_ADMIN_PASSWORD (mounted via Secret Manager — see below), PUBLIC_URL set to the Service URL, PLAYWRIGHT_URL=http://localhost:<port> / TRAFILATURA_URL=http://localhost:<port> (in-pod sidecars for topology α).
• serviceAccountName: phew-pipeline-sa@<project>.iam.gserviceaccount.com (reuse the existing per-tenant SA; existing roles/secretmanager.secretAccessor grant covers it).
• No /health probe. pullmd has no documented /health (RESEARCH §2.4). Use GET / for startupProbe/livenessProbe httpGet.path: / (the cocoindex manifest’s /health probe is wrong for pullmd — do NOT copy it). Set a generous startupProbe failureThreshold/initialDelaySeconds to accommodate the Playwright container warm-up.

Workflow wiring (.github/workflows/cloud-run-deploy.yml).

• Path-trigger allowlist (lines 39-45): add - "cloudrun/services/*pullmd*" (and - "cloudrun/cloudbuild-pullmd.yaml" only if that build file is added). NOTE: cloudrun/** already matches, so this addition is belt-and-braces / explicitness; keep it for clarity.
• NEW deploy step “Deploy pullmd Service” (after “Deploy cocoindex Services”): a non-looped single replace per env (phew-only): gcloud run services replace cloudrun/services/${ENV_PREFIX}-pullmd.yaml --project="$GCP_PROJECT" --region=europe-west2. Do NOT wrap in for TENANT in phew kpf.
• NEW secret mount on the pullmd Service: PULLMD_ADMIN_EMAIL, PULLMD_ADMIN_PASSWORD (and PULLMD_API_TOKEN if pullmd reads its own admin token from env at bootstrap) via gcloud run services update kh-pullmd-phew --set-secrets=....
• Extend COCOINDEX_SECRETS (line 345): add ,PULLMD_API_TOKEN=PULLMD_API_TOKEN:latest so the cocoindex Service (the adapter’s caller) gets the Bearer token. --set-secrets is declarative (replaces the whole set) — the new token must be appended to the existing string, not issued as a separate update.
• NEW smoke-verify step for pullmd: gcloud run services describe kh-pullmd-phew --format='value(status.url)' then a GET <url>/ 200 check (NOT /health). Assert the image tag is not :latest via gcloud run services describe kh-pullmd-phew --format='value(spec.template.spec.containers[0].image)' and grep-fail on a trailing :latest.

Secret Manager (both envs).

• CREATE PULLMD_API_TOKEN (the Bearer token, value pmd_<32-char-base62>), PULLMD_ADMIN_EMAIL, PULLMD_ADMIN_PASSWORD. These are created out-of-band (gcloud / console) — the workflow only references them by :latest.
• REPLACE (not add) PULLMD_SERVICE_URL with the real Cloud Run Service URL after first deploy (it currently holds the S258 placeholder https://pullmd-not-yet-deployed-staging.example.com). The cocoindex Service revision consuming the mount must be re-deployed so the running revision sees the new value.

WP-D — Migration (net-new provenance schema)

NEW file: supabase/migrations/<timestamp>_id42_pullmd_provenance.sql (timestamp from supabase migration new id42_pullmd_provenance). Authored via CLI ONLY (supabase migration new + supabase db push), never MCP execute_sql / apply_migration (CLAUDE.md). cat supabase/.temp/project-ref before push; relink if drift. Function-free (pure DDL) → no REVOKE-anon needed (REVOKE-guard lints CREATE FUNCTION only). Mirror the op_id precedent exactly:

SET search_path = public, extensions;

-- source_documents.pullmd_share_id — pullmd X-Share-Id permalink handle (8-hex).
ALTER TABLE public.source_documents ADD COLUMN IF NOT EXISTS pullmd_share_id text NULL;
CREATE INDEX IF NOT EXISTS idx_source_documents_pullmd_share_id
  ON public.source_documents (pullmd_share_id) WHERE pullmd_share_id IS NOT NULL;
COMMENT ON COLUMN public.source_documents.pullmd_share_id IS
  'pullmd X-Share-Id permalink (GET /s/:id round-trips); ID-42 (docs/specs/id-42-pullmd-deploy/TECH.md WP-D)';

-- source_documents.extraction_method — typed provenance column + CHECK enum.
ALTER TABLE public.source_documents ADD COLUMN IF NOT EXISTS extraction_method text NULL;
ALTER TABLE public.source_documents DROP CONSTRAINT IF EXISTS source_documents_extraction_method_check;
ALTER TABLE public.source_documents ADD CONSTRAINT source_documents_extraction_method_check
  CHECK (extraction_method IS NULL OR extraction_method = ANY (ARRAY[
    'rss_content','fetch','jina_reader','firecrawl','summary_fallback',
    'pullmd_readability','pullmd_playwright','pullmd_cloudflare','pullmd_reddit',
    'pullmd_trafilatura','docling'
  ]));
COMMENT ON COLUMN public.source_documents.extraction_method IS
  'Extractor that produced the markdown; pullmd_* mirrors X-Source, docling for binary; ID-42';

-- feed_articles.extraction_method CHECK extension (Surface A keeps writing this column;
-- pipeline.ts:395/480). Extend with pullmd_* + docling; NO destructive prune of firecrawl
-- (Ratified-S237 lockstep — prune is T7-gated).
ALTER TABLE public.feed_articles DROP CONSTRAINT IF EXISTS feed_articles_extraction_method_check;
ALTER TABLE public.feed_articles ADD CONSTRAINT feed_articles_extraction_method_check
  CHECK (extraction_method IS NULL OR extraction_method = ANY (ARRAY[
    'rss_content','fetch','jina_reader','firecrawl','summary_fallback',
    'pullmd_readability','pullmd_playwright','pullmd_cloudflare','pullmd_reddit',
    'pullmd_trafilatura','docling'
  ]));

Post-migration: regenerate types (supabase gen types typescript --project-id <ref> --schema public > supabase/types/database.types.ts) — commit the regenerated database.types.ts. The CHECK-redefinition is non-destructive (extends the admitted set; existing firecrawl rows stay valid — verified the live constraint admits exactly the legacy 5). schema-parity (schema-parity.yml) must show prod↔staging match after push.

CV update: docs/ontology/16-extraction-method.md baseline_values add the six new values (pullmd_readability, pullmd_playwright, pullmd_cloudflare, pullmd_reddit, pullmd_trafilatura, docling); do NOT remove firecrawl (extension only).

WP-E — Write-site wiring (flow.py — extraction_method + pullmd_share_id onto source_documents)

This is net-new (Surface C writes neither today). Two sub-problems:

1. Surface the headers from the adapter to the write site. convert_binary_to_markdown currently returns str and content_text = source.transform(...) carries that into the downstream extraction stages, which expect text. Changing the outer return to a struct would ripple into Stages 3-6. The recommended approach: keep convert_binary_to_markdown returning the markdown str for the content_text flow, and have the pullmd branch ALSO persist its provenance via a side-channel keyed by the source identity (the storage_path/url), OR add a second flow field (e.g. extraction_meta = source.transform(extract_pullmd_provenance)) that the source_documents bind reads. The exact mechanism depends on the cocoindex flow-graph field-fan-out API; specify it precisely at PLAN time once the Stage-6 write API is functional (see R1 / OQ-FOR-PARENT).
2. Add the columns to SOURCE_DOCUMENTS_SCHEMA (flow.py:560-568) and to the source.bind_target(sd_target, key_fields=("id",), ...) field set: extraction_method and pullmd_share_id (both ColumnDef(type="text", nullable=True)).

BLOCKER (R1): the Stage-6 source.bind_target(...) + flow["op_id"] write path is a documented non-functional placeholder in cocoindex 1.0.3 (# type: ignore; flow docstring lines 30-39). The schema/field additions here are READY (correct shape, correct columns), but the actual row write — and therefore the Inv-7/8/9 end-to-end proof through Surface C — cannot execute until the cocoindex flow-scope UPSERT capability lands (tracked in the canonical `docs/specs/id-28-cocoindex-flow-scaffolding/TECH.md §P-5 amendment queue). See OQ-FOR-PARENT and §Testing for how Inv-7/8/9 are gated.

WP-F — End-to-end ingest proof (Inv-7/8/9 verification harness)

A NON-mocked test/script that stages one real HTML source through the cocoindex .html/.htm→pullmd path against the DEPLOYED pullmd Service and asserts: (a) the produced content_items row body is non-empty markdown (Inv-7); (b) source_documents.extraction_method is a pullmd_* value consistent with the live X-Source (Inv-8); (c) source_documents.pullmd_share_id equals the live X-Share-Id and GET <url>/s/<share_id> re-serves the markdown (Inv-9). Per docs/reference/test-philosophy.md this exercises real behaviour, never a mock of pullmd. Gated on R1 — see §Testing.

WP-G — Retirement (Surface A + B HTML branches + npm deps) — GATED on Inv-10

All retirement is ordered strictly AFTER the Inv-7/8/9 proof is green in prod (Inv-10). The file:line surfaces to delete (spec’d here, NOT pre-ordered):

• Surface A (content-extractor.ts): delete Tier 2 (fetch + extractMainContentHtml, lines ~37, 196-240), Tier 2.5 (jina_reader, lines 242-272), Tier 3 (firecrawl, lines 274-315), and the Firecrawl guards checkFirecrawlApiKey (136) / isFirecrawlConfigured (27) / firecrawlWarningLogged (12) / firecrawlKeyMissing (19). Decision (Inv-12): Tier 1 rss_content (180) is PRESERVED (inline content:encoded, no URL fetch — no pullmd equivalent); Tier 4 summary_fallback (331) is PRESERVED as the all-tiers-failed degraded path. The HTML→markdown work for web/RSS-with-URL sources is re-pointed at pullmd (a single pullmd call replaces Tiers 2/2.5/3). The checkFirecrawlApiKey() call from pipeline.ts:777 (RESEARCH §1) must be removed in lockstep.
• Surface A (feed-poller.ts): pollWebSource() (336) Firecrawl scrape (~430-435) re-pointed at pullmd (or removed if the web-source path collapses into the pullmd call). The firecrawlCalled telemetry + the source-test route’s firecrawlCreditsExpected (route.ts:57-75) are updated to drop the Firecrawl-credit semantics.
• Surface A (types.ts) — EXTEND the union (load-bearing, not derivable from the DB CHECK). After Surface A re-points Tiers 2/2.5/3 at pullmd, extractContent() returns an ExtractionResult whose method is a pullmd_* value; without extending the TS union the tsc build breaks and pipeline.ts:395/:480 write an out-of-type value to feed_articles.extraction_method. This is a SEPARATE change from WP-D’s SQL CHECK (a constraint on a text column) — the DB enum does not imply the TS literal type. Extend the ExtractionResult.method inline union literal (types.ts:32-37) to add 'pullmd_readability' | 'pullmd_playwright' | 'pullmd_cloudflare' | 'pullmd_reddit' | 'pullmd_trafilatura' alongside the preserved 'rss_content', 'fetch', 'summary_fallback', and (until the T7 prune) 'jina_reader'. Remove the 'firecrawl' member only once no runtime path produces it. (The DB CHECK keeps firecrawl for historical rows — Inv-17; the TS-union and DB-CHECK lifecycles are independent.)
• Surface B (html.ts/url.ts): app/api/ingest/url HTML extraction re-pointed at pullmd; extractFromHtml (html.ts:28, JSDOM + Readability) retired. extractFromUrl’s PDF branch (extractPdfText/unpdf, url.ts:97) is PRESERVED.
• npm deps (package.json): remove @mendable/firecrawl-js, @mozilla/readability, jsdom. KEEP unpdf (Surface B PDF branch). Run bun install; commit the lockfile.
• Workflow FIRECRAWL_API_KEY secret mount (cloud-run-deploy.yml:324) — IN-SCOPE, remove as part of WP-G. PRODUCT Inv-15 lists this mount as a MUST-BE-ZERO item (not an exclusion), and the Inv-15 verification grep covers it. Remove the FIRECRAWL_API_KEY=FIRECRAWL_API_KEY:latest line from the SECRETS heredoc (line 324) in lockstep with the Surface-A retirement — it must NOT be deferred to {42.4}. (The PULLMD_API_TOKEN addition from WP-C is unaffected.)
• Cosmetic trailers (deferrable to {42.4}/PLAN clean-up — these ARE in PRODUCT Inv-15’s explicit-exclusion list): lib/env-server.ts:61 FIRECRAWL_API_KEY Zod entry and lib/intelligence/url-validation.ts:6 firecrawl comment — removed only once the runtime usage is gone (cosmetic; trails the runtime removal).

---

Proposed changes per invariant

One-to-one mapping. Each PRODUCT invariant (Inv-N) → the proposed change (WP) + the runnable verification the Checker uses. (PRODUCT owns the behaviour text; this maps it to implementation + a concrete check.)

Inv	Proposed change	Runnable verification
Inv-1 (Service resolves — staging)	WP-C: staging-pullmd.yaml deployed to kh-staging-494815.	gcloud run services describe kh-pullmd-phew --project kh-staging-494815 --format='value(status.url)' → one HTTPS URL; curl -so /dev/null -w '%{http_code}' <url>/ → 200. Two consecutive describes (pre/post no-op redeploy) return the same URL.
Inv-2 (Service resolves — prod)	WP-C: prod-pullmd.yaml deployed to kh-prod-494815.	Inv-1 commands against --project kh-prod-494815.
Inv-3 (Pinned image, not :latest)	WP-C: manifest pins aeternalabshq/pullmd:<2.x.y>.	gcloud run services describe kh-pullmd-phew --format='value(spec.template.spec.containers[0].image)' shows a concrete tag, not …:latest (smoke-verify step grep-fails on :latest).
Inv-4 (Deploy reproducible from source control)	WP-C: manifests under cloudrun/services/; path-trigger allowlist + deploy step in cloud-run-deploy.yml; no console-only steps.	ls cloudrun/services/{staging,prod}-pullmd.yaml; grep -n 'pullmd' .github/workflows/cloud-run-deploy.yml shows the deploy step + path glob; re-running the deploy job (workflow_dispatch) produces a healthy Service.
Inv-5 (PULLMD_SERVICE_URL reflects real URL — both envs)	WP-C: Secret Manager REPLACE + cocoindex Service re-deploy.	gcloud secrets versions access latest --secret=PULLMD_SERVICE_URL --project <env> returns the Inv-1/2 URL, contains no not-yet-deployed/example.com; the consuming cocoindex revision is re-deployed.
Inv-6 (cocoindex can reach pullmd)	WP-A (Bearer header) + WP-C (token in COCOINDEX_SECRETS + internal ingress).	A smoke call GET {PULLMD_SERVICE_URL}/api?url=<known-good-html> with Authorization: Bearer $PULLMD_API_TOKEN from the cocoindex Service context returns 200 + non-empty text/markdown.
Inv-7 (one HTML source ingests → non-empty body)	WP-A + WP-E + WP-F. GATED on R1.	WP-F harness: stage one HTML source through the deployed path; assert length(content_items.content_text) > 0 and markdown-shaped. No pullmd mock. Gate: executes once Stage-6 UPSERT lands — see OQ-FOR-PARENT.
Inv-8 (pullmd-sourced extraction method recorded)	WP-A (capture X-Source) + WP-D (source_documents.extraction_method enum) + WP-E (write) + WP-F. GATED on R1.	WP-F: produced source_documents.extraction_method equals a pullmd_* value consistent with the live X-Source. SQL probe: SELECT extraction_method FROM source_documents WHERE id=….
Inv-9 (pullmd_share_id present + round-trips)	WP-D (pullmd_share_id column + partial index) + WP-A (capture X-Share-Id) + WP-E (write) + WP-F. GATED on R1.	WP-F: source_documents.pullmd_share_id = live X-Share-Id; curl <url>/s/<share_id> → 200 + non-empty markdown.
Inv-10 (no deletion before verified end-to-end ingest)	Sequencing: WP-G commits ordered strictly after WP-F green in prod.	Checker bisects commit history: at any WP-G deletion commit’s parent, Inv-7/8/9 already pass; no single commit both deletes a legacy surface AND first introduces the pullmd path.
Inv-11 (Surface A HTML tiers retired)	WP-G: delete Tiers 2/2.5/3 + extractMainContentHtml + Firecrawl guards in content-extractor.ts; re-point/remove feed-poller.ts Firecrawl scrape; EXTEND the ExtractionResult.method union (types.ts:32-37) with the pullmd_* members (so tsc + the pipeline.ts:395/480 writes stay in-type).	grep -n 'extractMainContentHtml|r\.jina\.ai|@mendable/firecrawl-js' lib/intelligence/content-extractor.ts lib/intelligence/feed-poller.ts → zero hits; AND grep -nE "pullmd_readability|pullmd_playwright|pullmd_cloudflare|pullmd_reddit|pullmd_trafilatura" lib/intelligence/types.ts → the five pullmd_* members present in the union; bun run lint / tsc --noEmit clean.
Inv-12 (Tier 1 rss_content + Tier 4 summary_fallback disposition explicit)	WP-G decision: BOTH preserved (stated above).	An RSS source with inline content:encoded still ingests via Tier 1; the all-tiers-failed path still yields summary_fallback. Verified by the regression suite + a targeted unit test.
Inv-13 (Surface B HTML branch retired; PDF intact)	WP-G: re-point app/api/ingest/url at pullmd; retire extractFromHtml; keep extractPdfText/unpdf.	Ingest an HTML URL → non-empty pullmd extraction; ingest a PDF URL → non-empty via PDF branch; grep -n 'extractFromHtml' lib/extraction/url.ts shows it unreached for HTML.
Inv-14 (npm HTML-extraction deps removed; unpdf kept)	WP-G: remove @mendable/firecrawl-js, @mozilla/readability, jsdom; keep unpdf.	jq -r '.dependencies|keys[]' package.json | grep -E 'firecrawl-js|readability|jsdom' → empty; … | grep unpdf → present; bun install + bun build succeed.
Inv-15 (retirement grep gate — scoped)	WP-G: the in-scope live surfaces cleared.	grep -rIn -E 'firecrawl|FIRECRAWL|jina\.ai|extractMainContentHtml' lib/ app/ --include='*.ts' | grep -vE '__tests__|\.test\.' → zero in-scope hits. PLUS (Checker NIT honoured): grep -n FIRECRAWL_API_KEY .github/workflows/cloud-run-deploy.yml → zero (the workflow-YAML secret mount removed). Residual hits must fall in PRODUCT Inv-15’s explicit-exclusion list.
Inv-16 (cron still ingests post-swap)	WP-G keeps Tier 1 + pullmd path live; entrypoints unchanged.	app/api/cron/intelligence-poll + app/api/intelligence/trigger-poll each produce new feed_articles rows; admin source-test route returns a successful dry-run; no removed-symbol 500s; bun run test passes.
Inv-17 (existing feed_articles corpus not corrupted)	WP-D: CHECK extension is additive (no prune).	Pre-existing extraction_method='firecrawl' rows still present + readable; the redefined feed_articles_extraction_method_check still admits the legacy 5; CV still lists them. SQL: SELECT count(*) FROM feed_articles WHERE extraction_method='firecrawl' unchanged across the migration.
Inv-18 (tests verify real behaviour)	WP-A tests (behaviour, not shape); WP-F end-to-end hits the deployed Service.	Review WP-F asserts non-empty body + provenance + gate ordering against the live Service, not a mock (docs/reference/test-philosophy.md).

Coverage check: Inv-1 … Inv-18 each have exactly one row. No invariant is unmapped.

---

Testing and validation

• Adapter (Inv-6 substrate, WP-A): rewritten test_cocoindex_adapters.py unit tests (mocked httpx.AsyncClient) — GET /api?url=, Bearer header, raw-text body, header capture, fail-fast on missing env, error propagation. Run python3 -m pytest scripts/tests/test_cocoindex_adapters.py.
• Migration (Inv-8/9/17, WP-D): apply via supabase db push to staging first; verify \d source_documents shows the two new columns + partial index + CHECK; verify feed_articles CHECK admits both legacy + pullmd_*; verify pre-existing firecrawl rows survive; regenerate + commit database.types.ts; confirm schema-parity.yml green.
• End-to-end (Inv-7/8/9, WP-F) — GATED (R1): NON-mocked harness against the deployed pullmd Service. This is the load-bearing acceptance and it cannot run until the cocoindex Stage-6 UPSERT write path is functional (see Risks R1 + OQ-FOR-PARENT). The spec defines the assertions precisely so the harness is non-vacuous the moment Stage-6 lands; until then, Inv-7/8/9 are NOT verifiable through Surface C and Inv-10 (the deletion gate) therefore CANNOT open. This is the correct fail-safe: retirement (WP-G) is blocked by construction until the proof is real.
• Retirement regression (Inv-11-16, WP-G): the grep gates (Inv-11, Inv-14, Inv-15) plus bun run test (full regression) plus live-entrypoint smoke (cron poll, manual trigger, source-test, app/api/ingest/url). Each retirement commit lands only after the Inv-10 bisect precondition holds.
• Deploy (Inv-1-5, WP-C): the workflow’s NEW pullmd smoke-verify step (describe + GET / 200 + :latest grep-fail); manual workflow_dispatch reproduces a healthy Service from a clean checkout.

---

Risks and mitigations

• R1 — cocoindex Stage-6 write path is a non-functional 1.0.3 placeholder (LOAD-BEARING; see OQ-FOR-PARENT). flow.py:744-765 source.bind_target(...) + flow["op_id"] are spec-sketch placeholders (# type: ignore; flow docstring 30-39). Inv-7/8/9 (the end-to-end source_documents proof through Surface C) cannot execute until the cocoindex flow-scope UPSERT/op_id API ships (canonical `cocoindex-flow-scaffolding/TECH.md §P-5 amendment queue). Mitigation: WP-A/WP-D/WP-E are authored READY (correct adapter, correct columns, correct schema-field additions) so that when Stage-6 lands the proof is a thin wiring step, not a redesign; and Inv-10’s deletion gate is fail-safe (retirement is blocked until Inv-7/8/9 are genuinely green). The Orchestrator must decide whether ID-42 (a) accepts this gate and lands deploy + adapter + migration + schema-readiness now, retirement later; or (b) re-scopes the proof onto the functional Surface-A path — which would contradict the ratified OQ-2 table choice. See OQ-FOR-PARENT.
• R2 — httpx sync-in-async. The current adapter calls httpx.post(...) synchronously inside async def, blocking the event loop. WP-A switches to httpx.AsyncClient + await client.get(...). Verify no remaining httpx.post/httpx.get (sync) in adapters.py.
• R3 — --set-secrets is declarative (replaces the whole set). Adding PULLMD_API_TOKEN to COCOINDEX_SECRETS must APPEND to the existing line (line 345), not issue a separate partial update (which would drop the other 12 secrets). Mitigation: edit the COCOINDEX_SECRETS= string in-place; the existing single --set-secrets call carries the full set.
• R4 — Playwright 3.7 GB cold-start. Topology α with minScale:1 keeps the instance warm; verify the Service does not scale to zero (maxScale:1, minScale:1).
• R5 — schema-parity drift. Push the migration to staging AND prod; schema-parity.yml must show no diff. The DROP CONSTRAINT IF EXISTS + re-ADD CONSTRAINT for the two CHECK constraints is non-destructive but MUST run in both envs.
• R6 — source_documents.extraction_method CHECK admits NULL. The cocoindex flow writes Docling/passthrough rows too; the CHECK allows NULL and docling so non-pullmd rows are not rejected. Verify a Docling-extracted source_documents row (extraction_method=docling or NULL) is accepted.
• R7 — doc-drift (non-blocking). cloudrun/services/README.md:35 references a non-existent scripts/cocoindex_pipeline/_pullmd_to_markdown.py (the code lives in adapters.py::_pullmd_to_markdown). Fix the reference opportunistically when touching the README; not a gate.

---

OQ-FOR-PARENT (route to Liam — NOT covered by the three ratified S265 decisions)

OQ-42.3-A — Inv-7/8/9 end-to-end proof is blocked by the cocoindex Stage-6 write-path capability, not by the adapter. The ratified OQ-2 decision places extraction_method + pullmd_share_id on source_documents — the table written by the cocoindex Surface-C path. But that path’s row-write (flow.py source.bind_target(...) + flow["op_id"]) is a documented non-functional placeholder in cocoindex 1.0.3 (# type: ignore; flow docstring 30-39; canonical `cocoindex-flow-scaffolding/TECH.md §P-5 amendment queue). So PRODUCT Inv-7/8/9 (and therefore the Inv-10 deletion gate) cannot be satisfied through Surface C until that capability lands. This materially affects ID-42’s shape and is not one of the three S265 ratifications, so I surface it rather than invent an answer.

Two resolutions for Liam:

• (A) — Accept the gate (my best-justified assumption; spec written this way). ID-42 lands now: pullmd deploy (WP-C), adapter rewrite (WP-A), httpx pin (WP-B), migration + schema readiness (WP-D), write-site columns/fields READY (WP-E). Inv-7/8/9 (WP-F) and retirement (WP-G) are gated on the cocoindex Stage-6 UPSERT/op_id API shipping. This honours the ratified source_documents table choice and keeps Inv-10 fail-safe (retirement cannot proceed on an unproven path). Trade-off: ID-42 does not fully “retire the cascade” in one pass — retirement waits on a T8-adjacent capability.
• (B) — Re-scope the proof onto the functional Surface-A path. Surface A’s pipeline.ts→storeAsContentItem DOES functionally write feed_articles + content_items today (real sb() calls). If the pullmd-ingest proof is carried by re-pointing Surface A at pullmd (Tiers 2/2.5/3 → one pullmd call) and persisting provenance onto feed_articles.extraction_method (+ a new content_items / feed_articles pullmd_share_id column), Inv-7/8/9 become provable now WITHOUT waiting on cocoindex Stage-6 — but this contradicts the ratified OQ-2 table choice (source_documents) and would relocate the provenance contract to feed_articles / content_items. This needs Liam’s explicit re-ratification of OQ-2’s table.

Recommendation: (A). It preserves both ratified decisions, keeps the deletion gate honest, and converts the cocoindex-capability dependency into an explicit, tracked gate rather than a silent assumption that the adapter rewrite alone makes HTML ingest work (exactly the S264/Q-EX2 failure class this spec-time check exists to prevent).

---

Installed-reality divergences from PRODUCT/RESEARCH (and how handled)

1. cocoindex Stage-6 write path non-functional (the big one). PRODUCT Inv-7/8/9 presume the cocoindex write path works; RESEARCH §1 Surface C noted “writes neither today” but framed it as net-new wiring, not as blocked-on-cocoindex-capability. Installed reality: bind_target/flow["op_id"] are 1.0.3 placeholders. Handled via R1 + OQ-FOR-PARENT (gate, not silent spec-around).
2. staging vs production-readiness branch. PRODUCT/RESEARCH refer to “both envs”; the actual staging deploy trigger is the production-readiness branch (not staging). Handled — WP-C wires production-readiness→staging / main→prod, matching the existing workflow; no new branch introduced.
3. feed_articles.extraction_method IS still written post-swap. RESEARCH OQ-2 part 2 was conditional (“IF Surface A re-points and keeps writing feed_articles.extraction_method”). Installed reality: pipeline.ts:395 AND :480 both write it unconditionally, and Surface A re-points (not removed). Handled — WP-D extends the feed_articles CHECK as the ratified decision directs.
4. Existing adapter tests assert the WRONG contract. test_cocoindex_adapters.py:398-458 codify POST /extract + .json()["markdown"]. Handled — WP-A explicitly rewrites them (they would otherwise fail the build after the adapter change).
5. pullmd contract re-verified (OQ-3, 25/05/2026). Re-fetched the AeternaLabsHQ/pullmd README: GET /api?url= raw text/markdown; X-Source/X-Quality/X-Share-Id; PULLMD_AUTH_MODE=single-admin + Authorization: Bearer pmd_<…>; no /health; GET /s/:id; PLAYWRIGHT_URL/TRAFILATURA_URL; PULLMD_ADMIN_EMAIL/PULLMD_ADMIN_PASSWORD. SIGNATURE_DRIFT+BEHAVIOUR_DRIFT of the current adapter confirmed. httpx==0.28.1, cocoindex==1.0.3 pins confirmed. No ABSENT/blocking drift for THIS spec.

OQ-3 pre-ratification empirical-verification table (Q-EX2 forcing function)

#	Subject	Date	Pin / source	Symbol / shape checked	Result
O1	pullmd v2.x HTTP contract	25/05/2026	no pin (network service); WebFetch of AeternaLabsHQ/pullmd README (main)	GET /api?url= → raw response.text (text/markdown); headers X-Source/X-Quality/X-Share-Id; single-admin + Authorization: Bearer pmd_…; no /health; GET /s/:id	BEHAVIOUR_DRIFT+SIGNATURE_DRIFT of the EXISTING adapter (POST /extract + .json()["markdown"]). WP-A rewrites to the verified contract. Non-blocking for this spec (pullmd exists; correct shape documented).
O2	httpx (adapter HTTP client)	25/05/2026	transitive via cocoindex today — WP-B pins httpx==0.28.1	httpx.AsyncClient, httpx.Timeout, case-insensitive httpx.Headers, httpx.HTTPStatusError	PRESENT (0.28.1). Adapter rewrite (WP-A) uses httpx.AsyncClient.
O3	cocoindex engine + connectors	25/05/2026	cocoindex[postgres]==1.0.3 (requirements.txt)	cocoindex.__version__; cocoindex.connectors.postgres.{mount_table_target,ColumnDef,TableSchema}; cocoindex.connectorkits.target.ManagedBy; connectors.localfs.walk_dir	PRESENT (1.0.3, pin matches). BUT the flow-graph write API source.bind_target(...) / flow["op_id"] is a non-functional placeholder (R1 / OQ-42.3-A) — ABSENT as a usable write path despite the imports resolving.

---

Follow-ups

• The cosmetic firecrawl trailers (env-server.ts Zod entry, url-validation.ts comment) — land in {42.4}/PLAN clean-up once runtime usage is gone (PRODUCT Inv-15 explicit-exclusion list). NOTE: the cloud-run-deploy.yml FIRECRAWL_API_KEY mount is NOT in this list — it is in-scope WP-G (Inv-15 MUST-BE-ZERO), removed with the Surface-A retirement.
• unpdf removal + Surface B PDF disposition — separate later Task (out of ID-42 scope).
• firecrawl prune from both CHECK enums + CV — T7 corpus-reingest Task (Ratified-S237 lockstep).
• cloudrun/services/README.md:35 stale _pullmd_to_markdown.py reference — fix when next touching the README.

---

End of {42.3} TECH. Returns to Orchestrator for ratification. ONE OQ-FOR-PARENT (OQ-42.3-A — cocoindex Stage-6 write-path gate for Inv-7/8/9) carried for Liam; best-justified assumption (A: accept the gate) baked into the spec body. {42.4} PLAN, if dispatched, decomposes WP-A … WP-G against this TECH + the ratified PRODUCT.