Scoping memo — NEW standalone Task: Vercel-side HTML-extraction cutover + Surface A/B retirement

From: subo-42-retire (Cmux D) · For: parent O-of-O → Liam ratification Date: 2026-06-15 · Mode: research-and-scoping only (no code, no deletion executed) Evidence: intended-arch-evidence.md, survey-42-retire.md, OQ-42-retire-blocking.md (this events dir)

1. Why this Task exists (the premise correction)

ID-42 {42.11-13} were briefed as “retire Surfaces A/B by re-pointing them at pullmd.” That premise is broken: pullmd is host-local to the Coolify/B1 compose network, unreachable from Vercel, with no TS client. Crucially, the intended architecture confirms this is not a defect — it always intended HTML/URL extraction to run server-side in the cocoindex/B1 worker behind pullmd, with the Vercel app only reading results from Postgres (03-tech-stack.md §3.2, L50). So the correct move is not “make Vercel reach pullmd” (public ingress / proxy — both off-intent) but finish moving extraction into the worker and then delete the dead Vercel Surfaces A/B.

This work is already named and ratified: pullmd-wiring-design-s300.md (RATIFIED S300, Liam, banner L19-28) says “Option A (cocoindex remote/custom source) confirmed as primary, with B2 as the fallback… Land as a new Task.” This memo scopes that S300-named Task.

2. Disposition of ID-42 (recommended)

ID-42 → done on delivered scope. {42.5} adapter rewrite, {42.6}/{42.7} deploy (retooled to Coolify S298/S288), {42.8} migration, {42.9} provenance wiring, {42.10} E2E proof (closed by reference to the green {62.10} B1 staging run, S355) are all delivered. {42.11}/{42.12}/{42.13} are superseded (premise broke) and roll into the new Task below — they are NOT executed as retirement and NOT marked done. Per canonical-pipeline-sequencing.md (L60-62, L333-336) the {42.11-13} retirement was already explicitly deferred to a future session — this formalises that.

3. What the new Task is FOR (proposed scope)

Title (proposed): Cutover HTML/URL extraction to the cocoindex/B1 worker; retire Vercel Surfaces A/B + Firecrawl/Jina/readability deps

Scope (1 paragraph): Land the worker-side URL/feed ingest that the intended architecture ratified — a cocoindex remote URL source (S300 Option A primary; B2 fetch-worker fallback if cocoindex’s custom-source surface proves non-functional) so web/RSS + manual URLs are extracted by pullmd behind cocoindex on B1, never on Vercel. Then re-prove the HTML datapath against a real URL (the {42.10}-class proof, now runnable post-S288 co-location), and delete the now-dead Vercel extraction code: Surface A (lib/intelligence/content-extractor.ts Tier 2/2.5/3 + feed-poller.ts Firecrawl in pollWebSource) and Surface B (lib/extraction/url.ts HTML branch + lib/extraction/html.ts), then drop the now-unused npm deps (@mendable/firecrawl-js, @mozilla/readability, jsdom — verify JSDOM ontology-path callers first; keep unpdf unless PDF also moves to Docling/worker) and the cosmetic trailers (env-server.ts FIRECRAWL_API_KEY, url-validation.ts comment), passing the scoped lib/+app/ TS grep gate. Add the PullMD/URL wiring entry to deferral-register.md (S299+S300 both flag it missing). Surface A (live-feed) is the spine of this Task; cross-link — do not duplicate — ID-110/bl-298 (manual-URL producer) and ID-45 (bounded existing-corpus re-ingest).

Why (grounded in intent): every canonical doc ratifies retiring the Vercel cascade into pullmd-behind-cocoindex; the reachability “mitigation” is the already-ratified worker-side topology (S300), and worker→pullmd reachability is already solved by S288 co-location (host-local http://pullmd:3000, dissolved the {62.1} defect). The substrate is partly built: scripts/cocoindex_pipeline/url_source.py (FeedUrlSource/UrlItem) exists under ID-75 WP-C.

Spec tier (suggested): TECH+PLAN (or full chain). Behaviour is largely ratified upstream; the load-bearing work is technical (cocoindex source viability, deletion blast-radius, re-ingest) — EXCEPT the one open product question in §4, which a short PRODUCT slice should pin.

4. Open decisions Liam still owns (do NOT pre-decide in this Task)

(LOAD-BEARING) Surface B synchronous UX. Intended-arch models ingest as an async file-walked flow and never reconciles the manual POST /api/ingest/url route returning the created reference_items row synchronously with an async B1 pipeline. Decide: keep a synchronous Vercel fetch+write for the manual single-URL case (a deliberate, scoped exception to “extraction runs in the worker”), or make manual ingest async (enqueue URL → worker extracts → row appears later — a UX change). This question already lives at ID-110/bl-298 and ID-75 OQ-T1 — the new Task should consume that decision, not re-open it.
cocoindex custom/remote-source viability (Option A vs B2). S300 §5.2/§6 gated Option A on empirically confirming cocoindex accepts a usable remote source (historical bind_target placeholder risk). Pin has since moved (1.0.7) and url_source.py exists — likely retired, but the Task must confirm before committing to A over B2.
JSDOM retention. 07-collapse-list.md §7 (L208) flags jsdom [RATIFIED-RETIRE] ⚠ verify — may be retained by the ontology path. Confirm non-extraction callers before dropping.
unpdf / PDF branch. Surface B’s PDF branch (unpdf) is NOT in ID-42’s removal set; decide whether PDF extraction also moves to the worker (Docling) in this Task or stays Vercel-side.

5. Stale-doc note for the Planner

The 02-data-flow.md/03-tech-stack.md §4 “Cloud Run sidecar” framing is stale (Cloud Run torn down S298 → IONOS/Coolify). The placement (pullmd behind cocoindex, server-side) is unchanged, but the TECH spec must be written against IONOS/Coolify, not Cloud Run — S300 §5.1 cites exactly this as a reason for a fresh Task rather than amending ID-42.

6. Downstream

Cmux E / {46.9} re-gates on THIS new Task (not on the dead {42.11-13} chain) — the parent owns that sequencing and will not dispatch E this session.

7. Key cites (for the Planner)

07-collapse-list.md §5.1 (P1 L140, P4 L142), §5.2 (L152-154), §7 (firecrawl L210, readability L207, jsdom L208 ⚠verify, unpdf L209), §11.2 (firecrawl lockstep L280)
reference/decision-graph.md CX.6 (L228), CX.13 (L231), CX.8 (AGPL L313)
reference/pullmd-wiring-design-s300.md — RATIFIED banner (L19-28); Options A/B/C (L162-301); recommend A+B2 (L325-359); new-Task landing (L379-407); deferral-register gap (L448)
reference/pullmd-cocoindex-intent-s299.md — Liam steer “feeds are remote, never localfs” (L251-273); deferral-register gap (L388)
reference/canonical-pipeline-sequencing.md — {42.11-13} deferred (L60-62, L333-336); {62.1} dissolved by S288 co-location (L189-192)
03-tech-stack.md §3.2 (Vercel reads Postgres only, L50), §7.4 (AGPL HTTP boundary, L170)
code substrate: scripts/cocoindex_pipeline/url_source.py (FeedUrlSource/UrlItem), ID-75 WP-C