TECH -- HTML/URL extraction cutover: one in-house cleaner, two call sites (ID-112.3)

TECH — HTML/URL extraction cutover: one in-house cleaner, two call sites (ID-112.3)

Task: ID-112 — Cutover HTML/URL extraction to an in-house cocoindex-native cleaner; retire pullmd (worker) + Readability (manual) + the dormant Firecrawl/Jina tier. Subtask: {112.3} TECH (precedes {112.4} PLAN per the Q-PLANNER-2 fresh-per-Subtask discipline). Author date: 16/06/2026. Status: SPEC — technical plan. Fresh-Planner artefact (Q-PLANNER-2): authored by a different instance from the {112.2} PRODUCT author and the {112.1} RESEARCH author; both predecessors + the senior-staff review read in full this dispatch. This TECH maps 1:1 to PRODUCT’s seventeen PI-N invariants (one “Proposed changes” block per PI, each independently verifiable), and resolves the three PRODUCT-deferred TECH hand-offs (shared-seam mechanism, OQ-8 A-vs-B2 topology, concrete Trafilatura config). UK English throughout (DD/MM/YYYY, “colour”, “organisation”, “behaviour”). ASCII only.

Predecessors (read in full this dispatch): {112.1} RESEARCH, {112.2} PRODUCT (specs/id-112-html-url-extraction-cutover/), /tmp/claude/id112-staff-review.md (senior-staff review, findings F1-F6 authoritative).

Builds on (do NOT re-implement): ID-110 evidence-pair write contract (reference_ingest RPC, 20260614010200_id110_reference_ingest_rpc.sql); ID-66 B1 co-location compose (deploy/coolify/docker-compose.{staging,production}.yaml); ID-75 cocoindex URL source (url_source.py::FeedUrlSource); ID-42 additive provenance CHECK (20260526074944_id42_pullmd_provenance.sql); the aiohttp worker HTTP wrapper (scripts/cocoindex_pipeline/server.py).

Governing principle (PRODUCT PI-1/PI-4): “one in-house cleaner, defined once, two call sites.” The cutover changes only the body producer — never the write contract, the route topology, or the auth gate. PRODUCT pinned behaviour; this TECH decides the three deferred mechanisms and nothing else (it does NOT re-open any ratified PRODUCT decision).

---

Context

This Task replaces three divergent HTML-cleaning surfaces — pullmd (AGPL, worker), Readability (@mozilla/readability, manual route), and the dormant Firecrawl/Jina tier (content-extractor.ts) — with ONE permissively-licensed in-house cleaner (Trafilatura, Apache-2.0; optional Playwright JS-render, Apache-2.0) defined once and shared by both live call sites. PRODUCT defines the seventeen PI-N behaviour invariants; this TECH grounds each in current file:line reality and supplies the migration plan.

The single most consequential finding of this dispatch is that the shared seam and the OQ-8 topology collapse into one decision: the B1 compose network already runs a trafilatura sidecar AND a playwright sidecar, and the worker already runs an aiohttp HTTP wrapper (server.py) exposing /walk//stage//health. A new POST /extract endpoint on that wrapper is simultaneously the single shared cleaner (hand-off #1) AND the fetch/render runtime (hand-off #2) — so “Option A vs B2” resolves to B2 (a co-located extraction endpoint both the worker component and the TS manual route call), grounded in the container evidence below.

Code-intelligence orientation (cited verbatim, this dispatch, repo knowledge-hub, HEAD 7fe481f74)

Per the binding code-intel discipline, the URL/HTML extraction entrypoints + reference-ingest write path were oriented via gitnexus_query/gitnexus_context (TS corpus) supplemented with grep/direct read for the Python pipeline (scripts/cocoindex_pipeline/*.py), SQL migrations, and the B1 compose files (gitnexus indexes none of these). Raw outputs (verbatim, not paraphrased):

• gitnexus_query({query: "url html extraction trafilatura clean text reference ingest worker", repo: knowledge-hub}) -> top process proc_13_post = Function:app/api/ingest/url/route.ts:POST (startLine 53, endLine 240); process_symbols/definitions surfaced Function:lib/intelligence/content-extractor.ts:extractContent (169-333) + extractMainContentHtml (36-42), and Function:components/create-content/url-ingest-form.tsx:UrlIngestForm (59-362) / handleSubmit (106-180) — the manual route’s UI caller.
• gitnexus_context({name: "extractFromUrl", file_path: "lib/extraction/url.ts"}) -> uid: Function:lib/extraction/url.ts:extractFromUrl (46-127); incoming: {} (gitnexus does NOT index the dynamic import() at route.ts:108); outgoing.calls: extractFromHtml (lib/extraction/html.ts), extractOgMetadata (lib/extraction/og-metadata.ts), extractPdfText (lib/extraction/pdf.ts), validateUrl (lib/extraction/url-validation.ts).
• gitnexus_context({name: "extractFromHtml", file_path: "lib/extraction/html.ts"}) -> uid: Function:lib/extraction/html.ts:extractFromHtml (27-49); incoming.calls: extractFromUrl ONLY (sole caller — Readability retirement is contained); outgoing: {} (jsdom + Readability are lazy import()s, not graph-indexed).

Grounded current state (file:line verified live for this TECH):

• Manual route body producer: app/api/ingest/url/route.ts:108-109 -> dynamic import('@/lib/extraction/url') -> extractFromUrl(url); quality gate :113 (<100->422), :122 (<500->warn); provenance :166 (mimeType ternary on extractionMethod), :171-175 (extraction_metadata JSONB workaround), :191-205 RPC args (14 named params incl. p_extraction_metadata), :206-212 reference_ingest call. normaliseUrl imported route.ts:15 from @/lib/intelligence/content-extractor.
• HTML cleaner (manual): lib/extraction/html.ts:28-49 — lazy import('jsdom') (:33) + import('@mozilla/readability') (:34) -> new JSDOM -> Readability.parse() -> turndown.turndown(article.content) (:46, imports @/lib/extraction/turndown at :10). Returns extractionMethod: 'readability' at url.ts:125.
• Worker body producer: scripts/cocoindex_pipeline/flow.py:_ingest_url_body (2626-2801); HTML branch :2690 result = await _pullmd_fetch(item.url, item.content_epoch), :2692 extraction_method = _pullmd_extraction_method(result.x_source), :2693 pullmd_share_id = result.share_id; PDF branch :2683-2687 Docling (out of scope). declare_row sd :2715-2729 (storage_path = item.url :2719), ri :2733-2759 (body = markdown :2739). _pullmd_fetch is @coco.fn(memo=True) keyed on (url, content_epoch) (adapters.py:248-262).
• _pullmd_extraction_method mapper: flow.py:2448 — maps X-Source -> pullmd_<x_source> for the five known values else None-degrades; the admitted-set frozenset _PULLMD_X_SOURCE_METHODS = frozenset({"readability","playwright","cloudflare","reddit","trafilatura"}) lives at adapters.py:288-290 (mapper and frozenset are distinct sites — do not conflate).
• Worker HTTP wrapper: scripts/cocoindex_pipeline/server.py — aiohttp; _health_handler (GET /health, :182), _stage_handler (POST /stage, :198), POST /walk route (:19, update_blocking(live=False)). Listens on $PORT (default 8080).
• Firecrawl/Jina dormant tier: lib/intelligence/content-extractor.ts:extractContent (170-333) Tier 2.5 Jina + Tier 3 Firecrawl; checkFirecrawlApiKey (136-164, THROWS in production at :148 when key absent), called at startup lib/intelligence/pipeline.ts:557. feed-poller.ts:430 dynamic import('@mendable/firecrawl-js'). pipeline.ts:335/:420 write extraction_method to feed_articles/content_items.
• Provenance CHECK (both tables): 20260526074944_id42_pullmd_provenance.sql:24-26,35-37 admit 'rss_content','fetch','jina_reader','firecrawl','summary_fallback','pullmd_readability','pullmd_playwright','pullmd_cloudflare','pullmd_reddit','pullmd_trafilatura','docling' — neither trafilatura, playwright, nor unpdf (the reason the manual path writes NULL).
• RPC signature: 20260614010200_id110_reference_ingest_rpc.sql:64-79 — 14 named params (p_source_url … p_extraction_metadata jsonb DEFAULT '{}', p_op_id uuid DEFAULT NULL); :129 inserts extraction_method as a hardcoded NULL literal with the comment “ID-42 CHECK rejects readability/unpdf”; :130 records the true producer in p_extraction_metadata. Server-side uuid5 PKs :98-101 (namespace fbfaf1ff-1ee4-583c-9757-1674465b2ec1, sd:/ri: prefixes); idempotency :107-115 + ON CONFLICT (id) DO NOTHING :132/:144.
• Container reality (the OQ-8 decider): trafilatura==2.0.0 is a DIRECT requirements.txt dep (:11) — importable in-process in the cocoindex image; playwright is importable Python ({112.1} RESEARCH empirical, playwright==1.50.0 PRESENT) BUT the cocoindex image is buildpack-built with NO apt layer (requirements.txt:30 verbatim: “the cocoindex image is buildpack-built ({66.7}) with no apt layer, so a system pandoc is unavailable”), so Playwright’s ~400 MB browser binaries cannot be apt/playwright install-provisioned in-image without a Dockerfile rewrite. A pullmd-playwright sidecar (~3.7 GB image, :8002/render) AND a pullmd-trafilatura sidecar (:8001/extract) ALREADY run on the B1 compose network (docker-compose.production.yaml:240-262), currently wired as pullmd’s internal sidecars (PLAYWRIGHT_URL/TRAFILATURA_URL env on the pullmd service :211-212).

ccc search fallback was NOT required — the codebase is richly indexed and the survey above is a brownfield cutover with an existing, well-instrumented pipeline. No greenfield disclaimer applies.

---

TECH hand-off resolutions

Hand-off #1 — The shared-seam MECHANISM (PI-4/PI-5)

Decision: a POST /extract HTTP endpoint on the B1 cocoindex worker (extends server.py). One Trafilatura cleaner module (scripts/cocoindex_pipeline/extract.py); BOTH call sites resolve their cleaning behaviour from it — the worker component imports it directly (same Python process), the TS manual route reaches it over HTTP.

Rationale (constraints from the dispatch brief, each addressed):

• (b) shell-to-Python — REJECTED. Vercel serverless functions are Node-only; there is no Python runtime and no persistent process to shell into. route.ts runs on Vercel (maxDuration = 60, route.ts:18). Shelling out is impractical and is rejected.
• (c) a vetted TS Trafilatura-equivalent — REJECTED. This is the “second-implementation trap” D1 set out to kill (staff review S3): a TS reimplementation cannot be byte-parity-proven against Python Trafilatura’s text-density heuristics, and maintaining two cleaners re-creates exactly the two-divergent-stacks problem (PRODUCT Problem section). Rejected.
• (a) a B1 extract endpoint both paths call — ADOPTED. The cleaner config lives in ONE Python module (extract.py); the worker component calls it in-process; the manual route POSTs HTML (or a URL) to POST /extract and receives cleaned text. There is exactly one Trafilatura config literal and one quality-gate definition. This is the convergence the brief anticipated: the endpoint that gives the sync route access to Python Trafilatura IS the single shared seam.

Single-config invariant (how “defined once” is enforced): extract.py exports one TrafilaturaConfig (the config values below) and one apply_quality_gate(text) -> GateVerdict. The worker _ingest_url_body HTML branch calls clean_html(html, config) directly; the POST /extract handler calls the SAME clean_html + apply_quality_gate; the TS manual route calls the endpoint. No config literal is duplicated in TS. Golden fixtures (PI-6) assert byte-identical cleaned output for the in-process call and the over-HTTP call on the same fixture input — proving the seam is genuinely shared, not two copies.

Hand-off #2 — OQ-8 topology: Option A vs B2 (F3 is the real risk)

Decision (RATIFIED, Liam S366): /extract-on-B1 is a PURE CLEANER (HTML in -> clean text out), NOT a fetch/render service — so Option A (extract-in-@coco.fn) is rejected in favour of a co-located cleaner endpoint, and the FETCH/SSRF stays on each caller, not on B1. The decider is NOT the source enumeration (FeedUrlSource is viable + probe-proven, url_source.py:20-44) — it is keeping the SSRF surface single and the B1 endpoint attack-surface minimal.

The four ratified properties of POST /extract (S366):

1. Pure cleaner, no fetch/SSRF on B1. The Vercel manual route performs its OWN SSRF-gated fetch (its existing validateUrl at route.ts:76/url.ts:49) and POSTs the already-fetched HTML body to /extract; /extract runs Trafilatura extract() on that HTML and returns clean text. /extract does NO fetch_url, owns NO SSRF gate — there is exactly one SSRF surface per caller (the Vercel route’s, and the worker’s validate_url at flow.py:2658), never a second one on B1. The worker’s HTML branch runs in the SAME Python process as the handler, so it imports clean_html directly (no HTTP hop, no second fetch) — it already fetched upstream of _pullmd_fetch’s replacement.
2. Soft coupling — DEGRADE, do not fail. If B1 /extract is unreachable, the manual route returns a named HTTP 503 “extraction temporarily unavailable, retry shortly” (recoverable), NOT a 500. There is NO in-process Readability fallback — an in-process fallback would keep @mozilla/readability alive past the PI-12/PI-13 deletion, which is explicitly rejected. The 503 is the entire failure contract.
3. Dedicated bearer token, NOT CRON_SECRET. /extract is gated by its OWN bearer secret (e.g. EXTRACT_API_TOKEN), distinct from CRON_SECRET (which gates /walk’s token-burning corpus walks). Do not conflate privilege classes — a cleaner call and a corpus-walk trigger are different blast radii.
4. Hardened endpoint. /extract gets its OWN rate-limit + a per-route request-body cap. The 50 MB /stage cap was justified by “compose-internal only” (server.py / Traefik labels route /stage nowhere); that premise dies once /extract is reachable over Traefik, so /extract sets a tighter body cap appropriate to a single HTML page (aligned to the manual route’s 20 MB MAX_CONTENT_SIZE, url.ts:30).

Deciding container evidence (why a co-located cleaner endpoint, not in-@coco.fn):

• The HTTP wrapper substrate already exists. server.py already serves /walk//stage//health over aiohttp; adding POST /extract is a route delta, not new infrastructure.
• trafilatura==2.0.0 is a direct in-process dep (requirements.txt:11), so the cleaner runs in the cocoindex image with no new system packages — the buildpack-no-apt constraint (requirements.txt:30) is irrelevant to a pure-HTML cleaner (no browser binary needed).

Playwright / JS-render — RESOLVED, deferred to v1.1 (Liam S366; deferral-register V11-13). v1 ships static-only: Trafilatura extract() on caller-fetched HTML, NO Playwright, NO JS-render. This is a settled scope decision, not an open question. The v1.1 JS-render lane will need a STANDALONE render sidecar — the existing pullmd-playwright sidecar (docker-compose.production.yaml:248-249) is wired into pullmd and dies with pullmd when the worker retires it (Checker N1), so it cannot be the v1.1 render runtime. v1.1 must stand up its own render service; that is out of ID-112 scope and tracked in the deferral register, not flagged here as a Liam-bound OQ.

Hand-off #3 — Concrete Trafilatura config (PI-4)

extract.py TrafilaturaConfig (carry verbatim into the Executor brief):

trafilatura.extract(
    html,
    output_format="txt",        # clean TEXT, not markdown -- Markdown NOT load-bearing on the URL path (PI-1)
    favor_recall=True,          # prefer capturing the full article body over aggressive precision pruning
    include_comments=False,     # strip reader comments -- they are boilerplate for the reference layer
    include_tables=True,        # tables often ARE the article substance (procurement/spec pages)
    include_formatting=False,   # plain text; no inline markup needed downstream
    with_metadata=False,        # title/author/date come from extractOgMetadata (manual) / item ledger (worker)
    url=final_url,              # passed for relative-link + canonical resolution
)

• output_format="txt" — PI-1 reframes the URL path to “boilerplate-stripped clean text” with Markdown incidental: embedding, classification, and reference_items.body never parse Markdown structure, and there is no reference reader UI (ID-111 unbuilt). txt avoids the Markdown-serialisation degradation Trafilatura shows on borderline inputs and removes the Turndown tail entirely. (Markdown-fidelity stays load-bearing ONLY on the binary->Docling path, out of scope.)
• favor_recall=True — the {112.1} empirical caveat showed Trafilatura degrades toward near-full-text on tiny inputs but cleans correctly on realistically-sized articles; favor_recall biases toward keeping article body (better for the embedding/classification corpus than aggressive precision pruning that can drop short paragraphs).
• include_comments=False — comment threads are boilerplate for an evidence-layer reference.

Quality gate (PI-5, ported verbatim from route.ts:113/:122): apply_quality_gate(text) returns REJECT when len(text) < 100, WARN when 100 <= len(text) < 500, else OK. The manual route maps REJECT->HTTP 422 (preserving route.ts:114-121); the worker maps REJECT->structured per-item failure (BI-19 containment, NO partial rows — flow.py:2636-2638). WARN appends the existing warning string on the manual path / a structured log on the worker.

Fixtures (PI-6): golden fixtures use realistically-sized article HTML (8+ substantial paragraphs + nav/aside/footer chrome — the {112.1} empirical shape), NOT sub-paragraph synthetic fragments. Stored under scripts/tests/fixtures/extraction/ (Python) with a TS mirror reference for the over-HTTP parity assertion.

---

Migration plan (ordering — gated by PI-12)

1. Land extract.py + the hardened POST /extract endpoint (worker, behind no traffic switch — additive). The endpoint is a pure cleaner (HTML in -> clean text out): dedicated bearer (EXTRACT_API_TOKEN, not CRON_SECRET), own rate-limit, own body cap, and a Traefik route added alongside /walk+/health (docker-compose.{staging,production}.yaml labels). Worker HTML branch re-points _pullmd_fetch -> in-process clean_html behind the SAME surrounding contract (SSRF gate flow.py:2658, classification, embedding, declare_row all unchanged).
2. Land the append-only provenance migration (PI-10) BEFORE ID-45 re-ingests.
3. Re-point the manual route body producer to POST /extract; write typed extraction_method (PI-11).
4. Relocate normaliseUrl -> lib/extraction/url-normalise.ts (PI-14).
5. NO deletion of legacy surfaces until the ID-45 cutover window confirms zero legacy traffic (PI-12); the deletion set (PI-13/PI-15) lands as a SEPARATE gated change after prod-proving.

Migration discipline (supabase/CLAUDE.md): supabase migration new + foreground db push (never background — it prompts) + type regen via supabase gen types. Verify supabase/.temp/project-ref before any push (staging turayklvaunphgbgscat).

---

Proposed changes per PI (1:1 with PRODUCT)

A. Scope framing & route topology

PI-1 — One cleaner, two call sites; pullmd retirement is worker-only. Proposed change: extract.py clean_html() is the single cleaner; the worker (flow.py:2690, replacing _pullmd_fetch) and the manual route (route.ts:108, replacing extractFromUrl/Readability) both resolve from it. output_format="txt" (clean text, not hard Markdown). Commit messages name Readability (manual) and pullmd (worker) as the two retired cleaners; no artefact claims the manual route ran pullmd. Verify: exactly one clean_html definition; both sites call it; the body requirement is clean text not Markdown; commits name the two distinct retired cleaners.

PI-2 — The synchronous manual route is retained. Proposed change: NONE to topology. route.ts:POST stays synchronous; only step 6 (route.ts:108-109) changes — the route keeps its OWN SSRF-gated fetch (validateUrl + fetch, the existing url.ts:48-90 front matter) and POSTs the fetched HTML body to POST /extract, receiving clean text back (a thin TS wrapper cleanViaWorker(html, finalUrl) in lib/extraction/). B1 unreachable -> named 503 “extraction temporarily unavailable, retry shortly” (Hand-off #2 property 2), NOT a 500 and NOT an in-process Readability fallback. The route still returns { id, title, source_url, ... } synchronously (route.ts:224-233) on success. PDF still routes to unpdf in-process (out of scope). Verify: the manual route returns the landed reference id synchronously on success; a /extract outage yields a recoverable 503, not a 500 or a Readability fallback; no async job-handle shape.

PI-3 — No new async ingestion surface is built for v1. Proposed change: NONE. url_source.py stays feed_articles WHERE passed = true (url_source.py:64-69); no url_import producer, no job API, no async UI. The POST /extract endpoint is a stateless cleaner call, NOT an async ingestion producer (it lands no rows; the caller owns the write). Verify: no new url_import producer/source, job API, or async UI; worker source enumeration unchanged.

B. The single shared extractor seam (CORE)

PI-4 — The cleaner is defined exactly once. Proposed change: scripts/cocoindex_pipeline/extract.py exports one TrafilaturaConfig (Hand-off #3 values) + clean_html(html, *, url) -> str. Worker imports it in-process; POST /extract handler calls it; TS route reaches it over HTTP. No TS HTML-cleaning reimplementation; no duplicated config literal. Verify: exactly one Trafilatura config definition; both sites resolve cleaning from it; no parallel TS cleaner.

PI-5 — The quality gate is defined exactly once and shared. Proposed change: extract.py apply_quality_gate(text) -> REJECT(<100) / WARN(<500) / OK, applied inside /extract so the verdict travels in the endpoint response (e.g. { text, verdict, warnings }). Manual route maps REJECT->422 (preserving route.ts:113-121), WARN->warning push (route.ts:122-126); worker (calling clean_html+apply_quality_gate in-process) maps REJECT->BI-19 structured per-item failure. The 422 (content-too-short) is distinct from the 503 (endpoint-unreachable, Hand-off #2 property 2): a clean REJECT verdict is still a successful /extract response. Thresholds are NOT weakened. Verify: one gate definition referenced by both sites; manual route still 422-below-100 / warn-below-500; worker applies the same thresholds; a too-short page yields 422 (not 503), an endpoint outage yields 503 (not 422).

PI-6 — Cleaner parity is proven by golden fixtures. Proposed change: scripts/tests/fixtures/extraction/*.html realistically-sized fixtures; a Python test asserts clean_html(fixture) (in-process) == the POST /extract response body for the same fixture (over-HTTP), and the TS manual-route test asserts its received clean text matches the same expected output. Same-config -> same-output assertion; no sub-paragraph fragment fixtures. Verify: golden-fixture tests use realistically-sized HTML and assert byte-identical cleaned output across both call sites; no tiny-fragment fixture.

C. The ID-110 evidence-pair write contract (must NOT regress)

PI-7 — The reference_ingest RPC contract is preserved on the manual path. Proposed change: SIGNATURE unchanged. route.ts:191-212 keeps the 14-named-param call (p_source_url … p_op_id) and storage_path = source_url parity (RPC :126-127); sd inserted before ri (:121/:134). The PI-11 migration is a CREATE OR REPLACE over the IDENTICAL 14-arg signature — only the function BODY changes (the extraction_method insert derives from p_extraction_metadata->>'extractor' instead of a NULL literal); because the signature is byte-identical, grants persist on the same function identity (no DROP/re-GRANT). p_body now carries the Trafilatura clean text. Verify: same 14-param signature (no added/removed/renamed param); storage_path = source_url; sd-before-ri; authenticated/service_role grants unchanged (same function identity, not re-granted).

PI-8 — Deterministic uuid5 PKs + idempotency converge on both paths. Proposed change: NONE. uuid5 PKs minted server-side in the RPC (:98-101) on the manual path and in flow.py:2711-2712 on the worker, both from the normalised URL via the shared namespace; idempotency via :107-115 + ON CONFLICT (id) DO NOTHING. The cutover touches only the body producer. Verify: re-ingesting a landed URL on either path creates no new rows / reports already-existed; uuid5 PKs match across paths for the same normalised URL.

PI-9 — Auth gating is unchanged. Proposed change: NONE to the manual-route gate. route.ts:57-58 getAuthorisedClient(['admin','editor']) / auth.success / authFailureResponse(auth) runs BEFORE the /extract call (step 6 sits after step 1) — the owner-gate is untouched. The /extract endpoint has its OWN, SEPARATE transport gate: a dedicated bearer secret (e.g. EXTRACT_API_TOKEN, distinct from CRON_SECRET; Hand-off #2 property 3) + its own rate-limit + body cap (property 4). The two gates are layered, not conflated — the route’s owner-gate authorises the human; the endpoint’s bearer authorises the route-to-B1 transport. Verify: an unauthenticated/non-owner manual-route request is rejected via authFailureResponse(auth) before any extraction runs; /extract rejects requests without its dedicated bearer; /extract does NOT accept CRON_SECRET as its credential.

D. Provenance enum re-map (append-only)

PI-10 — Append-only provenance migration; no prune. Proposed change: a NEW migration supabase/migrations/<ts>_id112_inhouse_extraction_provenance.sql mirroring the ID-42 additive pattern (20260526074944...sql:22-26,33-37): DROP CONSTRAINT IF EXISTS then ADD CONSTRAINT ... CHECK on BOTH source_documents and feed_articles, extending the admitted ARRAY with 'trafilatura', 'playwright', 'unpdf'. All existing pullmd_*/firecrawl/docling values retained; no value removed. Type regen follows. Verify: the migration only ADDs to both CHECK arrays; pre-existing legacy-value rows still satisfy the constraint; no value removed.

PI-11 — The manual path writes a typed extraction_method and stops the NULL-into-metadata workaround. Proposed change (RESOLVED, Liam S366 — Option 2: derive server-side from metadata; the 14-arg signature is UNCHANGED): (i) RPC: a NEW migration CREATE OR REPLACE FUNCTION public.reference_ingest(...) keeps the EXISTING 14 named params verbatim (p_source_url … p_op_id) — ZERO signature change — and changes ONLY the body: replace the hardcoded NULL literal at 20260614010200...sql:129 with COALESCE(p_extraction_metadata->>'extractor', NULL) so the typed column is derived server-side from the JSONB the route ALREADY passes ({"extractor":"trafilatura"|"unpdf",...}, route.ts:171-175). Because it is a CREATE OR REPLACE over the identical signature, there is NO second overload, NO DROP-old-signature, NO re-GRANT dance — grants persist on the same function identity. (ii) route.ts: the route already sets extraction_metadata.extractor to the true producer (trafilatura for HTML via the new cleaner, unpdf for PDF) — the body-derivation picks it up; the existing extractionMethod value just needs to be 'trafilatura' on the HTML branch (was 'readability'). The extractor key is retained in extraction_metadata (it is now the SOURCE of the derivation, not a workaround). (iii) the typed column ends up populated WITHIN the single atomic RPC write — a follow-up UPDATE outside the RPC is REJECTED (it would break the atomic single-write evidence-pair seam). (iv) CV-16 ontology updated in lockstep. Labels are admin/analytics-facing only. Verify: a manual HTML ingest writes extraction_method = 'trafilatura' (PDF writes 'unpdf') into the typed column, not NULL, derived inside the RPC from p_extraction_metadata->>'extractor'; the reference_ingest signature is byte-identical (still 14 params); no out-of-RPC UPDATE writes the column; CV-16 reflects the new labels; no end-user surface renders the label.

E. Deletion set & gating

PI-12 — Legacy-surface deletion is gated behind the ID-45 cutover window. Proposed change: the migration plan above lands extract.py + the endpoint + the enum migration FIRST; the deletion set (PI-13/PI-15) is a SEPARATE change that does NOT merge until the worker path is prod-proven and a cutover window confirms zero legacy traffic. {112.4} PLAN sequences deletion as a terminal, gated Subtask. Verify: in-house cleaner + enum migration land first; no legacy-surface deletion commit merges before the ID-45 cutover-window/zero-traffic confirmation.

PI-13 — Deletion set: DROP exactly two deps; KEEP the shared survivors. Proposed change: package.json removes ONLY @mendable/firecrawl-js and @mozilla/readability. KEEP jsdom (Vitest environment + ~40 tests; sole extraction caller was html.ts:33), unpdf (PDF branch, out of scope), turndown (shared with lib/content/html-to-markdown.ts + app/api/upload/route.ts:17 — F4), normaliseUrl (relocated, PI-14). Deleted code: the Firecrawl/Jina tiers in content-extractor.ts/feed-poller.ts; the Readability path in extraction/url.ts+html.ts (the HTML branch — the PDF branch in url.ts:95-110 stays for the manual PDF path). Verify: only the two deps removed from package.json; jsdom/unpdf/turndown remain; no survivor import breaks.

PI-14 — normaliseUrl is relocated, not deleted. Proposed change: move normaliseUrl (content-extractor.ts:96) to a new lib/extraction/url-normalise.ts; update the two importers (route.ts:15, lib/intelligence/pipeline.ts:10) to direct-import the new home (no barrel re-export). Behaviour byte-identical. Use gitnexus_rename (never find-and-replace) for the move; run ast-dataflow-rename-sweep after to verify completeness. Verify: normaliseUrl resolves from lib/extraction/url-normalise.ts; both importers updated; behaviour unchanged; no barrel re-export.

PI-15 — The checkFirecrawlApiKey() startup fail-fast is removed in lockstep with the Firecrawl tier. Proposed change: in the SAME deletion change as the Firecrawl tier, remove checkFirecrawlApiKey() (content-extractor.ts:136-164) and its startup call (pipeline.ts:557), plus isFirecrawlConfigured (content-extractor.ts:27) and the firecrawlKeyMissing/firecrawlWarningLogged module state if no other caller survives. The app must boot in production without FIRECRAWL_API_KEY. Verify: after the tier is removed, no startup path calls checkFirecrawlApiKey(); the app boots in production without the Firecrawl env var.

F. Backlog & doc reconciliation

PI-16 — bl-308 closes on landing; no double-build. Proposed change: (Orchestrator/Curator write, NOT Planner/Executor — flagged in Reporting.) The landing change cites bl-308; the Curator moves bl-308 parked -> closed. No net-new TS PullMD client is introduced (the shared-seam mechanism is the B1 endpoint, not a TS client). Verify: bl-308 referenced by the landing change and moved to closed; no net-new TS PullMD client.

PI-17 — Descriptive ontology docs are updated as doc-only follow-ons. Proposed change: CV-33 (reference_items.body “PullMD/Docling markdown”) and CV-16 descriptions updated to “in-house Trafilatura/Docling body producer”. Documentation-only; no code rework. (Docs-site upkeep is automated in the docs-site repo per its own .claude/ lane.) Verify: CV-33/CV-16 no longer describe pullmd as the live HTML body producer; the update is documentation-only.

---

KH quality bars (inherited by every Executor Subtask)

• TS: auth.success + authFailureResponse(auth) (route.ts:57-58, unchanged); sb()/tryQuery() Supabase safety (route.ts:89/:206); no barrel re-exports (direct file imports — the url-normalise.ts relocation MUST be a direct import); TanStack Query only (no fetch surface added to the client); semantic design tokens only (no UI in this Task); bun run test (never bun test).
• Python: python3 -m pytest scripts/tests/; worktree pytest runs from the worktree CWD (namespace-package hazard, scripts/CLAUDE.md); PYTHONUNBUFFERED=1 for background output; pipeline service-account UUID a0000000-0000-4000-8000-000000000001 where a userId is needed.
• Tests: behaviour-first per reference/test-philosophy.md — the fixture-parity tests (PI-6) assert observable cleaned output, not Trafilatura internals; the shared Supabase mock for any TS test touching the route.
• Migrations: DDL via CLI only (supabase migration new + foreground db push); SET search_path = public, extensions on any new PL/pgSQL; verify project-ref before push; type regen after.
• Code-intel discipline: gitnexus_impact({target, direction: "upstream"}) before modifying extractFromUrl/extractFromHtml/extractContent/_ingest_url_body/normaliseUrl (warn on HIGH/CRITICAL); gitnexus_detect_changes() before commit; gitnexus_rename for the normaliseUrl move.
• Client discipline: literal <CLIENT> in any artefact; no client names in filenames/commands.

---

Empirical verification (Q-EX2 forcing function — external-library symbols)

Import-and-call check carried forward from {112.1} RESEARCH (16/06/2026, pinned versions per requirements.txt), re-affirmed for the symbols THIS TECH mandates:

Library (pinned)	Symbol path	Result
trafilatura==2.0.0 (Apache-2.0)	trafilatura.extract(html, output_format="txt", favor_recall=..., include_comments=..., include_tables=...)	PRESENT (extract confirmed {112.1}; the kwargs above are stable Trafilatura 2.x extract params). This is the ONLY Trafilatura call the cutover uses — /extract is a pure cleaner on caller-fetched HTML, so fetch_url is NOT in the v1 datapath.
cocoindex==1.0.7	@coco.fn / update_blocking(live=False) (worker substrate — unchanged)	PRESENT ({112.1})
playwright==1.50.0 (Apache-2.0)	(NOT used in v1)	N/A for v1 — static-only ships (RESOLVED, Liam S366); JS-render deferred to v1.1 behind a STANDALONE render sidecar (the pullmd-playwright sidecar dies with pullmd — Checker N1). No Playwright symbol is on the v1 datapath.

No ABSENT/SIGNATURE_DRIFT for any symbol this TECH relies on -> spec chain may proceed to {112.4} PLAN.

Verification caveat carried to PLAN/Executor (NOT a blocker): the exact output_format="txt" + favor_recall=True + include_tables=True kwarg COMBINATION must be import-and-call confirmed against trafilatura==2.0.0 at Executor time on a realistically-sized fixture (the {112.1} empirical confirmed output_format="markdown" behaviour but the spec now selects txt — a different serialiser path). Trafilatura’s small-input degradation ({112.1} BEHAVIOUR caveat) is mitigated by the quality gate (PI-5), not by the config alone.

---

TECH open questions — ALL RESOLVED (Liam-ratified S366)

No remaining Liam-bound OQ before {112.4} PLAN. The three TECH-surfaced questions were ratified by Liam at S366; their resolutions are encoded in the hand-offs and PI blocks above:

1. OQ-112-T1 — Playwright JS-render lane. RESOLVED: static-only v1. v1 ships Trafilatura extract() on caller-fetched HTML, NO Playwright, NO JS-render. SPA JS-render is deferred to v1.1 (deferral-register V11-13) and is OUT of ID-112 scope. The v1.1 lane needs a STANDALONE render sidecar — the existing pullmd-playwright sidecar dies with pullmd and cannot be reused (Checker N1). Not a flagged OQ; a settled scope boundary. (See Hand-off #2.)
2. OQ-112-T2 — POST /extract endpoint exposure + coupling. RESOLVED: pure cleaner, soft coupling. /extract is exposed over the existing Traefik ingress (joining /walk//health) but gated by a DEDICATED bearer secret (EXTRACT_API_TOKEN), NOT CRON_SECRET. It is a PURE cleaner (HTML in -> clean text out): no fetch, no SSRF on B1 (the caller fetches). It has its own rate-limit + body cap. A /extract outage degrades the manual route to a named recoverable 503, never a 500 and never a Readability fallback. (See Hand-off #2 properties 1-4 + PI-2/PI-9.)
3. OQ-112-T3 — typed extraction_method write. RESOLVED: derive server-side from metadata (Option 2). The 14-arg reference_ingest signature is UNCHANGED; a CREATE OR REPLACE redefines the body to derive extraction_method := p_extraction_metadata->>'extractor' (the route already passes it). The 15th-param approach is DROPPED entirely — PI-7 is honoured literally (zero signature change, no overload, no DROP/re-grant). No out-of-RPC UPDATE (it would break the atomic single-write seam). (See PI-7 + PI-11.)