Skip to content
Canonical

PRODUCT — HTML/URL extraction cutover: one in-house cleaner, two call sites (ID-112.2)

PRODUCT — HTML/URL extraction cutover: one in-house cleaner, two call sites (ID-112.2)

Section titled “PRODUCT — HTML/URL extraction cutover: one in-house cleaner, two call sites (ID-112.2)”

Task: ID-112 — Cutover HTML/URL extraction to an in-house cocoindex-native extractor; retire pullmd + Firecrawl + Readability. Subtask: {112.2} PRODUCT (precedes {112.3} TECH → {112.4} PLAN per the Q-PLANNER-2 fresh-per-Subtask discipline). Author date: 16/06/2026. Author: Task Planner (FRESH dispatch — Q-PLANNER-2; read {112.1} RESEARCH + the senior-staff review fresh, NOT the RESEARCH author). Tier: PRODUCT (of a RESEARCH → PRODUCT → TECH → PLAN chain). Predecessor: {112.1} RESEARCH (specs/id-112-html-url-extraction-cutover/RESEARCH.md) — ratified; /tmp/claude/id112-staff-review.md senior-staff review (findings F1–F6 authoritative).

Every invariant is numbered PI-N with a one-line Verify means a Checker can gate against acceptance criteria. UK English throughout (DD/MM/YYYY, “colour”, “organisation”, “behaviour”). Cross-references are by path; this document cites RESEARCH and the staff review — it does NOT re-dump them. It pins behaviour, not mechanism: HOW the synchronous route reaches the shared cleaner (shell to Python / call a worker extract endpoint / vetted TS-equivalent) and WHERE the fetch/render runs (in-component vs separate worker) are {112.3} TECH decisions, carried here as explicit hand-offs, NOT resolved.

Summary

Section titled “Summary”

Knowledge Hub cleans fetched web pages into embeddable, classifiable boilerplate-stripped text (Markdown serialisation incidental — see PI-1) for the reference layer on two call sites today, each on different, AGPL-encumbered or data-egressing tooling: the asynchronous cocoindex worker uses pullmd (AGPL-3.0), and the synchronous manual single-URL route uses Readability (@mozilla/readability). A third, dormant tier (Firecrawl + Jina) sits in the RSS-feed extractor. This Task replaces all three HTML-cleaning surfaces with one in-house, permissively-licensed cleaner (Trafilatura, Apache-2.0; optional Playwright JS-render, Apache-2.0), defined once and shared by both live call sites. The framing is deliberately “one in-house cleaner, two call sites” — NOT “converge both paths off pullmd”: the manual route was never on pullmd (the worker-only retirement). Everything downstream of the body producer — the ID-110 evidence-pair write contract, deterministic uuid5 PKs, idempotency, and the sync/async route split — is preserved unchanged.

Problem

Section titled “Problem”

Two motivations, both grounded in {112.1} RESEARCH §1: (a) licence/supply-chain — pullmd and Firecrawl’s self-host core are both AGPL-3.0, shipped into every per-client deploy, and pullmd’s vendor pushed two breaking body-format changes under moving tags inside one week (RESEARCH §1); (b) single-cleaner integrity — today the worker cleans HTML with pullmd and the manual route cleans HTML with Readability, two divergent extractor stacks producing the same surface (reference_items.body). The cutover’s real risk is re-creating that divergence (a Python cleaner in the worker + a separate TS reimplementation on the route), so the single-seam property below is load-bearing, not cosmetic (staff review §3, F-mitigation).

Code-intelligence orientation (cited verbatim, this dispatch, repo knowledge-hub, HEAD 7fe481f74):

  • gitnexus_query({query: "url html extraction reference ingest pipeline"}) → top process proc_13_post = Function:app/api/ingest/url/route.ts:POST (startLine 53, endLine 240) — the manual route; definitions surfaced Function:scripts/cocoindex_pipeline/flow.py:_ingest_url_body (startLine 2625, endLine 2800) and Function:lib/intelligence/content-extractor.ts:extractContent (startLine 169, endLine 333). process_symbols also returned Class:scripts/cocoindex_pipeline/adapters.py:SourceProvenance (293–305) — the worker provenance struct the enum re-map touches.
  • gitnexus_context({name: "extractFromUrl", file_path: "lib/extraction/url.ts"}) → uid: Function:lib/extraction/url.ts:extractFromUrl (46–127); incoming: {} (gitnexus does NOT index the dynamic import() at route.ts:108); outgoing.calls: extractFromHtml (lib/extraction/html.ts), extractOgMetadata, extractPdfText, validateUrl.
  • gitnexus_context({name: "extractFromHtml", file_path: "lib/extraction/html.ts"}) → uid: Function:lib/extraction/html.ts:extractFromHtml (27–49); incoming.calls: extractFromUrl ONLY (sole caller — Readability retirement is contained); outgoing: {} (jsdom + Readability are lazy import()s, not graph-indexed).
  • gitnexus_context({name: "extractContent", file_path: "lib/intelligence/content-extractor.ts"}) → uid: Function:lib/intelligence/content-extractor.ts:extractContent (169–333); incoming.calls: processFeedSource (lib/intelligence/pipeline.ts) ONLY (the dormant Firecrawl/Jina tier; single caller).
  • grep (Python/SQL, not gitnexus-indexed): worker write path verbatim — flow.py:2690 result = await _pullmd_fetch(item.url, item.content_epoch); :2692 extraction_method = _pullmd_extraction_method(result.x_source); :2719 "storage_path": item.url; :2726/:2727 write extraction_method/pullmd_share_id. The PDF (Docling) branch defaults extraction_method = "docling" (flow.py:2684) — out of scope.
  • grep + direct read (migration) — reference_ingest RPC (20260614010200_id110_reference_ingest_rpc.sql:65–78) has 14 named parameters (p_source_url … p_extraction_metadata, plus the easily-missed 14th p_op_id uuid DEFAULT NULL); line :130 inserts p_extraction_metadata with the comment “true producer recorded here, not the CHECKed column” — the NULL-into-metadata workaround, confirmed. (RESEARCH §3’s “14-arg” figure is correct — no discrepancy.)
  • grep (current CHECK admitted set, both tables, 20260526074944_id42_pullmd_provenance.sql:24–26,35–37, verbatim): 'rss_content','fetch','jina_reader','firecrawl','summary_fallback','pullmd_readability','pullmd_playwright','pullmd_cloudflare','pullmd_reddit','pullmd_trafilatura','docling' — admits neither readability nor unpdf (the reason the manual path writes NULL).
  • grep (turndown importers) — shared across survivors: lib/intelligence/content-extractor.ts, lib/intelligence/feed-poller.ts, lib/content/html-to-markdown.ts, lib/extraction/turndown.ts, lib/extraction/html.ts, app/api/upload/route.ts (F4 — turndown must NOT be in the deletion set).

Goals / Non-goals

Section titled “Goals / Non-goals”

Goals: replace the three HTML-cleaning surfaces with one shared in-house cleaner; append the typed provenance labels so the manual path stops writing NULL-into-metadata; relocate the normaliseUrl helper; gate legacy-surface deletion behind the ID-45 cutover window.

Non-goals (scope fence — do NOT fold in):

  • PDF / Docling extraction — untouched. The worker PDF branch stays on Docling; the manual PDF branch stays on unpdf. ID-112 touches only the HTML→clean-text cleaning step (OQ-4). Markdown-fidelity IS load-bearing for the binary→Docling path — which is exactly why that path is out of scope here.
  • The embedding zero-egress swap (LiteLLMEmbedder → SentenceTransformerEmbedder) — orthogonal; noted only as a sequencing sibling (both want to land before ID-45 re-ingest), NOT scoped here (RESEARCH §6, F6).
  • OQ-45-1 / ingestion_source forward-set — belongs to ID-45 / bl-106, NOT here.
  • A url_import cocoindex worker producer or async-ingestion UI — explicitly NOT built (PI-3; OQ-7 RATIFIED).
  • Surface-deletion mechanics (the exact removal steps) — TECH/PLAN concern; PRODUCT encodes only the gating invariant (PI-12).

Behaviour

Section titled “Behaviour”

A. Scope framing & route topology

Section titled “A. Scope framing & route topology”

  1. PI-1 — One cleaner, two call sites; pullmd retirement is worker-only. ID-112 replaces Readability on the synchronous manual route AND pullmd on the asynchronous worker with a single shared in-house cleaner. The load-bearing requirement on the HTML/URL path is boilerplate-stripped clean text — chrome (nav, aside, footer, ads) removed, leaving the article body — NOT a particular serialisation format. The work is framed as “one in-house cleaner, two call sites,” NOT “converge both paths off pullmd” — the manual path was never on pullmd (id-110 TECH §0 OQ-B explicitly rejected a PullMD TS client; the manual route shipped on extractFromUrl/Readability). Markdown-as-format is conventional, NOT load-bearing for the HTML/URL path: embedding (LiteLLMEmbedder), classification, and storage to reference_items.body never parse Markdown structure, and there is no reference reader UI yet (ID-111 unbuilt). Markdown-fidelity is load-bearing ONLY for the binary→Docling path (out of ID-112 scope). Trafilatura’s output_format (txt vs markdown) is therefore a free TECH choice. Verify: spec/code framing and commit messages name Readability (manual) and pullmd (worker) as the two distinct retired cleaners; the cleaner requirement is stated as boilerplate-stripped clean text (not a hard Markdown-output requirement); no artefact claims the manual route ran pullmd.

  2. PI-2 — The synchronous manual route is retained. The interactive single-URL paste route (app/api/ingest/url/route.ts:POST) remains synchronous: an owner pastes a URL, waits, and receives the landed reference id plus any warnings in the same request. The two-route split (interactive sync manual + batch async worker) is a correct latency/topology split, not historic debt (OQ-7 RATIFIED). Verify: the manual route still returns the landed reference id synchronously in its response body; it does not return an async “processing…” / job-handle shape.

  3. PI-3 — No new async ingestion surface is built for v1. This Task does NOT build a url_import cocoindex worker producer, a job-submit/poll API, or an async-ingestion UI. The deferred url_import join (url_source.py:59) stays deferred (OQ-7 RATIFIED). Verify: no new url_import producer/source, job API, or async UI is added; the worker source enumeration remains feed_articles WHERE passed = true.

B. The single shared extractor seam (CORE)

Section titled “B. The single shared extractor seam (CORE)”

  1. PI-4 — The cleaner is defined exactly once. The Trafilatura extraction configuration (output_format, recall/precision knobs, comment handling) is defined in a single shared definition and consumed by BOTH live call sites. There is no second, independent HTML-cleaning implementation. (HOW the TS route reaches the shared cleaner — shell to Python, call a worker extract endpoint, or run a vetted TS-equivalent that consumes the same config — is a TECH decision; PRODUCT mandates the “defined once” property, not the mechanism.) Verify: exactly one Trafilatura config definition exists; both the worker body producer and the manual route resolve their cleaning behaviour from it; no duplicated config literals or a parallel TS HTML-cleaning reimplementation.

  2. PI-5 — The quality gate is defined exactly once and shared. The content-length quality gate — < 100 characters ⇒ reject (HTTP 422 on the sync route / structured per-item failure on the worker); < 500 characters ⇒ retain with a warning — is defined once and applied at both call sites. The manual route’s existing thresholds (route.ts:113 < 100 ⇒ 422; route.ts:122 < 500 ⇒ warning) are preserved, not weakened. Verify: a single gate definition is referenced by both sites; the manual route still returns 422 below 100 chars and warns below 500; the worker applies the same thresholds.

  3. PI-6 — Cleaner parity is proven by golden fixtures. A set of golden fixtures (realistically-sized article HTML, not too-small synthetic fragments) demonstrates that the shared cleaner produces equivalent cleaned output for both call sites on identical inputs. Fixtures MUST be realistically sized: Trafilatura degrades toward near-full-text on tiny inputs (RESEARCH §Empirical-verification BEHAVIOUR caveat), so tiny fragments are not valid acceptance evidence. Verify: golden-fixture tests exist using realistically-sized article HTML and assert byte-identical cleaned output (the configured format) from the shared config across both call sites on the shared fixtures — the parity assertion is same-config→same-output, not hard-coded to Markdown; no fixture relies on a sub-paragraph synthetic fragment.

C. The ID-110 evidence-pair write contract (must NOT regress)

Section titled “C. The ID-110 evidence-pair write contract (must NOT regress)”

  1. PI-7 — The reference_ingest RPC contract is preserved on the manual path. The manual route continues to land the atomic source_documents + reference_items evidence pair via the owner-gated reference_ingest SECURITY DEFINER RPC with its existing parameter shape and storage_path = source_url parity. The cutover changes only the body producer (what fills reference_items.body), not the RPC, its arguments, or its grants. (RPC signature: 14 named params, 20260614010200…:65–78, incl. the 14th p_op_id uuid DEFAULT NULL.) Verify: the manual route still calls reference_ingest with the same parameter set; storage_path = source_url; sd inserted before ri; RLS grants (authenticated/service_role only) unchanged.

  2. PI-8 — Deterministic uuid5 PKs + idempotency converge on both paths. Both the manual route and the worker continue to mint the SAME deterministic uuid5 PKs from the normalised source URL, and a repeat URL remains idempotent (no duplicate evidence pair; already_existed/ON CONFLICT DO NOTHING behaviour preserved). Verify: re-ingesting an already-landed URL on either path creates no new rows and reports already-existed; the uuid5 PKs match across paths for the same normalised URL.

  3. PI-9 — Auth gating is unchanged. The manual route remains owner-gated (getAuthorisedClient() / auth.success / authFailureResponse(auth)); the in-house cleaner is invoked behind that gate and does not re-implement or relax auth. Verify: an unauthenticated/non-owner request to the manual route is rejected via the existing authFailureResponse(auth) path before any extraction runs.

D. Provenance enum re-map (append-only)

Section titled “D. Provenance enum re-map (append-only)”

  1. PI-10 — Append-only provenance migration; no prune. A new, append-only migration extends the extraction_method CHECK enum on BOTH source_documents and feed_articles to additionally admit trafilatura, playwright, AND unpdf. Existing pullmd_* / firecrawl / docling / other legacy values stay valid — no destructive prune (ID-42 additive precedent). Verify: the migration only ADDs to the CHECK arrays on both tables; pre-existing rows with legacy values still satisfy the constraint; no value is removed.

  2. PI-11 — The manual path writes a typed extraction_method and stops the NULL-into-metadata workaround. Once the typed labels exist, the manual route records the true producer in the typed extraction_method column (trafilatura for the HTML branch, unpdf for the PDF branch) instead of writing NULL into the column and stashing the producer in extraction_metadata (the current route.ts:171 / RPC :130 workaround). Provenance labels are platform/admin/analytics-facing only — never shown to an end user — so tool names (trafilatura/playwright/unpdf) are the correct vocabulary. The CV-16 ontology is updated in lockstep. Verify: a manual HTML ingest writes extraction_method = 'trafilatura' (and PDF writes 'unpdf') into the typed column, not NULL; CV-16 reflects the new labels; no end-user surface renders the label.

E. Deletion set & gating

Section titled “E. Deletion set & gating”

  1. PI-12 — Legacy-surface deletion is gated behind the ID-45 cutover window. No deletion of the legacy HTML-cleaning surfaces (the Firecrawl/Jina tiers in content-extractor.ts/feed-poller.ts; the Readability path in extraction/url.ts+html.ts) occurs until the worker path is prod-proven and a cutover window confirms zero legacy traffic. The in-house cleaner + the enum migration (PI-10) land BEFORE ID-45 re-ingests, so re-ingested rows carry consistent in-house provenance (OQ-2 RATIFIED). Verify: the in-house cleaner + enum migration land first; no legacy-surface deletion commit merges before the ID-45 cutover-window/zero-traffic confirmation.

  2. PI-13 — Deletion set: DROP exactly two deps; KEEP the shared survivors. The cutover DROPS @mendable/firecrawl-js and @mozilla/readability. It KEEPS: jsdom (Vitest environment + ~40 tests; its only extraction caller is html.ts), unpdf (PDF branch, out of scope), turndown (shared with lib/content/html-to-markdown.ts + app/api/upload/route.ts — F4), and normaliseUrl (relocated, see PI-14). Verify: only @mendable/firecrawl-js and @mozilla/readability are removed from package.json; jsdom, unpdf, turndown remain; no survivor import breaks.

  3. PI-14 — normaliseUrl is relocated, not deleted. The pure URL helper normaliseUrl (today exported from lib/intelligence/content-extractor.ts, imported by the manual route route.ts:84/:15 and lib/intelligence/pipeline.ts) is relocated to lib/extraction/url-normalise.ts and retained. It is unrelated to the Firecrawl/Jina tiers and must survive their removal (OQ-5 RATIFIED). Verify: normaliseUrl resolves from lib/extraction/url-normalise.ts; the manual route and pipeline.ts import it from the new home; behaviour is unchanged (direct file import, no barrel re-export).

  4. PI-15 — The checkFirecrawlApiKey() startup fail-fast is removed in lockstep with the Firecrawl tier. When the Firecrawl tier is deleted, the checkFirecrawlApiKey() call at pipeline startup (pipeline.ts:557, which THROWS in production when the key is absent — content-extractor.ts:136) is removed in the SAME change. It must not survive the tier as a production boot-failure on a now-dead env var (F5). Verify: after the Firecrawl tier is removed, no startup path calls checkFirecrawlApiKey(); the app boots in production without the Firecrawl env var.

F. Backlog & doc reconciliation

Section titled “F. Backlog & doc reconciliation”

  1. PI-16 — bl-308 closes on landing; no double-build. The PullMD-parity backlog item bl-308 (currently parked, “Superseded by ID-112 D1 … no TS PullMD client build warranted”) is formally CLOSED when ID-112 lands and is cited so the manual-path PullMD-parity work is not double-built. (Reconciliation is an Orchestrator/Curator write, not a Planner write — flagged, see Reporting.) Verify: bl-308 is referenced by the landing change and moved to closed; no net-new TS PullMD client is introduced.

  2. PI-17 — Descriptive ontology docs are updated as doc-only follow-ons. The CV-33 (reference_items.body “PullMD/Docling markdown”) and CV-16 descriptions are updated to reflect the in-house Trafilatura/Docling body producer. These are descriptive doc updates, not code rework (F1). Verify: CV-33/CV-16 no longer describe pullmd as the live HTML body producer; the update is documentation-only.

TECH hand-offs (carried, NOT resolved here)

Section titled “TECH hand-offs (carried, NOT resolved here)”

These are explicitly handed to {112.3} TECH; PRODUCT records them as open so they are not lost, but does not decide them:

  • OQ-8 / F3 — fetch/render topology (the real open risk). The cocoindex custom-source enumeration via FeedUrlSource is viable and probe-proven, but the unvalidated part is where Trafilatura’s fetch_url + the OPTIONAL Playwright JS-render run. Replacing the thin httpx _pullmd_fetch (adapters.py:249) with in-process Trafilatura+Playwright changes the worker’s runtime footprint (browser binaries, memory). TECH MUST validate the Playwright-in-container runtime/memory cost on B1 and choose (A) extract-in-component vs (B2) a separate fetch/render worker the component calls, BEFORE committing the topology.
  • Shared-seam mechanism (PI-4/PI-5). HOW the TS manual route reaches the single shared cleaner (shell to Python / call a worker extract endpoint / vetted TS-equivalent) is a TECH decision, constrained only by “defined once + fixture-pinned.”
  • Trafilatura config values (PI-4). The concrete output_format (a free choice per PI-1 — txt or markdown; not load-bearing for the HTML/URL path), favor_recall vs favor_precision, and include_comments settings (RESEARCH §Empirical-verification caveat).