RESEARCH — HTML/URL extraction cutover to cocoindex-native; retire pullmd + Firecrawl (ID-112.1)

Task: ID-112 — Cutover HTML/URL extraction to cocoindex/B1 worker; retire Vercel Surfaces A/B + Firecrawl/Jina/readability deps. Subtask: {112.1} RESEARCH (precedes {112.2} PRODUCT → {112.3} TECH → {112.4} PLAN per the Q-PLANNER-2 fresh-per-Subtask discipline). Status: DECISION-SUPPORT / findings. This consolidates the ratified D1 decision memo (/tmp/claude/id112-d1-investigation.md), the S355 code survey + intended-arch evidence (this dir), and a fresh code-intelligence + empirical-verification pass into one cited findings brief. It records the current state and constraints the in-house cutover must honour; it does not author PRODUCT/TECH decisions (those are gated downstream). Primary input: D1 is RATIFIED — retire pullmd (AGPL-3.0) AND Firecrawl (OSS core also AGPL-3.0) → converge BOTH URL datapaths onto ONE in-house, cocoindex-native extractor.

Context (code-intelligence orientation — raw outputs, cite-verifiable)

Per the binding code-intel discipline, the URL/HTML extraction entrypoints + reference-ingest write path were oriented via gitnexus_query / gitnexus_context (TS corpus) supplemented with grep for the Python pipeline (scripts/cocoindex_pipeline/*.py) and SQL migrations (supabase/migrations/*.sql), which gitnexus/ast-dataflow do not index. Raw outputs (verbatim, not paraphrased):

gitnexus_query({query: "url extraction content pipeline reference ingest", repo: knowledge-hub}) surfaced the canonical execution flow symbols:

Function:app/api/ingest/url/route.ts:POST (startLine 53, endLine 240) — manual single-URL route.
Function:scripts/cocoindex_pipeline/flow.py:_ingest_url_body (startLine 2625, endLine 2800) — worker URL body writer.
Function:scripts/cocoindex_pipeline/flow.py:_ingest_content_branch (startLine 1966, endLine 2388) — file-walked content branch.
Function:lib/intelligence/content-extractor.ts:extractContent (startLine 169, endLine 333) + checkFirecrawlApiKey (135–163) — Surface-A.
Function:components/create-content/url-ingest-form.tsx:UrlIngestForm (59–362) / handleSubmit (106–180) — the manual route’s UI caller.

gitnexus_context({name: "extractFromUrl", file_path: "lib/extraction/url.ts"}) → uid: Function:lib/extraction/url.ts:extractFromUrl (startLine 46, endLine 127); incoming: {} (gitnexus does NOT index the dynamic import() at route.ts:108); outgoing.calls: extractFromHtml (lib/extraction/html.ts), extractOgMetadata (lib/extraction/og-metadata.ts), extractPdfText (lib/extraction/pdf.ts), validateUrl (lib/extraction/url-validation.ts).

gitnexus_context({name: "extractFromHtml", file_path: "lib/extraction/html.ts"}) → uid: Function:lib/extraction/html.ts:extractFromHtml (startLine 27, endLine 49); incoming.calls: extractFromUrl only; outgoing: {} (jsdom + Readability are lazy import()s, not graph-indexed).

gitnexus_context({name: "_ingest_url_body", file_path: "scripts/cocoindex_pipeline/flow.py"}) → incoming.calls: ingest_url; outgoing.calls: _docling_to_markdown, _pullmd_fetch (adapters.py), extract_classification (extraction.py), embed_content_text (flow.py), _pullmd_extraction_method (flow.py), _fetch_url_bytes, _url_is_pdf, _url_filename, _backlink_feed_articles, validate_url (url_validation.py), _log_ssrf_rejection. This is the canonical worker write path; the in-house extractor re-points the _pullmd_fetch call site behind the SAME surrounding contract.

grep (Python/SQL, not gitnexus-indexed):

flow.py:2129 — V-1 trap, verbatim: RecursiveSplitter is the cocoindex-native splitter (cocoindex.functions.SplitRecursively is ...); live symbol imported at flow.py:2144 (RecursiveSplitter), instantiated flow.py:2147.
flow.py:2690 result = await _pullmd_fetch(item.url, item.content_epoch); 2692 extraction_method = _pullmd_extraction_method(result.x_source); 2693 pullmd_share_id = result.share_id; 2706 embedding = await embed_content_text(markdown); 2719 "storage_path": item.url (= source_url = normalised URL).
flow.py:2448 _pullmd_extraction_method(x_source) — maps the five known X-Source values to pullmd_<x_source>, else None (avoids CHECK-enum violation).
adapters.py:288 _PULLMD_X_SOURCE_METHODS = frozenset(...) ∈ {reddit, cloudflare, readability, trafilatura, playwright}; provenance struct SourceProvenance at adapters.py:305.
url_source.py:59-60 — url_import join deferred (“joins when a manual-URL route ships (post-v1)”); INGESTION_SOURCE_RSS_FEED = "rss_feed"; _PassedUrlItems enumerates feed_articles WHERE passed = true (url_source.py:67-68), hardcoding ingestion_source='rss_feed' (url_source.py:166).

ccc search fallback was NOT required — the codebase is richly indexed; no greenfield disclaimer applies. This is a brownfield cutover with an existing, well-instrumented pipeline.

1. Driver + license posture (the AGPL axis — ratified D1)

The decision driver is not “best URL→Markdown tool” but “what cleans a fetched web page into embeddable/classifiable text for the reference layer, without importing copyleft or third-party data egress.” Findings:

pullmd = AGPL-3.0 (image aeternalabshq/pullmd:2.0.0, pinned in both deploy/coolify/docker-compose.{staging,production}.yaml). Network-isolated use is compliant, but it ships AGPL into every client deploy and some security teams reflex-reject it (ID-112 status_note S361).
Firecrawl’s OSS/self-host core is ALSO AGPL-3.0 (verified 16/06/26 on the firecrawl repo) — self-hosting it trades one AGPL for another and loses features; hosted Firecrawl dodges AGPL but sends client ingest URLs to a third-party SaaS (data-egress + recurring cost + lock-in). Firecrawl is rejected for the platform direction on both branches.
In-house permissive stack: Trafilatura (Apache-2.0) + Playwright (Apache-2.0). Both are already trusted in the deploy: Trafilatura is a direct requirements.txt dep (trafilatura==2.0.0) AND a pullmd sidecar; Playwright runs as a pullmd sidecar container AND is importable as a Python dep (playwright==1.50.0 — see §Empirical verification). The serialise tail (Turndown / MarkItDown / kreuzberg) is NOT needed — Trafilatura emits Markdown directly.
Supply-chain corroboration: pullmd shipped a BREAKING body-format change in v3.0.0 (10/06/26) + v3.1.0 (15/06/26) under moving :2/:latest tags from a single low-eyeball vendor; KH’s :2.0.0 pin protects today but means deliberately staying several breaking versions behind. Dated evidence for the supply-chain motivation, not speculation.

Minimal cutover stack (carry verbatim from D1 §7.4): [cocoindex-native, ALREADY WIRED: chunking via RecursiveSplitter + embedding via the existing LiteLLMEmbedder seam] + [NEW custom: (i) Trafilatura boilerplate-strip — REQUIRED, cocoindex has NO article-cleaning op; (ii) Playwright JS-render — OPTIONAL, only hydrated SPAs]. cocoindex’s text ops are syntactic/codebase-oriented (RecursiveSplitter splits structurally; detect_code_language maps filename→programming-language) — neither performs semantic boilerplate removal, so Trafilatura is the irreducible new surface.

2. Current-state inventory — BOTH URL datapaths (at HEAD 7fe481f74)

Two separate URL datapaths exist, on DIFFERENT tooling, both feeding reference_items.body + embedding + classification:

Surface-A — RSS-feed / web-source poll (Vercel, live)

Firecrawl tier: lib/intelligence/feed-poller.ts::pollWebSource (337–533); dynamic import('@mendable/firecrawl-js') at feed-poller.ts:430, firecrawl.scrape(source.url, …) at :433; firecrawlCalled telemetry (Sentry breadcrumbs intelligence.web-source.firecrawl-call). Read by app/api/intelligence/workspaces/[id]/sources/[sourceId]/test/route.ts:57 (firecrawlCreditsExpected).
Tier cascade: lib/intelligence/content-extractor.ts::extractContent (170–333) — Tier 1 rss_content, Tier 2 direct fetch + extractMainContentHtml, Tier 2.5 Jina (https://r.jina.ai/${url} at :244), Tier 3 Firecrawl (:278), Tier 4 summary_fallback.
Exports of content-extractor.ts (170–333 + helpers): extractContent, checkFirecrawlApiKey (136), isFirecrawlConfigured (27), isGoogleNewsUrl (46), resolveGoogleNewsUrl (60), normaliseUrl (96).
DELETION-COUPLING (flag): normaliseUrl is imported by the manual route (app/api/ingest/url/route.ts:84) and by lib/intelligence/pipeline.ts. So Surface-A is not wholesale-deletable while the manual route lives — normaliseUrl (a pure URL helper, unrelated to Firecrawl/Jina) must be retained or relocated, not deleted with the extraction tiers. checkFirecrawlApiKey is called at pipeline startup (pipeline.ts:557); pipeline.ts:335/420 write extraction_method to feed_articles/content_items.

Surface-B — manual single-URL (Vercel, live; ID-110)

app/api/ingest/url/route.ts:108-109 → dynamic import('@/lib/extraction/url') → extractFromUrl(url).
lib/extraction/url.ts::extractFromUrl (46–127): validateUrl → fetch (15 s timeout, 20 MB cap) → re-validate final URL → route by content-type: PDF branch extractPdfText (unpdf) → extractionMethod: 'unpdf'; HTML branch extractFromHtml(html, finalUrl) + extractOgMetadata → extractionMethod: 'readability'.
lib/extraction/html.ts::extractFromHtml (27–49): lazy import('jsdom') + import('@mozilla/readability'); new JSDOM → new Readability → reader.parse() → turndown.turndown(article.content). Only caller is extractFromUrl.

Worker — cocoindex canonical (B1; the convergence target)

scripts/cocoindex_pipeline/flow.py::ingest_url → _ingest_url_body (2625–2800): _pullmd_fetch(item.url, item.content_epoch) (adapters.py:249, httpx.AsyncClient, @coco.fn(memo=True) epoch-keyed) → extract_classification(markdown) (2704) → embed_content_text(markdown) (2706) → write with storage_path=item.url (2719), extraction_method (2724), pullmd_share_id (2727). Chunking via RecursiveSplitter (2129–2155) on the content branch.
Source: url_source.py::FeedUrlSource / _PassedUrlItems — feed_articles WHERE passed = true, ingestion_source hardcoded 'rss_feed'; the manual url_import producer is NOT built (url_source.py:59).

Out-of-scope (D1 explicit — keep):

unpdf + lib/extraction/pdf.ts — binary→Markdown PDF path; D1 leaves PDF alone (worker uses Docling, MIT, docling==2.95.0). NOT in this cutover’s removal set.
jsdom — flagged [RATIFIED-RETIRE] ⚠verify in S355; it is a Vitest env dependency and may have non-extraction callers. Confirm before dropping (Open Question).

3. The ID-110 evidence-pair WRITE CONTRACT (must NOT regress)

The cutover re-points the extractor behind the SAME write contract — it does not touch the contract itself. Both paths land an atomic source_documents + reference_items evidence pair:

Manual route: route.ts:206-212 calls the reference_ingest SECURITY DEFINER RPC (migration 20260614010200_id110_reference_ingest_rpc.sql). 14-arg signature; mints both PKs server-side as uuid5: sd namespace + 'sd:'||source_url, ri namespace fbfaf1ff-1ee4-583c-9757-1674465b2ec1 + 'ri:'||source_url (migration:100-101). Idempotent: a repeat URL returns already_existed=true and writes nothing (migration:104-115); ON CONFLICT (id) DO NOTHING belt-and-braces. Atomic: both INSERTs in the caller’s transaction, sd FIRST (FK target). RLS-P4: REVOKE … FROM anon, PUBLIC; GRANT … TO authenticated, service_role.
Worker path: mints the SAME uuid5 PKs (parity cited at migration:179 → flow.py:1601/1884); storage_path = source_url.
Constraint for the cutover (NON-NEGOTIABLE): the in-house extractor changes only the body producer. It MUST preserve: (a) the reference_ingest RPC call + 14-arg shape on the manual path; (b) the deterministic uuid5 PKs + idempotent ON-CONFLICT converge on both paths; (c) storage_path = source_url parity; (d) the evidence-pair atomicity. The Checker verifies the in-house path against these.
Auth note: the manual route is owner-gated (getAuthorisedClient() / auth.success / authFailureResponse(auth)); the in-house extractor is invoked behind that gate — it does not re-implement auth.

4. Provenance enum re-map (bounded; not a blocker)

Confirmed present: supabase/migrations/20260526074944_id42_pullmd_provenance.sql. It defines source_documents.extraction_method (text + CHECK) and pullmd_share_id (text + partial index), and mirrors the CHECK onto feed_articles.extraction_method.
Current CHECK admitted set (both tables, verbatim): 'rss_content','fetch','jina_reader','firecrawl','summary_fallback','pullmd_readability','pullmd_playwright','pullmd_cloudflare','pullmd_reddit','pullmd_trafilatura','docling'.
Code-side mapper: flow.py:2448 _pullmd_extraction_method emits pullmd_<x_source> only for the five _PULLMD_X_SOURCE_METHODS (adapters.py:288), else None-degrades with a structured warning. The manual route writes extraction_method = NULL and records the true producer in extraction_metadata (route.ts:171-175, migration:129-130) — because the ID-42 CHECK currently admits neither readability nor unpdf.
Re-map needed (cutover): a NEW append-only migration adding in-house labels (e.g. trafilatura, playwright) to BOTH CHECK enums, so the in-house worker writes a typed extraction_method (not the NULL-into-metadata workaround). Append-only (parity with ID-42’s additive, non-destructive CHECK redefinition) — pre-existing pullmd_*/firecrawl rows stay valid; no prune. Bounded schema change, NOT a blocker. Requires the standard supabase migration new + foreground db push + type regen (supabase/CLAUDE.md).

5. Surface deletion gating (BEHIND the ID-45 cutover window)

Both Surface-A and Surface-B remain LIVE at HEAD (carry production traffic — S355 survey Q5). Deletion is gated behind the ID-45 re-ingest cutover window: no deletion until the worker feed-ingest path is prod-proven and a cutover window confirms zero live traffic on the legacy surfaces.
Sequencing (from D1 §5 + the 0.8.4 §10.4 logic): land the in-house extractor + the extraction_method enum migration BEFORE ID-45 re-ingests, so re-ingested reference rows carry consistent in-house provenance, not a legacy pullmd_*/new-label mix.
Keep (do NOT delete in this cutover): jsdom (Vitest env; verify non-extraction callers first); unpdf + lib/extraction/pdf.ts (PDF path, Docling-scoped, out of D1); normaliseUrl (pure URL helper coupled to the live manual route — §2 DELETION-COUPLING).
ID-110 backlog reconciliation (flag to Orchestrator): the S352 “PullMD-parity backlog follow-up” for the manual path is subsumed/superseded by D1 — instead of bringing the manual path UP to pullmd, D1 brings BOTH paths onto the in-house cleaner. The backlog item must be reconciled (not double-built).

6. Orthogonal dependency — embedding zero-egress (NOT this Task’s lever)

The embedding step is cocoindex-native (LiteLLMEmbedder("text-embedding-3-large"), flow.py:1087) but routes to OpenAI’s hosted API → NOT zero-egress today. The local option, SentenceTransformerEmbedder (verified PRESENT at cocoindex.ops.sentence_transformers, §Empirical verification), is a different op the pipeline does not currently use. Swapping LiteLLMEmbedder → SentenceTransformerEmbedder is a separate cocoindex-native lever orthogonal to D1’s extraction question — being spiked in parallel; noted here only to keep the “cocoindex-native” claim honest. It should land before ID-45 if true zero-egress is a goal, but it is NOT in ID-112’s scope and must not be conflated with the extractor cutover.

Empirical verification (Q-EX2 forcing function — external-library symbols)

Import-and-call check, 16/06/2026, against pinned versions (requirements.txt). Scope: external-library symbols the spec chain will rely on (not internal KH symbols).

Library (pinned)	Symbol path	Result
`trafilatura==2.0.0` (Apache-2.0)	`trafilatura.extract`, `.fetch_url`, `.bare_extraction`, `.extract_metadata`	PRESENT (all four)
`trafilatura==2.0.0`	`extract(html, output_format="markdown")` boilerplate-strip behaviour	PRESENT / BEHAVIOUR CONFIRMED — see caveat
`cocoindex==1.0.7`	`cocoindex.ops.text.RecursiveSplitter`	PRESENT
`cocoindex==1.0.7`	`cocoindex.functions.SplitRecursively` / `cocoindex.ops.text.SplitRecursively` (docs name)	ABSENT (V-1 trap empirically confirmed — cite `RecursiveSplitter`)
`cocoindex==1.0.7`	`cocoindex.ops.litellm.LiteLLMEmbedder`	PRESENT
`cocoindex==1.0.7`	`cocoindex.ops.sentence_transformers.SentenceTransformerEmbedder`	PRESENT (the local/zero-egress option; orthogonal)
`playwright==1.50.0` (Apache-2.0)	`import playwright` (Python binding)	PRESENT — Playwright is importable in-process, not only a sidecar container
`docling==2.95.0` (MIT)	installed (PDF path, out of D1 scope)	PRESENT

BEHAVIOUR caveat (recorded honestly, surfaced to TECH): on a too-small synthetic HTML fragment, trafilatura.extract retained nav text and emitted no Markdown headings (text-density heuristics fall back to near-full-text on tiny inputs). On a realistically-sized article (8 substantial paragraphs + nav/aside/footer chrome), extract(output_format="markdown") correctly stripped all chrome and emitted proper Markdown (# The Real Headline, ## A Subheading) — favor_recall/favor_precision knobs available. Implication for the spec chain (not a blocker): the in-house extractor’s acceptance tests MUST use realistically-sized fixtures; very short pages degrade toward full-text and the existing manual-route quality gate (route.ts:113 <100 chars → 422; <500 chars → warning) should be preserved/ported. No ABSENT/SIGNATURE_DRIFT → spec chain may proceed.

Open Questions for PRODUCT/TECH ratification (Liam owns)

These do NOT block authoring {112.2} PRODUCT, but each needs a ratified answer before/within the spec chain:

OQ-1 (CONFIRM — provenance-enum migration). Ratify the bounded, append-only migration adding in-house extraction_method labels (trafilatura, playwright?) to BOTH the source_documents and feed_articles CHECK enums. Decide the exact label vocabulary (do we mirror Trafilatura/Playwright as the producers, and does the manual path stop writing NULL-into-metadata once the typed labels exist?).
OQ-2 (CONFIRM — deletion gated behind ID-45). Ratify the invariant: Surface-A (content-extractor.ts tiers + feed-poller.ts Firecrawl) and Surface-B (extraction/url.ts + html.ts) deletion stays BEHIND the ID-45 cutover window (no deletion until worker feed-ingest is prod-proven + zero-traffic confirmed). Land the in-house extractor + enum migration BEFORE ID-45 re-ingests.
OQ-3 (sync vs async — consume, don’t re-open). D2 = async/in-worker canonical + a THIN sync fallback for the manual route until the url_import worker producer ships, both calling the SAME in-house extractor library (avoid two divergent extractor stacks). Confirm this is the shape PRODUCT encodes (the underlying UX question already lives at ID-110/bl-298 + ID-75 OQ-T1 — consume that decision, do not re-open).
OQ-4 (jsdom retention). Confirm jsdom has only Vitest-env / non-extraction callers before any drop; D1 keeps it. (unpdf + pdf.ts confirmed KEEP — PDF/Docling out of scope.)
OQ-5 (normaliseUrl relocation). normaliseUrl (content-extractor.ts:96) is coupled to the live manual route (route.ts:84) + pipeline.ts. Confirm it is retained/relocated (not deleted) when the Firecrawl/Jina tiers retire.
OQ-6 (ID-110 backlog reconciliation — Orchestrator routing). The S352 PullMD-parity manual-path backlog follow-up is subsumed by D1; flag for reconciliation so it is not double-built.
OQ-7 (Trafilatura config + quality gate, INFORMS TECH). Decide the Trafilatura extraction config (favor_recall vs favor_precision, include_comments, output_format) and that the in-house path preserves the manual route’s <100-char/<500-char quality gate; acceptance tests use realistically-sized fixtures (per the §Empirical-verification caveat).
OQ-8 (Option A vs B2 viability — from S355). S300 gated cocoindex remote/custom-source (Option A) on empirically confirming a usable remote source; pin moved to 1.0.7 and url_source.py exists. Confirm Option A viability vs the B2 fetch-worker fallback before TECH commits a topology.