PLAN -- HTML/URL extraction cutover: decomposition into implementation Subtasks (ID-112.4)

PLAN — HTML/URL extraction cutover: decomposition into implementation Subtasks (ID-112.4)

Task: ID-112 — Cutover HTML/URL extraction to an in-house cocoindex-native cleaner; retire pullmd (worker) + Readability (manual) + the dormant Firecrawl/Jina tier. Subtask: {112.4} PLAN (final spec-chain artefact, per the Q-PLANNER-2 fresh-per-Subtask discipline). Author date: 16/06/2026. Author: Task Planner (FRESH dispatch — Q-PLANNER-2; a different instance from the {112.1} RESEARCH / {112.2} PRODUCT / {112.3} TECH authors; read the ratified PRODUCT + TECH in full this dispatch). Predecessors (read in full this dispatch): {112.2} PRODUCT (17 ratified PI-N invariants), {112.3} TECH (1:1 proposed-change blocks, all three hand-offs RESOLVED S366). UK English throughout (DD/MM/YYYY).

This PLAN decomposes the ratified PRODUCT+TECH pair into nine TM-shape implementation Subtasks {112.5}-{112.13}. The Subtask JSON the Orchestrator feeds to bun scripts/ledger-cli.ts add-subtasks 112 --file - lives at /tmp/claude/id112-subtasks.json (ids start at “5”). PLAN records the cut, the dependency chain, the PI coverage matrix, and the gating; it does NOT re-dump the specs.

---

Code-intelligence orientation (cited verbatim, this dispatch, repo knowledge-hub, HEAD 7fe481f74)

The decomposition was grounded against live file:line reality (every TECH anchor re-verified this dispatch; gitnexus indexes only the TS corpus, so the Python pipeline / SQL migrations / compose files were grep + direct-read confirmed):

• gitnexus_context({name: "normaliseUrl", file_path: "lib/intelligence/content-extractor.ts", repo: "/Users/liamj/Documents/development/knowledge-hub"}) -> uid: Function:lib/intelligence/content-extractor.ts:normaliseUrl (startLine 95, endLine 120); incoming.calls: Function:app/api/ingest/url/route.ts:POST, Function:lib/intelligence/pipeline.ts:isDuplicate, Function:lib/intelligence/pipeline.ts:processFeedSource — THREE call sites across TWO files (route.ts + pipeline.ts, two callers in pipeline.ts). This refines PI-14’s “two importers”: the relocation Subtask {112.11} must update route.ts:15 AND the pipeline.ts import feeding both isDuplicate and processFeedSource.
• gitnexus_context({name: "extractFromUrl", file_path: "lib/extraction/url.ts", repo: ...}) -> uid: Function:lib/extraction/url.ts:extractFromUrl (46-127); incoming: {} (the dynamic import() at route.ts:108 is NOT graph-indexed — confirms TECH); outgoing.calls: extractFromHtml, extractOgMetadata, extractPdfText, validateUrl.
• grep (Python/SQL/compose, not gitnexus-indexed) re-verified this dispatch:
  • flow.py: HTML branch :2690 _pullmd_fetch, :2692 _pullmd_extraction_method, :2693 pullmd_share_id; SSRF validate_url :2658; declare_row sd :2715-2729 (storage_path=item.url :2719); _pullmd_extraction_method :2448; _PULLMD_X_SOURCE_METHODS frozenset adapters.py:288-290.
  • server.py: build_app() route registration :512-515 (add_get('/health'), add_post('/stage'), add_post('/walk')); _walk_handler bearer pattern :419-432 (CRON_SECRET, fail-closed 401); client_max_size=50MB :512.
  • route.ts: auth :57-58; normaliseUrl import :15; dynamic extractFromUrl import :108-109; gate :113(<100->422)/:122(<500->warn); provenance :166/:172; RPC args :191-212.
  • url.ts: extractionMethod: 'readability'|'unpdf' :24; MAX_CONTENT_SIZE = 20MB :30; :106 unpdf / :125 readability return shapes; PDF branch :95-110.
  • RPC 20260614010200_id110_reference_ingest_rpc.sql: CREATE OR REPLACE :64; 14 params :64-78; NULL literal :129; grants in-migration :169; uuid5 :98-101; ON CONFLICT DO NOTHING :132/:144.
  • ID-42 pattern 20260526074944_id42_pullmd_provenance.sql: DROP CONSTRAINT IF EXISTS -> ADD CONSTRAINT ... CHECK (... = ANY(ARRAY[...])) on BOTH source_documents :22-26 and feed_articles :33-37; 11 admitted values verbatim.
  • requirements.txt: trafilatura==2.0.0 :11 (DIRECT dep, in-process); buildpack-no-apt :30; cocoindex[postgres]==1.0.7 :54.
  • package.json: @mendable/firecrawl-js :67, @mozilla/readability :70, turndown :126, unpdf :128, jsdom :153.
  • Deletion surfaces: content-extractor.ts checkFirecrawlApiKey :136, isFirecrawlConfigured :27, module state :12/:19; pipeline.ts startup call :557; feed-poller.ts @mendable import :430; html.ts Readability :28-49.
  • Compose: docker-compose.production.yaml Traefik walk rule :81 (PathPrefix(/walk)||PathPrefix(/health)); /stage deliberately un-routed :62 (Inv-13); CRON_SECRET env :138.

ccc search fallback was NOT required — this is a brownfield cutover with a richly-indexed pipeline; no greenfield disclaimer applies.

---

Decomposition: nine Subtasks, chain-ordered

Subtask	Title	PI(s)	Deps	Status	Size
{112.5}	extract.py shared Trafilatura cleaner module	PI-4, PI-5	—	pending	S
{112.6}	POST /extract pure-cleaner endpoint + Traefik route	PI-4, PI-9	5	pending	M
{112.7}	Re-point worker HTML branch to in-process clean_html	PI-1, PI-8	5	pending	M
{112.8}	Append-only provenance enum migration (both tables)	PI-10	—	pending	S
{112.9}	reference_ingest RPC body — Option-2 derivation	PI-7, PI-11	8	pending	S
{112.10}	Rewire manual route body producer to /extract	PI-1, PI-2, PI-5, PI-9, PI-11	6, 9	pending	M
{112.11}	Relocate normaliseUrl -> lib/extraction/url-normalise.ts	PI-14	10	pending	S
{112.12}	Golden-fixture in-process==over-HTTP parity tests	PI-6	6	pending	S
{112.13}	DELETION set (DROP 2 deps + dead tiers) — GATED	PI-12, PI-13, PI-15	6, 7, 10, 11	deferred	L

Dependency chain (topological)

{112.5} extract.py ──┬──> {112.6} POST /extract ──┬──> {112.10} route rewire ──> {112.11} normaliseUrl reloc ─┐
   (cleaner)         │       (endpoint)            │        (manual path)            (serialise route.ts)      │
                     ├──> {112.7} worker re-point ─┼─────────────────────────────────────────────────────────┤
                     │       (flow.py)             │                                                          │
                     └──> {112.12} parity tests ───┘                                                          │
                                                                                                              │
{112.8} enum migration ──> {112.9} RPC body Option-2 ──> (feeds {112.10})                                     │
                                                                                                              ▼
                                          {112.13} DELETION (deferred) <─── deps 6, 7, 10, 11 ────────────────┘
                                          + EXTERNAL gate: ID-45 zero-traffic window

Two independent roots can start in parallel: {112.5} (cleaner) and {112.8} (enum migration). They converge at {112.10} (which needs both the endpoint {112.6} and the RPC body {112.9}).

Cut rationale (where the natural slices were finalised)

• The brief’s 9 slices map 1:1 onto the 9 Subtasks — the cut held; no slice needed splitting or merging.
• {112.6} and {112.7} are SEPARATE (both dep only {112.5}, not each other): the worker calls clean_html in-process (same Python process, no HTTP hop — TECH Hand-off #2 prop 1), so it does NOT depend on the POST /extract endpoint. Only the TS manual route reaches the endpoint over HTTP. Keeping them independent maximises parallelism and reflects the true coupling.
• {112.11} depends on {112.10} purely to serialise route.ts edits: {112.10} rewires route.ts:108 heavily; {112.11} changes route.ts:15 (the import line). The dependency is a merge-collision-avoidance ordering, not a logical one — noted so the Orchestrator understands it is not a hard data dependency.
• {112.9} depends on {112.8} (logical): the enum must admit trafilatura/unpdf before the RPC body derives and writes them, else the CHECK rejects the write.
• {112.13} is TERMINAL + deferred with deps [6,7,10,11] (every live-path Subtask proven). The deps express in-branch ordering; the ID-45 zero-traffic cutover window is the EXTERNAL release gate the Orchestrator holds before un-deferring (PI-12).

---

PI coverage matrix (all 17 PRODUCT invariants accounted for)

Every PI-N is covered by a Subtask OR is a doc/backlog reconciliation routed to the Orchestrator/Curator (NOT an implementation Subtask — per the brief and PRODUCT Reporting):

PI	Covered by	PI	Covered by
PI-1	{112.7} + {112.10}	PI-10	{112.8}
PI-2	{112.10}	PI-11	{112.9} + {112.10}
PI-3	(no-build invariant — nothing added; verified by {112.7} leaving url_source.py untouched)	PI-12	{112.13} (gating)
PI-4	{112.5} + {112.6}	PI-13	{112.13}
PI-5	{112.5} (gate def) + {112.10} (route mapping)	PI-14	{112.11}
PI-6	{112.12}	PI-15	{112.13} (lockstep)
PI-7	{112.9}	PI-16	Orchestrator/Curator (bl-308 close — NOT a Subtask)
PI-8	preserved by {112.7}/{112.9}/{112.10} (no producer touches PKs)	PI-17	Curator (CV-33/CV-16 doc-only — NOT a Subtask)
PI-9	{112.6} (endpoint bearer) + {112.10} (route auth layering)

PI-3 and PI-8 are “must-NOT-regress” invariants with no net-new code; they are encoded as testStrategy/details assertions inside {112.7}/{112.9}/{112.10} rather than as standalone Subtasks (there is nothing to build, only something to not break).

---

Migration ordering (gated by PI-12)

Mirrors TECH §Migration-plan exactly:

1. {112.5} + {112.6} + {112.7} — land the cleaner + endpoint + worker re-point (additive, behind no traffic switch).
2. {112.8} — append-only provenance migration BEFORE ID-45 re-ingests.
3. {112.9} — RPC body Option-2 (lands AFTER {112.8} so labels are admitted).
4. {112.10} + {112.11} — re-point the manual route; relocate normaliseUrl.
5. {112.12} — parity tests prove the shared seam.
6. {112.13} — NO deletion until the ID-45 cutover window confirms zero legacy traffic (deferred).

Migrations {112.8} + {112.9} land BEFORE ID-45 (PRODUCT PI-12 / TECH Migration-plan). Migration discipline (supabase/CLAUDE.md): supabase migration new + foreground db push (never background — it prompts) + supabase gen types; verify supabase/.temp/project-ref (staging turayklvaunphgbgscat) before any push.

---

Sibling-only dependency constraint — HONOURED (no escalation)

All nine Subtasks’ dependencies are sibling-only (ids within {112.5}-{112.13}, validated programmatically). No cross-Task dependency surfaced during decomposition:

• The ID-45 cutover-window gate on {112.13} is an external release gate the Orchestrator holds, NOT a Subtask dependency — it is expressed via status: "deferred" + the gate text in details, not via a dependencies edge to an ID-45 Subtask. This is the correct encoding (a cross-Task dep would violate sibling-only; the deferred status + external gate is the sanctioned alternative).
• The ID-110 RPC, ID-66 compose, ID-75 source, ID-42 migration are all build-on predecessors already landed — they are file:line anchors, not pending dependencies.

No Task split/merge needed. The 9-Subtask decomposition is well within the 25-Subtask soft ceiling.

---

Out-of-Subtask reconciliations (routed to Orchestrator/Curator, NOT {112.5+})

Per the brief and PRODUCT PI-16/PI-17, these are NOT implementation Subtasks:

• PI-16 — bl-308 closure. The landing change cites bl-308; the Curator moves bl-308 parked -> closed via update-roadmap-backlog. No net-new TS PullMD client is introduced (the shared seam IS the B1 endpoint). Orchestrator action: route to Curator at Task-close.
• PI-17 — CV-33 + CV-16 doc updates. Descriptive ontology updates (CV-33 reference_items.body “PullMD/Docling markdown” -> in-house Trafilatura/Docling; CV-16 labels). Documentation-only; automated in the docs-site repo’s own .claude/ lane. Orchestrator action: route to Curator as a doc-only follow-on.
• PI-11 (iv) CV-16 lockstep is the same CV-16 update — folded into the PI-17 reconciliation above.

---

Risks and mitigations

Risk	Impact	Mitigation
output_format="txt" kwarg combo behaves differently from the {112.1}-confirmed "markdown" path	Med	{112.5} details mandates an Executor-time import-and-call check of the exact txt+favor_recall+include_tables combo against trafilatura==2.0.0 on a realistically-sized fixture (TECH §Empirical-verification caveat — NOT a blocker, but DO it).
Trafilatura small-input degradation produces near-full-text	Med	Quality gate (PI-5, {112.5}) + realistically-sized fixtures only (PI-6, {112.12}); tiny-fragment fixtures are explicitly invalid acceptance evidence.
{112.10} and {112.11} both edit route.ts -> merge collision	Low	{112.11} deps {112.10} to serialise the two route.ts edits (encoded).
{112.13} deletion merged before zero-traffic confirmed	High	status: "deferred" + the ID-45 gate spelled in details; the Orchestrator holds the external release gate before un-deferring.
Deleting a pullmd_* symbol still referenced by a survivor	Med	{112.13} details mandates gitnexus_impact upstream on every deleted symbol BEFORE removal (warn on HIGH/CRITICAL) + full bun run test regression.

---

Open questions for Liam (before implementation)

None. All three TECH hand-offs were RATIFIED by Liam at S366 (static-only v1; pure-cleaner /extract with EXTRACT_API_TOKEN; Option-2 server-side extraction_method derivation), and the TECH §“TECH open questions” section records zero remaining Liam-bound OQ. This PLAN surfaced no new OQ — the spec pair is internally consistent and every PI maps cleanly to a Subtask or a routed reconciliation. Implementation may proceed once the Orchestrator adds the Subtasks.