Extraction Method

Extraction Method

Definition

Closed enumeration capturing which extraction route produced the body text. Originally 5 values on feed_articles.extraction_method per source §2.1 (added in S235 WP1); extended to 11 values in ID-42 (pullmd deploy + HTML retirement) — the six new values (pullmd_* + docling) also govern the net-new source_documents.extraction_method column. Relevant to 0.9-context.md §7.5 “drop Firecrawl” planning — that decision will retire one value from this CV (T7-gated, no prune yet).

Synonyms / Acronyms

None.

Baseline values

key	label	provenance
rss_content	RSS Content	core
fetch	Fetch	core
jina_reader	Jina Reader	core
firecrawl	Firecrawl	core
summary_fallback	Summary Fallback	core
pullmd_readability	pullmd Readability	core
pullmd_playwright	pullmd Playwright	core
pullmd_cloudflare	pullmd Cloudflare	core
pullmd_reddit	pullmd Reddit	core
pullmd_trafilatura	pullmd Trafilatura	core
docling	Docling	core

Relationships

• feed_articles.extraction_method — CHECK constraint feed_articles_extraction_method_check (column declared as extraction_method: string | null in supabase/types/database.types.ts:1157).
• source_documents.extraction_method — net-new column + CHECK constraint source_documents_extraction_method_check (ID-42, migration 20260526074944_id42_pullmd_provenance.sql); the cocoindex Surface-C path writes pullmd_* (mirroring the live X-Source header) or docling for binary extraction. Adjacent net-new source_documents.pullmd_share_id carries the pullmd X-Share-Id permalink.
• Adjacent to feed_articles.ai_summary (separate column from the now-dropped content_items.summary, pre-ID-131, per CLAUDE.md gotcha — do NOT “fix” the naming).
• Sector-intelligence pipeline (feed_articles) + source-document ingestion (source_documents).

Consumers

• lib/intelligence/pipeline.ts:407, 492 — pipeline writes extraction_method per feed-article ingestion path.
• lib/intelligence/starter-packs.ts, lib/intelligence/summary.ts — intelligence surfaces read extraction_method.
• scripts/kb_pipeline/extract.py:34, 260, 322, 379 — Python extractor sets extraction_method to trafilatura, jina_reader, pdfplumber (note: those Python values diverge from the 5-value CV — flagged for ratifier reconciliation).
• scripts/backfill-classify-content-items.ts, scripts/batch-rescore-articles.ts, scripts/verify-intelligence-classification.ts — batch/analytics scripts read extraction_method.
• feed_articles analytics / monitoring surfaces.

Notes

• Added to source §2.1 in S235 WP1 (commit 53659515) — recently inventoried.
• Closed CHECK enum.
• firecrawl retired-S236, prune-on-migration timing Ratified-S237 (Liam). §7.5 “drop Firecrawl” decision has landed; prod content (27 articles currently using extraction_method='firecrawl') will be migrated or replaced. Liam ratified S237 that firecrawl is pruned from this CV’s baseline_values when the CHECK migration drops it from the DB enum — not before. This keeps the CV in lockstep with the live schema (same pattern as CV 04 capability). The migration is out of WP6 scope; this CV records the retirement decision so downstream tooling stops emitting the value.
• Python extractor divergence — Ratified-S237 retire trafilatura + pdfplumber on Docling adoption (Liam). scripts/kb_pipeline/extract.py currently emits trafilatura, jina_reader, pdfplumber — these diverge from the 5-value CV. Per docs/plans/phase-0-investigation/10-feedback-investigation-findings/00-synthesis-v2.md §3.2 + §3.3, Docling becomes the primary PDF/DOCX/XLSX extractor (Cloud Run sidecar) and pullmd is retained for HTML; trafilatura + pdfplumber paths retire on Docling adoption. Detailed write-path reconciliation (which extractors emit which extraction_method values once Docling lands) belongs in the WP4 architecture sub-docs 02-data-flow.md + 03-tech-stack.md. The CV records the retirement direction here so the live divergence is documented, not lost.
• ID-42 extension (pullmd deploy + HTML retirement). Six values added (pullmd_readability, pullmd_playwright, pullmd_cloudflare, pullmd_reddit, pullmd_trafilatura, docling) — extension only, no prune of firecrawl or any legacy value (Ratified-S237 lockstep; the firecrawl prune stays T7-gated and DB-driven). The pullmd_* values mirror the live pullmd X-Source header; docling covers binary (PDF/DOCX/XLSX) extraction via the Cloud Run sidecar. The same 11-value set governs both feed_articles.extraction_method (CHECK extended, additive) and the net-new source_documents.extraction_method column (TECH §WP-D, migration 20260526074944_id42_pullmd_provenance.sql).
• This CV now governs both the sector-intelligence pipeline (feed_articles) and source-document ingestion (source_documents, ID-42); content_items (which this CV was never wired to) was DROPPED at ID-131 M6.