Skip to content
Canonical

Knowledge Hub Legacy Extraction Surfaces Survey — ID-42 Retirement Planning

Knowledge Hub Legacy Extraction Surfaces Survey — ID-42 Retirement Planning

Section titled “Knowledge Hub Legacy Extraction Surfaces Survey — ID-42 Retirement Planning”

Scope: Surfaces A (lib/intelligence/content-extractor.ts + firecrawl) and B (lib/extraction/url.ts + extractFromHtml)
Date: 2026-06-15
Worktree: /Users/liamj/Documents/development/knowledge-hub/.claude/worktrees/subo-42-retire

Q1 — Surface A: lib/intelligence/content-extractor.ts Structure

Section titled “Q1 — Surface A: lib/intelligence/content-extractor.ts Structure”

Tier Cascade (exact line ranges)

Section titled “Tier Cascade (exact line ranges)”

Tier 1 — RSS content:encoded (lines 180–193)

  • Function: extractContent() → checks item.contentEncoded
  • Condition: word count >= MIN_CONTENT_WORDS (100)
  • Success logs: [Extraction] {url} — Tier 1 (rss_content), {word count} words
  • Returns: ExtractionResult with method: 'rss_content'

Tier 2 — Direct fetch + extractMainContentHtml (lines 196–240)

  • Function: extractContent() → direct fetch(item.url)
  • Calls: extractMainContentHtml(html) at line 220
  • Validates content-type includes text/html or application/xhtml
  • Success logs: [Extraction] {url} — Tier 2 (fetch), {word count} words
  • Returns: method: 'fetch'

Tier 2.5 — Jina Reader / r.jina.ai (lines 243–272)

  • Function: extractContent() → calls https://r.jina.ai/{url}
  • No API key required
  • Accepts: text/markdown header
  • Success logs: [Extraction] {url} — Tier 2.5 (jina_reader), {word count} words
  • Returns: method: 'jina_reader'

Tier 3 — Firecrawl / @mendable/firecrawl-js (lines 275–315)

  • Function: extractContent() → dynamic import at line 276
  • Instantiates: new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY })
  • Call: firecrawl.scrape(item.url, { formats: ['html'] }) at line 278
  • Reads metadata.sourceURL (Firecrawl-resolved publisher URL) at lines 288–290
  • Success logs: [Extraction] {url} — Tier 3 (firecrawl), {word count} words
  • Returns: method: 'firecrawl', optional resolvedUrl if Firecrawl metadata differs from input URL
  • Error logs: [Extraction] {url} — Tier 3 (firecrawl) failed

Tier 4 — summary_fallback (lines 318–334)

  • Function: extractContent() → fallback to item.summary ?? item.title
  • Logs WARN with fallback reason (firecrawlKeyMissing status or all 4 tiers failed)
  • Returns: method: 'summary_fallback'
Section titled “Firecrawl-related symbols (all line numbers)”
  • Line 12: let firecrawlWarningLogged = false; — module-level state flag
  • Line 19: let firecrawlKeyMissing = false; — module-level state flag for health check
  • Line 27–29: isFirecrawlConfigured() export — reads \!firecrawlKeyMissing && Boolean(process.env.FIRECRAWL_API_KEY)
  • Line 136–164: checkFirecrawlApiKey() export — sets firecrawlKeyMissing, throws in production if key missing
  • Line 156–162: Firecrawl warning log (once per process)
  • Line 37–43: extractMainContentHtml(html) — helper; extracts <article> or <main> tag content
  • Line 276: const { default: Firecrawl } = await import('@mendable/firecrawl-js');
  • Line 277: const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
  • Line 278: const doc = await firecrawl.scrape(item.url, { formats: ['html'] });
  • Line 297: Log: [Extraction] {url} — Tier 3 (firecrawl), {word count} words
  • Line 305: method: 'firecrawl' set in return
  • Line 313: Error log: [Extraction] {url} — Tier 3 (firecrawl) failed
  • Line 322–324: Fallback reason: firecrawlKeyMissing check

Exported symbols from content-extractor.ts

Section titled “Exported symbols from content-extractor.ts”
  1. extractContent(item: ParsedFeedItem) — line 170–334
  2. checkFirecrawlApiKey() — line 136
  3. isFirecrawlConfigured() — line 27
  4. isGoogleNewsUrl(url) — line 46–53
  5. resolveGoogleNewsUrl(url) — line 60–93
  6. normaliseUrl(url) — line 96–121

Imports across lib/ and app/

Section titled “Imports across lib/ and app/”

Callers of exported symbols:

  • lib/intelligence/pipeline.ts:

    • Line 11: imports checkFirecrawlApiKey, isGoogleNewsUrl, resolveGoogleNewsUrl from content-extractor
    • Line 14: imports normaliseUrl from content-extractor
    • Line 557: calls checkFirecrawlApiKey() at pipeline startup
    • Lines 335, 420: write extraction_method: extraction.method to database

  • lib/intelligence/feed-poller.ts:

    • Line 7: re-exports validateWebUrl (from url-validation, NOT content-extractor)
    • Firecrawl import: dynamic import in pollWebSource() at line 430

  • app/api/intelligence/workspaces/[id]/sources/[sourceId]/test/route.ts:

    • Imports pollWebSource and pollFeed from feed-poller (NOT direct content-extractor use)
    • Line 57: reads result.firecrawlCalled from pollWebSource return

  • app/api/ingest/url/route.ts:

    • Line 15: imports normaliseUrl from content-extractor
    • Line 84: calls normaliseUrl(url) for reference_items URL dedup

pullmd references in content-extractor.ts

Section titled “pullmd references in content-extractor.ts”

No references to pullmd, PULLMD_SERVICE_URL, or PULLMD_API_TOKEN found in this file.

Q2 — Surface A Collaborators

Section titled “Q2 — Surface A Collaborators”

lib/intelligence/pipeline.ts

Section titled “lib/intelligence/pipeline.ts”
  • Line 11: Imports checkFirecrawlApiKey from content-extractor
  • Line 557: Calls checkFirecrawlApiKey() once at pipeline startup in runPipeline()
  • Lines 335, 420: Writes extraction_method: extraction.method to content_items table
    • Line 335: in processArticle() → content insertion path
    • Line 420: in processFeedArticle() → RSS feed article path

lib/intelligence/feed-poller.ts

Section titled “lib/intelligence/feed-poller.ts”

  • pollWebSource() function (lines 336–533):

    • Line 430: const { default: Firecrawl } = await import('@mendable/firecrawl-js');
    • Line 431: const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
    • Line 433: const doc = await firecrawl.scrape(source.url, { formats: ['html'] })
    • Line 432: Sets firecrawlCalled = true BEFORE attempt (counts on billing semantics)
    • Lines 439, 458, 507: Sentry breadcrumbs with category: 'intelligence.web-source.firecrawl-call'

  • firecrawlCalled field in WebPollResult (lines 312–313):

    • Definition: /** Whether firecrawl.scrape() was actually invoked. False on HEAD-304 short-circuit, false on validateWebUrl failure, true otherwise. */
    • Write sites:
      • Line 358: initialized firecrawlCalled: false in early-return error path
      • Line 404: firecrawlCalled: false in validateWebUrl error path
      • Line 416: firecrawlCalled: false on HEAD-304 short-circuit
      • Line 428: local var let firecrawlCalled = false
      • Line 432: set to true on Firecrawl instantiation (before attempt)
      • Line 447, 496, 515, 530: returned in WebPollResult
    • Read sites:
      • Line 505: conditional if (firecrawlCalled) to emit Sentry breadcrumb on error
      • Returned in all code paths at lines 496, 530

app/api/intelligence/workspaces/[id]/sources/[sourceId]/test/route.ts

Section titled “app/api/intelligence/workspaces/[id]/sources/[sourceId]/test/route.ts”
  • Line 5: Imports pollWebSource from feed-poller
  • Line 51: Calls pollWebSource(source, { dryRun: true })
  • Line 57: Reads result.firecrawlCalled and assigns to firecrawlCreditsExpected (0 or 1)
  • Line 65, 75: Returns firecrawlCreditsExpected in JSON response

Q3 — Surface B: lib/extraction/url.ts + lib/extraction/html.ts

Section titled “Q3 — Surface B: lib/extraction/url.ts + lib/extraction/html.ts”

lib/extraction/url.ts::extractFromUrl

Section titled “lib/extraction/url.ts::extractFromUrl”

Full structure (lines 47–128):

  1. URL validation (lines 48–52): calls validateUrl(url) (from ./url-validation, not firecrawl)

  2. Fetch with timeout/size limits (lines 54–70):

    • Fetch timeout: 15 seconds (FETCH_TIMEOUT_MS = 15_000)
    • Max content size: 20 MB (MAX_CONTENT_SIZE)

  3. Final URL re-validation after redirects (lines 85–90): validateUrl(finalUrl)

  4. Route by content-type (lines 93–127):

    PDF branch (lines 95–110):

    • Condition: contentType.includes('application/pdf')
    • Calls: extractPdfText(buffer) from ./pdf (line 97)
    • Returns: extractionMethod: 'unpdf' (line 106)

    HTML branch (lines 112–127):

    • Calls: extractFromHtml(html, finalUrl) at line 115
    • Calls: extractOgMetadata(html) at line 114
    • Returns: extractionMethod: 'readability' (line 125)
    • Merges og-metadata fields (ogImage, ogDescription, ogDate, ogAuthor)

lib/extraction/html.ts::extractFromHtml

Section titled “lib/extraction/html.ts::extractFromHtml”

Full structure (lines 28–50):

  • Lazy imports (lines 33–34):

    • const { JSDOM } = await import('jsdom');
    • const { Readability } = await import('@mozilla/readability');

  • DOM construction & Readability parse (lines 36–38):

    • const dom = new JSDOM(html, { url });
    • const reader = new Readability(dom.window.document);
    • const article = reader.parse();

  • Turndown conversion (line 46):

    • turndown.turndown(article.content ?? '')

  • Return shape (lines 44–49):

    {
      title: article.title || '',
      content: turndown.turndown(...),
      author: article.byline || '',
      excerpt: article.excerpt || '',
    }

Callers of extractFromHtml

Section titled “Callers of extractFromHtml”
  • lib/extraction/url.ts (line 115): calls within extractFromUrl() HTML branch
  • NO OTHER CALLERS FOUND — extractFromHtml is local to url.ts extraction flow

Who calls extractFromUrl

Section titled “Who calls extractFromUrl”
  • app/api/ingest/url/route.ts:
    • Line 108: const { extractFromUrl } = await import('@/lib/extraction/url');
    • Line 109: const extracted = await extractFromUrl(url);
    • Context: Manual single-URL ingest landing into reference_items + source_documents (ID-75 OQ-D §2)

Q4 — CRITICAL: Is pullmd reachable from the Vercel app?

Section titled “Q4 — CRITICAL: Is pullmd reachable from the Vercel app?”

grep: PULLMD_SERVICE_URL, PULLMD_API_TOKEN across entire repo

Section titled “grep: PULLMD_SERVICE_URL, PULLMD_API_TOKEN across entire repo”

TypeScript/JavaScript references (lib/, app/, scripts/):

  1. scripts/cocoindex_pipeline/adapters.py:27 — HTML/pullmd AGPL boundary (O-Q3) comment
  2. scripts/cocoindex_pipeline/adapters.py:32 — docs/specs reference + “URL via PULLMD_SERVICE_URL”
  3. scripts/cocoindex_pipeline/adapters.py:69 — ingest_url → _pullmd_fetch comment
  4. scripts/cocoindex_pipeline/adapters.py:153 — “Structured pullmd extraction result”
  5. scripts/cocoindex_pipeline/adapters.py:156 — “pullmd v2.x provenance headers”
  6. scripts/cocoindex_pipeline/adapters.py:169 — async def _pullmd_http_get(url: str) -> PullmdResult:
  7. scripts/cocoindex_pipeline/adapters.py:172–175 — memoization + HTTP contract docs
  8. scripts/cocoindex_pipeline/adapters.py:181 — pullmd_url = os.environ.get("PULLMD_SERVICE_URL")
  9. scripts/cocoindex_pipeline/adapters.py:182 — check: if not pullmd_url:
  10. scripts/cocoindex_pipeline/adapters.py:187 — api_token = os.environ.get("PULLMD_API_TOKEN")
  11. scripts/cocoindex_pipeline/adapters.py:199 — f"{pullmd_url}/api"
  12. scripts/cocoindex_pipeline/adapters.py:249 — async def _pullmd_fetch(url: str, content_epoch: str) -> PullmdResult:
  13. scripts/cocoindex_pipeline/flow.py:2690 — result = await _pullmd_fetch(item.url, item.content_epoch)
  14. scripts/cocoindex_pipeline/flow.py:2693 — pullmd_share_id = result.share_id

Python test references (scripts/tests/):

  • Multiple test mocks and assertions on pullmd functions in test_verify_driver_url.py and test_cocoindex_adapters.py

TypeScript pullmd client in lib/?

Section titled “TypeScript pullmd client in lib/?”

NO TypeScript pullmd client found. _pullmd_fetch() and _pullmd_http_get() are PYTHON-ONLY functions in scripts/cocoindex_pipeline/adapters.py.

PULLMD_SERVICE_URL / PULLMD_API_TOKEN declaration

Section titled “PULLMD_SERVICE_URL / PULLMD_API_TOKEN declaration”
  • NO entries in lib/env-client.ts (NEXT_PUBLIC_* only)
  • NO entries in lib/env-server.ts (checked lines 1–137)
  • Deployment only: Defined in deploy/coolify/docker-compose.{staging,production}.yaml as environment variables for the cocoindex-{staging,production} service

Deployment context (docker-compose)

Section titled “Deployment context (docker-compose)”

Is pullmd a service in compose? YES.

  • deploy/coolify/docker-compose.staging.yaml:206–250 — service pullmd-staging

    • Image: aeternalabshq/pullmd:2.0.0
    • Host-local alias: http://pullmd-staging:3000
    • Depends-on: playwright-staging, trafilatura-staging

  • deploy/coolify/docker-compose.production.yaml:188–234 — service pullmd

    • Image: aeternalabshq/pullmd:2.0.0
    • Host-local alias: http://pullmd:3000

Network reachability:

  • pullmd is INTERNAL to the compose network only
  • Reachable only via host-local compose DNS alias (pullmd:3000 or pullmd-staging:3000)
  • NO public ingress; NOT exposed to the Vercel app
  • Vercel app runs in a separate deployment context and cannot reach local aliases

Conclusion:

pullmd is network-isolated from the Vercel app. Only the cocoindex worker (scripts/cocoindex_pipeline/) can reach pullmd, via compose host-local DNS.

Q5 — ID-75 reconciliation: Has the reference-layer landing shipped?

Section titled “Q5 — ID-75 reconciliation: Has the reference-layer landing shipped?”

reference_items table schema

Section titled “reference_items table schema”

Found in migration 20260606121451_id75_reference_items_layer.sql:

  • Columns include: source_url, title, body, embedding, ingestion_source, extraction_method (NEW), pullmd_share_id (NEW)
  • Provenance tracking: source_documents table linked via foreign key

Ingestion path: extractFromUrl → reference_items

Section titled “Ingestion path: extractFromUrl → reference_items”

app/api/ingest/url/route.ts (lines 40–241):

  1. Fetch & extract (lines 108–109): extractFromUrl(url) returns ExtractedContent with extractionMethod: 'readability' | 'unpdf'
  2. Classify (lines 149–160): classifyText() populates primary_domain/primary_subtopic
  3. Ingest via RPC (lines 206–212): calls reference_ingest RPC with:
    • p_source_url: normalised
    • p_body: extracted.content
    • p_extraction_metadata: { extractor: extracted.extractionMethod, via: 'app_sync_url_import', ... }
  4. NO content_items written — reference_items + source_documents only

Status: SHIPPED — ID-75 app-side landing is active.

Has Surface A/B traffic been rerouted to pullmd?

Section titled “Has Surface A/B traffic been rerouted to pullmd?”

NO — Surfaces A and B still carry live traffic:

  1. Surface A (Firecrawl): Active in lib/intelligence/feed-poller.ts::pollWebSource() for web sources

    • Tier 3 Firecrawl called on every web-source poll (unless HEAD-304 short-circuit)
    • Firecrawl credits counted in Sentry breadcrumbs (telemetry only, no DB aggregate yet per line 535–546)

  2. Surface B (extractFromUrl): Active in app/api/ingest/url/route.ts

    • Manual URL ingest path lands in reference_items (NEW via ID-75)
    • REPLACED old content_items path, but SURFACE B CODE ITSELF is unchanged (readability/unpdf extraction still active)

  3. Python cocoindex: Calls pullmd via _pullmd_fetch() for URL-sourced markdown (ID-75 WP-C)

    • Lands in source_documents + reference_items

Conclusion:

Surfaces A and B have NOT been retired. They are still in active use:

  • Surface A (Firecrawl) carries web-source polling load
  • Surface B (extractFromUrl) carries manual URL ingest load (reference layer destination, not content layer)
  • Both are ELIGIBLE for retirement ONLY after a cutover window confirms zero live traffic and successful pullmd-based alternative landing in production

Q6 — npm deps + trailers

Section titled “Q6 — npm deps + trailers”

package.json dependencies (exact lines)

Section titled “package.json dependencies (exact lines)”
  • Line 67: "@mendable/firecrawl-js": "^4.23.0",
  • Line 70: "@mozilla/readability": "^0.6.0",
  • Line 126: "turndown": "^7.2.4",
  • Line 127: "turndown-plugin-gfm": "^1.0.2",
  • Line 128: "unpdf": "^1.6.2",
  • Line 147: "@types/turndown": "^5.0.6",
  • Line 153: "jsdom": "^28.1.0",

Import sites across lib/ app/ scripts/

Section titled “Import sites across lib/ app/ scripts/”

@mendable/firecrawl-js:

  • lib/intelligence/content-extractor.ts:276 — dynamic import in extractContent()
  • lib/intelligence/feed-poller.ts:430 — dynamic import in pollWebSource()

@mozilla/readability:

  • lib/extraction/html.ts:34 — dynamic import in extractFromHtml()

jsdom:

  • lib/extraction/html.ts:33 — dynamic import in extractFromHtml()

turndown:

  • lib/extraction/turndown.ts — singleton initialization
  • lib/intelligence/content-extractor.ts:5 — import from extraction/turndown
  • lib/extraction/html.ts:10 — import from extraction/turndown
  • lib/content/html-to-markdown.ts:1 — import from extraction/turndown

unpdf:

  • lib/extraction/pdf.ts — dynamic import in extractPdfText()

@types/turndown, @types/jsdom:

  • Type imports in respective modules (no dynamic usage)

lib/env-server.ts: FIRECRAWL_API_KEY

Section titled “lib/env-server.ts: FIRECRAWL_API_KEY”

Line 61:

FIRECRAWL_API_KEY: z.string().optional().or(z.literal('')),

Shape: Optional string (empty string allowed as no-op). NOT required. Non-production environments can run degraded.

lib/intelligence/url-validation.ts: firecrawl comment

Section titled “lib/intelligence/url-validation.ts: firecrawl comment”

Line 6: // firecrawl dependencies (S222 W3-A §2.3.4 D-4).

Q7 — Baseline grep gate (raw command output)

Section titled “Q7 — Baseline grep gate (raw command output)”

Command 1: TS firecrawl/jina references (lib/ app/)

Section titled “Command 1: TS firecrawl/jina references (lib/ app/)”
lib/env-server.ts:61:  FIRECRAWL_API_KEY: z.string().optional().or(z.literal('')),
lib/intelligence/content-extractor.ts:12:let firecrawlWarningLogged = false;
lib/intelligence/content-extractor.ts:19:let firecrawlKeyMissing = false;
lib/intelligence/content-extractor.ts:28:  return \!firecrawlKeyMissing && Boolean(process.env.FIRECRAWL_API_KEY);
lib/intelligence/content-extractor.ts:37:function extractMainContentHtml(html: string): string {
lib/intelligence/content-extractor.ts:131: *   set firecrawlKeyMissing so health/status endpoints can surface the
lib/intelligence/content-extractor.ts:137:  if (process.env.FIRECRAWL_API_KEY) {
lib/intelligence/content-extractor.ts:138:    firecrawlKeyMissing = false;
lib/intelligence/content-extractor.ts:142:  firecrawlKeyMissing = true;
lib/intelligence/content-extractor.ts:149:      '[SI Pipeline] FIRECRAWL_API_KEY is not set — refusing to start pipeline in production. ' +
lib/intelligence/content-extractor.ts:150:        'Set FIRECRAWL_API_KEY in the environment, or explicitly run with NODE_ENV \!= production ' +
lib/intelligence/content-extractor.ts:156:  if (\!firecrawlWarningLogged) {
lib/intelligence/content-extractor.ts:158:      '[SI Pipeline] WARNING: FIRECRAWL_API_KEY is not set — Firecrawl extraction tier will be unavailable. ' +
lib/intelligence/content-extractor.ts:162:    firecrawlWarningLogged = true;
lib/intelligence/content-extractor.ts:220:        const contentHtml = extractMainContentHtml(html);
lib/intelligence/content-extractor.ts:244:    const jinaUrl = `https://r.jina.ai/${item.url}`;
lib/intelligence/content-extractor.ts:276:    const { default: Firecrawl } = await import('@mendable/firecrawl-js');
lib/intelligence/content-extractor.ts:277:    const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
lib/intelligence/content-extractor.ts:278:    const doc = await firecrawl.scrape(item.url, {
lib/intelligence/content-extractor.ts:297:        `[Extraction] ${item.url} — Tier 3 (firecrawl), ${wordCount(text)} words`,
lib/intelligence/content-extractor.ts:305:        method: 'firecrawl',
lib/intelligence/content-extractor.ts:313:      `[Extraction] ${item.url} — Tier 3 (firecrawl) failed`,
lib/intelligence/content-extractor.ts:322:    const fallbackReason = firecrawlKeyMissing
lib/intelligence/content-extractor.ts:323:    ? 'all extraction tiers failed and Firecrawl is not configured (FIRECRAWL_API_KEY missing)'
lib/intelligence/content-extractor.ts:324:    : 'all extraction tiers failed (rss_content, fetch, jina_reader, firecrawl)';
lib/intelligence/feed-poller.ts:304: * `pipeline_runs.result.firecrawl_credits_consumed` aggregation (AC-12).
lib/intelligence/feed-poller.ts:311:  /** Whether `firecrawl.scrape()` was actually invoked. False on HEAD-304
lib/intelligence/feed-poller.ts:313:  firecrawlCalled: boolean;
lib/intelligence/feed-poller.ts:358:      firecrawlCalled: false,
lib/intelligence/feed-poller.ts:396:      category: 'intelligence.web-source.firecrawl-call',
lib/intelligence/feed-poller.ts:404:        firecrawlCalled: false,
lib/intelligence/feed-poller.ts:416:      firecrawlCalled: false,
lib/intelligence/feed-poller.ts:428:  let firecrawlCalled = false;
lib/intelligence/feed-poller.ts:430:    const { default: Firecrawl } = await import('@mendable/firecrawl-js');
lib/intelligence/feed-poller.ts:431:    const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
lib/intelligence/feed-poller.ts:432:    firecrawlCalled = true; // count the call attempt (matches Firecrawl billing)
lib/intelligence/feed-poller.ts:433:    const doc = await firecrawl.scrape(source.url, {
lib/intelligence/feed-poller.ts:439:        category: 'intelligence.web-source.firecrawl-call',
lib/intelligence/feed-poller.ts:447:          firecrawlCalled: true,
lib/intelligence/feed-poller.ts:458:      category: 'intelligence.web-source.firecrawl-call',
lib/intelligence/feed-poller.ts:466:        firecrawlCalled: true,
lib/intelligence/feed-poller.ts:496:      firecrawlCalled,
lib/intelligence/feed-poller.ts:500:    // here for SDK-thrown errors (4xx/5xx wrapped by @mendable/firecrawl-js,
lib/intelligence/feed-poller.ts:504:    // `firecrawlCalled` flag tracks attempts, not successes.
lib/intelligence/feed-poller.ts:505:    if (firecrawlCalled) {
lib/intelligence/feed-poller.ts:507:        category: 'intelligence.web-source.firecrawl-call',
lib/intelligence/feed-poller.ts:515:          firecrawlCalled: true,
lib/intelligence/feed-poller.ts:530:      firecrawlCalled,
lib/intelligence/feed-poller.ts:535:// TODO(roadmap §8.3): `pipeline_runs.result.firecrawl_credits_consumed`
lib/intelligence/feed-poller.ts:544:// embedding token aggregation; firecrawl-credit wiring rides along. Until
lib/intelligence/feed-poller.ts:546:// `firecrawlCalled` flag returned from this function.
lib/intelligence/types.ts:36:    | 'firecrawl'
lib/intelligence/url-validation.ts:6:// firecrawl dependencies (S222 W3-A §2.3.4 D-4).
app/api/intelligence/workspaces/[id]/sources/[sourceId]/test/route.ts:57:      const firecrawlCreditsExpected: 0 | 1 = result.firecrawlCalled ? 1 : 0;
app/api/intelligence/workspaces/[id]/sources/[sourceId]/test/route.ts:65:          firecrawlCreditsExpected,
app/api/intelligence/workspaces/[id]/sources/[sourceId]/test/route.ts:75:        firecrawlCreditsExpected,

Command 2: Python firecrawl/jina references (scripts/)

Section titled “Command 2: Python firecrawl/jina references (scripts/)”

No matches found.

Command 3: GitHub Workflows check

Section titled “Command 3: GitHub Workflows check”
No cloud-run workflow found (expected — deleted S298)

Command 4: Test files exercising these surfaces

Section titled “Command 4: Test files exercising these surfaces”

Found 19 test files:

  • __tests__/integration/si-google-news-dedup.integration.test.ts
  • __tests__/components/feed-source-form.test.tsx
  • __tests__/scripts/embedding-smoke-test.test.ts
  • __tests__/validation/url-normalisation-parity.test.ts
  • __tests__/integration/cocoindex/url-landing-set.integration.test.ts
  • __tests__/api/intelligence/sources.test.ts
  • __tests__/api/intelligence/sources-test-poll-web.test.ts
  • __tests__/api/intelligence/sources-test-poll.test.ts
  • __tests__/api/ingest/url-reference.test.ts
  • __tests__/lib/intelligence/pipeline.test.ts
  • __tests__/lib/intelligence/web-source-poller.test.ts
  • __tests__/lib/intelligence/si-gap-fixes.test.ts
  • __tests__/lib/intelligence/feed-validation.test.ts
  • __tests__/lib/intelligence/google-news-dedup.test.ts
  • __tests__/lib/intelligence/content-extractor.test.ts
  • __tests__/lib/intelligence/feed-poller-web.test.ts
  • __tests__/lib/intelligence/embedding-cache.test.ts
  • __tests__/lib/intelligence/feed-poller.test.ts
  • __tests__/lib/extraction/html.test.ts

Summary: Retirement Decision Points

Section titled “Summary: Retirement Decision Points”

Surfaces still in active use?

Section titled “Surfaces still in active use?”

YES — Both still carry live traffic:

SurfaceActive PathConditionDocs
A (Firecrawl)lib/intelligence/feed-poller.ts::pollWebSource()Every web source poll (unless HEAD-304)Tier 3 in cascade
A (Jina)lib/intelligence/content-extractor.ts Tier 2.5RSS articles < 100 words after Tier 2Tier 2.5 fallback
B (readability)app/api/ingest/url/route.tsManual URL ingest into reference_itemsID-75 landing shipped
B (unpdf)app/api/ingest/url/route.tsPDF URL ingest into reference_itemsID-75 landing shipped

Pre-retirement checklist

Section titled “Pre-retirement checklist”

  1. Coverage audit: Ensure pullmd can handle all Surface A/B extraction use cases

    • Jina tier (free, no auth) → covered by pullmd’s trafilatura/playwright
    • Firecrawl tier (paid, JS-heavy) → covered by pullmd’s playwright sidecar
    • Readability (CSS selectors) → covered by pullmd’s readability-like algorithm
    • unpdf (PDF extraction) → covered by pullmd’s Docling integration

  2. Network access: Vercel app must be reachable to pullmd over HTTP (currently impossible)

    • Current: pullmd isolated to compose network (host-local alias only)
    • Required: Either expose pullmd via public ingress OR move extraction logic server-side

  3. Telemetry migration: Surface A tracks Firecrawl credits via Sentry breadcrumbs

    • Line 535–546: TODO(roadmap §8.3) for DB-persisted aggregation
    • Migration needed before retiring Surface A

  4. Test coverage: 19 test files mock/exercise these surfaces

    • Mock pullmd responses or modify tests post-retirement

Files to retain/delete summary

Section titled “Files to retain/delete summary”

RETAIN (core to pullmd integration)

Section titled “RETAIN (core to pullmd integration)”
  • scripts/cocoindex_pipeline/adapters.py — _pullmd_fetch, _pullmd_http_get
  • scripts/cocoindex_pipeline/flow.py — _pullmd_fetch calls
  • deploy/coolify/docker-compose.{staging,production}.yaml — pullmd services
  • Migration 20260526074944_id42_pullmd_provenance.sql — extraction_method, pullmd_share_id columns
  • Migration 20260606121451_id75_reference_items_layer.sql — reference_items layer

DELETE (Surface A/B)

Section titled “DELETE (Surface A/B)”
  • lib/intelligence/content-extractor.ts — entire file OR extract to archive
  • lib/intelligence/feed-poller.ts::pollWebSource() — Firecrawl tier (lines 275–315 in content-extractor)
  • lib/extraction/url.ts — entire file
  • lib/extraction/html.ts — entire file
  • lib/extraction/pdf.ts — unpdf extraction
  • lib/extraction/turndown.ts — Turndown singleton (if pullmd handles Markdown generation)
  • package.json deps: @mendable/firecrawl-js, @mozilla/readability, jsdom, unpdf, turndown, @types/*
  • Test files using Surface A/B mocks (or refactor to pullmd mocks)