P0-WEB: SI Web Feed Handler via Firecrawl

P0-WEB: SI Web Feed Handler via Firecrawl

1. Intent

Problem

lib/intelligence/feed-poller.ts has no source_type branching. The pollFeed() function unconditionally routes all sources through rss-parser, which throws on HTML content. The feed_sources schema already supports source_type = 'web' (DB CHECK constraint), the Zod validation schema accepts it (FeedSourceCreateSchema at lib/validation/schemas.ts:778), and the starter-pack type definitions allow it. But creating a web source and waiting for the next cron poll produces an unhandled rss-parser exception, marking the source as errored indefinitely.

Firecrawl is already integrated at the article-extraction tier (lib/intelligence/content-extractor.ts:276-309). FIRECRAWL_API_KEY is required in production (fail-fast enforcement at content-extractor.ts:135-163). The infrastructure exists; only the poll-layer branching and web-specific validation are missing.

Outcome

An admin can create a source_type = 'web' feed source pointing at any HTML page. The SI pipeline polls it on schedule, extracts content via Firecrawl, synthesises a ParsedFeedItem, and proceeds through the standard dedup, scoring, and storage path. The RSS path is completely untouched.

2. Scope

In scope

1. Branch processFeedSource() on source.source_type
2. New pollWebSource() function using Firecrawl .scrape()
3. ParsedFeedItem synthesis from Firecrawl response
4. validateWebUrl() for source creation (HTML equivalent of validateFeedUrl)
5. Wire validation in POST /api/intelligence/workspaces/[id]/sources/route.ts
6. Add source_type to FeedSource interface in pipeline.ts
7. Source creation route sets polling_interval_minutes = 360 for web sources
8. PATCH handler resets consecutive_failures = 0 on is_active false-to-true
9. Unit and integration tests for the new branch

Non-goals

• Adding api source type (no consumer pre-launch)
• Changing any RSS/Atom polling behaviour
• Multi-page crawl or sitemap discovery (single-URL scrape only)
• Changing the Firecrawl integration in content-extractor.ts
• UI changes to the source creation form (it already has a source_type field)
• Starter pack changes (all current packs are RSS; web sources are user-created)
• Modifying FeedSourceRef in feed-poller.ts (used only by pollFeed)

3. Design

3.1 Pipeline branching

processFeedSource() at pipeline.ts:280 calls pollFeed(source) without checking source_type. The fix adds a branch before the poll call:

if source.source_type === 'web':
  pollResult = await pollWebSource(source)
else:
  pollResult = await pollFeed(source)    // existing RSS path, unchanged

The FeedSource interface at pipeline.ts:32 must gain source_type: string. The get_due_feed_sources RPC already returns SELECT * from feed_sources, so the column is already present in the response — it just needs to be typed.

3.2 pollWebSource() function

New export in lib/intelligence/feed-poller.ts. Signature:

pollWebSource(source: { id: string; url: string; source_type: 'web' }): Promise<PollResult>

Steps:

1. Import Firecrawl (dynamic import, matching content-extractor.ts:278)
2. Call firecrawl.scrape(source.url, { formats: ['html'] })
3. If scrape fails or returns empty HTML, return PollResult with status: 'error'
4. If scrape succeeds, synthesise one ParsedFeedItem:
   • title: doc.metadata?.title ?? doc.metadata?.ogTitle ?? 'Untitled'
   • url: source.url (canonical; Firecrawl may return a different metadata.sourceURL after redirects — see D2 below)
   • guid: null (web pages have no GUID; dedup is by URL)
   • publishedAt: doc.metadata?.publishedTime ?? doc.metadata?.dcDate ?? null
   • summary: doc.metadata?.description ?? doc.metadata?.ogDescription ?? null
   • contentEncoded: Firecrawl HTML output (see S3.3)
   • categories: [] (no structured categories from HTML)
5. Return PollResult with status: 'success', items: [item], etag/ lastModified from Firecrawl response headers if available (likely null)

The existing pipeline loop at pipeline.ts:342 then processes this single item through the standard path: Google News URL resolution (no-op for non-Google URLs), URL normalisation, isDuplicate() check against feed_articles, content extraction, relevance scoring, and content-item promotion.

3.3 Content extraction for web-polled items

Decision: Request formats: ['html'] from Firecrawl at the poll layer.

When a web-polled item reaches extractContent() at content-extractor.ts:169, the Firecrawl-extracted HTML is in item.contentEncoded. The tier 1 check (item.contentEncoded at line 179) runs it through Turndown (turndown.turndown()), which converts HTML to markdown — exactly what Turndown is designed for. If word count passes, the item returns as method: 'rss_content'. This avoids a second Firecrawl call at the extraction tier. The method label is slightly misleading but acceptable for an internal field.

If the HTML is too short post-conversion, the standard tier 2-4 cascade applies (which includes Firecrawl again at tier 3 — this is harmless and provides a retry opportunity).

Rejected alternatives:

• Request formats: ['markdown'] from Firecrawl (original draft): Turndown is an HTML-to-markdown converter. Fed raw markdown, it double-escapes heading markers (# Heading becomes \# Heading) and link syntax. Requesting HTML from Firecrawl keeps each tool doing what it was designed for.
• Add preExtractedContent field to ParsedFeedItem (Option B): Cleaner semantically but changes the shared type and all consumers for a single use case. Not justified at this scale.

3.4 Dedup and re-poll semantics for web sources

RSS dedup in isDuplicate() at pipeline.ts:220 checks two columns:

• feed_articles.external_url (normalised URL, UNIQUE per workspace)
• feed_articles.external_id (GUID)

For web sources, the URL is the sole dedup key (no GUID). On re-poll, the normalised URL matches the existing feed_articles row, and the item is skipped. This is the correct behaviour: unchanged pages produce no new items.

“Monitor-only after first ingest” semantic: After the first successful poll creates a feed_articles row and (if relevance passes) promotes it to a content_item, every subsequent poll of the same web source calls Firecrawl, receives the page content, URL-dedupes against the existing row, and discards the result. This means each poll costs one Firecrawl API credit with no new output. At current scale (single-digit web sources, 360-minute intervals), this cost is negligible (~4 credits/source/day). Optimisation deferred to post-launch: if credit cost exceeds 500 calls/month, implement a last_successful_content_at check on the feed_sources row — skip the Firecrawl call entirely when a promoted content_item already exists for the source. This is tracked in the product backlog.

If the page content changes between polls, the URL still matches and the new content is discarded. This is acceptable for the initial implementation — web sources are for monitoring static reference pages (policy documents, org pages), not rapidly-changing content. Content-change detection is a future enhancement (see S8.Q1).

Content-level dedup (checkExactDuplicate at pipeline.ts:599) runs when the item is promoted to content_items. This catches cross-source duplicates (e.g. the same page ingested via RSS and web) using the MD5 content hash. No changes needed here.

3.5 validateWebUrl() function

New export in lib/intelligence/feed-poller.ts:

validateWebUrl(url: string): Promise<FeedValidationResult>

Steps:

1. fetch(url) with timeout (FEED_FETCH_TIMEOUT_MS), User-Agent header, redirect: 'follow'
2. Check response.ok (2xx)
3. Check Content-Type includes text/html or application/xhtml
4. Check response body is non-empty (read first 1KB)
5. Return { valid: true } or { valid: false, error: '...' }

This is deliberately lightweight — it confirms the URL serves HTML content, not that the content is extractable. Extraction quality is the pipeline’s responsibility, not the validation gate’s.

3.6 Source creation route update

app/api/intelligence/workspaces/[id]/sources/route.ts:70-111 currently has:

if (parsed.data.source_type === 'rss' || !parsed.data.source_type) {
  // RSS validation + insert
}
// Non-RSS sources -- skip feed validation

Change to three branches:

if source_type === 'rss' or not set:
  validateFeedUrl() -- existing path, unchanged
else if source_type === 'web':
  validateWebUrl() -- new path
  set polling_interval_minutes = 360 unless admin provided an override
else:
  skip validation -- future source types (api)

The web branch returns the same 400 shape on failure. On success, it inserts the row with polling_interval_minutes = 360 (unless admin provided an explicit override). The Zod/DB default (30) is NOT changed — it remains correct for RSS. The override is applied in the route handler after Zod parsing. No feed_title or initial_article_count in the response (web sources have no feed metadata).

3.7 Type changes

• FeedSource in pipeline.ts:32: add source_type: string
• No changes to FeedSourceRef in feed-poller.ts. pollFeed() has one caller (processFeedSource() at pipeline.ts:304). After branching, pollFeed() only handles RSS and does not need source_type. The new pollWebSource() takes its own typed argument (see S3.2).
• No changes to ParsedFeedItem or PollResult (see S3.3)

3.8 Error handling and failure reset

Web source errors follow the existing pattern:

• Firecrawl 4xx/5xx or network error: PollResult.status = 'error'
• updateSourceAfterPoll() records the error in last_polled_error
• consecutive_failures increments (existing column, existing logic)
• After MAX_CONSECUTIVE_FAILURES (10), get_due_feed_sources excludes the source (existing WHERE consecutive_failures < 10)
• Success resets consecutive_failures to 0 (existing logic in updateSourceAfterPoll)

is_active toggle resets consecutive_failures (F-3): The PATCH handler for /api/intelligence/workspaces/[id]/sources/[sourceId] must detect an is_active false-to-true transition and reset consecutive_failures = 0 in the same update. Currently, updateSourceAfterPoll() at pipeline.ts:725 only resets on successful poll, meaning a source excluded at consecutive_failures = 10 stays excluded even after an admin toggle. The PATCH handler change applies to ALL source types (RSS and web alike):

if body.is_active === true:
  fetch current row
  if current.is_active === false:
    set consecutive_failures = 0 in the update payload

This is in scope for this spec (item 8 in S2).

4. Decisions

D1: Firecrawl fetch cadence

DECIDED: Always fetch + rely on URL dedup. Firecrawl .scrape() has no conditional-request support (no ETag/Last-Modified). isDuplicate() prevents duplicate processing. Cost: one API call per source per poll interval; negligible at current scale. See S3.4 for deferred optimisation threshold.

D2: URL-change detection / redirects

DECIDED: Use Firecrawl’s resolved URL as canonical. Firecrawl follows redirects internally. feed_articles.external_url is keyed on the original source URL (normalised), not the redirect target — same-domain and cross-domain redirects both work transparently. Known limitation: normaliseUrl() does not resolve redirects. Two sources pointing at different URLs in the same redirect chain create separate feed_articles rows. Acceptable — admin-created web sources are few and manually curated.

D3: Error handling

DECIDED: Mark as errored + retry next poll. Existing consecutive_failures mechanism pauses after 10 failures. last_polled_status / last_polled_error visible in Settings > Sources UI. No new notification surface needed.

D4: content_text_hash derivation

DECIDED: No hash at poll time; existing two-layer dedup applies. URL dedup at feed_articles (isDuplicate), content-hash dedup at content_items (checkExactDuplicate). Web sources participate in both without changes. See S3.4 for re-poll semantics.

D5: ParsedFeedItem mapping

DECIDED: Null acceptable for missing metadata. Pipeline tolerates null publishedAt, summary, and categories. Only title is required (fallback chain: metadata.title -> metadata.ogTitle -> 'Untitled').

D6: validateWebUrl scope

DECIDED: Status 200 + HTML content-type + non-empty body. A richer check would require a Firecrawl call at validation time — too expensive for a pre-insert gate. Content quality is the pipeline’s responsibility.

D7: Existing broken web row recovery

DECIDED: No migration needed. All starter packs are RSS. Any manually created web source with consecutive_failures >= 10 is recoverable via is_active toggle (PATCH handler resets counter — see S3.8).

5. Acceptance Criteria

AC-1: Admin creates a source_type = 'web' source via Settings > Sources. The creation request validates the URL (HTTP 200, HTML content-type, non-empty). Invalid URLs return 400 with a descriptive error.

AC-2: The next pipeline poll fetches the web source via Firecrawl, creates a feed_articles row with the extracted content, and (if relevance passes) promotes it to a content_item.

AC-3: Polling the same web source twice does not create a duplicate feed_articles row (URL dedup).

AC-4: Firecrawl failure (4xx/5xx/network) marks the source as errored with last_polled_status = 'error' and increments consecutive_failures. The next poll retries.

AC-5: After MAX_CONSECUTIVE_FAILURES (10) consecutive errors, the source is excluded from polling. An admin can recover it by toggling is_active off then on, which resets consecutive_failures to 0.

AC-6: The RSS polling path is completely unchanged. Existing RSS sources continue to work identically.

AC-7: FIRECRAWL_API_KEY absence in production causes the existing fail-fast behaviour (pipeline refuses to start). No new env var required.

AC-8: Web source creation defaults to polling_interval_minutes = 360 unless the admin provides an explicit override.

6. Test Plan

Unit tests

• pollWebSource() with mocked Firecrawl returning valid HTML + metadata -> returns PollResult with status success and one ParsedFeedItem
• pollWebSource() with mocked Firecrawl returning empty HTML -> returns PollResult with status error
• pollWebSource() with mocked Firecrawl throwing -> returns PollResult with status error
• extractContent() with a ParsedFeedItem whose contentEncoded contains realistic Firecrawl HTML output verifies Turndown produces valid markdown without double-escaping (headings, links, emphasis preserved correctly)
• validateWebUrl() with HTML response -> { valid: true }
• validateWebUrl() with non-HTML content-type -> { valid: false }
• validateWebUrl() with 404 -> { valid: false }
• validateWebUrl() with timeout -> { valid: false }
• processFeedSource() branching: source_type rss calls pollFeed, source_type web calls pollWebSource
• PATCH source with is_active: true on a source with is_active: false and consecutive_failures: 10 -> resets consecutive_failures to 0

Integration tests

• Source creation POST with source_type: 'web' and valid HTML URL -> 201, row has polling_interval_minutes = 360
• Source creation POST with source_type: 'web' and non-HTML URL -> 400
• Source creation POST with source_type: 'rss' -> existing behaviour (no regression)

E2E

• si-starter-pack-seeding.spec.ts: No fixture changes needed (all packs are RSS). Add a comment documenting that web sources are validated separately.

7. Rollback Plan

git revert the implementation commit(s). Web-sourced feed_articles rows can remain or be cleaned via DELETE WHERE feed_source_id IN (SELECT id FROM feed_sources WHERE source_type = 'web'). No schema migration to revert.

8. Open Questions for Liam

Q1: Content-change detection for web sources. Currently, re-polling a web source that has changed content produces no update (URL dedup skips it). Should we add content-change detection in a future iteration? E.g. compare the new Firecrawl HTML hash against the stored feed_articles.raw_content and create a new article row if changed. DECIDED: defer to post-launch. The initial use case is monitoring static reference pages. Override if your web sources are expected to change frequently.

9. Known-Good Invariants

1. RSS path untouched. Branch in processFeedSource() before pollFeed(). Existing RSS/Atom sources use the identical code path.
2. FIRECRAWL_API_KEY already live. Required in production; no new env var.
3. Dedup formula alignment. Same two-layer dedup as RSS: URL at feed_articles, content-hash at content_items. No divergence.
4. get_due_feed_sources returns all source types. No source_type filter in the RPC. Web sources returned alongside RSS without changes.
5. Zod + DB already accept web. FeedSourceCreateSchema enum and feed_sources_source_type_check both include web. No changes needed.
6. C-7 gate dependency. This spec clears C-7, enabling SI UI to expose the web source type to users.