Sector Intelligence Pathway Analysis

Sector Intelligence Pathway Analysis

Date: 29 March 2026 (updated 30 March 2026) Author: Claude Code (analysis for Liam) Input: Sector Intelligence Brief (Phew Design Limited, February 2026), existing analysis, Liam’s notes, current platform state Status: Recommendation for review — prerequisites substantially complete (S123-S129)

Historical context note (April 2026): Sections 1-14 below are the pre-implementation analysis written in March 2026. Phase 1 has been fully implemented across sessions S132-S163. The analysis remains valuable as architectural rationale and decision history, but for the current shipped architecture, see Shipped Architecture (Phase 1) at the end of this document.

---

1. Client Requirements Summary

Distilled from the original brief (docs/client-documentation/markdown/Sector-Intelligence-Brief-Liam-Final.md), not the earlier analysis document.

The problem

Phew operates in education, safeguarding, and audit sectors where policy, legislation, and competitor activity directly impact their products and sales conversations. They have tried RSS aggregation, AI digests, AI feed review, and automated competitor reports. Each tool works in isolation. Nothing feeds into a single knowledge base, nothing has a feedback loop, and the team does not trust the output enough to use it consistently.

What they need (business outcomes, not technical solutions)

1. A single system that ingests sector and competitor news, filters it with AI for relevance, and produces clean, usable output the team actually trusts.

2. Two intelligence streams:

   • Sector Intelligence — MAT activity (leadership appointments, mergers, splits, audits, Ofsted outcomes), public and health audits, legislation and policy (KCSIE, Education Act, DfE, health and social care, GDPR, funding), safeguarding and training updates.
   • Competitor Intelligence — products and services (launches, features, pricing, G-Cloud activity), market activity (contract wins, partnerships, case studies), leadership and strategy (key hires, direction, funding, LinkedIn activity).

3. AI-filtered ingestion — articles scored for relevance against a content tree. The goal is to ingest more but surface less: scaling from ~90 articles/week to 500+ while reducing what the team sees to 30-60 high-relevance items per week.

4. Team-controllable filtering — non-technical team members must be able to influence how the AI filters content. The brief frames this as “team-editable prompts with version history and rollback.” The underlying need is for the team to improve filtering quality over time.

5. Feedback loop — the team can flag false positives (irrelevant article got through) and false negatives (relevant article was filtered out). Flags must be visible alongside the filtering configuration so the team knows what to refine. Basic metrics: articles ingested, articles passed, articles flagged.

6. Audit trail — all articles (passed and filtered) stored for review, not just the ones that pass. The brief is explicit about this: the value is closing the loop so rejected articles can inform future filtering.

7. RSS output — two standard RSS feeds (one per stream), embeddable in their intranet. Each item includes title, source, date, AI-generated summary, and topic tags.

8. Knowledge base integration — filtered articles stored with structured metadata (date, source, stream, topic tags, relevance score) as a searchable archive feeding into the product knowledge base.

9. Scale trajectory — 5 feeds now, 25-45 feeds target. ~90 articles/week now, 500+ target with AI filtering reducing output to 30-60/week.

Phase boundaries from the brief

The brief explicitly defines Phase 1 as: AI-filtered ingestion, team-editable prompts, feedback flagging, two RSS output feeds, article storage with metadata in KB, basic filtering metrics, intranet-embeddable output.

Phase 2 (not in scope but noted for architectural awareness): native RSS syndication, direct website scraping, LinkedIn monitoring, email digest/newsletter, automated prompt improvement from flags, Slack/Teams notifications, trend analysis.

Business value (from the brief)

The brief articulates specific value across three departments: Marketing (content pipeline, webinar topics, positioning, credibility), Sales (MAT intelligence, legislation triggers, bid intelligence, conversation starters, retention), and Product & Strategy (roadmap input, competitive awareness, market awareness). This is not a “nice to have” — it directly supports sales conversations, bid writing, and product decisions.

---

2. Current Platform Capabilities Mapping

2.1 Content ingestion and storage

What exists: Three ingestion pathways are already operational. The Python CLI pipeline (scripts/kb_pipeline/) provides Extract, Dedup, Classify, Embed, Summarise, Store. The web UI (/item/new) allows manual content creation with auto-classification. The file upload path handles PDF/DOCX/MD/TXT with text extraction.

How it serves the need: The existing pipeline already handles URL-to-content-item conversion, including content extraction (trafilatura primary, Jina Reader fallback), duplicate detection, AI classification against a taxonomy, embedding generation, and structured storage. RSS article URLs can flow through this pipeline with minimal modification — the extraction step already handles arbitrary web pages.

2.2 Content types and classification

What exists: 15 content types including article, research, blog, policy, document. DB-driven taxonomy with 12 domains and 38 subtopics (all domain names normalised to lowercase slugs with display_name columns, S127). AI classification using Claude with confidence scoring, domain slug validation in the classification pipeline. ai_keywords for topic tagging. 12 entity types including standard and methodology (S127).

How it serves the need: The article content type is directly applicable to intelligence items. The taxonomy system is extensible — new domains and subtopics can be added via the admin UI with automatic slug-based naming and human-readable display names. The content tree categories from Phew’s brief (MAT activity, legislation, safeguarding, etc.) can map to taxonomy subtopics, or to ai_keywords, or both. The Classification Phase 4 validation (S129) confirmed the pipeline is ready for sector intelligence content — see Section 13 for details.

2.3 Workspaces

What exists: Generic containers with a type discriminator (currently bid and kb_section). The workspaces table has domain_metadata JSONB for type-specific configuration. The workspaces launcher at /workspaces already shows typed cards with “Coming soon” for unreleased types. The workspace type registry was implemented in S125 — lib/workspace-types.ts provides a data-driven type config mapping (type to label/icon/route/schema), replacing the previously hardcoded WORKSPACE_TYPES array.

How it serves the need: Workspaces provide the container abstraction. Intelligence can be a workspace type, giving it a dedicated launcher card, navigation entry point, and type-specific configuration. The existing workspace infrastructure (creation, archival, content assignment via junction table) would transfer. The type registry (S125) means adding intelligence as a workspace type requires no hardcoding — it follows the established pattern.

2.4 Guides

What exists: 7 guides defined with slug routing, ordered sections, content-type and subtopic filtering per section, progress tracking, and a research feed. The guides table supports types: sector, product, company, research, custom. Secondary domain guide matching (S124) ensures content appears in guides for both primary and secondary domains.

How it serves the need: Guides define “what content should exist” for a topic area. An intelligence guide could define the expected coverage structure — e.g., sections for each content tree category (MAT activity, legislation, safeguarding). As intelligence articles are ingested and classified, they would populate guide sections, giving the team a structured view of what intelligence they have and where gaps exist.

2.5 Change Reports (digests)

What exists: AI-generated change digests summarising recent activity. The digests table stores period-based summaries with domain breakdowns, theme clusters, and narrative summaries. The /digest page shows history and generation controls.

How it serves the need: The client’s Phase 2 mentions “email digest / newsletter generation.” The existing Change Reports infrastructure provides the pattern. A sector intelligence digest (“What happened this week in education policy?”) would be a natural extension — same generation mechanism, different content scope.

2.6 MCP tools and Claude integration

What exists: 38 MCP tools across search, content, bids, entities, governance, quality, templates, and dashboard domains. ClaudePromptButton bridge for contextual AI entry points. 5 MCP prompts including reorient and coverage_analysis. 2 MCP Apps (Coverage Matrix, Bid Dashboard). Content effectiveness tracking via get_content_effectiveness tool with win-rate analytics (S126).

How it serves the need: MCP tools already provide search_knowledge_base, get_content_items, find_similar_items, get_coverage_gaps, and get_freshness_report. Intelligence articles stored as content items would be immediately searchable via existing MCP tools. New tools could add intelligence-specific queries (“What happened this week in MAT activity?”). An Intelligence Feed MCP App would give Claude users a visual feed viewer.

2.7 Governance and review

What exists: Governance review workflow (governance_review_status), quality scoring, freshness tracking, review assignments, verification history, quality-to-governance and freshness-to-governance bridges, review cadence dashboard. The review queue at /review supports speed triage with keyboard shortcuts. Win-rate feedback loop (S126) — bid outcomes feed back into content effectiveness scoring, enabling queries like “Which KB content contributed to won bids?”

How it serves the need: The feedback loop requirement (flagging false positives/negatives) maps conceptually to the governance review workflow. However, the existing governance system reviews content quality, not ingestion filtering accuracy. The feedback loop is a distinct workflow — it evaluates whether the AI filter made the right decision, not whether the content itself is high-quality. The win-rate feedback loop (S126) demonstrates the pattern: outcome data feeding back into content scoring. Intelligence feedback follows the same principle — team flags feeding back into filter prompt refinement.

2.8 Entity graph

What exists: 999 entities across 12 types (organisation, certification, regulation, framework, capability, person, technology, project, sector, product, standard, methodology — expanded S127). 913 relationships. Case-insensitive dedup via functional index on LOWER(canonical_name) (S127). Context snippets populated for 91.6% of mentions (S127). Certification metadata propagated with holder awareness (S127). Entity metadata bridge function (S129) connecting entity mentions to content item metadata.

How it serves the need: Intelligence articles mentioning MATs, legislation, competitors, and sector bodies would enrich the entity graph. Competitor intelligence would create entity mentions for competitor organisations, building a structured record of competitor activity over time. The expanded entity types (S127) — particularly standard and methodology — are directly relevant to sector intelligence content about industry standards and frameworks. The entity metadata bridge (S129) ensures entity enrichment flows correctly through the classification pipeline.

2.9 Firecrawl

What is available: Firecrawl is available as a Claude Code skill (firecrawl:firecrawl-cli). It provides scraping, crawling, mapping, and structured extraction capabilities. The evaluation (/.planning/.archive/.research/.tool-evaluations/firecrawl-evaluation.md) rates it as production-grade with JavaScript rendering, LLM-ready markdown output, and proxy rotation.

How it serves the need: Firecrawl addresses two gaps in the current pipeline: (a) it handles JavaScript-rendered sites that the current trafilatura/Jina Reader extraction chain cannot, and (b) it provides website crawling for sources that lack RSS feeds (relevant for Phase 2 scraping, but useful now for extracting full article content from RSS link URLs).

---

3. Gap Analysis

3.1 Gaps that require new functionality

Gap	Description	Severity
Feed source management	No concept of “feed sources” — the platform ingests individual URLs, not subscribed feeds that are polled on a schedule.	Critical
Scheduled polling	No scheduled/automated ingestion. All ingestion is manually triggered (CLI or web UI).	Critical
AI relevance scoring	Classification assigns taxonomy domains. Relevance scoring against a configurable content tree with pass/fail threshold is different — it answers “is this relevant to our business?” not “what domain does this belong to?”	Critical
Configurable filter prompts	AI prompts are developer-managed (classification prompt in code/docs). No mechanism for end users to influence prompt content, view version history, or roll back.	Critical
Feedback flagging (filter accuracy)	Governance review tracks content quality, not filter accuracy. No mechanism to flag “this article should not have passed” or “this article should not have been filtered out.”	High
Rejected article storage	Only passed articles enter content_items. No mechanism to store articles that were filtered out for audit/feedback purposes.	High
RSS output generation	No API routes generating RSS XML.	Medium
Filter metrics	No aggregation of ingestion/pass/flag counts per stream.	Medium
Stream routing	No concept of routing content to different streams. Content is classified into taxonomy domains, not assigned to business-defined streams.	Medium

3.2 Gaps addressed by planned roadmap items

Gap	Roadmap item	Assessment
Workspace type for intelligence	S5 (Workspaces Future Phases) — type registry, shared shell, additional types	S5-B1 DONE (S125). Workspace type registry implemented. Intelligence as a workspace type can now be added without hardcoding.
Publication lifecycle for intelligence	S4 (Document Control — Unified Lifecycle) — formal status transitions with visibility gating	Intelligence articles that pass AI filtering but need human review before being surfaced to the team would benefit from draft -> in_review -> published states. However, this is not strictly required for Phase 1 — the feedback loop serves the same purpose.

3.3 What is NOT a gap (things the analysis document incorrectly identified)

• Content extraction from article URLs — the existing extract.py already handles this well.
• Embedding generation — embed.py works for any text content.
• Duplicate detection — dedup.py prevents re-processing.
• Semantic search — intelligence articles stored as content items are immediately searchable.
• Coverage tracking — intelligence articles populate the coverage dashboard automatically.
• MCP access — existing search_knowledge_base and get_content_items tools work with any content type.
• Entity extraction for intelligence content — the classification pipeline (Phases 1-3, S127) supports all 12 entity types including standard and methodology, with case-insensitive dedup and context snippet extraction. No additional entity work needed for intelligence.
• Taxonomy extensibility — confirmed via Phase 4 validation (S129). New intelligence domains can be created via admin UI without code changes.

---

4. Data Model Options

Three approaches for how intelligence content fits into the Knowledge Hub data model.

Option A: Intelligence as a workspace type

Intelligence becomes a new workspace type alongside bid and kb_section. Each intelligence stream (Sector, Competitor) becomes a workspace instance. Intelligence articles are content items assigned to the relevant workspace via content_item_workspaces.

Schema changes:

• Add intelligence to workspaces.type CHECK constraint.
• New tables: feed_sources, feed_prompts, feed_flags, feed_articles (all articles, passed and filtered).
• Passed articles also enter content_items (with content_type = 'article' and workspace assignment).
• workspaces.domain_metadata stores stream-specific configuration (content tree, polling interval, active status).

Pros:

• Aligns with the platform’s container model — workspaces are where you act on KB content.
• Naturally supports the two-stream structure (one workspace per stream).
• Leverages existing workspace infrastructure (creation, archival, content assignment, workspace filters in browse).
• Scales to additional intelligence streams without schema changes.
• The workspaces launcher gets a third card, reinforcing the “KB is the product, workspaces are applications” model.
• Validates the S5 workspace type registry pattern (implemented S125) — intelligence is application number two.

Cons:

• Couples intelligence to the workspace abstraction, which is designed for project-like containers, not ongoing streams.
• Workspaces have a status field (draft, active, won, lost) designed for project lifecycle, not continuous feeds.
• The two-stream structure is fixed by the client’s brief but may evolve. Workspace-per-stream could become unwieldy at 10+ streams.

Option B: Intelligence as content types

Add intelligence and/or news as valid content types in the CHECK constraint. Intelligence articles are content items differentiated by content_type, not by workspace assignment. Stream routing is handled via metadata.stream or a new column.

Schema changes:

• Add intelligence to content_items.content_type CHECK constraint.
• New tables: feed_sources, feed_prompts, feed_flags, feed_articles.
• Passed articles enter content_items with content_type = 'intelligence'.
• Stream (sector/competitor) tracked via a new column or metadata.stream.
• Topic tags from the content tree map to ai_keywords.

Pros:

• Simpler model — intelligence is just another content type in the KB.
• No dependency on workspace infrastructure or S5.
• Browse filters already support content type filtering — users can filter to intelligence items.
• Q&A Library set the precedent: content type determines interaction model (Q&A pairs have their own page at /library).

Cons:

• Content types are a flat dimension. “Intelligence” is really a content source/purpose, not a content format. An intelligence article is still an article — it just arrived via automated feed ingestion rather than manual import.
• Conflates content format with content provenance. A policy document from GOV.UK ingested via the intelligence pipeline is semantically the same as a policy document ingested manually — adding a different content type creates artificial separation.
• Does not naturally provide a container for stream-level configuration (prompts, feed sources, metrics). Those would need separate tables with no clean FK relationship to the content type value.
• Long-term, other “applications” (tender monitoring, market research) would each need new content types, fragmenting the taxonomy.

Option C: Hybrid — intelligence streams as workspace instances, articles as standard content types

Intelligence streams are workspace instances (type intelligence). Feed sources, prompts, and metrics belong to the workspace. Articles are standard content items (type article) assigned to the intelligence workspace via the junction table. A new feed_articles table stores ALL ingested articles (passed and filtered) for the audit trail.

Schema changes:

• Add intelligence to workspaces.type CHECK constraint.
• New tables: feed_sources (FK to workspace), feed_prompts (FK to workspace), feed_flags, feed_articles.
• Passed articles enter content_items with content_type = 'article' (or research, policy, etc., based on what they are) and get assigned to the intelligence workspace.
• The workspace’s domain_metadata holds stream configuration.

Pros:

• Clean separation of concerns: the workspace is the container/application, content items are the content. Content retains its true type.
• An article about KCSIE legislation is classified as content_type = 'article', primary_domain = 'Policy & Legislation' — which is correct regardless of how it entered the system. It enriches the KB as a first-class article.
• Intelligence workspace provides the management surface: configure feeds, edit prompts, review flagged articles, view metrics. Standard content browsing provides the consumption surface.
• Content from intelligence feeds shows up in semantic search, bid matching, coverage dashboards, and entity graphs without any special treatment — it is just KB content.
• Scales cleanly: each new intelligence stream or monitoring application is another workspace instance, not a schema change.
• Validates both the workspace type pattern (S125 type registry) and the “KB is the product” principle simultaneously.

Cons:

• Two-table storage for articles (feed_articles for audit, content_items for passed articles) adds complexity.
• The workspace-content junction adds a join for stream-filtered queries.

---

5. Recommended Data Model

Option C (Hybrid) is the right approach. Here is the rationale.

Why Option C

The fundamental question is: when an intelligence pipeline ingests an article about new KCSIE legislation, is that article’s identity “an intelligence item” or “an article about KCSIE legislation that was discovered via intelligence monitoring?”

The answer is clearly the latter. The content’s intrinsic nature — what it is about, how it should be classified, who would find it useful — is independent of how it was discovered. An article about KCSIE changes is equally valuable to bid writers searching for “keeping children safe in education” whether it was manually imported or automatically ingested from an RSS feed.

This aligns directly with the Knowledge Hub’s core principle: “One record, many views.” Intelligence is a view (a way of organising and managing content), not a content type. The workspace provides the management surface; the content item provides the KB integration.

Long-term implications

The Knowledge Hub is designed to expand beyond bids. Intelligence is application number two. Tender monitoring, market research, and client relationship tracking are plausible future applications. Option C establishes the pattern: each application is a workspace type with its own configuration and management surface, but the content it produces or curates is standard KB content that enriches the shared knowledge base.

This prevents the fragmentation problem: if intelligence articles are siloed as content_type = 'intelligence', they do not naturally surface in bid searches, coverage analysis, or entity graphs without special-case handling. As standard articles, they participate in all KB capabilities automatically.

Recommended schema outline

New tables:

feed_sources
├── id (uuid, PK)
├── workspace_id (uuid, FK → workspaces.id)
├── name (text) -- human label for this source
├── url (text) -- RSS feed URL
├── source_type (varchar) -- CHECK: 'rss', 'web', 'api'
├── polling_interval_minutes (int) -- how often to poll
├── last_polled_at (timestamptz)
├── last_polled_status (varchar) -- 'success', 'error', 'timeout'
├── article_count (int, default 0) -- running count
├── is_active (boolean, default true)
├── created_at, updated_at, created_by

feed_prompts
├── id (uuid, PK)
├── workspace_id (uuid, FK → workspaces.id)
├── prompt_text (text) -- the filtering prompt
├── version (int) -- monotonically increasing
├── is_active (boolean) -- only one active per workspace
├── change_notes (text) -- what changed and why
├── performance_snapshot (jsonb) -- metrics at time of version (pass rate, flag rate)
├── created_at, created_by

feed_articles
├── id (uuid, PK)
├── workspace_id (uuid, FK → workspaces.id)
├── feed_source_id (uuid, FK → feed_sources.id)
├── external_url (text) -- original article URL
├── external_id (text) -- RSS guid or computed hash for dedup
├── title (text)
├── raw_content (text) -- extracted article text
├── ai_summary (text) -- AI-generated summary
├── relevance_score (numeric) -- 0-1 AI confidence
├── relevance_reasoning (text) -- AI explanation
├── matched_categories (text[]) -- content tree categories matched
├── passed (boolean) -- whether it passed the filter
├── prompt_version_id (uuid, FK → feed_prompts.id) -- which prompt version scored it
├── content_item_id (uuid, FK → content_items.id, nullable) -- link to KB item if passed
├── published_at (timestamptz) -- original publication date
├── ingested_at (timestamptz, default now())
├── created_at

feed_flags
├── id (uuid, PK)
├── feed_article_id (uuid, FK → feed_articles.id)
├── flag_type (varchar) -- CHECK: 'false_positive', 'false_negative'
├── flagged_by (uuid, FK → auth.users)
├── notes (text)
├── resolved (boolean, default false)
├── resolved_at (timestamptz)
├── resolved_by (uuid)
├── created_at

Workspace type extension:

• Add intelligence to workspaces.type CHECK constraint.
• workspaces.domain_metadata for intelligence type stores: content tree definition, default polling interval, scoring model, scoring threshold.

Content items:

• Passed articles stored as standard content items with content_type set based on the content’s actual nature (article, research, policy, etc.).
• metadata.intelligence_stream to indicate which stream sourced the item (for filtering in browse).
• metadata.feed_source_id and metadata.feed_article_id for provenance tracking back to the ingestion audit trail.

---

6. Ingestion Strategy

RSS vs Firecrawl vs hybrid

The brief specifies RSS feeds as the primary input. The question from Liam’s notes is whether RSS is the best approach or if Firecrawl offers something better.

Assessment:

RSS and Firecrawl serve different purposes and are complementary, not competing:

Capability	RSS	Firecrawl
New article discovery	Automatic — feed provides new items	Manual — must crawl or monitor pages
Content extraction	Partial — RSS items often truncated	Full — renders JavaScript, strips boilerplate
Scheduling	Built into RSS readers (poll at intervals)	Requires external scheduling
Source coverage	Only sources with RSS feeds	Any public web page
Cost	Free	API credits per page
Anti-bot handling	Not applicable	Proxy rotation, Chromium rendering

Recommendation: RSS for discovery, Firecrawl for extraction.

The ingestion pipeline should:

1. Poll RSS feeds for new article URLs (lightweight, free, reliable).
2. Use Firecrawl (via the Claude Code skill or direct API) to extract full article content from those URLs. This replaces the current trafilatura/Jina Reader chain with something more robust for modern, JavaScript-heavy sites.
3. Fall back to the existing extraction pipeline (extract.py with trafilatura/Jina) for sources where Firecrawl is unavailable or unnecessary.

This gives the best of both worlds: RSS provides the real-time discovery mechanism (which is what the client already has and understands), while Firecrawl provides higher-quality content extraction than the current pipeline.

For Phase 2 (website scraping for non-RSS sources), Firecrawl becomes the primary tool — its crawl and map capabilities can discover pages on competitor websites without RSS feeds.

IMS repository patterns

The CLAUDE.md references an IMS project (ngsxwlaeybexlgsurnhy, read-only). This is a separate Supabase project. The relevant pattern from Knowledge Hub’s own ingestion review (/.planning/.archive/.research/content-ingestion-review.md) is the three-path ingestion model: CLI pipeline for bulk/automated, web UI for manual, file upload for documents.

The intelligence pipeline would be a fourth ingestion path — automated, scheduled, and filtered. It should follow the same architectural pattern as the CLI pipeline (modular stages, pipeline result tracking via pipeline_runs, quality logging) but with the additional AI relevance scoring stage.

Polling infrastructure

For 5 feeds at 15-30 minute intervals, Vercel Cron is sufficient. For 45+ feeds, dedicated infrastructure would be needed. The recommended approach:

• Phase 1: Vercel Cron route (app/api/cron/intelligence-poll/route.ts) called every 15 minutes. The route queries active feed sources, checks last_polled_at against each source’s polling_interval_minutes, and processes due sources sequentially.
• Scale trigger: When polling takes longer than the Vercel function timeout (60 seconds on Pro, configurable with maxDuration), split into a dispatcher/worker pattern: the cron route enqueues jobs in processing_queue, a separate endpoint processes them.

---

7. Intelligence Stream Mapping

How sector news and competitor intelligence map to platform concepts

Client concept	Platform mapping	Rationale
Intelligence stream (Sector / Competitor)	Workspace instance (type: intelligence)	Each stream is a workspace with its own feed sources, prompts, and metrics
Content tree categories (MAT activity, legislation, safeguarding, etc.)	Taxonomy subtopics + ai_keywords	Categories map to the existing classification system. New subtopics added via admin UI
Feed sources (RSS URLs)	feed_sources table (FK to workspace)	Each source belongs to a stream/workspace
Filtering prompt	feed_prompts table (FK to workspace)	Versioned prompts per stream
Passed articles	content_items + content_item_workspaces	Standard KB content assigned to the intelligence workspace
All articles (audit)	feed_articles table	Complete ingestion record for feedback and audit
False positive/negative flags	feed_flags table	Distinct from governance review (different workflow, different purpose)
RSS output feed	API route (/api/feeds/{workspace-id}/rss)	Public route generating RSS 2.0 XML from passed articles in the workspace
Filtering metrics	Aggregation queries on feed_articles + feed_flags	Per-stream counts derived from existing data

Guides integration

Each intelligence workspace should have an associated guide that mirrors its content tree structure. For the Sector Intelligence stream:

• Section: MAT Activity (subtopics: leadership appointments, mergers, splits, audits, Ofsted)
• Section: Legislation & Policy (subtopics: KCSIE, Education Act, DfE, health & social care, GDPR, funding)
• Section: Safeguarding & Training (subtopics: practice updates, CPD, sector guidance)
• Section: Public & Health Audits (subtopics: education audits, health audits, local authority inspections)

As intelligence articles are classified and tagged, they populate guide sections. The team can see at a glance: “We have strong coverage of KCSIE changes but nothing on MAT mergers this month.” Secondary domain guide matching (S124) ensures articles with multiple domain relevance appear in the appropriate guide sections.

Navigation

The intelligence management UI deserves its own navigation entry. The interaction model is genuinely distinct from existing nav items:

• Browse = filter and read content
• Q&A Library = copy-to-bid
• Coverage = gap analysis
• Guides = structured reading
• Workspaces = project management
• Review = speed triage
• Intelligence = configure + monitor + flag + refine

The intelligence page would show: stream overview (pass rates, recent articles, flag counts), feed source management, prompt editor, and article review/flagging interface. This is a monitoring and configuration interface, unlike anything else in the current navigation.

---

8. Prompt Management Approach

The question

The brief asks for “team-editable prompts” where non-technical staff can view and edit AI filtering prompts. Liam’s notes ask: “Would it not be better to have Claude review the prompts?”

Analysis

The client’s underlying need is not “edit raw prompts” — it is “improve filtering quality over time.” The team does not care about prompt engineering; they care about getting fewer irrelevant articles and missing fewer relevant ones.

Raw prompt editing has significant risks:

• Non-technical users may break prompt structure, causing AI scoring to degrade unpredictably.
• Version proliferation without understanding of cause and effect.
• No way to know if a prompt change actually improved results until many articles have been scored.

Recommendation: Claude-assisted prompt refinement, implemented as a skill

Instead of exposing raw prompts, build a guided refinement workflow. The “Review and Refine” process should be implemented as a skill — the user invokes it via the UI or Claude Desktop, and the skill handles the analysis, recommendation, and prompt update transparently.

1. The prompt is stored in feed_prompts with version history, but the primary UI is not a text editor — it is a structured refinement interface.

2. The team interacts via the feedback loop. They flag articles as false positives or false negatives. Each flag includes a brief note (“This is about a school building project, not MAT activity” or “This KCSIE update is directly relevant”).

3. When the team wants to improve filtering, they invoke the “Review and Refine” skill. The skill:

   • Loads the current prompt from feed_prompts.
   • Fetches all flags since the last refinement, grouped by flag type.
   • Generates a Claude analysis: “Based on 7 false positive flags, the current prompt is too broad on [topic]. Based on 3 false negative flags, it is missing [category]. Here are recommended changes.”
   • Presents the analysis and recommended changes for the team to approve, modify, or reject.
   • On approval, creates a new prompt version with change notes auto-populated.

   This follows the same pattern as the existing claude-md-management:revise-claude-md skill — AI analyses current state, proposes improvements based on evidence, user approves or adjusts.

4. For advanced users who want raw editing, the full prompt text is accessible via an “Advanced” toggle. But the default workflow is review-and-approve, not write-from-scratch.

5. Prompt testing — before a new prompt version goes live, the skill can re-score the last N articles with the new prompt and show a comparison: “With the new prompt, 3 previously-passed articles would be filtered, and 2 previously-filtered articles would pass.” This gives the team confidence before committing.

Version history and rollback

Regardless of how prompts are edited, version history with rollback is essential. The feed_prompts table stores every version. Rolling back activates a previous version and creates a new version record that references it (maintaining an unbroken audit trail). Performance snapshots stored with each version allow the team to see: “Version 3 had a 15% false positive rate; version 4 reduced it to 8%.“

---

9. Audit Trail and Feedback Loop

The core insight

The client is explicit: all articles (passed and filtered) must be stored. This is not just for compliance — the real value is closing the loop so rejected articles inform future filtering.

Integration with governance review

The feedback loop should NOT reuse the existing governance review workflow. They serve different purposes:

Dimension	Governance review	Intelligence feedback
Question answered	”Is this content accurate and current?"	"Did the AI filter make the right decision?”
Applies to	Content already in the KB	Articles at the ingestion boundary (before or after KB entry)
Actions	Verify, flag, revert	Flag as false positive / false negative
Outcome	Content quality improvement	Filter prompt improvement
Actor	Content owner / reviewer	Intelligence team member

Attempting to merge these workflows would conflate two distinct concerns and make both harder to use.

Recommended feedback loop architecture

1. feed_articles table stores every ingested article with its relevance score, pass/fail decision, and the prompt version used to score it.

2. Article review UI shows two views:

   • Passed articles — sorted by date, with flag-as-false-positive button.
   • Filtered articles — sorted by relevance score (highest-scoring filtered articles first, as they are most likely false negatives), with flag-as-false-negative button.

3. feed_flags table records every flag with the user, timestamp, notes, and resolution status.

4. Metrics dashboard shows per-stream, per-period:

   • Articles ingested vs passed (filter ratio).
   • False positive flags (precision problem).
   • False negative flags (recall problem).
   • Flags per prompt version (to track improvement over time).

5. Feedback-to-prompt bridge — when the team invokes the “Review and Refine” skill (see Section 8), unresolved flags are presented as evidence for Claude to analyse. When a new prompt version is created, outstanding flags are marked as “addressed by version N” for audit continuity.

Closing the loop

The architecture ensures every flag ultimately feeds back into the filtering system:

• Flag is created (with notes explaining why the decision was wrong).
• Flag appears in the refinement skill’s analysis alongside the current prompt.
• Claude analyses flags and proposes prompt changes.
• Team approves changes; new prompt version is created.
• Flag is resolved with a reference to the prompt version that addressed it.
• Metrics track whether the new version actually reduced the flag rate.

This is the perpetual improvement cycle the client needs.

---

10. Dependency Analysis

Relationship to S4 (Document Control — Unified Lifecycle)

Assessment: Not a prerequisite for intelligence, but would enhance it.

S4 proposes formal status transitions (draft, in_review, published, archived) with visibility gating. Intelligence articles that pass AI filtering could benefit from a “pending review” state before becoming fully visible to the team. However:

• The feedback loop already serves this purpose. If an article passes but is irrelevant, the team flags it. This is lighter-weight than a formal publication gate.
• The brief does not ask for human approval before articles are surfaced. The team trusts the AI filter; the feedback loop corrects it when it gets things wrong.
• Adding a publication gate would increase latency (articles sit in a queue) and create reviewer burden, which contradicts the “ingest more, surface less” goal.

Recommendation: Do not implement S4 as a prerequisite. If the client later wants a human approval step (e.g., for competitor intelligence that needs verification before being shared internally), S4 can be added as an optional enhancement.

Relationship to S5 (Workspaces Future Phases)

Assessment: S5-B1 (type registry) is DONE (S125). No longer a blocker.

The workspace type registry (lib/workspace-types.ts config mapping type to label/icon/route/schema) was implemented in S125. Adding intelligence as a workspace type now follows the established, data-driven pattern — no hardcoding required.

S5-B2 (shared shell) is useful but not critical for intelligence — the intelligence UI will have a distinct enough interaction model that a shared shell may not apply directly.

Recommendation: S5-B1 is complete. Proceed directly to intelligence schema work. S5-B2 remains a future enhancement.

---

11. Phased Implementation Plan

Phase 0: Foundation (1-2 sessions, ~6-8 hours)

Goal: Establish the intelligence workspace schema.

Note: The workspace type registry (previously part of Phase 0) is already complete (S125).

Work package	Description	Effort	Dependencies
Workspace type registry	lib/workspace-types.ts with type config (S5-B1). DONE (S125)	2-3h 0h	None
Intelligence schema	Migration: add intelligence to workspaces.type CHECK, create feed_sources, feed_prompts, feed_articles, feed_flags tables with RLS.	3-4h	None
Type generation	Regenerate TypeScript types from updated schema. Update lib/validation/schemas.ts.	30min	Schema
Intelligence taxonomy domains	Create new domains via admin UI (legislation-policy, market-intelligence, sector-news — see Phase 4 validation). Run bun run sync:plugin-taxonomy.	30min	Schema

Revised Phase 0 effort: 4-5 hours (down from 6-8).

Phase 1a: Ingestion Pipeline (1-2 sessions, ~10-14 hours)

Goal: RSS feeds are polled, articles are extracted and scored, results are stored.

Work package	Description	Effort	Dependencies
RSS feed parser	Python module to poll RSS feeds, parse XML, extract article metadata (title, URL, published date, guid). Use feedparser library.	2-3h	Schema
Article extraction	Integrate Firecrawl for full content extraction from article URLs, with fallback to existing extract.py.	2-3h	Parser
AI relevance scoring	New Python module: load active prompt from feed_prompts, score article content against it, return score + categories + reasoning. Use Claude Haiku for cost efficiency.	3-4h	Extraction
Pipeline orchestrator	Orchestrate: poll -> extract -> dedup -> score -> route -> (if passed: classify + embed + summarise + store as content_item + entity extraction) -> store feed_article. Track in pipeline_runs.	3-4h	All above
Cron endpoint	Vercel Cron route at app/api/cron/intelligence-poll/route.ts. Query due feed sources, process sequentially, update last_polled_at.	1-2h	Orchestrator

Note on entity extraction: Passed articles flow through the standard classification pipeline, which now handles all 12 entity types (S127), case-insensitive dedup (S127), context snippet extraction (91.6% coverage, S127), and the entity metadata bridge (S129). No additional entity work is needed — the pipeline is confirmed ready (Phase 4 validation, S129).

Phase 1b: Management UI (2-3 sessions, ~14-18 hours)

Goal: Team can manage feed sources, view articles, and flag incorrect decisions.

Work package	Description	Effort	Dependencies
Intelligence workspace creation	Page to create intelligence workspace (name, description, stream type). Workspace launcher card for intelligence type.	2-3h	Phase 0
Feed source management	CRUD UI for adding/editing/removing RSS feed sources within a workspace. Test polling button to verify feed URL.	3-4h	Phase 1a
Article review and flagging	Two-tab view (Passed / Filtered) with article cards showing title, source, score, summary. Flag buttons for false positive/negative with notes.	4-5h	Phase 1a
Prompt refinement skill	”Review and Refine” skill: load current prompt, fetch flags, generate Claude analysis, present recommendations, approve/modify/reject workflow. Version history sidebar. Advanced toggle for raw editing.	4-5h	Phase 1a
Navigation integration	New nav item for Intelligence. Route at /intelligence with workspace-scoped sub-pages.	1-2h	Pages built

Phase 1c: Output and Metrics (1 session, ~6-8 hours)

Goal: RSS output feeds are live and filtering metrics are visible.

Work package	Description	Effort	Dependencies
RSS output API	Two API routes generating RSS 2.0 XML from passed articles per workspace. Add to publicRoutes in proxy.ts.	2-3h	Phase 1a
Metrics dashboard	Per-workspace metrics: ingested/passed/flagged counts by period. Filter ratio trend chart. Flags per prompt version.	2-3h	Phase 1b
Intelligence guide auto-creation	When an intelligence workspace is created, auto-generate a guide with sections matching the content tree.	1-2h	Phase 1b

Phase 1d: KB Integration and MCP (1 session, ~6-8 hours)

Goal: Intelligence content is fully integrated into the knowledge base and accessible via Claude.

Work package	Description	Effort	Dependencies
Classification + entity extraction	Run standard classification and entity extraction on passed articles (already part of Phase 1a pipeline, verify integration).	1-2h	Phase 1a
MCP tool extension	Add stream filter parameter to search_knowledge_base and get_content_items. New get_intelligence_summary tool for “What happened this week?” queries.	2-3h	Phase 1a
Intelligence MCP App	Visual feed viewer inside Claude — article cards with relevance scores, source attribution, flag buttons. Build pattern from Coverage Matrix and Bid Dashboard apps.	2-3h	MCP tools
E2E tests	Critical path tests: workspace creation, feed source CRUD, article review, flag creation, RSS output validation.	2-3h	All above

Summary

Phase	Effort	Calendar	Dependencies
Phase 0: Foundation	4-5h (was 6-8h)	1 session	None
Phase 1a: Ingestion Pipeline	10-14h	1-2 sessions	Phase 0
Phase 1b: Management UI	14-18h	2-3 sessions	Phase 1a
Phase 1c: Output and Metrics	6-8h	1 session	Phase 1a + 1b
Phase 1d: KB Integration and MCP	6-8h	1 session	Phase 1a
Total	40-53h	5-8 sessions

Phases 1c and 1d can run in parallel after Phase 1a completes. Total elapsed time is approximately 4-6 sessions with parallelisation. Phase 0 is reduced by ~2-3 hours because the workspace type registry (S125) is already done.

---

12. Risks and Mitigations

Technical risks

Risk	Likelihood	Impact	Mitigation
Vercel Cron timeout at scale — 45+ feeds may exceed function execution time	Medium	Medium	Start with sequential polling. Monitor execution time. Split to dispatcher/worker pattern when needed.
AI scoring cost at scale — 500 articles/week scored by Claude	Low	Low	Use Claude Haiku for scoring (estimated under USD 10/month at 500 articles/week). Scoring is simpler than full classification.
Content extraction quality — RSS items often have truncated content	Medium	Medium	Use Firecrawl for full extraction. Fall back to existing pipeline. RSS provides the URL; extraction fetches the full page.
Prompt quality drift — team edits may degrade scoring accuracy	Medium	High	Claude-assisted refinement skill prevents direct prompt editing by default. Re-scoring comparison before committing new versions. Performance snapshots on each version.
Feed availability — RSS feeds may change URLs, go offline, or rate-limit	Medium	Low	Track last_polled_status per source. Alert on repeated failures. Sources are individually configurable.
Duplicate articles across feeds — same article from multiple sources	High	Low	Existing dedup.py handles URL-based dedup. Add external_id (RSS guid) dedup in feed_articles. Embedding similarity for content-level dedup.

Product risks

Risk	Likelihood	Impact	Mitigation
Team does not use the feedback loop — filtering stays static	Medium	High	Make flagging low-friction (one click + optional note). Show flag impact in metrics. Celebrate improvement (“False positive rate dropped from 15% to 8% since last prompt update”).
Intelligence content floods the KB — 30-60 articles/week may overwhelm existing 257 items	Medium	Medium	Intelligence articles are classified and tagged like any content. Browse filters (content type, domain, tags, workspace) allow users to include or exclude intelligence. Default browse view can exclude intelligence workspace items.
Scope creep into Phase 2 — LinkedIn monitoring, email digests, trend analysis	Medium	Low	Phase boundaries are clearly defined in the brief. Architecture supports Phase 2 without rework, but Phase 1 is scoped to RSS ingestion only.
Client expectations on prompt editing — brief says “team-editable prompts” but recommendation is guided refinement	Low	Medium	Present the guided refinement approach as an improvement over raw editing. Show how it reduces the risk of breaking the filter. Offer raw editing as an advanced option. Discuss with client before building.

Dependency risks

Risk	Likelihood	Impact	Mitigation
S5-B1 (type registry) takes longer than expected	Low	High	RESOLVED. S5-B1 completed in S125. No longer a dependency risk.
V1 prompts not ready from client — Matthew drafting initial prompts per brief	Medium	Medium	Can start with a well-structured default prompt based on the content tree in the brief. Refine with client prompts when available. The whole point of the system is iterative refinement.
Firecrawl availability — external dependency for content extraction	Low	Medium	Firecrawl is a fallback enhancement, not a hard dependency. The existing extract.py pipeline works for most sites. Firecrawl improves quality for JavaScript-heavy sources.

---

13. What S123-S129 Unlocked

This section summarises the prerequisite work completed since the original pathway analysis was written in S123.

Classification pipeline readiness (S127, validated S129)

The classification pipeline enhancement (Phases 1-3, S127) and Phase 4 validation (S129) confirmed the pipeline is ready for sector intelligence content:

• 12 entity types including standard and methodology — directly relevant for intelligence articles about industry standards, frameworks, and best practices.
• Case-insensitive entity dedup — functional index on LOWER(canonical_name) prevents duplicate entities from intelligence articles using different casing (e.g., “Ofsted” vs “OFSTED”).
• Domain slug normalisation — all taxonomy domains use lowercase slugs with display_name for UI rendering. New intelligence domains follow this convention automatically.
• Identifier exclusion patterns — SIC codes, VAT numbers, DUNS numbers excluded from entity extraction, preventing false positives from intelligence articles that mention company identifiers.

Taxonomy extensibility confirmed (S127, S129)

The Phase 4 validation confirmed three proposed intelligence domains can be created via the admin UI without code changes:

Proposed slug	Display name	Content type
legislation-policy	Legislation & Policy	KCSIE, Education Act, DfE guidance
market-intelligence	Market Intelligence	Competitor activity, market trends
sector-news	Sector News	MAT activity, safeguarding updates

After creation, bun run sync:plugin-taxonomy updates the classifier. No code changes required.

Entity graph enrichment (S127-S129)

The entity graph is now substantially richer and ready to absorb intelligence content:

• Context snippets at 91.6% coverage (S127) — entity mentions have surrounding context for intelligence synthesis.
• Certification metadata propagation (S127) — expiry dates, issuing bodies populated with holder awareness. Intelligence articles about certification changes will further enrich this data.
• Entity metadata bridge function (S129) — ensures entity enrichment data flows correctly between entity mentions and content item metadata during classification. This was the last piece needed for intelligence articles to enrich the entity graph on ingestion.

Data flow integration tests (S129)

S129 delivered 48 new data flow integration tests across Phases 2 and 3:

• Phase 2 tests: Bridge functions (quality-to-governance, freshness-to-governance), context snippet propagation, temporal reconciliation.
• Phase 3 tests: Golden path E2E tests covering the full content lifecycle from creation through classification, entity extraction, and governance.

These tests provide confidence that intelligence articles flowing through the standard pipeline will correctly populate all downstream systems (entity graph, coverage dashboard, governance workflows, semantic search).

Other prerequisites completed (S125-S126)

• Workspace type registry (S125) — lib/workspace-types.ts provides data-driven workspace type configuration. Adding intelligence requires a config entry, not code changes.
• Win-rate feedback loop (S126) — bid outcomes feed back into content effectiveness scoring. Demonstrates the feedback loop pattern that intelligence filtering will follow (team flags feeding back into prompt refinement).
• TanStack Query migration — 28/40 hooks migrated through Waves 1-5 (S122b-S128). The intelligence UI can use TanStack Query patterns from the start.

---

14. Remaining Prerequisites Before Phase 1 Implementation

#	Prerequisite	Status	Notes
1	Taxonomy extensibility	DONE (S127, validated S129)	Admin UI + slug system + display_name
2	Entity type expansion	DONE (S127)	12 types including standard/methodology
3	Entity dedup	DONE (S127)	Case-insensitive functional index
4	Domain slug normalisation	DONE (S127)	All domains use slugs, display_name for UI
5	Workspace type registry	DONE (S125)	lib/workspace-types.ts data-driven config
6	Win-rate feedback loop	DONE (S126)	Gap E complete, pattern validated
7	Entity metadata bridge	DONE (S129)	Entity enrichment flows through pipeline
8	Data flow integration tests	DONE (S129)	48 tests across Phases 2-3, golden path E2E
9	Classification Phase 4 validation	DONE (S129)	Pipeline confirmed ready for intelligence
10	Company profile schema	Not started	Needed for relevance scoring
11	Intelligence stream config	Not started	Domain-to-stream mapping
12	Relevance scoring pipeline	Not started	Separate AI stage
13	Feedback loop as skill	Not started	Review & Refine process (see Section 8)

Items 1-9 are complete. Items 10-13 are the scope of Sector Intelligence Phases 0-1 (see Section 11). The platform is ready for implementation.

---

Security Considerations

RSS Feed Exposure

The sector intelligence feature exposes two RSS feed endpoints per workspace:

1. Passed feed (/api/rss/[workspaceId]/passed) — articles that passed the relevance filter. This feed is intentionally public (no authentication) to support embedding in intranet RSS readers, Slack integrations, and other consumption tools that do not support authenticated feeds.

2. Filtered feed (/api/rss/[workspaceId]/filtered) — all scored articles with relevance reasoning. This feed exposes AI-generated relevance scores and reasoning text, which could reveal competitive intelligence priorities (e.g. which competitor products or market activities the organisation monitors and considers relevant).

Current Security Model

• Obscurity, not authentication. Both feeds are protected only by the workspace UUID in the URL path. Anyone with the UUID can access the feed. UUIDs are not guessable, but they are not credentials.
• No rate limiting. The RSS endpoints have no rate limiting beyond Vercel’s default function concurrency limits.
• No access logging. RSS feed access is not logged beyond standard Vercel function logs, which rotate and are not queryable for audit purposes.

Risk Assessment

• Passed feed: Low risk. The articles themselves are public web content; the feed merely aggregates them. The selection criteria (which articles passed) is the only non-public information, and this is low sensitivity.
• Filtered feed: Medium risk. The relevance reasoning reveals the organisation’s intelligence priorities. A competitor accessing this feed would learn which topics the organisation considers important and why. For most SMBs this is acceptable, but organisations with sensitive competitive positioning should be aware.

Future Recommendations

• Consider adding optional authentication (API key or Supabase auth token) to the filtered feed endpoint. The passed feed can remain public.
• Add audit logging for RSS feed access if clients request it.
• Document the security model in client-facing workspace configuration so administrators can make informed decisions about sharing feed URLs.

---

Summary of Recommendations

1. Data model: Hybrid approach (Option C) — intelligence streams as workspace instances, articles as standard content types. This validates the “KB is the product” principle and prevents content fragmentation.

2. No remaining prerequisite blockers. S5-B1 (workspace type registry, S125), classification pipeline (S127), entity graph enrichment (S127-S129), data flow tests (S129), and Phase 4 validation (S129) are all complete. Phase 0 can begin immediately.

3. RSS for discovery, Firecrawl for extraction. The two are complementary, not competing. RSS provides the real-time feed; Firecrawl provides robust content extraction.

4. Claude-assisted prompt refinement as a skill instead of raw prompt editing. The team interacts via the feedback loop; the “Review and Refine” skill analyses flags and proposes improvements. Raw editing available as an advanced option.

5. Separate feedback loop from governance review. They answer different questions and serve different workflows. Intelligence feedback is about filter accuracy; governance review is about content quality.

6. Start building. All platform prerequisites are met. Phase 0 (intelligence schema + taxonomy domains) can begin in the next session, unblocked by any external dependency except the V1 prompts (which can be bootstrapped from the content tree in the brief).

---

Shipped Architecture (Phase 1)

Added April 2026. Reflects the implemented system as of session S163.

Phase 1 was implemented across sessions S132-S163, following the Option C (Hybrid) data model recommended in Section 5 above. Intelligence workspaces are instances of the workspaces table with type = 'intelligence'. Feed articles are stored in dedicated SI tables; articles that pass relevance scoring are also promoted to standard content_items and participate in the full KB (search, coverage, entity graph, governance).

Library structure (lib/intelligence/)

All server-side intelligence logic lives in lib/intelligence/. The module is TypeScript-only (the original analysis proposed Python; implementation used TypeScript to stay within the Next.js runtime).

File	Purpose
pipeline.ts	Orchestration — polls due feed sources, extracts content, scores relevance, stores results
feed-poller.ts	RSS feed polling via rss-parser, with conditional requests (ETag / Last-Modified), rate-limit awareness
content-extractor.ts	Article content extraction — Jina Reader primary, Firecrawl fallback, Google News URL resolution
relevance-scorer.ts	AI relevance scoring (Claude) against workspace prompt + company context, with embedding pre-filter
article-summariser.ts	AI article summarisation (Claude Haiku) for passed articles — 2-3 sentence summaries for RSS and scanning
guide-generator.ts	Auto-generates hierarchical SI guides from company profile sectors and topics
starter-packs.ts	Pre-configured feed + guide packs per sector (Education, Health & Social Care, etc.)
topic-mappings.ts	Sector-to-topic/keyword mappings for hierarchical guide section nesting
flag-analyser.ts	Claude-backed flag pattern analysis for prompt refinement recommendations
health.ts	Workspace and pipeline health metrics (last run, failure counts, source status)
rate-limiter.ts	Per-domain rate limiting with exponential backoff for feed polling and content extraction
rss-generator.ts	RSS 2.0 XML generation (pure functions) with custom kh:relevanceScore namespace element
relevance-display.ts	Relevance score display helpers — maps 0-1 scores to category labels (Strong/Partial/Weak/Off-topic)
summary.ts	Intelligence summary data fetcher — aggregates articles, sources, flags for MCP tool
types.ts	TypeScript type definitions (PollResult, ExtractionResult, RelevanceResult, PipelineRunResult, etc.)

API routes (app/api/intelligence/)

All routes use getAuthorisedClient() for auth and return JSON unless noted otherwise.

Workspace management

Route	Methods	Purpose
workspaces/route.ts	GET, POST	List all intelligence workspaces; create new one
workspaces/[id]/route.ts	GET, PATCH, DELETE	Get, update, or delete a workspace

Feed sources

Route	Methods	Purpose
workspaces/[id]/sources/route.ts	GET, POST	List feed sources; add new source
workspaces/[id]/sources/[sourceId]/route.ts	PATCH, DELETE	Update or remove a feed source
workspaces/[id]/sources/[sourceId]/test/route.ts	POST	Test-poll a feed source to verify URL

Articles and flags

Route	Methods	Purpose
workspaces/[id]/articles/route.ts	GET	List articles (passed/filtered) with pagination
workspaces/[id]/articles/[articleId]/flag/route.ts	POST	Flag a specific article as false positive/negative
workspaces/[id]/flags/route.ts	GET	List all flags for a workspace
workspaces/[id]/flags/analyse/route.ts	POST	Trigger Claude-backed flag pattern analysis
workspaces/[id]/flags/resolve/route.ts	POST	Resolve flags (addressed or dismissed)

Metrics and health

Route	Methods	Purpose
workspaces/[id]/metrics/route.ts	GET	Aggregated workspace metrics (ingested, passed, flagged)
workspaces/[id]/metrics/trend/route.ts	GET	Time-series trend data for filter ratio charts
workspaces/[id]/metrics/prompt-performance/route.ts	GET	Per-prompt-version performance comparison
workspaces/[id]/health/route.ts	GET	Pipeline and source health status

Prompts

Route	Methods	Purpose
workspaces/[id]/prompts/route.ts	GET, POST	List prompt versions; create new version
workspaces/[id]/prompts/preview/route.ts	POST	Preview re-scoring results with a candidate prompt

Other

Route	Methods	Purpose
workspaces/[id]/seed-starter-pack/route.ts	POST	Seed a workspace with a pre-configured starter pack
profiles/route.ts	GET, POST	List/create company profiles (relevance scoring context)
profiles/[id]/route.ts	GET, PATCH, DELETE	Get, update, or delete a company profile
trigger-poll/route.ts	POST	Manually trigger a poll run (admin action)

RSS output routes (app/api/feeds/)

Public, unauthenticated RSS 2.0 XML endpoints for intranet embedding:

Route	Purpose
[workspaceId]/rss/route.ts	Passed articles feed — articles that cleared the relevance filter
[workspaceId]/rss/filtered/route.ts	Filtered (near-miss) articles — for reviewing potential false negatives

Both return Content-Type: application/rss+xml. No authentication — protected by workspace UUID obscurity only. See Security Considerations (Section above) for the risk assessment.

Cron jobs

Route	Schedule	Purpose
app/api/cron/intelligence-poll/route.ts	*/15 * * * *	Poll due feed sources, extract, score, store (maxDuration: 120s)
app/api/cron/intelligence-cleanup/route.ts	0 3 * * 0	Clean up filtered articles older than 90 days via cleanup_filtered_articles RPC (weekly, Sunday 03:00)

Both verify CRON_SECRET via bearer token. Configured in vercel.json.

Database tables

Five new tables plus one addition to an existing table. Full column definitions in docs/reference/SCHEMA-QUICK-REFERENCE.md (sections 24-28).

Table	Schema ref	Purpose
company_profiles	§24	Client business context (sectors, services, competitors, key topics) for relevance scoring
feed_sources	§25	RSS feed URLs, polling config, HTTP caching headers (ETag/Last-Modified), failure tracking
feed_prompts	§26	Versioned filtering prompts per workspace with performance snapshots
feed_articles	§27	Every ingested article (passed and filtered) with relevance scores and audit trail
feed_flags	§28	False positive/negative feedback on feed articles, with resolution tracking

The workspaces table CHECK constraint includes intelligence as a valid type. Workspace-level configuration (relevance threshold, polling settings) is stored in workspaces.domain_metadata JSONB.

Key relationships:

• feed_sources.workspace_id → workspaces.id
• feed_prompts.workspace_id → workspaces.id
• feed_articles.workspace_id → workspaces.id
• feed_articles.feed_source_id → feed_sources.id
• feed_articles.content_item_id → content_items.id (for passed articles promoted to KB)
• feed_articles.prompt_version_id → feed_prompts.id
• feed_flags.feed_article_id → feed_articles.id

Existing tables reused without modification: workspaces, content_items, content_item_workspaces (junction), guides, guide_sections.

MCP integration

Tool

Tool	Category	Description
get_intelligence_summary	intelligence	Aggregated workspace summary — ingestion totals, filter ratios, category/source breakdowns, top articles, flag counts. Parameters: workspace_id, period? (7d/14d/30d/90d), limit?

Registered in lib/mcp/tools/intelligence.ts. Uses lib/intelligence/summary.ts for data fetching and lib/mcp/formatters/intelligence.ts for formatting.

MCP App

App	Resource URI	Purpose
show_intelligence_feed	ui://intelligence-feed/app.html	Visual feed viewer in Claude Desktop/Claude.ai — article cards with relevance scores, source attribution

Source in mcp-apps/intelligence-feed/ (Vite single-file build). Bundled into lib/mcp/app-bundles.ts via bun run build:mcp-apps.

Resource

The kb://quality-briefing resource includes intelligence pipeline health data alongside other quality signals.

UI pages (app/intelligence/)

Route	Page
/intelligence	Workspace list with creation dialog
/intelligence/profiles	Company profiles management
/intelligence/[workspaceId]	Workspace overview (article counts, health, recent items)
/intelligence/[workspaceId]/articles	Article list (passed/filtered tabs) with flagging
/intelligence/[workspaceId]/sources	Feed source management (add, edit, test, remove)
/intelligence/[workspaceId]/filter-rules	Prompt editor, version history, refinement panel
/intelligence/[workspaceId]/metrics	Metrics dashboard (filter ratio trend, prompt performance)
/intelligence/[workspaceId]/settings	Workspace settings (threshold, polling, starter packs)

Navigation: dedicated “Intelligence” entry in the main nav sidebar. Sub-navigation within workspace via components/intelligence/workspace-sub-nav.tsx.

Components (components/intelligence/)

35 components across workspace management, article display, feed source management, prompt refinement, and metrics visualisation. Key groups:

• Workspace: intelligence-workspace-card.tsx, workspace-creation-dialog.tsx, workspace-settings.tsx, workspace-sub-nav.tsx
• Articles: article-card.tsx, article-list.tsx, flag-dialog.tsx
• Feed sources: feed-source-card.tsx, feed-source-form.tsx, feed-source-test-dialog.tsx
• Prompts: prompt-editor.tsx, prompt-version-sidebar.tsx, prompt-performance-table.tsx
• Prompt refinement: prompt-refinement/refinement-panel.tsx, prompt-refinement/flag-analysis-view.tsx, prompt-refinement/prompt-diff-view.tsx, prompt-refinement/rescoring-preview.tsx
• Metrics: metrics-dashboard.tsx, metrics-panel.tsx, filter-ratio-chart.tsx, pipeline-runs-panel.tsx
• Health: health-panel.tsx
• Company profiles: company-profile-card.tsx, company-profile-form.tsx
• RSS: rss-feed-panel.tsx
• Starter packs: seed-starter-pack-dialog.tsx

Hooks (hooks/intelligence/)

19 TanStack Query hooks providing data fetching and mutations:

Hook	Purpose
use-intelligence-workspaces.ts	List/create/update intelligence workspaces
use-feed-sources.ts	Feed source CRUD and test-poll
use-feed-articles.ts	Article list with pagination and pass/filter toggle
use-feed-prompts.ts	Prompt versions, create new version, activate
use-workspace-flags.ts	List flags, create flag
use-analyse-flags.ts	Trigger Claude flag analysis
use-resolve-flags.ts	Resolve flags (addressed/dismissed)
use-intelligence-metrics.ts	Workspace metrics (ingested, passed, flagged counts)
use-metrics-trend.ts	Time-series trend data for charts
use-prompt-performance.ts	Per-prompt-version performance metrics
use-workspace-health.ts	Pipeline and source health status
use-company-profiles.ts	Company profile CRUD
use-seed-starter-pack.ts	Seed workspace with starter pack
use-rescoring-preview.ts	Preview re-scoring with candidate prompt

Pipeline data flow

The intelligence pipeline follows this sequence per cron invocation:

1. Source selection — get_due_feed_sources RPC returns sources where now() - last_polled_at > polling_interval_minutes (with exponential backoff for failed sources)
2. Feed polling — feed-poller.ts fetches RSS XML with conditional requests (ETag/Last-Modified). Returns ParsedFeedItem[]
3. Dedup — Articles already in feed_articles (by external_url per workspace) are skipped
4. Content extraction — content-extractor.ts fetches full article content. Jina Reader primary, Firecrawl fallback (requires FIRECRAWL_API_KEY), Google News URL resolution for news.google.com links
5. Embedding pre-filter — Fast cosine similarity check against workspace relevance embedding to skip clearly irrelevant articles before expensive AI scoring
6. AI relevance scoring — relevance-scorer.ts scores content against workspace prompt + company context. Returns score (0-1), category (high/medium/low/irrelevant), reasoning, and matched categories
7. Storage — All articles stored in feed_articles with score and pass/fail decision
8. Passed article promotion — Articles above the relevance threshold are also stored as content_items (with AI summary, embedding, classification, entity extraction) and linked via feed_articles.content_item_id

Deviations from original analysis

Original recommendation	What was shipped
Python pipeline modules	TypeScript throughout — stays within Next.js runtime, simpler deployment
Firecrawl as primary content extractor	Jina Reader primary, Firecrawl as fallback — lower cost, adequate quality for most sources
feed_flags table name	feed_flags as proposed (name unchanged, but columns gained resolved_notes, resolution_type, prompt_version_id for richer resolution tracking)
Company profile as domain_metadata on workspace	Dedicated company_profiles table (§24) — cleaner separation, reusable across workspaces
Prompt refinement as a Claude Desktop skill	Built as in-app UI (prompt-refinement/ component group) with API-backed flag analysis
No cleanup mechanism discussed	Weekly intelligence-cleanup cron removes filtered articles older than 90 days
No health monitoring discussed	health.ts + health API endpoint + health panel component for pipeline observability
No starter packs discussed	starter-packs.ts provides one-click workspace seeding with sector-appropriate feeds and guides
No embedding pre-filter	Two-stage scoring: fast embedding similarity check before expensive AI scoring