Data Entry Point Reference

Data Entry Point Reference

DRIFT WARNING — added 2026-05-06. The Phase 0 schema investigation (docs/plans/content-items-cleanup-phase-0-investigation.md) confirmed substantial drift across §1–§12 of this document: stale line numbers, fields tables incomplete, post-S205 governance_review_status → publication_status rename not propagated, embed-cap values stale (5,000 → 24,000), claims of “no content_history” wrong (DB trigger writes v1 server-side from ingest_source), claims of “no dedup” wrong on multiple EPs, function/route names stale (e.g. §11 RSS uses intelligence-poll not intelligence/run, processFeedSource not processSource). Per-path drift cited in docs/plans/phase-0-investigation/0.1-*.md. Phase 0.4 (drift correction) is PAUSED pending Phase 0.6 synthesis output. Do not rely on field tables, line numbers, or “fields-not-written” lists in this document for code accuracy until refresh complete. (S35 W4 smoke-test subsection in §1 IS current — added post-banner 2026-05-06.)

Last validated: 28/04/2026 (S205C reference-doc refresh — confirms S205 WP-A1 typed provenance + WP-A2 pipeline_runs on EP9 still wired; no entry-point pipeline drift since fd1c41f5)

S184 WP1 (21/04/2026): TS-side cross-system dedup complete. All insert-capable EPs now soft-block on exact-hash match by stamping content_items.dedup_status='suspected_duplicate' and writing the existing id into metadata.suspected_duplicate_of. Spec §6 decisions:

• D1 soft-block — insert proceeds, no 409. Humans reconcile via UI.
• D2 admin-only skip_dedup=true override — exposed on EPs 3/4/5/6/9/10. Non-admins silently ignored; EP11 is automated (no override).
• D3 RSS many-to-many — same source_url → insert junction only (content_item_workspaces), no duplicate item. Different URL + same content hash → stamp suspected_duplicate.
• EP10 variant — exact match on bid-outcome new_entry is skip-and-log (not stamp), because bid-outcome is a post-won admin workflow where re-integration of an existing response would be wasteful.

Full dedup matrix refreshed below.

S174 refresh scope: S164b content_items.ai_summary → content_items.summary column rename propagated throughout (feed_articles.ai_summary is intentionally unchanged — it stores the RSS-filter LLM summary). S166–168 Plan A/B/C/D markdown canonicalisation and structural chunking reflected in the per-entry-point pipelines. S167 POST /api/items chunking gap fix verified wired. New Entry Point 11 (RSS Feed Promotion) added — articles that pass the intelligence pipeline’s relevance filter are promoted from feed_articles to content_items via lib/intelligence/pipeline.ts storeAsContentItem(). Compliance matrix rebuilt with chunking steps; known-gap added for Batch Item Creation API (Entry Point 6) which does not currently invoke regenerateChunks.

This document is the canonical reference for every path that writes to content_items. It records what each entry point actually does — which fields it sets, which processing steps it runs, and what it omits.

---

Quick Comparison Matrix

Entry point	Source file	Classification	Entity extraction	Relationship extraction	Temporal references	Date extraction (regex)	Embedding	AI summary	Layer inference	Quality score	Dedup check	Admin skip_dedup	Content history	Source document tracking
1. Python URL Ingestion	scripts/kb_pipeline/pipeline.py	Yes (Python)	Yes	Yes	Yes	No	Yes	Yes	Yes (S134)	No	URL hard-skip + content-hash soft-block	n/a (pipeline)	Yes (S153)	No
2. Python Markdown Ingestion	scripts/ingest_markdown.py	Yes (Python)	Yes	Yes	Yes	No	Yes	Yes	Yes (S134)	No	Content-hash soft-block + title-norm	n/a (pipeline)	Yes (S153)	No
3. File Upload API	app/api/upload/route.ts	Yes (TS shared)	Yes (via classify)	Yes (via classify)	Yes (via classify + regex)	Yes	Yes	Yes	Yes	Yes	detect_reupload + content-hash soft-block	Yes (form field)	Yes	Yes
4. URL Ingest API	app/api/ingest/url/route.ts	Yes (TS shared)	Yes (via classify)	Yes (via classify)	Yes (via classify + regex)	Yes	Yes	Yes	Yes	Yes	URL hard-skip + content-hash soft-block	Yes	Yes	No
5. Manual Item Creation API	app/api/items/route.ts	Optional (TS shared)	Optional (via classify)	Optional (via classify)	Optional (via classify)	No	Optional	Optional	Yes	Yes	Content-hash soft-block	Yes	Yes	No
6. Batch Item Creation API	app/api/items/batch/route.ts	Yes (TS shared)	Yes (via classify)	Yes (via classify)	Yes (via classify)	No	Yes	Yes	Yes	Yes	Content-hash soft-block per item	Yes (batch-wide)	Yes	Optional (param)
7. Batch Reclassification	scripts/batch-reclassify.ts	Yes (direct Anthropic)	Yes	Yes	Yes	No	Yes (regenerated)	Yes (summary field)	Yes	No	n/a (UPDATE only)	n/a	No	No
8. Q&A Import	scripts/import_bid_library.py	Yes (keyword classifier)	Optional (--entities)	Optional (--entities)	No	No	Yes	No (truncated answer only)	Yes	No	Content-hash soft-block + idempotency check	n/a (pipeline)	No	No
9. MCP create_content_item	lib/mcp/tools/content.ts	Yes (TS shared)	Yes (via classify)	Yes (via classify)	Yes (via classify)	No	Yes (skip for drafts)	Yes (skip for drafts)	Yes (skip for drafts)	No	Content-hash soft-block	Yes (tool arg)	Yes (S186)	Yes (S205 WP-A1 typed)
10. Bid Outcome KB Integration	app/api/procurement/[id]/outcome/integrate/route.ts	No	No	No	No	No	Yes	No (static template)	No	No	Content-hash skip-and-log (bid-outcome variant)	Yes	No	No
11. RSS Feed Promotion	lib/intelligence/pipeline.ts (S160-series)	Yes (via classifyContent)	Yes (via classify)	Yes (via classify)	Yes (via classify)	No	Yes (via classify)	Yes (via classify)	No (not called)	No	Source-url M2M + content-hash soft-block	n/a (pipeline)	No	No
12. Markdown Batch Ingest UI	app/api/ingest/markdown/route.ts + lib/ingest/markdown-orchestrator.ts	Yes (TS shared)	Yes (via classify)	Yes (via classify)	Yes (via classify)	No	Yes	Yes	Yes	No	Pre-flight content-hash soft-block + filename match	Yes (per-file, admin-only)	Yes (trigger)	No (markdown — no source_documents row)

Legend: “TS shared” = delegates to lib/ai/classify.ts classifyContent(). “Python” = uses kb_pipeline/classify.py. “via classify” = performed as part of the shared classification step. Chunking via regenerateChunks() is listed in the per-entry-point Processing Pipeline sections below and in Appendix B — it is not a column in this summary matrix to keep the width manageable.

---

1. Python URL Ingestion

Name	Python URL Ingestion
Trigger	CLI: python3 scripts/ingest.py <url>
Source file	scripts/kb_pipeline/pipeline.py — process_url() (line 44)
Language	Python
Auth requirement	None (uses SUPABASE_SECRET_KEY directly)

Smoke-test mode (no DB writes, exit 0)

A no-op trigger path short-circuits main() before any URL collection or DB connection — used to validate env-mount + import chain without performing any ingest:

• --smoke-test argparse flag (S34 commit 268ea58f) — local dev / manual invocation: python3 scripts/ingest.py --smoke-test. Logs smoke-test: env + imports OK and exits 0.

This path runs AFTER parser.parse_args() (so the --env=prod SUPABASE_URL fragment assertion still fires when the smoke-test flag is not set) and BEFORE URL collection / DB connection, so it requires no Supabase credentials, no taxonomy snapshot, and no network access beyond image-bundled deps.

Fields Written

Column	Value / Source
title	Extracted from page (or override_title)
content	Full extracted text
source_url	Input URL
source_domain	Extracted from URL
thumbnail_url	Extracted from page OG tags
content_type	Extracted or override_content_type
platform	Extracted or override_platform
author_name	Extracted or override_author
captured_date	Extracted from page
metadata	Extracted metadata dict + extra_metadata
primary_domain	From classification
primary_subtopic	From classification
secondary_domain	From classification
secondary_subtopic	From classification
classification_confidence	From classification
suggested_title	From classification
summary	From classification (overwritten by summary executive if generated)
ai_keywords	From classification
classification_reasoning	From classification
classified_at	datetime.now(utc).isoformat()
embedding	OpenAI text-embedding-3-large (1024d)
summary_data	Structured summary JSON (executive, detailed, takeaways, model, tokens)

Fields NOT Written

created_by, updated_by, brief, detail, reference, source_document_id, file_path, user_tags, priority, governance_review_status, answer_standard, answer_advanced, source_file, layer, quality_score, quality_score_updated_at, lifecycle_type, expiry_date.

Processing Pipeline

1. Extract — extract_url() fetches and parses the page (Jina Reader)
2. Apply overrides — title, content_type, platform, author, extra_metadata
3. Dedup (pre-embed) — URL-based duplicate check via is_duplicate(source_url=url)
4. Classify — Claude Opus 4.6 via kb_pipeline/classify.py (returns entities, relationships, temporal references)
5. Embed — build_embedding_text() + generate_embedding() via OpenAI
6. Dedup (post-embed) — Embedding similarity check via is_duplicate(embedding=embedding)
7. Summarise — generate_summary() produces structured executive/detailed/takeaways
8. Store — insert_content_item(record) — single INSERT
9. Store entities — store_entities() into entity_mentions table
10. Store relationships — store_relationships() into entity_relationships table
11. Store temporal references — merge_item_metadata() stores ai_temporal_references in metadata
12. Layer inference — infer_layer() + update_content_item() (non-blocking)
13. Quality logging — Logs to ingestion_quality_log for missing thumbnails, short content, low confidence, review flags
14. Markdown chunking — store_chunks() from scripts/kb_pipeline/chunk.py:215 (S167 Plan C parity); splits markdown at heading boundaries (mirrors TS chunkByHeadings()) into content_chunks rows with per-chunk embeddings, invoked at scripts/kb_pipeline/pipeline.py:259. Chunk embedding input truncated to MAX_EMBEDDING_CHARS (24,000) per chunk.

Notable Omissions

• No created_by / updated_by (runs as service role)
• No content history versioning
• No quality score calculation
• No source document tracking
• No guide section suggestion
• No topic suggestion
• No date extraction (regex-based)

Content Truncation

No explicit truncation limit on content. The build_embedding_text() function truncates for embedding purposes only. Chunk embedding input is truncated to MAX_EMBEDDING_CHARS (24,000) per chunk in chunk.py.

---

2. Python Markdown Ingestion

Name	Python Markdown Ingestion
Trigger	CLI: python3 scripts/ingest_markdown.py <dir>
Source file	scripts/ingest_markdown.py — process_markdown_file() (line 250)
Language	Python
Auth requirement	None (uses SUPABASE_SECRET_KEY directly)

Fields Written

Column	Value / Source
title	Extracted from H1 heading, bold title, or filename
content	Cleaned markdown (MDX tags stripped)
source_url	None
source_domain	None
content_type	'article' (hardcoded)
platform	'manual' (hardcoded)
author_name	From --author CLI flag
captured_date	File modification time
source_file	Relative path from base directory
metadata	{ingestion_source: 'markdown_file', source_folder, original_format: 'markdown'}
primary_domain	From classification
primary_subtopic	From classification
secondary_domain	From classification
secondary_subtopic	From classification
classification_confidence	From classification
suggested_title	From classification
summary	From classification (overwritten by summary executive if generated)
ai_keywords	From classification + folder tag + extra --tag
classification_reasoning	From classification
classified_at	datetime.now(utc).isoformat()
embedding	OpenAI text-embedding-3-large (1024d)
summary_data	Structured summary JSON (via post-insert UPDATE)

Fields NOT Written

created_by, updated_by, source_domain, thumbnail_url, brief, detail, reference, source_document_id, file_path, user_tags, priority, governance_review_status, answer_standard, answer_advanced, layer, quality_score, quality_score_updated_at, lifecycle_type, expiry_date.

Processing Pipeline

1. Read file — UTF-8 read from disk
2. Extract title — H1 heading, bold title after “Article N”, or filename fallback
3. Clean MDX tags — Strips <Note>, <CodeGroup>, etc. and documentation index blocks
4. Skip-existing check — Optional (--skip-existing): queries source_file column
5. Classify — Claude Opus 4.6 via kb_pipeline/classify.py
6. Embed — build_embedding_text() + generate_embedding() via OpenAI
7. Dedup (post-embed) — Embedding similarity check only (no URL dedup — no URL)
8. Build keywords — Classification keywords + folder tag + user tag
9. Store — insert_content_item(record) — single INSERT
10. Entity storage — store_entities() + store_relationships() (non-blocking)
11. Temporal references — merge_item_metadata() stores ai_temporal_references (non-blocking)
12. Layer inference — infer_layer() + update_content_item() (non-blocking)
13. Summarise — generate_summary() + update_content_item() (post-insert UPDATE)
14. Quality logging — Short content, low confidence, review flags

Notable Omissions

• No content history versioning
• No quality score calculation
• No source document tracking
• No created_by / updated_by
• No date extraction (regex-based)
• No markdown chunking (GAP). scripts/ingest_markdown.py does not invoke scripts/kb_pipeline/chunk.py store_chunks(), so items ingested by this path have content_items rows but no content_chunks rows. Search via search_content_chunks MCP tool will not return these items. Backfill-only fix available: bun run scripts/backfill-chunks.ts.

Content Truncation

No explicit content truncation. MDX cleanup may reduce character count.

---

3. File Upload API

Name	File Upload API
Trigger	POST /api/upload (multipart form data)
Source file	app/api/upload/route.ts — POST() (line 154)
Language	TypeScript
Auth requirement	Authenticated user with admin or editor role

Fields Written

Initial INSERT (line 293):

Column	Value / Source
title	Form field title or derived from filename
content	'' (empty — updated after extraction)
suggested_title	Same as title
content_type	Form field content_type or inferred from MIME type
platform	'manual'
metadata	{original_filename, file_size, mime_type, ingestion_source: 'upload'}
governance_review_status	'draft' if draft=true form field
author_name	Form field author
created_by	Authenticated user ID

Post-extraction UPDATE (line 508):

Column	Value / Source
content	Extracted text (PDF via unpdf, DOCX via mammoth, MD/TXT passthrough)
file_path	{itemId}/{filename} in Supabase Storage
metadata	Extended with page_count, tables, temporal_references, extraction_failed
expiry_date	From regex date extraction (if high/medium confidence)
lifecycle_type	'date_bound' (if expiry date found)
updated_by	Authenticated user ID

AI processing updates (via service client):

Column	Value / Source
embedding	OpenAI text-embedding-3-large (1024d) via lib/ai/embed
primary_domain	Via classifyContent()
primary_subtopic	Via classifyContent()
secondary_domain	Via classifyContent()
secondary_subtopic	Via classifyContent()
classification_confidence	Via classifyContent()
classified_at	Via classifyContent()
suggested_title	Via classifyContent()
summary	Via classifyContent() + generateSummary()
ai_keywords	Via classifyContent()
classification_reasoning	Via classifyContent()
summary_data	Via generateSummary()
quality_score	Via calculateAndRoundQualityScore()
quality_score_updated_at	Timestamp
source_document_id	From source_documents table insert
layer	Via inferLayer() (direct column .update())

Processing Pipeline

1. Auth + role check — getAuthorisedClient(['admin', 'editor'])
2. Validate file — Size (max 50 MB), MIME type, magic bytes
3. Detect re-upload — detect_reupload RPC (filename + user + content hash)
4. Create pipeline_run — Progress tracking record
5. INSERT content_item — Empty content, basic metadata
6. Upload to storage — Supabase Storage documents bucket
7. Create source_document — Lineage tracking (version, parent, storage path)
8. Link source_document — UPDATE source_document_id on content item
9. Extract text — PDF (unpdf), DOCX (mammoth), MD/TXT (passthrough)
10. Date extraction — extractTemporalReferences(), findExpiryDate(), extractDates() (regex)
11. UPDATE content_item — Extracted text, file_path, temporal refs, expiry date
12. Update source_document — Extracted text, extraction metadata
13. Content history — INSERT version 1
14. Embed — generateEmbedding() via OpenAI
15. Dedup check — checkForDuplicates() (informational warning only)
16. Classify — classifyContent() (entities, relationships, temporal refs via AI)
17. Summarise — generateSummary()
18. Quality score — calculateAndRoundQualityScore()
19. Layer inference — inferLayer() + direct column .update()
20. Topic suggestion — suggestTopic() + merge_item_metadata RPC
21. Guide section suggestion — suggestGuideSections() (returned in response, not stored)
22. Mark source_document processed — Status update
23. Diff computation — For re-uploads: computeDocumentDiff() + analyseDocumentImpact() + notifications
24. Complete pipeline_run — Final status update

Notable Omissions

• All processing steps are present — this is the most complete entry point

Content Truncation

Maximum file size: 50 MB. No explicit text truncation after extraction.

---

4. URL Ingest API

Name	URL Ingest API
Trigger	POST /api/ingest/url (JSON body with url)
Source file	app/api/ingest/url/route.ts — POST() (line 12)
Language	TypeScript
Auth requirement	Authenticated user with admin or editor role

Fields Written

Column	Value / Source
title	Extracted from page (or "Imported from {domain}")
content	Extracted page content via extractFromUrl()
content_type	Request body content_type or detectContentType(url)
platform	'web'
source_url	Input URL
source_domain	Parsed hostname
author_name	Extracted from page
thumbnail_url	OG image from page
captured_date	new Date().toISOString()
created_by	Authenticated user ID
user_tags	From request body user_tags
embedding	OpenAI text-embedding-3-large (1024d)
metadata	{ingestion_source: 'url_import', extraction_method, page_count, og_description}

Post-insert via classifyContent() + generateSummary():

Column	Value / Source
primary_domain	Via classifyContent()
primary_subtopic	Via classifyContent()
secondary_domain	Via classifyContent()
secondary_subtopic	Via classifyContent()
classification_confidence	Via classifyContent()
classified_at	Via classifyContent()
suggested_title	Via classifyContent()
summary	Via classifyContent() + generateSummary()
ai_keywords	Via classifyContent()
classification_reasoning	Via classifyContent()
summary_data	Via generateSummary()
layer	Via inferLayer() (direct column .update())
updated_by	Via classifyContent()

Processing Pipeline

1. Auth + role check — getAuthorisedClient(['admin', 'editor'])
2. Rate limit — 10 requests/minute per user
3. Validate body — Zod schema IngestUrlBodySchema
4. SSRF validation — validateUrl() blocks internal/private IPs
5. Existing URL check — Queries content_items for matching source_url (returns early if found)
6. Extract content — extractFromUrl() (Jina Reader or Readability fallback)
7. Quality check — Reject if < 100 chars; warn if < 500 chars
8. Detect content type — From request body or URL pattern
9. Embed — generateEmbedding() via OpenAI
10. Dedup check — checkForDuplicates() (informational warning only)
11. INSERT content_item — Single insert with embedding
12. Content history — INSERT version 1
13. Classify — classifyContent() (entities, relationships, temporal refs via AI)
14. Summarise — generateSummary()
15. Layer inference — inferLayer() + direct column .update()
16. Topic suggestion — suggestTopic() + merge_item_metadata RPC
17. Guide section suggestion — suggestGuideSections() (returned in response, not stored)
18. Fetch final state — Re-query for response

Notable Omissions

• No source document tracking

Content Truncation

Rejects pages with < 100 characters of extracted text (HTTP 422).

---

5. Manual Item Creation API

Name	Manual Item Creation API
Trigger	POST /api/items (JSON body)
Source file	app/api/items/route.ts — POST() (line 18)
Language	TypeScript
Auth requirement	Authenticated user with admin or editor role

Fields Written

Column	Value / Source
title	Request body
content	Request body (may contain HTML)
content_type	Request body
suggested_title	Same as title
platform	'manual'
captured_date	new Date().toISOString()
created_by	Authenticated user ID
metadata	{ingestion_source: ingestion_source ?? 'manual'}
primary_domain	Request body (optional)
primary_subtopic	Request body (optional)
secondary_domain	Request body (optional)
secondary_subtopic	Request body (optional)
priority	Request body (optional)
user_tags	Request body (optional)
ai_keywords	Request body (optional)
author_name	Request body (optional)
source_url	Request body (optional)
brief	Request body (optional)
detail	Request body (optional)
reference	Request body (optional)
embedding	OpenAI (if auto_embed=true)
governance_review_status	Request body (optional)
source_document_id	Request body (optional)

Post-insert AI processing (conditional on auto_classify / auto_summarise):

Column	Value / Source
primary_domain	Via classifyContent() (if auto_classify)
primary_subtopic	Via classifyContent() (if auto_classify)
secondary_domain	Via classifyContent() (if auto_classify)
secondary_subtopic	Via classifyContent() (if auto_classify)
classification_confidence	Via classifyContent() (if auto_classify)
classified_at	Via classifyContent() (if auto_classify)
suggested_title	Via classifyContent() (if auto_classify)
summary	Via classifyContent() / generateSummary()
ai_keywords	Via classifyContent() (if auto_classify)
classification_reasoning	Via classifyContent() (if auto_classify)
summary_data	Via generateSummary() (if auto_summarise)
layer	Via inferLayer() (always)
quality_score	Via calculateAndRoundQualityScore()
quality_score_updated_at	Timestamp
updated_by	Via classifyContent()

Processing Pipeline

1. Auth + role check — getAuthorisedClient(['admin', 'editor'])
2. Rate limit — 20 requests/minute per user
3. Validate body — Zod schema ItemCreateBodySchema
4. Embed — generateEmbedding() (if auto_embed=true)
5. Dedup check — checkForDuplicates() (informational warning only)
6. INSERT content_item — Single insert with all provided fields + embedding
7. Content history — INSERT version 1 (includes brief, detail, reference)
8. Classify — classifyContent() (if auto_classify=true) + pipeline_runs logging
9. Summarise — generateSummary() (if auto_summarise=true) + pipeline_runs logging
10. Layer inference — inferLayer() (always runs)
11. Quality score — calculateAndRoundQualityScore()
12. Topic suggestion — suggestTopic() + merge_item_metadata RPC
13. Guide section suggestion — suggestGuideSections() (returned in response, not stored)

Notable Omissions

• No source document tracking (though source_document_id can be passed)
• No date extraction (regex-based)
• Classification, summary, and embedding are all optional (controlled by auto_classify, auto_summarise, auto_embed flags)

Content Truncation

Content field max: 500,000 characters (Zod validation). Title max: 500 characters.

---

6. Batch Item Creation API

Name	Batch Item Creation API
Trigger	POST /api/items/batch (JSON body with items array)
Source file	app/api/items/batch/route.ts — POST() (line 69)
Language	TypeScript
Auth requirement	Authenticated user with admin or editor role

Fields Written (per item)

Column	Value / Source
title	From item in array (truncated at word boundary to 120 chars by caller)
content	From item in array (Q: {question}\n\nA: {answer} format)
content_type	'q_a_pair'
platform	'extraction'
suggested_title	Same as title
captured_date	new Date().toISOString()
created_by	Authenticated user ID
metadata	{ingestion_source: 'upload_autosplit', autosplit_batch_id, section_name, detection_source, detection_confidence}
source_document_id	From request body (optional, shared across all items)
answer_advanced	From item in array (optional)

Post-insert AI processing (per item):

Column	Value / Source
embedding	OpenAI text-embedding-3-large (1024d)
primary_domain	Via classifyContent()
primary_subtopic	Via classifyContent()
secondary_domain	Via classifyContent()
secondary_subtopic	Via classifyContent()
classification_confidence	Via classifyContent()
classified_at	Via classifyContent()
suggested_title	Via classifyContent()
summary	Via classifyContent() + generateSummary()
ai_keywords	Via classifyContent()
classification_reasoning	Via classifyContent()
summary_data	Via generateSummary()
layer	Via inferLayer() (direct column .update())
quality_score	Via calculateAndRoundQualityScore()
quality_score_updated_at	Timestamp

Processing Pipeline (per item, sequential)

1. Auth + role check — getAuthorisedClient(['admin', 'editor'])
2. Validate body — Zod schema (max 100 items per batch)
3. Batch token enforcement — SHA-256 hash checked against pipeline_runs to prevent re-submission
4. Create pipeline_run — qa_autosplit progress tracking
5. INSERT content_item — Via service client (bypasses RLS)
6. Content history — INSERT version 1
7. Embed — generateEmbedding() via OpenAI
8. Classify — classifyContent() (entities, relationships, temporal refs via AI)
9. Summarise — generateSummary()
10. Layer inference — inferLayer() + direct column .update()
11. Topic suggestion — suggestTopic() + merge_item_metadata RPC
12. Quality score — calculateAndRoundQualityScore()
13. Update pipeline_run progress — After each item

Notable Omissions

• No dedup check (items created sequentially to avoid race conditions, but no explicit dedup)
• No guide section suggestion
• No date extraction (regex-based)
• No source document creation (only links via source_document_id if provided)
• No markdown chunking (GAP). app/api/items/batch/route.ts does not invoke regenerateChunks(). Items created via this path (Q&A autosplit, S116 workflow) have content_items rows but no content_chunks rows. S167 Plan C wired chunking into single-item routes but batch was missed — flag for a follow-up WP. Backfill via scripts/backfill-chunks.ts.

Content Truncation

Content field max: 500,000 characters per item (Zod validation). Title max: 500 characters. Max 100 items per batch.

---

7. Batch Reclassification

Name	Batch Reclassification + Entity Extraction
Trigger	CLI: bun run scripts/batch-reclassify.ts --execute
Source file	scripts/batch-reclassify.ts (line 411, main function)
Language	TypeScript (Bun)
Auth requirement	None (uses SUPABASE_SECRET_KEY directly)

Fields Updated (not INSERT — UPDATE only)

Column	Value / Source
primary_domain	From Claude Sonnet classification
primary_subtopic	From Claude Sonnet classification
secondary_domain	From Claude Sonnet classification
secondary_subtopic	From Claude Sonnet classification
ai_keywords	From Claude Sonnet classification
summary	From Claude Sonnet classification
suggested_title	From Claude Sonnet classification
classification_confidence	From Claude Sonnet classification
classification_reasoning	From Claude Sonnet classification
classified_at	new Date().toISOString()
embedding	Regenerated via OpenAI (using updated suggested_title + content)

Processing Pipeline

1. Load taxonomy — Fetches taxonomy_domains + taxonomy_subtopics from DB
2. Fetch items — Queries content_items with optional domain filter, ordered by content_type priority
3. Data quality report — Identifies garbled keywords, editorial notes, duplicate titles (dry-run report)
4. For each item: a. Classify — Direct Anthropic API call with return_classification_with_entities tool use b. Canonicalise entities — canonicalise() from lib/entities/entity-dedup c. UPDATE content_item — Classification fields + regenerated embedding (skip if --entities-only) d. Delete + re-insert entity_mentions — Clean slate per item e. Delete + re-insert entity_relationships — Clean slate per item

Notable Omissions

• Does not create new items — only updates existing ones
• No content history versioning
• No quality score calculation
• No topic suggestion
• No guide section suggestion
• No summary generation (only sets summary from classification output, not structured summary_data)
• No source document tracking
• No date extraction (regex-based)

Content Truncation

Content is truncated for the classification prompt (first ~8000 tokens passed to Claude). The summary field from classification is a 1-2 sentence summary (20-50 words).

---

8. Q&A Import

Name	Q&A Import (Bid Library)
Trigger	CLI: python3 scripts/import_bid_library.py <dir>
Source file	scripts/import_bid_library.py — build_content_record() (line 209) + main() (line 281)
Language	Python
Auth requirement	None (uses SUPABASE_SECRET_KEY directly)

Fields Written

Column	Value / Source
title	Question text truncated at word boundary to 120 chars
content	"Q: {question}\n\n{answer_standard}\n{answer_advanced}"
answer_standard	From DOCX extraction
answer_advanced	From DOCX extraction
content_type	'q_a_pair'
platform	'extraction'
source_url	'' (empty string)
source_domain	'' (empty string)
primary_domain	From keyword classifier
primary_subtopic	From keyword classifier
secondary_domain	From keyword classifier
secondary_subtopic	From keyword classifier
classification_confidence	From keyword classifier
summary	Answer text truncated at word boundary to 200 chars
ai_keywords	Deterministic keyword extraction (domain + subtopic + frequent words, 3-5 items)
source_file	From DOCX extraction (source_file field)
user_tags	From --batch-tag CLI flag
metadata	{section_name, table_index, row_index, has_standard, has_advanced, import_batch, has_tracked_changes}
embedding	OpenAI text-embedding-3-large (1024d) (unless --skip-embed)

Processing Pipeline

1. Find .docx files — Non-recursive directory scan
2. Detect Track Changes — has_tracked_changes() on each file
3. Extract Q&A pairs — extract_qa_from_docx() (tables, lists, headings)
4. Exact dedup — MD5 of normalised question text
5. Near-duplicate detection — Similarity scoring (flagged, not removed)
6. Classify — Keyword-based classifier (keyword_classifier.py) — no AI
7. Quality validation — Empty content and fragment detection
8. Idempotency check — Queries existing content_items by title match (unless --force)
9. Embed — build_embedding_text() + generate_embedding() via OpenAI (unless --skip-embed)
10. Store — insert_content_item(record) — single INSERT per pair

Notable Omissions

• No AI classification (uses keyword classifier only)
• No entity extraction (unless --entities flag is used)
• No relationship extraction (unless --entities flag is used)
• No temporal reference storage
• No AI summary generation (only truncated answer text)
• No content history versioning
• No quality score calculation
• No source document tracking
• No created_by / updated_by
• No topic suggestion
• No guide section suggestion
• No date extraction (regex-based)

Content Truncation

Title: truncated at word boundary to 120 characters. summary: truncated at word boundary to 200 characters.

---

9. MCP create_content_item

Name	MCP create_content_item Tool
Trigger	MCP tool call create_content_item via Claude Desktop / Claude.ai
Source file	lib/mcp/tools/content.ts — tool registration at line 113
Language	TypeScript
Auth requirement	MCP OAuth bearer token with admin or editor role

Fields Written

Column	Value / Source
title	Tool input title
suggested_title	Same as title
content	Tool input content
content_type	Tool input content_type (enum validated)
platform	'manual'
captured_date	new Date().toISOString()
created_by	MCP user ID
primary_domain	Tool input (optional)
primary_subtopic	Tool input (optional)
priority	Tool input (optional)
embedding	OpenAI (skip for drafts) — uses title + ' ' + content.slice(0, 5000)
governance_review_status	'draft' if specified
source_url	Tool input (optional, S205 WP-A1) — typed column, was metadata blob
source_file	Tool input (optional, S205 WP-A1) — typed column, was metadata blob
source_document_id	Tool input (optional, S205 WP-A1) — FK to source_documents.id
metadata	{batch_tag} (if provided); legacy source_document removed S205 WP-A1

Post-insert AI processing (skip for drafts):

Column	Value / Source
layer	Via inferLayer()
primary_domain	Via classifyContent()
primary_subtopic	Via classifyContent()
secondary_domain	Via classifyContent()
secondary_subtopic	Via classifyContent()
classification_confidence	Via classifyContent()
classified_at	Via classifyContent()
suggested_title	Via classifyContent()
summary	Via classifyContent() + generateSummary()
ai_keywords	Via classifyContent()
classification_reasoning	Via classifyContent()
summary_data	Via generateSummary()

Processing Pipeline

1. Auth + role check — checkMcpRole(extra.authInfo, ['admin', 'editor'])
2. Embed — generateEmbedding() (skip for drafts — saves API cost)
3. Build metadata — batch_tag and source_document if provided
4. INSERT content_item — Single insert
5. Layer inference — inferLayer() + UPDATE layer column (skip for drafts)
6. Classify — classifyContent() (skip for drafts)
7. Summarise — generateSummary() (skip for drafts)
8. Guide section suggestion — suggestGuideSections() (returned in response, not stored; skip for drafts)

Notable Omissions

• No content history versioning
• No dedup check
• No quality score calculation
• No topic suggestion
• No date extraction (regex-based)
• All AI processing skipped for draft items

Pipeline Run

recordPipelineRun({ pipelineName: 'mcp_create_content_item' }) is emitted once per invocation (S205 WP-A2 / spec §3.2). Status mapping:

• success → completed
• partial AI-step failure (any warnings) → completed_with_errors
• content_items insert failure → failed
• draft branch → completed with result.skipped_reason='draft'

Content Truncation

Embedding input truncated: content.slice(0, 5000). Content field max: 500,000 characters. Title max: 500 characters.

---

10. Bid Outcome KB Integration

Name	Bid Outcome KB Integration
Trigger	POST /api/bids/:id/outcome/integrate (JSON body)
Source file	app/api/procurement/[id]/outcome/integrate/route.ts — POST() (line 22)
Language	TypeScript
Auth requirement	Authenticated user with admin or editor role

Fields Written (new_entry action)

Column	Value / Source
title	Integration title or question text (first 200 chars)
suggested_title	Same as title
content	Bid response text (HTML)
content_type	Integration content_type or 'q_a_pair'
platform	'extraction'
source_url	null
embedding	OpenAI text-embedding-3-large (1024d)
primary_domain	From bid workspace domain_metadata.domain
summary	"Response to bid question: {question_text}" (static template, max 200 chars)
captured_date	new Date().toISOString()
created_by	Authenticated user ID
metadata	{source_bid_id, source_bid_name, source_question_id, source_question_text, integrated_at}

Fields Updated (update_existing action)

Column	Value / Source
content	Bid response text
summary	"Updated from winning bid response: {question_text}" (static template)
updated_by	Authenticated user ID
updated_at	new Date().toISOString()
embedding	Regenerated (best-effort)

Processing Pipeline

1. Auth + role check — getAuthorisedClient(['admin', 'editor'])
2. Rate limit — 10 requests/minute per user
3. Validate bid — Must exist and be in won state
4. Fetch questions + responses — From bid_questions and bid_responses tables
5. For each integration: a. Skip — If action is skip or no response text b. New entry — Generate embedding, INSERT content_item c. Update existing — UPDATE content + summary, regenerate embedding (best-effort)

Notable Omissions

• No AI classification (only sets primary_domain from bid metadata)
• No entity extraction
• No relationship extraction
• No temporal reference storage
• No AI summary generation (uses static template string)
• No content history versioning
• No layer inference
• No quality score calculation
• No source document tracking
• No dedup check
• No topic suggestion
• No guide section suggestion
• No date extraction (regex-based)
• No classified_at, classification_confidence, classification_reasoning, ai_keywords

Content Truncation

Title derived from question text: first 200 characters. summary template: question text truncated to 200 chars (new) or 150 chars (update).

Notable Omissions (S174)

• No markdown chunking (GAP). app/api/procurement/[id]/outcome/integrate/route.ts does not invoke regenerateChunks(). Content items created by this path have no content_chunks rows. Backfill via scripts/backfill-chunks.ts if these items need to be discoverable via search_content_chunks.

---

11. RSS Feed Promotion

Name	RSS / Atom Feed Promotion (Sector Intelligence pipeline)
Trigger	Cron: app/api/intelligence/run/route.ts → runPipeline() → processSource()
Source file	lib/intelligence/pipeline.ts — storeAsContentItem() (line 521)
Language	TypeScript
Auth requirement	Service-role client (runs under pipeline service user PIPELINE_SYSTEM_USER_ID)

Trigger semantics

feed_articles rows are inserted for every polled RSS/Atom item and scored by an LLM filter (lib/intelligence/relevance-scorer.ts). Only articles where the filter returns passed = true are promoted to content_items via storeAsContentItem(). The feed_articles.content_item_id column is updated to link the source feed article to the promoted content item. feed_articles.ai_summary (intentionally keeps the ai_ prefix) stores the filter-generated LLM summary of the RSS article — separate column from the content_items.summary written by generateSummary() after promotion.

Fields Written (initial INSERT at line 534)

Column	Value / Source
title	extraction.title or raw feed item.title
content	extraction.content (full extracted article text via reader / fallback)
content_type	'article' initially; may be re-inferred post-classify (see pipeline step 6 below)
source_url	item.url (resolved redirect URL)
metadata	{source: 'intelligence_pipeline', feed_source_id, feed_source_name, published_at, thumbnail_url}

Fields Written (post-classify via classifyContent() + inferLayer() downstream)

Column	Value / Source
primary_domain	Via classifyContent()
primary_subtopic	Via classifyContent()
secondary_domain	Via classifyContent()
secondary_subtopic	Via classifyContent()
classification_confidence	Via classifyContent()
classified_at	Via classifyContent()
suggested_title	Via classifyContent()
summary	Via classifyContent() + generateSummary() (if called)
ai_keywords	Via classifyContent()
classification_reasoning	Via classifyContent()
embedding	Via classifyContent() (OpenAI text-embedding-3-large 1024d)
content_type	Re-inferred via inferContentType() from classification result (if not 'article')

Fields Written (side-table INSERT at line 616)

Table	Columns set
content_item_workspaces	workspace_id = feed source’s workspace, content_item_id
feed_articles	content_item_id UPDATE (line 557)

Processing Pipeline

1. Cron poll + dedup — upstream in runPipeline() / processSource() (lib/intelligence/pipeline.ts); upserts feed_articles with normalised URL deduplication (code 23505 no-op).
2. LLM filter — lib/intelligence/relevance-scorer.ts scores each new article; only passed = true rows progress to storeAsContentItem().
3. INSERT content_item — single INSERT without classified_at / primary_domain etc. (line 534).
4. Link feed_article — UPDATE feed_articles.content_item_id (line 557, by workspace_id + normalised external_url).
5. Classify — classifyContent({ force: true, userId: PIPELINE_SYSTEM_USER_ID }) (line 564). Generates entities, relationships, temporal refs, embedding.
6. Content-type inference — re-reads classified fields; if still 'article', calls inferContentType() and UPDATEs content_type if the inferred type is more specific (line 585).
7. Workspace assignment — INSERT into content_item_workspaces (line 616).

Notable Omissions (S174)

• No suggestedTitle / UI-visible fallback title logic; relies on extraction.title -> feed item.title.
• No quality score, layer inference, topic suggestion, guide section suggestion, source document tracking, or content history versioning at this entry point.
• No markdown chunking (GAP). storeAsContentItem() does not invoke regenerateChunks(). Content items promoted from RSS feeds have no content_chunks rows; they are discoverable via item-level semantic search but not via search_content_chunks. Backfill via scripts/backfill-chunks.ts.
• No date extraction (regex-based); relies on AI temporal extraction only.
• No created_by / updated_by (service-role run; PIPELINE_SYSTEM_USER_ID is used as the userId argument to classifyContent but not written to the content_items row).

Content Truncation

No explicit truncation. extraction.content is full body text from the feed’s enclosed article. LLM filter (step 2) and classification (step 5) each apply their own 5,000-char input truncation downstream.

---

12. Markdown Batch Ingest UI

Name	Markdown Batch Ingest UI (EP2 §1.11; admin/editor drag-and-drop)
Trigger	UI: /item/new “Upload file” tab → multi-file .md drop → phase=analyse then phase=import POST
Source file	app/api/ingest/markdown/route.ts + lib/ingest/markdown-orchestrator.ts
Language	TypeScript
Auth requirement	getAuthorisedClient(['admin', 'editor']) (spec §5.3 D-1)

Trigger semantics

The route is a dedicated EP2 surface (Liam D-D9, S199): EP3 /api/upload/route.ts (maxDuration=60) is untouched; this route runs at maxDuration=300 to accommodate batch imports of up to 10 markdown files. Two-phase POST on the same endpoint via the multipart phase field:

• phase=analyse — read-only pre-flight. Returns { analysis: MarkdownIngestAnalysis[] } per file (front-matter parse, title extraction, diff-marker scan, dedupVerdict.{isDuplicate, existingId, existingTitle} from checkExactDuplicate, sourceFileMatch from filename collision query). NO DB writes.
• phase=import — full pipeline. Orchestrator opens a pipeline_runs row via startPipelineRun() (Pattern E Step 1), per-file loop with mid-flight updatePipelineProgress() UPDATEs at each file boundary (Pattern E Step 2), terminal UPDATE via finaliseRun() using createServiceClient() internally (chokepoint per S213 fix — admin-only RLS on pipeline_runs required service-role client; closed S214 by adding admin UPDATE/DELETE policies, but service-role pattern retained for cron / orchestrator parity).

The UI fires GET /api/pipeline-runs/:id every 1–2 s in parallel during the import POST so users see progress (per spec §7.2 Pattern E framing).

Fields Written (per content_items row)

Column	Value / Source
title	extractMarkdownTitle() priority chain: frontMatter.title > body H1 > bold-after-Article-N > filename
content	Cleaned body via cleanMdxTags() (PascalCase tag stripping; Python parity)
content_type	'article' initially; may be re-inferred post-classify
source_file	Original filename
ingest_source	'upload' (per S205-S207 schema widening; spec §3.4)
content_owner_id	Resolved via resolveContentOwnerId({ explicit, role, userId })
publication_status	Per D-A: draft_or_final='draft' | 'unknown' → 'draft'; 'final' → 'in_review'
governance_review_status	NULL on insert (publication-management event, not governance event; spec §9.1)
dedup_status	'clean' for unique; 'suspected_duplicate' for soft-block matches
metadata	Includes front_matter_fields, title_provenance, diff_markers, original_filename
primary_domain, primary_subtopic, secondary_domain, secondary_subtopic, classification_confidence, classified_at, suggested_title, summary, ai_keywords, classification_reasoning, embedding	Via classifyContent() post-insert

Fields NOT Written

• content_text_hash — GENERATED ALWAYS (Postgres auto-computes; CLAUDE.md gotcha).
• source_url — N/A for markdown file ingest.
• source_document_id — markdown files don’t go through source_documents (no PDF/DOCX storage path).
• quality_score, lifecycle_type, expiry_date — set by downstream cron (cron/quality-score, cron/freshness-transitions).

Processing Pipeline

1. Auth + multipart parse + per-file validation (app/api/ingest/markdown/route.ts:90-194) — file count ≤10, per-file size ≤1 MB (early-return 413 on first violation), total batch ≤5 MB, .md extension only (mixed batch → 400, all-non-.md → 415), UTF-8 only via TextDecoder('utf-8', { fatal: true }) (415 on decode failure BEFORE orchestrator runs).
2. Phase routing — phase='analyse' → orchestrator returns MarkdownIngestAnalysis[] with no DB writes; phase='import' → full pipeline.
3. pipeline_runs row UPSERT (producer-side, Pattern 2 caller-allocated UUID) with pipeline_name='upload_markdown_batch', status='running', progress.detail='Queued N file(s); awaiting worker claim…'. The route .upsert(..., { onConflict: 'id', ignoreDuplicates: true }) against the service-role client (app/api/ingest/markdown/route.ts:295-319) — compiles to PostgreSQL ON CONFLICT (id) DO NOTHING. Mirrors the worker-side Path B UPSERT in lib/pipeline/start-run.ts:142-145 for symmetry: when AC-3 (same-day re-enqueue with stable pipeline_run_id) or AC-4 (next-day re-enqueue) POSTs the same UUID twice, neither side clobbers the other — the worker’s terminal UPDATE handles status. Pattern E preservation per §5.4.4 D-10 + D-11 ratified.
4. Per-file loop (with mid-flight updatePipelineProgress after each file) — extract title + clean MDX tags + dedup pre-check + INSERT content_items (with dedup_status stamped from pre-check) → classifyContent() → regenerateChunks() → record outcome to stored[] and conditionally to dedup_flagged[] (subset relationship — see spec §5.4 dedup_flagged note).
5. pipeline_runs terminal UPDATE via finaliseRun() (service-role client) — sets status='completed' \| 'completed_with_errors' \| 'failed', result=results_summary, completed_at.

Per-batch admin overrides

Field	Description
skip_dedup	Per-file boolean in per_file_overrides[]. Admin-only (editor request silently ignored). When true on admin request, dedup soft-block bypassed; dedup_status='clean'.
auto_supersede	Admin-only batch-wide flag (forwards as no-op until orchestrator wires setSupersession post-EP2; reserved for §1.18 Python parity follow-on).
excluded	Per-file boolean to skip a file from import (any role). Listed in metadata.results_summary.skipped_excluded.
draft_or_final override	Per-file override for the filename heuristic; values 'draft' | 'final' | 'unknown'.

Notable Omissions

• No source_documents linkage. Markdown files don’t carry binary storage; no row inserted into source_documents. (Contrast with EP3 PDF/DOCX upload which always creates a source_documents row + FK.)
• No re-upload / version detection. detect_reupload RPC is not invoked because there’s no canonical filename versioning convention for markdown (auto-supersede heuristic via filename DRAFT/final pairs is the rough equivalent and is opt-in).
• No parent_id chain. Each markdown file is a leaf; supersession is the explicit relationship.

Content Truncation

No batch-level content truncation. MAX_EMBEDDING_CHARS = 24_000 applies inside classifyContent() for the embedding step (TS+Python parity per S168). Classification input is similarly truncated at ~8,000 tokens inside the LLM call.

Pipeline Run

pipeline_runs.pipeline_name='upload_markdown_batch'. Lifecycle uses the Pattern E primitives (startPipelineRun() + updatePipelineProgress() + finaliseRun()); contrast with EP3 single-file which uses recordPipelineRun() as a one-shot terminal write — no opId is passed, so the helper takes the plain-insert path (see Appendix G for the opId-carrying terminal-transition path the cocoindex webhook route uses to keep one pipeline_runs row per invocation, Inv-16).

---

Appendix A: “Complete” Content Item Definition

Every field a fully-processed content item should have. “Fully-processed” means the item has been through the most complete pipeline (File Upload API, entry point 3).

Field	Expected type	How populated	Required/Optional	Default
id	uuid	Auto-generated (PK)	Required	gen_random_uuid()
title	text	Set on creation	Required	—
content	text	Set on creation / extracted	Required	—
source_url	text	From source / null for uploads	Optional	null
source_domain	varchar	Parsed from URL	Optional	null
thumbnail_url	text	OG image extraction	Optional	null
content_type	varchar	Set on creation (CHECK constraint)	Required	—
platform	varchar	Set on creation (CHECK constraint)	Optional	null
parent_id	uuid	Manual linkage	Optional	null
author_name	varchar	Extracted or user-provided	Optional	null
metadata	jsonb	Set on creation, extended during processing	Optional	null
primary_domain	varchar	AI classification	Optional	null
primary_subtopic	varchar	AI classification	Optional	null
secondary_domain	varchar	AI classification	Optional	null
secondary_subtopic	varchar	AI classification	Optional	null
classification_confidence	numeric	AI classification (0.0-1.0)	Optional	null
classified_at	timestamptz	Set during classification	Optional	null
suggested_title	text	AI classification	Optional	null
classification_reasoning	text	AI classification	Optional	null
summary	text	AI classification or summary generation	Optional	null
ai_keywords	text[]	AI classification	Optional	null
embedding	vector(1024)	OpenAI text-embedding-3-large	Optional	null
summary_data	jsonb	AI summary generation (executive, detailed, takeaways)	Optional	null
user_tags	text[]	User-provided	Optional	null
priority	varchar	User-provided (high/medium/low)	Optional	null
file_path	text	Upload pipeline (Supabase Storage path)	Optional	null
captured_date	timestamptz	Extraction or creation time	Optional	null
created_at	timestamptz	Auto-generated	Required	now()
updated_at	timestamptz	Auto-updated on changes	Optional	null
created_by	uuid	Authenticated user ID	Optional	null
updated_by	uuid	Last modifier	Optional	null
brief	text	Manual (progressive depth layer)	Optional	null
detail	text	Manual (progressive depth layer)	Optional	null
reference	text	Manual (progressive depth layer)	Optional	null
source_document	text	Manual (originating document name)	Optional	null
source_bid	uuid	Bid workspace linkage	Optional	null
freshness	varchar	Computed by governance cron	Optional	'fresh'
previous_freshness	varchar	Set on freshness transition	Optional	null
freshness_checked_at	timestamptz	Governance cron	Optional	null
lifecycle_type	varchar	Date extraction or manual	Optional	'evergreen'
expiry_date	timestamptz	Regex date extraction or manual	Optional	null
verified_at	timestamptz	SME verification	Optional	null
verified_by	uuid	SME who verified	Optional	null
governance_review_status	text	Governance workflow	Optional	null
governance_review_due	timestamptz	Governance workflow	Optional	null
governance_reviewer_id	uuid	Governance workflow	Optional	null
answer_standard	text	Q&A pair standard answer	Optional	null
answer_advanced	text	Q&A pair advanced answer	Optional	null
archived_at	timestamptz	Soft-delete	Optional	null
archived_by	uuid	Archiver	Optional	null
archive_reason	text	Reason for archiving	Optional	null
content_owner_id	uuid	Ownership assignment	Optional	null
source_document_id	uuid	FK to source_documents	Optional	null
notes	text	Free-form notes	Optional	null
quality_score	int	calculateAndRoundQualityScore() (0-100)	Optional	null
quality_score_updated_at	timestamptz	Set during quality calculation	Optional	null
previous_quality_score	int	Stored before quality update	Optional	null
citation_count	int	Trigger-maintained	Required	0
source_file	text	Python ingestion (promoted from metadata)	Optional	null
layer	varchar	inferLayer() or manual	Optional	null
starred	boolean	User toggle via toggle_star() RPC	Required	false
content_text_hash	text	Dedup detection	Optional	null

---

Appendix B: Canonical Pipeline Steps

Added S136 per the pipeline-parity spec, Phase 2.

This section defines the canonical set of processing steps that constitute a “full pipeline” for content ingestion, and records which steps each entry point implements. It serves as the reference for future pipeline changes and the basis for the automated drift detection test (Phase 3).

Required Steps (19)

Every entry point that performs AI classification should implement all 19 required steps. Entry points that skip AI classification entirely (e.g. Q&A Import with keyword classifier, Bid Outcome KB Integration) are exempt from classification-dependent steps.

#	Step name	Description
1	content_extraction	Extract text from source (HTML, URL, .md, .docx)
2	content_truncation	Truncate to 5,000 chars for classification input
3	classification	AI classification via Claude
4	taxonomy_validation	Validate domain/subtopic against taxonomy
5	keyword_normalisation	Normalise AI keywords (proper nouns, singular form)
6	entity_extraction	Extract entities (AI, optionally keyword-based)
7	entity_canonicalisation	Normalise entity names (12-rule canonicalise)
8	entity_alias_resolution	Resolve entity aliases from DB
9	entity_exclusion_filtering	Filter out non-entity identifiers (SIC codes, VAT, etc.)
10	entity_context_extraction	Extract surrounding text snippet for each entity
11	entity_storage	Upsert into entity_mentions
12	relationship_extraction	Extract entity relationships (AI)
13	relationship_storage	Insert into entity_relationships
14	temporal_reference_extraction	Extract temporal references (AI)
15	temporal_reference_storage	Store in content_items.metadata
16	temporal_entity_bridge	Bridge temporal refs to entity mention metadata
17	embedding_generation	Generate vector embedding (OpenAI)
18	deduplication	Check for duplicates (URL and/or embedding similarity)
19	layer_inference	Suggest content layer (7-rule deterministic)

Optional Steps (6)

These steps are not required for parity. Some are intentionally limited to one pipeline.

#	Step name	Implemented by	Notes
20	summary_generation	Both	Renamed S164b from ai_summary_generation; writes content_items.summary + summary_data.
21	regex_date_extraction	TS only	Supplementary to AI extraction; low priority for Python port
22	temporal_reconciliation	TS only	Depends on regex date extraction
23	quality_score_calculation	TS only	Intentionally TS-only (depends on web app fields)
24	content_history_versioning	TS only	Intentionally TS-only (CLI creates, doesn’t update)
25	keyword_entity_extraction	Python only	Intentionally Python-only (compensates for different prompt strategy)
26	markdown_chunking	Both (with gaps)	Added S166–168 (Plan A/B/C). Split the canonical markdown body at H2 headings into content_chunks rows. TS: lib/content/chunk-store.ts regenerateChunks(). Python: scripts/kb_pipeline/chunk.py store_chunks(). Currently NOT invoked by Entry Points 2 (Python MD), 6 (Batch Items), 10 (Bid Outcome), 11 (RSS). Items ingested via those paths are not findable via search_content_chunks.
27	chunk_embedding_generation	Both (with gaps)	Per-chunk OpenAI text-embedding-3-large (1024d) with heading-path prefix. Coupled with step 26 — runs iff markdown_chunking runs.

Per-Entry-Point Compliance Matrix

Each cell shows whether the entry point implements the canonical step. Legend:

• Y = Yes, implemented
• N = Not implemented (gap)
• ~ = Partial or conditional (see notes)
• n/a = Not applicable (entry point does not perform AI classification)

#	Step	Python URL	Python MD	Q&A Import	URL Ingest API	File Upload API	Manual Create API	Batch Reclass	MCP Create
1	content_extraction	Y	Y	Y (DOCX)	Y (URL)	Y (PDF/DOCX/MD)	n/a (user provides)	n/a (existing)	n/a (user provides)
2	content_truncation	Y (5,000 + "...")	Y (5,000 + "...")	n/a	Y (5,000)	Y (5,000)	~ (if auto_classify)	Y (~8,000 tokens)	~ (if not draft)
3	classification	Y (Opus)	Y (Opus)	~ (keyword only)	Y (Sonnet)	Y (Sonnet)	~ (if auto_classify)	Y (Sonnet)	~ (if not draft)
4	taxonomy_validation	~ (warn only)	~ (warn only)	n/a	Y (auto-correct)	Y (auto-correct)	~ (if auto_classify)	Y (auto-correct)	~ (if not draft)
5	keyword_normalisation	Y	Y	n/a (deterministic keywords)	Y	Y	~ (if auto_classify)	Y	~ (if not draft)
6	entity_extraction	Y (AI + keyword)	Y (AI + keyword)	~ (--entities flag)	Y (AI)	Y (AI)	~ (if auto_classify)	Y (AI)	~ (if not draft)
7	entity_canonicalisation	Y	Y	~ (via --entities)	Y	Y	~ (if auto_classify)	Y	~ (if not draft)
8	entity_alias_resolution	Y	Y	~ (via --entities)	Y	Y	~ (if auto_classify)	Y	~ (if not draft)
9	entity_exclusion_filtering	Y	Y	~ (via --entities)	Y	Y	~ (if auto_classify)	Y	~ (if not draft)
10	entity_context_extraction	N	N	N	Y	Y	~ (if auto_classify)	Y	~ (if not draft)
11	entity_storage	Y	Y	~ (--entities flag)	Y	Y	~ (if auto_classify)	Y	~ (if not draft)
12	relationship_extraction	Y	Y	~ (--entities flag)	Y	Y	~ (if auto_classify)	Y	~ (if not draft)
13	relationship_storage	Y	Y	~ (--entities flag)	Y	Y	~ (if auto_classify)	Y	~ (if not draft)
14	temporal_reference_extraction	Y	Y	N	Y	Y	~ (if auto_classify)	Y	~ (if not draft)
15	temporal_reference_storage	Y	Y	N	Y	Y	~ (if auto_classify)	Y	~ (if not draft)
16	temporal_entity_bridge	Y	Y	N	Y	Y	~ (if auto_classify)	Y	~ (if not draft)
17	embedding_generation	Y	Y	~ (--skip-embed)	Y	Y	~ (if auto_embed)	Y	~ (skip drafts)
18	deduplication	Y (URL + embed)	Y (embed only)	Y (MD5 + idempotency)	Y (URL + embed)	Y (informational)	~ (if auto_embed)	N	N
19	layer_inference	Y	Y	Y	Y	Y	Y	Y	~ (skip drafts)
26	markdown_chunking	Y (S167 Plan C)	N (GAP)	N (intentional — atomic)	Y (S167 Plan C)	Y (S167 Plan C)	Y (S167 Plan C)	N (update-only; chunks stale after reclassify — use backfill-chunks.ts)	Y (S167 Plan C, skip drafts)
27	chunk_embedding_generation	Y	N (step 26 gap)	n/a	Y	Y	Y	N	Y (skip drafts)

S174 Additional Entry-Point Chunking Coverage

Per-entry-point chunking coverage for the entry points not in the matrix above:

Entry point	Chunking invoked?	Note
6. Batch Item Creation API	N (GAP)	app/api/items/batch/route.ts never calls regenerateChunks(). S167 wired single-item routes but batch was missed. Backfill script available.
10. Bid Outcome KB Integration	N (GAP)	app/api/procurement/[id]/outcome/integrate/route.ts never calls regenerateChunks(). Lower priority (won-bid responses are surfaced via bid workspace UI, not chunk search).
11. RSS Feed Promotion	N (GAP)	lib/intelligence/pipeline.ts storeAsContentItem() never calls regenerateChunks(). All SI-ingested articles rely on item-level search only.
12. Markdown Batch Ingest UI	Y (S211)	lib/ingest/markdown-orchestrator.ts:674 calls regenerateChunks(supabase, inserted.id, cleanedBody) after every successful insert. Closes the markdown_chunking gap for the UI markdown path (the Python EP2 path remains gapped — see §1.18 roadmap entry for the parity follow-on).

Known Gaps Summary

Gap	Affected entry points	Severity	Phase
entity_context_extraction missing in Python	Python URL, Python MD, Q&A Import	Medium	Not yet planned
taxonomy_validation warn-only in Python	Python URL, Python MD	Low	Not yet planned
content_truncation ellipsis divergence	Python URL, Python MD (append "...")	Very low	Cosmetic
deduplication missing in batch reclassify	Batch Reclass	None (intentional — operates on existing items)	n/a
deduplication missing in MCP create	MCP Create	Low	Not yet planned
Q&A Import skips most steps	Q&A Import	None (intentional — keyword classifier)	See Appendix C
markdown_chunking not invoked	Python MD (EP 2), Batch Items (EP 6), Bid Outcome (EP 10), RSS (EP 11)	Medium — items not findable via search_content_chunks	Not yet planned
markdown_chunking stale after reclassify	Batch Reclass (EP 7)	Medium — chunks and embedding diverge from new classification	Fix via backfill-chunks.ts post-reclassify

Parity Constants

The following constants must match between TS and Python pipelines. These are verified by the drift detection test (Phase 3).

Constant	TS location	Python location	Current value
Classification truncation limit	lib/ai/classify.ts (inline 5000)	kb_pipeline/classify.py build_user_prompt() (inline 5000)	5,000 chars
Entity types (12)	lib/ai/classify.ts tool schema	kb_pipeline/classify.py VALID_ENTITY_TYPES	organisation, certification, regulation, framework, capability, person, technology, project, sector, product, standard, methodology
Temporal entity types	lib/entities/entity-metadata-bridge.ts TEMPORAL_ENTITY_TYPES	kb_pipeline/temporal_bridge.py	certification, framework, regulation
Excluded entity patterns (5)	lib/ai/classify.ts EXCLUDED_PATTERNS	kb_pipeline/classify.py _EXCLUDED_PATTERNS	SIC Code, VAT, DUNS, numeric, VAT format
Layer inference rules (7)	lib/layer-inference.ts	kb_pipeline/layer_inference.py	7 rules, same priority order
Layer keys (4)	lib/client-config.ts	kb_pipeline/layer_inference.py	sales_brief, bid_detail, company_reference, research
Canonicalisation rules (12)	lib/entities/entity-dedup.ts	kb_pipeline/classify.py	12 rules, same order
Proper noun allowlist (19)	lib/validation/schemas.ts TAG_PROPER_NOUN_ALLOWLIST	kb_pipeline/classify.py PROPER_NOUN_ALLOWLIST	19 entries, same set
Abbreviations lookup (39)	lib/entities/entity-dedup.ts ABBREVIATIONS	kb_pipeline/classify.py _ABBREVIATIONS	39 entries, matching

---

Appendix C: Processing Pipeline Comparison

Rows represent processing steps. Columns represent entry points. Marks indicate whether the step is performed.

Processing step	1. Python URL	2. Python MD	3. File Upload	4. URL Ingest	5. Manual Create	6. Batch Create	7. Batch Reclass	8. Q&A Import	9. MCP Create	10. Bid Outcome
Text extraction	Yes	Yes (file read)	Yes (PDF/DOCX/MD)	Yes (URL fetch)	No (user provides)	No (user provides)	No (existing)	Yes (DOCX tables)	No (user provides)	No (existing response)
SSRF validation	No	No	No	Yes	No	No	No	No	No	No
URL dedup check	Yes	No	No	Yes (existing URL)	No	No	No	No	No	No
Embedding dedup check	Yes	Yes	Yes (informational)	Yes (informational)	Yes (informational)	No	No	No	No	No
Exact dedup (MD5)	No	No	No	No	No	No	No	Yes	No	No
File upload to storage	No	No	Yes	No	No	No	No	No	No	No
Source document tracking	No	No	Yes	No	No	Optional	No	No	No	No
Re-upload detection	No	No	Yes	No	No	No	No	No	No	No
Diff computation	No	No	Yes (re-uploads)	No	No	No	No	No	No	No
AI classification	Yes (Python)	Yes (Python)	Yes (TS shared)	Yes (TS shared)	Optional	Yes (TS shared)	Yes (Anthropic direct)	No (keyword only)	Yes (TS shared)	No
Entity extraction	Yes	Yes	Yes (via classify)	Yes (via classify)	Optional	Yes (via classify)	Yes	Optional (--entities)	Yes (via classify)	No
Relationship extraction	Yes	Yes	Yes (via classify)	Yes (via classify)	Optional	Yes (via classify)	Yes	Optional (--entities)	Yes (via classify)	No
Temporal references (AI)	Yes	Yes	Yes (via classify)	Yes (via classify)	Optional	Yes (via classify)	Yes	No	Yes (via classify)	No
Temporal-to-entity bridge	Yes	Yes	Yes (via classify)	Yes (via classify)	Optional	Yes (via classify)	Yes	No	Yes (via classify)	No
Date extraction (regex)	No	No	Yes	Yes	No	No	No	No	No	No
Embedding generation	Yes	Yes	Yes	Yes	Optional	Yes	Yes (regenerated)	Optional	Yes (skip drafts)	Yes
AI summary generation	Yes	Yes	Yes	Yes	Optional	Yes	No (classification only)	No	Yes (skip drafts)	No
Layer inference	Yes (S134)	Yes (S134)	Yes	Yes	Yes	Yes	Yes	Yes (S134)	Yes (skip drafts)	No
Quality score	No	No	Yes	Yes	Yes	Yes	No	No	No	No
Topic suggestion	No	No	Yes	Yes	Yes	Yes	No	No	No	No
Guide section suggestion	No	No	Yes (response only)	Yes (response only)	Yes (response only)	No	No	No	Yes (response only)	No
Content history v1	Yes (S153)	Yes (S153)	Yes	Yes	Yes	Yes	No	No	No	No
Quality logging	Yes	Yes	No	No	No	No	No	No	No	No
Pipeline run tracking	No	No	Yes	No	Yes (classify/summarise)	Yes	No	No	No	No
Batch token enforcement	No	No	No	No	No	Yes	No	No	No	No
Rate limiting	No	No	No	Yes (10/min)	Yes (20/min)	No	No	No	No	Yes (10/min)
created_by set	No	No	Yes	Yes	Yes	Yes	No (update only)	No	Yes	Yes
updated_by set	No	No	Yes	Yes (via classify)	Yes (via classify)	No	No	No	No	Yes (update action)
Markdown chunking	Yes (S167)	No (GAP)	Yes (S167)	Yes (S167)	Yes (S167)	No (GAP)	No (update-only)	n/a (atomic Q&A)	Yes (skip drafts)	No (GAP)
Chunk embedding generation	Yes	No	Yes	Yes	Yes	No	No	n/a	Yes (skip drafts)	No

Note (S174): Entry Point 11 (RSS Feed Promotion) is not represented as a column in this comparison table. Its profile mirrors Entry Point 4 (URL Ingest API) for AI classification, embedding, and entity extraction (all “Yes via classifyContent()”), but No for date extraction (regex), AI summary generation (generateSummary() not called from storeAsContentItem()), layer inference, quality score, topic suggestion, content history, content chunking, and created_by / updated_by. See §11 above for the authoritative entry-point detail and Appendix B “S174 Additional Entry-Point Chunking Coverage” for chunking gap status.

---

Appendix D: Canonical content_history.change_reason Values

Added in S152B WP3 (migration 20260407220000_add_content_history_change_reason.sql), wired across all write paths in S153. The column is free-text (no CHECK constraint) but every production write path uses one of the values below. See supabase/migrations/20260407220000_add_content_history_change_reason.sql COMMENT for background on the WHY/WHAT/HOW-categorised distinction between change_reason, change_summary, and change_type.

Value	Written by	Meaning
initial_ingest	trg_content_items_ensure_v1_history trigger (S207 WP-A4) — fires when content_items.ingest_source IS NOT NULL (Option D §4.4). Pre-WP-A4: written app-side from ingest/url, upload, items/batch, items POST, MCP create_content_item, Python URL + Markdown pipelines (S153).	First version created by any ingest path. S207 WP-A4 (28/04/2026): v1 history is now trigger-derived from content_items.ingest_source (typed column). Explicit app-level v1 writes were deleted in plan Task 3.4 — the trigger is the SOLE authority. Granular per-path observability lives on content_history.metadata->>'ingest_source' (one of the 11 values in Appendix F).
auto_v1_on_insert	trg_content_items_ensure_v1_history trigger fallback branch — fires when content_items.ingest_source IS NULL (legacy/back-compat).	Fallback for rows where ingest_source was never populated (pre-WP-A4 legacy rows post-backfill, or future writes that omit the typed column).
reclassify	/api/items/[id]/classify, classification_quality cron (future wiring)	Classification pipeline re-run
entity_enrichment	Entity-mention / relationship backfill scripts (future wiring)	Entity or relationship data updated
template_coverage_refresh	Template coverage job (future wiring)	New version produced by template coverage refresh
source_document_accepted	Source-document workflow (future wiring)	Accepted a diff from an uploaded source document
owner_change	app/api/items/[id]/owner PATCH	Content owner reassigned
rollback_to_v<N>	app/api/items/[id]/rollback POST	Version rollback from item detail UI
archive	lib/mcp/tools/governance.ts (soft-delete branch)	Item archived via MCP governance tool
hard_delete	lib/mcp/tools/governance.ts (hard-delete branch)	Item hard-deleted via MCP governance tool
status_change_publish	lib/mcp/tools/governance.ts (status-change branch)	Draft → live promotion via MCP governance tool
status_change_draft	lib/mcp/tools/governance.ts (status-change branch)	Live → draft demotion via MCP governance tool
bulk_approve	app/api/review/publication-bulk-action/route.ts (action=‘approve’)	§5.3 publication approval gate — bulk 'in_review' → 'published' transition. Distinct from per-row PATCH 'Transition from in_review to published' so audit queries can filter bulk-vs-singleton approvals. S220 W2.
bulk_return_to_draft	app/api/review/publication-bulk-action/route.ts (action=‘return_to_draft’)	§5.3 publication approval gate — bulk 'in_review' → 'draft' transition. Audit-trail counterpart to bulk_approve. S220 W2.
rollback_legacy	Migration backfill only (historical change_type='rollback' rows where we lack the target version)	Not written by any live path — backfill-only value
backfill_owner_assign_wp_a3	Migration 20260428145733_backfill_content_owner_id_from_created_by.sql (S206)	One-time backfill: content_owner_id auto-assigned from created_by for human-authored content_items (WP-A3)
NULL	app/api/items/[id] PATCH when admin UI “Why change?” field is empty	Acceptable when a caller cannot or does not supply a reason

S207 WP-A4 contract (28/04/2026): change_reason for v1 rows is trigger-derived from content_items.ingest_source (not an app-level literal); explicit app-level v1 writes are no longer accepted (the inverted guard at __tests__/validation/content-items-v1-history-guard.test.ts enforces absence across lib/, app/, scripts/). Per spec §3.4 AC4.9 + plan Task 4.2.

Adding a new value: update this table, the canonical list in the migration COMMENT, the TS call site comment, and __tests__/api/items.test.ts regression tests. No DB constraint change is required — the column is intentionally free-text.

---

Appendix E: Operational data writes (one-shot SQL, not driven by an entry point)

This appendix records mass DML writes that happen outside the canonical entry-point pipeline — typically migrations or backfill scripts that touch production data once. Each entry must record date, target table+column, row count, and rationale so the write is navigable from the audit trail.

Date	Operation	Table.column	Rows	Rationale
27/04/2026	S200 WP5 §5.5 Phase 1 review-cadence backfill	content_items.review_cadence_days + content_items.next_review_date	440	Three cohorts per spec §18.1 OQ-1.2: regulatory/compliance (24 @ 365d), Q&A (395 @ 180d), methodology/case-study (21 @ 365d). Articles/blogs/research not backfilled. WHERE review_cadence_days IS NULL predicate preserved any owner-set values (pre-snapshot was 0 owner-set rows; post-snapshot match expected).
28/04/2026	S205 WP-A1 Phase 1 metadata→typed source-column backfill	content_items.source_url + content_items.source_file (from metadata->>'source_document')	23 (prod) / 0 (staging)	Migration 20260428131822_backfill_metadata_source_document_to_typed_columns.sql. Disambiguation rule: values matching ^https?:// (case-insensitive) copy to source_url; non-URL (file paths, opaque tokens) copy to source_file. Idempotent (WHERE … IS NULL clauses). Pre-flight on prod rovrymhhffssilaftdwd: 23 rows had metadata.source_document set, all 23 were filenames → source_file, 0 URLs. Legacy JSONB key preserved per AC1.4 for rollback. Pairs with EP9 typed-column writes.
28/04/2026	S207 WP-A4 Phase 3 ingest_source backfill	content_items.ingest_source	23 (prod) / 0 (staging)	Migration 20260428180945_backfill_ingest_source.sql. Maps pre-WP-A4 rows from metadata->>'ingestion_source' (legacy JSONB key) onto the typed column using the canonical 11-value vocabulary in Appendix F. Idempotent (WHERE ingest_source IS NULL). Legacy JSONB key preserved for rollback. Required so the rewritten ensure_v1_history_at_commit() trigger emits change_reason='initial_ingest' (vs the legacy 'auto_v1_on_insert' fallback) on subsequent re-versions of pre-existing rows.

---

Appendix F: Canonical content_items.ingest_source Values

Added in S207 WP-A4 (migration 20260428174512_add_ingest_source_to_content_items.sql). The column is typed text with no DB CHECK constraint applied in S207 (CHECK deferred to a future phase per spec §10.4 — NOT VALID + VALIDATE CONSTRAINT to avoid table-rewriting locks on prod). Every production INSERT-time write site maps to one of the 10 canonical values below; the 11th value ('batch_reclassify') is reserved for an EP7 UPDATE path and is intentionally NOT written by any INSERT-time wire today.

The DB trigger trg_content_items_ensure_v1_history reads NEW.ingest_source and emits content_history.change_reason='initial_ingest' when non-NULL, falling back to 'auto_v1_on_insert' when NULL (legacy/back-compat). Granular per-path observability lives on content_history.metadata->>'ingest_source'. See docs/specs/ingest-path-consistency-spec.md §3.4 + §4.4.

Value	INSERT-time write site	Maps to entry point
manual	app/api/items/route.ts:163 (web-form POST default; ingestion_source body field overrides)	EP3
url_import	app/api/ingest/url/route.ts:179	EP4
upload	app/api/upload/route.ts:344	EP5
upload_autosplit	app/api/items/batch/route.ts:256	EP6 (autosplit branch)
mcp_create	lib/mcp/tools/content.ts:491 (MCP create_content_item tool)	EP9
rss_feed	lib/intelligence/pipeline.ts:653 (RSS feed promotion)	EP11
bid_outcome_integration	app/api/procurement/[id]/outcome/integrate/route.ts:220	EP10
python_url	scripts/kb_pipeline/pipeline.py (Python URL ingest pipeline)	EP1
python_markdown	scripts/ingest_markdown.py + scripts/ingest_stage2_markdown.py (Python Markdown ingest pipelines)	EP2
qa_import	scripts/import_bid_library.py (Q&A pair imports from .docx)	EP8
batch_reclassify	Reserved — for an EP7 UPDATE path. NOT written by any INSERT-time wire today (out of scope per S207 plan §3.3).	EP7 (deferred, UPDATE-only)

Adding a new value: widen any TS literal-union type referencing the canonical set (feedback_db_check_ts_union_paired_widening), update this table, update the COMMENT on the column in the migration that introduced the value, and add a wire-site test under __tests__/api/ or __tests__/mcp/. If a CHECK constraint is later applied, the new value MUST be added to the constraint expression and the migration paired with bunx tsc --noEmit verification.

---

Appendix G: Canonical pipeline_runs.pipeline_name Values

Single source of truth for the pipeline_name column on pipeline_runs. All callers of recordPipelineRun() (lib/pipeline/record-run.ts) MUST use one of the values below. The column is free-text (no DB CHECK) but every production write path uses one of these labels — keeping the set small is what makes /admin/pipeline-runs and Sentry alert routing tractable.

The Python ingest pipelines do not flow through recordPipelineRun() — they write to pipeline_runs directly via scripts/kb_pipeline/pipeline_log.py (start_run()/finish_run()). Their pipeline_name value ('ingest') is listed below for completeness.

Value	Written by	Purpose
mcp_create_content_item	lib/mcp/tools/content.ts (S205 WP-A2)	MCP create_content_item tool invocation audit (success, validation-fail, and error branches all log)
publish_classify	app/api/items/[id]/route.ts (PATCH publish branch) + lib/mcp/tools/governance.ts (status-change path)	Draft → live promotion classification audit
background_classify	app/api/items/route.ts (POST after item creation)	Background classification job kicked off post item insert
background_summarise	app/api/items/route.ts (POST after item creation)	Background summarisation job kicked off post item insert
quality_score	app/api/cron/quality-score/route.ts	Daily KB quality-score recompute cron
freshness_transitions	app/api/cron/freshness-transitions/route.ts	Daily content freshness-state transition cron
content_gaps	app/api/cron/content-gaps/route.ts	Daily content-gap detection cron
classification_quality	app/api/cron/classification-quality/route.ts	Daily classification-quality eval cron
coverage_alert	app/api/cron/coverage-alerts/route.ts	Daily template-coverage alert cron
review_cadence	app/api/cron/review-cadence/route.ts (S200 WP5 §5.5 Phase 2)	Daily review-cadence scheduling cron (writes one run per phase: schedule, transitions, alerts)
taxonomy_sync	app/api/admin/taxonomy-sync/route.ts (constant PIPELINE_NAME)	Admin-triggered taxonomy sync dispatch + GitHub Actions status
provenance_audit_pdf	app/api/admin/provenance/export/verification-history/route.ts	Provenance audit PDF export run
ingest	scripts/kb_pipeline/pipeline_log.py (start_run('ingest'))	Python URL + Markdown ingest pipelines (does not flow through recordPipelineRun())

Adding a new value: add a new row above, search for the literal in lib/, app/, and scripts/ to confirm the wire-site, and prefer reusing an existing value when the new caller is a peer of an existing one (e.g. another cron of the same kind). Per feedback_record_pipeline_run_signature, the helper’s terminal status set is 'completed' | 'completed_with_errors' | 'failed' | 'cancelled'; it also accepts 'in_progress' as a lifecycle status for the cocoindex flow-start emission (ID-28.11 FX-1) — a terminal status carrying an opId transitions the invocation’s existing in_progress row to the terminal state (Inv-16: one row per invocation; Inv-12: op_id resolves to exactly one row) rather than inserting a sibling. Non-cocoindex callers (crons, MCP tools, queue telemetry) omit opId and keep plain-insert semantics.

Sync drift — 02/07/2026

Live-code-verified drift sweep against kh_code_sources. The doc’s own DRIFT WARNING banner at the top already documents the Phase 0 investigation findings; this section adds confirmations against the current main of the public repo. Counts by category:

• [path] 8 cited source files no longer exist on main: scripts/kb_pipeline/ pipeline.py, scripts/kb_pipeline/classify.py, scripts/kb_pipeline/chunk.py (whole scripts/kb_pipeline/ dir retired — ID-46), scripts/ingest.py, scripts/ingest_markdown.py, scripts/import_bid_library.py, scripts/backfill-chunks.ts, lib/ingest/markdown-orchestrator.ts, and scripts/batch-reclassify.ts (moved to lib/queue/handlers/batch-reclassify.ts). Affects §1, §2, §7, §8, §12 and Quick Comparison Matrix rows 1, 2, 7, 8.
• [route] app/api/intelligence/run/route.ts (§11 trigger) does not exist; the live cron route is app/api/cron/intelligence-poll/route.ts (banner already flags this).
• [route] app/api/ingest/markdown/route.ts (§12) does not exist; live ingest routes are app/api/ingest/url/route.ts and app/api/ingest/folder-drop/route.ts.
• [lib] lib/intelligence/pipeline.ts exports processFeedSource and runPipeline (no processSource or storeAsContentItem symbols); §11 cites storeAsContentItem() at line 521 and processSource() in the trigger semantics — both stale (banner already flags processSource).
• [route] §10 trigger line still reads POST /api/bids/:id/outcome/integrate; live route is app/api/procurement/[id]/outcome/integrate/route.ts (the Source file column was updated in S248 but the trigger line was not).