Content Management — Technical Reference

Content Management — Technical Reference

Last verified: Session 215 W4 T9 (30 April 2026). S215 update: EP12 markdown batch UI ingest entry point added (app/api/ingest/markdown/route.ts

• lib/ingest/markdown-orchestrator.ts; reuses ingest_source='upload' enum value alongside EP3; pipeline_name='upload_markdown_batch' Pattern E lifecycle). S210 update: ingest_source typed column shipped (S205 WP-A1 backfill + S207 WP-A4 forward-write at all 8 INSERT entry points + S209 OPS-41 residual cleanup → 100% non-null on prod); source_document_id FK promotes legacy metadata.source_document to typed column (S205); content_text_hash is GENERATED ALWAYS and must be omitted from payloads; Source Information accordion now content-type-aware via SourceMetadata branching (S197 §1.19); Q&A inline editor uses dynamically- imported Tiptap ContentEditor for answer_standard/answer_advanced (S198 §1.5); MCP create_content_item emits pipeline_runs audit row (S208 OPS-40); the find tool’s chunk branch (backed by the search_content_chunks RPC) widened with cadence filters (S208 §5.5 Phase 4); content owner auto-assigned at all 6 ingest entry points (S206); tag morphology canonicalisation library-adoption (S197 §1.17). Prior baseline: S188 (22 April 2026).

Overview

Content management is the foundational feature area of Canonical. It handles the full lifecycle of content items — creation, ingestion (8 entry points), classification, embedding, versioning, archival, source document management, deduplication, supersession, recurring-review cadence, and heading-based chunking. The content_items table is the central entity, with supporting tables for history, workspaces, templates, source documents, read marks, pipeline runs, tag morphology drift triage, and content_chunks (heading- bounded sub-document records with per-chunk embeddings). Column count is volatile — see docs/reference/SCHEMA-QUICK-REFERENCE.md for the live list.

Ingest Path Consistency (S205 + S207 + S209)

The platform supports 8 INSERT-time entry points across TS and Python. Each populates the typed content_items.ingest_source column with one of 11 canonical values. The deferred constraint trigger trg_content_items_ensure_v1_history is the SOLE authority for v1 content_history rows — app-level v1 inserts have been removed.

Canonical ingest_source values

Value	Wired By	Entry Point
manual	TS — app/api/items/route.ts	EP5 web-form create
url_import	TS — app/api/ingest/url/route.ts	EP4 URL ingest (Readability)
upload	TS — app/api/upload/route.ts; also app/api/ingest/markdown/route.ts (EP12 batch markdown UI)	EP3 file upload (PDF/DOCX/MD/TXT) + EP12 markdown batch UI
upload_autosplit	TS — app/api/items/batch/route.ts	EP6 batch / auto-split upload
mcp_create	TS — lib/mcp/tools/content.ts	EP9 MCP create_content_item
rss_feed	TS — lib/intelligence/pipeline.ts::storeAsContentItem	EP11 RSS feed promotion
bid_outcome_integration	TS — app/api/procurement/[id]/outcome/integrate/route.ts	EP10 form-outcome integration (pipeline-name literal unchanged)
python_url	Python — scripts/kb_pipeline/pipeline.py	EP1 Python URL ingest (trafilatura)
python_markdown	Python — scripts/ingest_markdown.py, ingest_stage2_markdown.py	EP2/EP2b markdown directory ingest
qa_import	Form-library import path (key literal unchanged)	EP8 Q&A form-library import
batch_reclassify	Reserved (UPDATE path)	Out-of-scope per spec; not wired to INSERT

v1 History Trigger (S207 WP-A4)

Migration 20260428174512_add_ingest_source_to_content_items.sql updated ensure_v1_history_at_commit() to read NEW.ingest_source and emit content_history.change_reason='initial_ingest' when set, falling back to 'auto_v1_on_insert' for legacy NULL rows. The trigger writes metadata.ingest_source on the v1 row for granular per-path observability.

App-level v1 inserts have been removed from all TS routes (/api/items, /api/upload, /api/ingest/url, /api/items/batch) and from MCP create_content_item. Inverted guard at __tests__/validation/content-items-v1-history-guard.test.ts enforces the trigger-sole-authority contract.

Backfill (S207 + S209)

• 20260428180945_backfill_ingest_source.sql — initial typed-column backfill from metadata.ingestion_source keys; pipeline-derived rows.
• 20260428235042_refine_ingest_source_backfill_residuals.sql (S209 OPS-41) — resolved 71 residual NULLs (UI Q&A entry / batch upload / E2E fixtures / admin notes) to 'manual'. Prod project rovrymhhffssilaftdwd post-apply: 0 NULLs.

pipeline_runs.pipeline_name Canonical List

Audit trail for ingest invocations, recorded via recordPipelineRun():

Pipeline Name	Emitted By
mcp_create_content_item	MCP create_content_item (S206 WP4 + S208 OPS-40)
file_upload	/api/upload
url_ingest	/api/ingest/url
batch_create	/api/items/batch
upload_markdown_batch	/api/ingest/markdown (EP12 EP2 markdown batch UI; Pattern E lifecycle — startPipelineRun + per-file updatePipelineProgress + terminal finaliseRun)
background_classify	Awaited classification step inside /api/items and /api/items/[id]/classify
background_summarise	Awaited summary step
review_cadence	Daily 03:45 UTC cron
python_pipeline	Python kb_pipeline/pipeline.py

OPS-40 (S208) closed the only remaining gap by routing MCP create_content_item through recordPipelineRun() across success / partial / draft / auth-fail / catch-all paths via a service-role client (editor RLS would otherwise drop the audit row).

Source Document Linkage (S205 WP-A1)

content_items.source_document_id (UUID FK → source_documents.id) is the canonical column for joining content items to source files. It promoted the legacy metadata.source_document JSONB blob:

• Migration 20260428131822_backfill_metadata_source_document_to_typed_columns.sql copy-forwarded the legacy values to typed columns: URL-shaped (regex ^https?://) → source_url; otherwise → source_file. 23 prod rows migrated. Idempotent (WHERE source_url IS NULL etc.).
• The legacy metadata.source_document JSONB key is preserved on disk per spec (S205 WP-A1 AC1.4).
• MCP create_content_item rejects source_document Zod input — callers must use one of source_url, source_file, or source_document_id.

Generated Columns (S202)

content_items.content_text_hash is GENERATED ALWAYS — Postgres auto- computes via md5(normalised content). Explicit insert/update values are rejected with cannot insert a non-DEFAULT value into column. Omit the field from any payload writing content_items.

Markdown Canonical Format

Markdown is the canonical storage format for content_items.content. The extraction layer normalises every ingest path to markdown before dedup, embedding, classification, or chunking runs:

Source	Path	Library
Web pages	lib/extraction/url.ts	Readability + Turndown (with gfm plugin)
PDFs (TS)	lib/extraction/pdf.ts	unpdf
PDFs (Python)	scripts/kb_pipeline/extract.py	pdfplumber
DOCX	app/api/upload/route.ts::extractDocxText	mammoth → Turndown (gfm)
Markdown files	scripts/ingest_markdown.py	Direct read (no transformation)
HTML normalisation	lib/content/html-to-markdown.ts	Turndown
Stripping	lib/content/strip-markdown.ts (stripMarkdown())	marked + tag stripper

Mammoth convertToMarkdown() drops tables — DOCX uploads use a two-step convertToHtml() → Turndown (with turndown-plugin-gfm) so tables survive.

Heading-Based Chunking

Markdown content is split at H2 boundaries (H1 fallback) by chunkByHeadings() in lib/content/chunking.ts (uses marked.lexer()). Sub-MIN_CHUNK_CHARS chunks merge with siblings; sub-MIN_DOCUMENT_CHARS documents stay as a single chunk. Each chunk gets its own vector(1024) embedding via lib/content/chunk-store.ts (buildChunkEmbeddingText prefixes the heading_path breadcrumb).

Chunking fires automatically on insert/edit at five entry points:

• POST /api/upload
• POST /api/ingest/url
• POST /api/items (web-form create, non-draft)
• PATCH /api/items/[id] (content edit + Q&A rebuild)
• MCP create_content_item (non-draft items)

The Python pipeline has a parallel implementation in scripts/kb_pipeline/chunk.py using regex-based splitting respecting code blocks. Backfill CLI: scripts/backfill-chunks.ts (--limit, --dry-run, --item-id) and scripts/backfill_chunks_stage2.py.

MCP retrieval: the find tool’s chunk-granularity branch (backed by the search_content_chunks RPC) returns heading-bounded chunks ranked by cosine similarity; the get tool appends a Document Sections list summarising the item’s chunks.

MCP filter widening (S208 §5.5 Phase 4)

The search_content_chunks RPC (called by the find tool’s chunk branch) accepts two cadence-aware filters at the RPC level:

• overdue_review: boolean — when true, restrict to chunks from items where governance_review_status = 'review_overdue'; when false, exclude overdue items.
• review_due_within_days: integer (1-365) — restrict to chunks from items whose next_review_date falls within N days from now.

Migration 20260428212936_extend_search_content_chunks_review_filters.sql recreated the RPC with filter_overdue_review + filter_review_due_within_days params (both DEFAULT NULL), preserving LANGUAGE plpgsql STABLE SECURITY DEFINER + SET search_path = public, extensions and existing GRANTs. Existing 4-arg callers are unchanged.

Pipeline Extraction Parity (S168 Plan D)

lib/extraction/extraction-result.ts (TS factory) and scripts/kb_pipeline/extraction_result.py (Python dataclass) produce parity PipelineExtractionResult output for every extraction path. The factory derives content_plain via stripMarkdown, extracts headings via ^#{1,6}\s+ regex (multiline), detects tables and code fences, and emits quality warnings: very short content (< 50 words), no headings detected (no headings AND > 200 words), no tables detected in PDF, high markdown- to-plain ratio (> 1.25), empty title.

Cross-language parity tests at __tests__/lib/extraction-result-parity.test.ts and scripts/tests/test_extraction_result_parity.py use 8 shared fixtures with set-equality assertions on warnings and exact-match on heading counts.

Embedding Truncation

Both languages truncate embedding input at MAX_EMBEDDING_CHARS = 24_000 chars (TS: lib/ai/embed.ts; Python: scripts/kb_pipeline/embed.py). Parity guard at scripts/tests/test_embed.py::test_max_embedding_chars_matches_ts_constant.

Tag Morphology Canonicalisation (S197 §1.17)

content_items.ai_keywords morphology is canonicalised at the classify write boundary. The in-house _to_singular rules were replaced with pluralize@8 (TS) and inflect==7.5.0 (Python). Domain carve-outs run as a pre-pass override before the libraries are invoked:

• 11 -ics fields-of-study (mathematics, physics, economics, politics, etc.) — preserved as-is.
• Unchanged plurals: news, means, series, species.
• Allowlisted proper nouns: DUNS, URN, etc.
• Compound last-token guard prevents Latin/Greek regressions (inspection data does NOT become inspection datum).

Shared cross-language fixture __tests__/fixtures/keyword-normalisation-cases.json (74 cases) enforces TS↔Py parity. Web-form-submitted keywords are normalised at the write boundary in app/api/items/route.ts to match classify-time canonicalisation:

const normalisedAiKeywords = ai_keywords?.length
  ? [...new Set(ai_keywords.map(normaliseTag).filter((k) => k.length > 0))]
  : undefined;

Drift triage table

Migration 20260424222432_tag_morphology_drift_flags.sql provisions tag_morphology_drift_flags (admin/editor RLS) — pending decisions surfaced by scripts/eval-tag-morphology-adoption.ts against production tags. Admin triage UI lives at /settings?section=tag-morphology with Accept / Add-override / Dismiss actions and a rationale dialog.

Content Owner Auto-Assignment (S206 WP-A Phase 2)

content_items.content_owner_id is auto-assigned at all 6 ingest entry points via lib/auth/owner-default.ts::resolveContentOwnerId({ explicit, role, userId }):

• Admin caller + explicit override → use the explicit UUID
• Non-admin caller + explicit override → silently force to caller’s userId
• No explicit override → use caller’s userId

5-of-6 ingest payload Zod schemas widened with optional content_owner_id field; KBIntegrationBodySchema (EP10) is hard-coded route-side. Service- account UUID list canonical: ['a0000000-0000-4000-8000-000000000001'].

Migration 20260428145733_backfill_content_owner_id_from_created_by.sql backfilled legacy rows (content_owner_id := created_by). The /api/upload route Zod-validates the form-data UUID via a dedicated narrow schema before passing it to resolveContentOwnerId (S206 M-2 fix).

Source Information Accordion (S197 §1.19)

The /item/[id] Source Details accordion was renamed “Source Information” and rebuilt to render content-type-specific fields. SourceMetadata (components/reader/source-metadata.tsx, ~465 lines) dispatches in this priority order:

1. platform === 'email' → EmailFields
2. contentType === 'pdf' → PdfFields
3. contentType === 'q_a_pair' → QAPairFields (renders source_file raw + section_name + “Imported on DD/MM/YYYY” parsed from import_batch)
4. feedArticle != null (or legacy JSONB fallbacks) → FeedArticleFields (renders feed name + published date via feed_articles → feed_sources join)
5. detectMarkdownIngest(meta) → MarkdownFields (renders ingestion-source label via INGESTION_SOURCE_LABELS map covering 10 production values)
6. Default → GenericArticleFields

The accordion is collapsed by default. Pure helpers in components/reader/source-metadata-helpers.ts:

• INGESTION_SOURCE_LABELS — maps markdown_file/markdown_pipeline/ markdown_import/stage2_markdown to “Markdown upload”; url_import to “URL import”; upload to “File upload”; upload_autosplit to “Auto-split upload”; manual to “Manual entry”; bid_library/bid_library_import (key literals unchanged) to “Procurement library import”.
• getIngestionSourceLabel(raw, hasFeedArticle) — returns “RSS feed” when raw is null/undefined and feedArticle exists; otherwise looks up the map with raw-string fallback.
• parseImportBatchDate(importBatch) — parses trailing -YYYYMMDD-HHMMSS tail into a UTC Date with calendar-validity check.
• formatConfidencePercent(0..1) — integer percentage, no decimals.
• truncateUrl(url, maxLen) — text truncation only; full URL stays in the href.
• detectMarkdownIngest(metadata) — fires on metadata.ingestion_source === 'markdown_file' OR metadata.original_format === 'markdown'.

AI-visibility carve-out

The accordion exposes role-gated classification_confidence rendering via useUserRole() — only admin + editor roles see the plain-text percentage row. Viewers and anonymous users see nothing. The accordion has no AI branding, no colour coding, and no accompanying AI-mechanism fields (model names, reasoning, tokens, cost). Decision: Liam, 24/04/2026; full policy at docs/reference/ai-visibility-policy.md §Editor+admin Source Information surface.

Q&A ContentEditor (S198 §1.5)

The 387 Q&A items on /item/[id] edit answer_standard / answer_advanced via the same dynamically-imported Tiptap ContentEditor already shipped for the canonical content field — replacing the previous plain <textarea> in the QAInlineEditor subcomponent of components/qa/qa-answer-display.tsx.

Edit pattern is single-field-at-a-time per the post-S178 F-1 useInlineFieldEdit shape. Per-field save-safety guard composes inside the active editor’s handleSave against the per-field baseline; ≥20% shrink blocks the save. Per-field regen-embedding checkbox in the editor footer threads regenerate_embedding: true into the PATCH body when ticked; the flag resets on startEdit / cancelEdit / successful save so it cannot leak across the shared useInlineFieldEdit instance.

PATCH preserves the Q: {question}\n\n prefix in the rebuilt content_items.content. Read mode renders both fields via QAPairRenderer → ContentRenderer. Stable data-testid="qa-answer-panel-{standard,advanced}" on the panel divs.

Content-Hash Deduplication (S183 + S184)

The dedup system prevents duplicate content from entering the knowledge base across all 11 ingest entry points (4 Python + 7 TypeScript). Two-phase approach: content-hash exact matching (MD5 of normalised text via content_items.content_text_hash) and cosine similarity near-match (threshold 0.92). The system soft-blocks — duplicates are stamped with dedup_status='suspected_duplicate' and a pointer to the matching item in metadata.suspected_duplicate_of, but the insert still succeeds.

Coverage

Layer	Helper	Entry Points
TypeScript	resolveDedupStamp(existingId, { skipDedup })	EP3 upload / EP4 URL / EP5 manual / EP6 batch /
	(lib/dedup/content-dedup.ts)	EP9 MCP / EP10 form outcome / EP11 RSS
Python	check_content_hash_duplicate() +	EP1 URL / EP2 markdown / EP2b stage 2 / EP8 Q&A
	normalise_title_for_dedup()
	(scripts/kb_pipeline/dedup.py)

All Python regex uses re.ASCII flag to match JS + PG regex semantics.

Admin override

skip_dedup?: boolean optional field on 5 Zod schemas (ItemCreateBodySchema, IngestUrlBodySchema, BatchCreateBodySchema, KBIntegrationBodySchema, MCP create_content_item inputSchema). Non-admin callers are silently ignored (no 403) per spec §6 D2.

RSS many-to-many (EP11 D3)

When the same source_url promotes into a second workspace, no duplicate content_items row is created. Instead, a new row is inserted into the content_item_workspaces junction table only. A different URL with the same content hash stamps suspected_duplicate.

Supersession Model (S186)

Migration 20260421222059_add_superseded_by_to_content_items.sql:

• content_items.superseded_by — UUID FK to content_items(id), ON DELETE SET NULL.
• CHECK content_items_superseded_by_not_self.
• Partial index on WHERE superseded_by IS NOT NULL.
• Widens dedup_status CHECK to include 'superseded'.

Three setting mechanisms (admin UI / MCP supersede_content_item / Python --auto-supersede) converge on shared helpers lib/supersession/set.ts::setSupersession() (TS) and scripts/kb_pipeline/supersede.py::set_supersession() (Python). All MCP retrieval tools default include_superseded=false. Direct ID lookup is unchanged — superseded rows are always returned.

Recurring Review Cadence (§5.5 Phases 1-5)

content_items.next_review_date date NULL + content_items.review_cadence_days integer NULL (CHECK 1-1095). Partial index idx_content_items_next_review_date excludes superseded + archived rows.

• Daily cron at app/api/cron/review-cadence/route.ts (03:45 UTC) flags next_review_date < CURRENT_DATE items as 'review_overdue' (only when current status is NULL or 'approved').
• Auto-renewal in the 'approve' branch via shared lib/governance/cadence-renewal.ts::computeNextReviewDate(...).
• §5.5 Phase 4: MCP filter widening (see Heading-Based Chunking section above and whats_in_my_queue (facet: governance) in docs/product-functionality/quality-governance/).
• §5.5 Phase 5: cadence-compliance scorer at lib/quality/quality-score.ts::cadenceCompliancePenalty(nextReviewDate, now) returns 0/5-10/15/25/40 per spec §9.3 schedule. Only applies when next_review_date is non-null (preservation rule §9.4).

Publication Lifecycle (§5.2 Phases 1+2+2.5+1f)

content_items.publication_status (NOT NULL DEFAULT 'published', CHECK enum {'draft', 'in_review', 'published', 'archived'}) is the four-state lifecycle column.

• Three partial indexes (idx_content_items_publication_status_published, idx_content_items_published_recent, idx_content_items_archived).
• Bidirectional enforce_archive_state_consistency PL/pgSQL trigger enforces archived_at <-> publication_status='archived' invariant.
• PATCH route at app/api/items/[id]/route.ts accepts field='publication_status' with optimistic-concurrency guard (.eq('publication_status', fromStatus) → 409 PGRST116 on stale write).
• MCP update_publication_status tool mirrors the route semantics.
• Both consume shared helper lib/governance/publication-transitions.ts (VALID_PUBLICATION_STATUSES, computeAllowedTransitions(state, role), applyTransitionSideEffects(...)).
• AC6.5 grep guard test enforces zero write-position regressions on the 6 production writers.

Phase 3 (RPC visibility flip) + Phase 4 (UI surfaces) + Phase 5 (supersession

• cron-exclusion) remain on the roadmap.

Post-Insert Helper (Python)

scripts/kb_pipeline/post_insert.py::run_post_insert(...) runs 8 post-insert side-effects in canonical order: content_history v1 (NO-OP — DB trigger is the authority) → chunks → entity aliases → entity_mentions → entity_relationships → metadata.ai_temporal_references → temporal-to-entity bridge → layer inference. Wired into all 4 Python ingest scripts.

Post-Ingest Quality Gate (S185)

scripts/quality-gate.ts (OPS-5) — read-only observer that inspects DB state after ingestion. Profiles: re-ingest, batch, onboarding, audit-content. Content-management-relevant checks include corpus_counts, dedup_status_reconciled, suspected_duplicate_backlog, history_v1_present, chunk_coverage. Exit codes: 0 (passed), 1 (failed), 2 (operational failure). Output formats: markdown, JSON. Config sidecars live under scripts/config/quality-gate/.

API Routes

Core CRUD

Method	Route	Auth	Rate Limit	Purpose
POST	/api/items	Admin/Editor	20/min	Create content item
PATCH	/api/items/[id]	Admin/Editor	—	Update single field
DELETE	/api/items/[id]	Admin only	—	Delete (CASCADE)

Archive, History, Rollback

Method	Route	Auth	Purpose
POST	/api/items/[id]/archive	Admin/Editor	Soft-archive with reason
GET	/api/items/[id]/history	All authed	List versions (paginated)
GET	/api/items/[id]/history/[verId]	All authed	Single version with full content
POST	/api/items/[id]/rollback	Admin/Editor	Non-destructive rollback

Classification and AI

Method	Route	Auth	Rate Limit	Purpose
POST	/api/items/[id]/classify	Admin/Editor	20/min	AI classification
POST	/api/items/[id]/vision	Admin/Editor	10/min	PDF visual analysis
POST	/api/summaries/generate	Admin/Editor	10/min	AI summary generation
POST	/api/embed	Admin/Editor	30/min	Generate embedding
POST	/api/extract	Admin/Editor	10/min	Structured extraction

Metadata, Priority, Layers, Owner

Method	Route	Auth	Purpose
PATCH	/api/items/[id]/metadata	Admin/Editor	Update metadata JSONB
PATCH	/api/items/[id]/priority	Admin/Editor	Set priority level
PATCH	/api/items/[id]/owner	Admin/Editor	Assign content owner
GET	/api/items/[id]/layers	All authed	Get layer assignments
GET	/api/items/[id]/effectiveness	All authed	Win rate via RPC

Files, Images, Workspaces

Method	Route	Auth	Purpose
POST	/api/items/[id]/files	Admin/Editor	Upload PDF (max 32MB)
DELETE	/api/items/[id]/files	Admin/Editor	Remove file
POST	/api/items/[id]/images	Admin/Editor	Extract PDF images
GET	/api/items/[id]/images	All authed	Retrieve extracted images
GET	/api/items/[id]/workspaces	All authed	List workspace assignments
POST	/api/items/[id]/workspaces	Admin/Editor	Assign to workspace

Batch Operations

Method	Route	Auth	Purpose
POST	/api/items/batch	Admin/Editor	Batch create Q&A (maxDur 120s)
POST	/api/items/batch-review	Admin/Editor	Batch governance (‘pending’ only)
POST	/api/items/batch-workspaces	All authed	Batch fetch workspace assignments

Ingestion

Method	Route	Auth	Rate Limit	Purpose
POST	/api/upload	Admin/Editor	—	File upload (max 50MB)
POST	/api/ingest/url	Admin/Editor	10/min	URL import (SSRF-validated)

Supporting Routes

Method	Route	Auth	Purpose
POST	/api/dedup/check	Admin/Editor	MD5 exact + cosine 0.92 near
GET	/api/read-marks	All authed	Fetch read marks
POST	/api/read-marks	All authed	UPSERT read mark
POST	/api/content-owners/bulk-assign	Admin only	Bulk assign (max 500)
GET	/api/content-owners/stats	All authed	Ownership statistics
GET	/api/content-suggestions	All authed	Priority-ranked suggestions
POST	/api/freshness/calculate	Admin/Editor	Calculate freshness
POST	/api/freshness/recalculate-all	Admin only	Recalculate all items

Source Document Routes

Method	Route	Auth	Purpose
GET	/api/source-documents/[id]	All authed	Document with linked items
GET	/api/source-documents/[id]/versions	All authed	Version chain via RPC
GET	/api/source-documents/[id]/diff	All authed	Retrieve diff
POST	/api/source-documents/[id]/diff	Admin/Editor	Compute diff
PATCH	/api/source-documents/[id]/diff	Admin/Editor	Update diff status
POST	/api/source-documents/[id]/send-to-review	Admin/Editor	Trigger governance review

Handoff Routes (owned by other feature areas)

Route	Owner	Relationship
GET/PATCH /api/quality	Quality Governance	Reads quality_score, governance_review_status
GET /api/quality/summary	Quality Governance	Aggregates content quality metrics

Components

For current counts, see docs/generated/codebase-stats.md.

Browse (components/browse/)

Content browsing, filtering, and list views. Includes filter panels (filter-bar, filter-panel, domain/subtopic/content-type/platform filters), preset bars (preset-bar, manage-presets-dialog, save-preset-dialog), bulk action toolbars (bulk-actions, bulk-action-toolbar), the search bar, and the persona prompt cards (prompt-card-chip-composite, search-prompt-cards — driven by the discriminated-union card data model from S197 §1.20).

Create Content (components/create-content/)

4-tab consolidated creation UI at /item/new. The four tabs (Write / URL / Upload / Batch) live in app/item/new/new-item-tabs.tsx. Sub-components include classification-fieldset, provenance-fieldset, progressive-depth-fieldset, template-selector, url-ingest-form, upload-tab-content, upload-review-step, ingestion-progress, ingestion-success-card, mobile-step-indicator, save-actions-bar, and file-upload.

Q&A (components/qa/)

Q&A content type with specialised editing and display: qa-answer-display (S198 ContentEditor mount), qa-pair-renderer, qa-row, qa-preview-list, batch-qa-preview-table.

Item Detail (components/item-detail/)

Full content item view. Notable components: item-action-bar (mounts SupersedeContentDialog), item-title-section, metadata-sidebar (mounts ReviewCadenceEditor from S206), collapsible-section, content-tabs, content-renderer, content-editor + inline-content-editor, organise-section, entity-badges, version-history, version-diff, item-completeness-checklist, temporal-references-section, related-content-section, related-by-tags, related-by-entities, reader-view.

Reader (components/reader/)

Distraction-free reading. Components: iframe-viewer, image-gallery, pdf-reader-view, pdf-viewer, reader-panel, reader-view, source-metadata + source-metadata-helpers, transcript-reader. The floating overlay reader was removed in S183 P1-7 (react-rnd dropped).

Content (components/content/)

Shared display and action components: content-card (mounts ReviewCadenceBadge from S206), content-grid, content-list, content-row, delete-content-dialog, supersede-content-dialog, content-owner-selector, content-owner-badge, quick-assign-button, quick-review-actions, review-cadence-editor (S206), claude-prompt-button, citation-panel, layer-suggestion-banner, content-layer-selector, content-library-drawer, content-library-result.

Source Documents (components/source-document/)

Document upload, version comparison, and diff review: source-document-info, source-document-history, source-document-diff-review, diff-highlighted-text, reupload-banner.

Hooks

Browse Hooks (hooks/browse/)

use-browse-data, use-browse-filters, use-debounced-preview, use-filter-data, use-filter-draft, use-filter-presets, use-library-filters, use-top-domains (S197 §1.20 chip composite).

Detail Hooks

Hook	Purpose
use-item-detail-data	Master data hook for item detail view
use-inline-field-edit	Single-field inline editing with PATCH (drives Q&A answer-field edits S198)
use-item-detail-shortcuts	Keyboard shortcuts for detail view

Creation Hooks

Hook	Purpose
use-file-upload-pipeline	Multi-step file upload with progress
use-batch-create	Batch Q&A creation with pipeline tracking
use-content-templates	Template selection and application

Source Document Hooks

Hook	Purpose
use-diff-review	Diff computation, display, and review actions

Database Tables

content_items (live column count: see SCHEMA-QUICK-REFERENCE)

The central table. Column groups:

Group	Key Columns
Identity	id, title, content, content_type, suggested_title
Classification	primary_domain, primary_subtopic, secondary_domain, secondary_subtopic, classification_confidence, classification_reasoning, classified_at
Content body	brief, detail, reference, answer_standard, answer_advanced
AI	summary, ai_keywords, embedding vector(1024), summary_data JSONB
Provenance	platform, source_url, source_domain, source_file, source_document_id FK (S205), source_bid FK, author_name, parent_id FK, ingest_source (S207), captured_date
Freshness	freshness, freshness_checked_at, previous_freshness, lifecycle_type, expiry_date
Cadence	next_review_date, review_cadence_days (§5.5 Phase 1)
Quality	quality_score, quality_score_updated_at, previous_quality_score, content_text_hash (GENERATED ALWAYS)
Dedup	dedup_status (CHECK: clean/suspected_duplicate/confirmed_duplicate/confirmed_unique/superseded), superseded_by (UUID FK → self, ON DELETE SET NULL)
Lifecycle	publication_status (CHECK: draft/in_review/published/archived, NOT NULL DEFAULT published)
Governance	governance_review_status, governance_review_due, governance_reviewer_id, verified_at, verified_by
Organisation	layer, priority, user_tags, starred
Ownership	content_owner_id (S206 — auto-assigned at all 6 ingest entry points), citation_count
Archive	archived_at, archived_by, archive_reason
Files	file_path, thumbnail_url
Metadata	metadata JSONB, notes
Audit	created_at, updated_at, created_by, updated_by

content_history

Version history for content items. v1 rows are written exclusively by the deferred trigger trg_content_items_ensure_v1_history (S207 — single authority). The trigger emits change_reason='initial_ingest' when ingest_source is set, falling back to 'auto_v1_on_insert' for legacy rows.

content_chunks

Heading-bounded sub-document records with per-chunk vector(1024) embeddings. Populated by regenerateChunks() at 5 entry points and the parallel Python chunk.py from the post-insert helper.

content_item_workspaces

Junction table — many-to-many for RSS feed promotion (S184 EP11 D3).

content_templates

Template definitions. Currently code constants drive creation (see limitations below).

source_documents

Uploaded source files with version chain via parent_id FK. Status enum includes processing and failed.

read_marks

Per-user read tracking. Unique constraint on (user_id, content_item_id).

pipeline_runs

Ingestion pipeline execution records. Insert via recordPipelineRun() from @/lib/pipeline/record-run (never raw insert). Two indexes for Pipeline Health queries (Provenance Pipeline tab).

tag_morphology_drift_flags (S197)

Triage queue for tag morphology drift surfaced by scripts/eval-tag-morphology-adoption.ts. RLS: admin/editor read/insert/ update/delete.

MCP Tools

For current counts and full inventory, see docs/generated/mcp-inventory.md.

Content management tools:

Under ID-71 several of these consolidated: get (one-or-many) replaced get_content_item + get_content_items; assign (one-or-many, with a scope-filter branch) replaced assign_content_owner + bulk_assign_owner; the former audit_content folded into where_are_we_exposed; chunk search is now the find tool’s chunk-granularity branch.

Tool	Type	Purpose
get (id | ids)	Read	Fetch one or many items (replaced get_content_item + get_content_items)
create_content_item	Write	Create new item (S206 + S208 OPS-40 audit emission)
update_content_item	Write	Update item fields
get_workspace_items	Read	Items by workspace
assign (one-or-many)	Write	Set content owner; scope-filter branch + dry-run replaced bulk_assign_owner
get_document_versions	Read	Source document version chain
get_document_diff	Read	Document diff
classify_content	Write	AI classification
generate_summary	Write	AI summary generation
delete_content_item	Destructive	Delete item
update_governance_status	Write	Set governance review status
where_are_we_exposed	Read	Five-layer exposure report (absorbed the former audit_content)
suggest_content_creation	Read	Gap-based creation suggestions
supersede_content_item	Destructive	Mark old item superseded by new (admin-only)
find (granularity: "chunk")	Read	Heading-bounded chunk search + cadence filters (S208); backed by the search_content_chunks RPC

Current Limitations

1. Q&A content field is derived — auto-rebuilt from answer_standard + answer_advanced; direct edits are overwritten.
2. PATCH cannot trigger reclassification directly — warns the user; must call /classify separately.
3. Batch-review only supports ‘pending’ — cannot batch-set other governance statuses.
4. Content templates use code constants — not yet driven by the content_templates DB table.
5. Expiry date UI gap — two-store architecture with no sync and no edit UI.
6. quality_score handles both ‘ageing’ and ‘aging’ spellings — legacy normalisation.
7. Supersession chains not followed — A superseded_by B superseded_by C queries do not walk the pointer.
8. Near-duplicate dedup UI not yet built — OPS-3 Phase 2 (near-dedup review dashboard + merge UI) is scheduled post-re-ingestion. Currently only exact-hash duplicates are stamped; cosine-similarity near-matches are detected by POST /api/dedup/check but not wired to the stamp flow.
9. TS-side batch/form/RSS entry points do not chunk — EP6 batch item creation, EP10 form-outcome integration, and EP11 RSS feed promotion do not call regenerateChunks().
10. §5.2 Phases 3-5 not yet shipped — RPC visibility flip, UI surfaces (publication-review queue tab), and supersession + cron-exclusion integration remain on the roadmap.
11. MCP-EMBED-1 — lib/mcp/tools/content.ts truncates at 5,000 chars; align to MAX_EMBEDDING_CHARS once stable for MCP-created items. Product-backlog entry §6.

Architecture Decisions

Decision	Rationale	Alternative Considered
Single PATCH route per field	Granular updates with targeted embedding regeneration	Bulk update endpoint
Warnings envelope pattern	Non-fatal pipeline failures don’t block saves	Strict fail-fast
MD5 + cosine deduplication	Catches both exact and near-duplicate content	Content hash only
Non-destructive rollback	Creates new version rather than deleting history	Destructive revert
auto_embed flag for drafts	Controls whether embedding is generated; PATCH route generates embedding when publishing	Automatic skip based on status
CASCADE delete	Simplifies cleanup; admin-only gate provides safety	Soft-delete everywhere
Soft-block dedup (not hard)	Insert succeeds with suspected_duplicate stamp; operators review and resolve. No data loss from false positives.	Hard-block (reject the insert)
resolveDedupStamp helper	Single function for all 7 TS entry points ensures consistent stamp format	Per-route inline logic
Single superseded_by column	One-column pointer + default search filter covers both DRAFT/final and future revision use cases without a chain model	Separate supersession table
Admin-only supersession	Supersession hides content from search — too impactful for editor self-service. Matches the gated dedup override pattern	Editor+ access
ON DELETE SET NULL for superseded_by	If successor is deleted, old row becomes current again rather than orphaning	ON DELETE CASCADE
Quality gate as read-only observer	Does not block writes or trigger ingestion — decoupled from the pipeline so it can be re-run at any time	Inline validation in pipeline
Typed ingest_source column	One typed value per ingest path; trigger reads it and emits granular change_reason. Replaces JSONB metadata.ingestion_source audit	Continue with JSONB metadata
DB trigger as v1 history sole authority	Eliminates “did the app forget to write v1?” failure mode; structurally guarantees coverage. Inverted guard test enforces the contract	App-level write per ingest path
resolveContentOwnerId silent-force	Admin can override; non-admin override is silently ignored (no 403). Mirrors skip_dedup pattern	Hard 403 for non-admin override
content_text_hash GENERATED ALWAYS	Postgres computes deterministically; eliminates app-side hash drift between TS and Python	App-computed hash on insert
Library tag morphology + carve-outs	pluralize@8 / inflect==7.5.0 plus pre-pass override for domain-specific terms; compound last-token guard for Latin/Greek	In-house morphology rules
Content-type-aware Source Information	SourceMetadata branches on content_type/platform/feedArticle/markdown-ingest detection so each surface shows only relevant fields	Generic source_url row only