Skip to content

Knowledge Organisation — Workflows

Last verified: Session 210 (29 April 2026) Pending updates: None

Knowledge organisation workflows cover the data flows for taxonomy management, entity lifecycle, layer vocabulary, tag governance, tag morphology triage, guide configuration, and coverage gap analysis. Key automated processes include entity extraction during content classification, coverage matrix computation, unified gap aggregation, entity metadata propagation, and the four-step taxonomy sync chain.

Workflow 1: Taxonomy Domain + Subtopic CRUD

Section titled “Workflow 1: Taxonomy Domain + Subtopic CRUD”

Trigger: Admin submits POST / PATCH to /api/taxonomy/domains or /api/taxonomy/subtopics Owner: app/api/taxonomy/domains/*/route.ts + app/api/taxonomy/subtopics/*/route.ts

Validate (TaxonomyDomainCreateSchema) → Auto-assign display_order if not provided → Insert taxonomy_domains (provenance='client', is_active=true) → Return 201
Validate (TaxonomySubtopicCreateSchema) → Verify domain_id exists → Auto-assign display_order → Insert taxonomy_subtopics → Return 201
  • Auto-assign display_order — query max existing + 10 when omitted
  • Provenance — created rows always get provenance='client'; the AI pipeline writes provenance='recommended' directly, never via this route
  • Unique constraints — domains by name, subtopics by (domain_id, name); both return 409 on duplicate
  • Update pathPATCH only updates the changed fields (name, colour, description, display_order, is_active, key_signal, accepted_at)
  • Optional description field on subtopics added in P1-37 (S185)
Error ConditionHandlingUser Feedback
Validation failure400 returnedField-level errors
Domain not found (subtopic create)400 returned”Domain not found”
Duplicate name409 returned”already exists” message
DB error500 returnedGeneric error message

Trigger: Admin submits POST /api/taxonomy/reorder Owner: app/api/taxonomy/reorder/route.ts

Validate (TaxonomyReorderSchema) → Validate subtopic ownership (if type=subtopic) → Update display_order for each item → Return success count
  1. Validate input — Fields: type (domain or subtopic), domain_id (for subtopics), items (array of {id, display_order}).
  2. Validate subtopic ownership — For subtopic reordering, verify all IDs belong to the specified domain. Returns 400 if any subtopic belongs to a different domain.
  3. Apply updates — Update each item’s display_order individually; return count of successfully updated items.

useTaxonomyAdmin implements optimistic updates for reordering:

  1. Cancel any in-flight queries for the affected data
  2. Snapshot current data as rollback target
  3. Apply new display_order values to the cache
  4. Sort cached data by display_order
  5. On success: invalidate queries to refetch
  6. On error: restore snapshot, show error toast

Trigger: Admin submits POST /api/entities/merge Owner: app/api/entities/merge/route.ts

Validate (EntityMergeBodySchema) → Call merge_entities RPC (atomic transaction) → Return merge results

The merge is atomic via the merge_entities database function:

  1. Update canonical_name on all entity_mentions rows for source entities to the target name
  2. Set entity_type_override to the chosen entity type
  3. Update entity_relationships — both source_entity and target_entity references updated from source names to target name
  4. Remove duplicate mention rows where (canonical_name, entity_type, content_item_id) collides after the merge
Error ConditionHandlingUser Feedback
Validation failure400 returnedField-level errors
RPC failure500 returnedMerge error message
Rate limit429 returned (10/min)Rate limit response

Trigger: Admin submits POST /api/entities/split Owner: app/api/entities/split/route.ts

Validate (EntitySplitBodySchema) → Validate names differ → Update entity_mentions canonical_name for selected variants → Check if all mentions moved → Conditionally update entity_relationships → Return split results
  1. Validate input — Fields: canonical_name, variant_names (array), new_canonical_name. Rejects identical old/new canonical names.
  2. Move selected variant mentions — Update canonical_name on entity_mentions rows matching both the old canonical name AND the specified variant names. Uses service client to bypass RLS.
  3. Check remaining mentions — Query whether any mentions still have the old canonical name.
  4. Update relationships — Only when ALL mentions were moved. Updates source_entity and target_entity on entity_relationships from old to new canonical name.
ConditionRelationship UpdateRationale
Some mentions remainNo updateOld entity still exists
All mentions movedUpdate referencesOld entity has been fully renamed

Trigger: Admin submits PATCH /api/entities/[canonical_name]/type — either from the entity list type-edit dialog or inline via the entity detail panel type-change <Select> (P1-22). Owner: app/api/entities/[canonical_name]/type/route.ts

Validate (EntityTypeOverrideBodySchema) → Update entity_type_override on all mentions → Return count of updated mentions
  • Sets entity_type_override on all entity_mentions rows matching the canonical name
  • Override takes precedence via COALESCE(entity_type_override, entity_type) in all queries
  • Uses service client for atomic update
  • Returns 404 if no mentions found
PathComponentHook
Entity list type dialogEntityListLocal useMutation
Entity detail inline (P1-22)EntityDetailPaneluseEntityDetail.changeType

Both call the same API endpoint. The detail panel path uses optimistic update with rollback; the list path refreshes the list on success.


Workflow 6: Entity Metadata Update with Reverse Bridge

Section titled “Workflow 6: Entity Metadata Update with Reverse Bridge”

Trigger: Editor or admin submits PATCH /api/entities/[canonical_name]/metadata Owner: app/api/entities/[canonical_name]/metadata/route.ts

Validate (EntityMetadataUpdateSchema) → Find first mention → Shallow merge metadata → Update mention → Reverse bridge: propagate expiry_date to content items → Return updated metadata + warnings
  1. Find mention — Query entity_mentions for the canonical name, limit 1. 404 if not found.
  2. Merge metadata — Shallow merge: new keys override existing, existing keys preserved.
  3. Reverse bridge propagation (conditional)
    • Only when expiry_date is present in the merged metadata
    • Only for entity types: certification, regulation, standard
    • Finds all content items linked via entity_mentions.content_item_id
    • Updates expiry_date and sets lifecycle_type to date_bound on those content items
    • Non-fatal: failures added to warnings array, not blocking
Error ConditionHandlingUser Feedback
Entity not found404 returned”Entity not found”
Metadata update fails500 returnedError message
Reverse bridge failsWarning added, metadata still savedWarning in response

Workflow 7: Holder Metadata Derivation (Classification Pipeline)

Section titled “Workflow 7: Holder Metadata Derivation (Classification Pipeline)”

Trigger: Content classification — runs at the end of every successful classification with validate=true or in batch reclassification. Owner: lib/ai/classify.ts::deriveHolderMetadata (TS) + scripts/kb_pipeline/classify.py::derive_holder_metadata (Python).

Pass 1: scan relationships for `holds` → build holdsRelsByTarget map
Pass 2 (synonym fallback): scan relationships for `complies_with`/`evidences` → only accept when target is certification AND source is client org or extracted org AND no canonical `holds` exists for that target
For each certification mention: if holdsRelsByTarget has a source for canonical_name → set metadata.holder = 'self' (if source == client org lower) or {holder: 'supplier', supplier_name: source}

The classifier sometimes emits complies_with or evidences when the content phrases a certification differently (e.g. “our ISO 27001 compliance”, “evidenced by our DBS check”). Both are valid relationship-type enum members, but the holder rule was originally only acting on holds. The S196 fix adds a synonym fallback that is conservative:

  • (a) target must be a certification entity (preserves semantic meaning of complies_with/evidences in non-cert contexts — e.g. “our org complies_with GDPR” where GDPR is a regulation)
  • (b) source must be the client organisation OR an extracted organisation entity in this batch (prevents garbage rels like “ISO 27001 complies_with Cyber Essentials Plus” from being mis-derived as cert-held-by-cert)
  • (c) no canonical holds rel already exists for that target (holds wins over synonyms on tie)

Each certification row that matched a holds (or eligible synonym) writes metadata JSONB:

Source matches client org?metadata payload
Yes{ "holder": "self" }
No{ "holder": "supplier", "supplier_name": <source> }

Consumers:

  • app/api/certifications/route.ts filters certs by holder == 'self' AND source matches BRANDING.organisationName.toLowerCase()
  • The MCP where_are_we_exposed tool (certification layer — formerly the standalone get_certification_status tool) applies the same filter

Cross-language parity:

  • TS test: __tests__/lib/ai/classify-derive-holder-metadata.test.ts
  • Python test: scripts/tests/test_classify_holder_rule.py
  • Eval: scripts/kb_pipeline/eval_holder_rule.py

Workflow 8: ISO Family Type Override (Classification Pipeline)

Section titled “Workflow 8: ISO Family Type Override (Classification Pipeline)”

Trigger: Storage step in classification — runs after canonicalise + alias

  • filter, before upsert. Owner: lib/ai/classify.ts Step 15b (and Python parity in scripts/kb_pipeline/classify.py).
For each filtered entity row → If canonical_name (lowercased) ∈ {iso 9001, iso 14001, iso 22301, iso 27001, iso 45001, iso 50001} AND entity_type != 'certification' → Force entity_type = 'certification' (log override)

UK SMB form-response libraries overwhelmingly mention these six ISO families in certification-context (i.e. “we hold ISO 27001”), but classifier Pass 2 sometimes retypes them to standard when the document discusses the published document. The override eliminates cross-item type flip-flop that broke entity dedup. CREST is deliberately NOT in the set (genuinely ambiguous between professional body and credential).

Source-of-truth for entity types: docs/reference/entity-type-taxonomy-spec.md.


Workflow 9: Tag Morphology Drift Flag Disposition

Section titled “Workflow 9: Tag Morphology Drift Flag Disposition”

Trigger: Admin or editor submits PATCH /api/admin/tag-morphology/flags/[id] (S196 §1.17) Owner: app/api/admin/tag-morphology/flags/[id]/route.ts

Validate (TagMorphologyFlagDecisionSchema) → Update tag_morphology_drift_flags row with decision + decided_by + decided_at + rationale → Return updated row
DecisionEffectFollow-up
acceptLibrary output is correct; tag should be backfilled to proposed_canonicalAdmin runs scripts/apply-tag-morphology-backfill.ts against accepted IDs
add_overrideStored form should be preserved; library should NOT apply this transformationAdd to TAG_PLURAL_LOOKING_SINGULARS or TAG_PROPER_NOUN_ALLOWLIST
dismissLibrary agrees, or noise — no actionNone

Backfill mutates production tags across content_items.ai_keywords. Liam’s spec decision (recorded in docs/specs/p1-tag-morphology-library-adoption-spec.md §3.5.3) is that humans must accept each disagreement before backfill runs. This avoids regressing existing curated tags when the library’s morphology output diverges from established usage.

OperationTableAuth
UPDATEtag_morphology_drift_flagsAdmin, Editor

Drift flags are populated by scripts/eval-tag-morphology-adoption.ts. The eval iterates content_items.ai_keywords, applies normaliseTag() to every stored value, and inserts (or updates the usage_count of) a flag for each disagreement via POST /api/admin/tag-morphology/flags. UNIQUE on (stored_tag, proposed_canonical) makes the bulk-insert idempotent.


Workflow 10: Tag Mutations (Rename / Merge / Delete / Bulk)

Section titled “Workflow 10: Tag Mutations (Rename / Merge / Delete / Bulk)”

Trigger: Admin submits to one of the /api/tags/* write endpoints Owner: app/api/tags/*/route.ts

All tag mutations route through DB RPCs to keep the operation atomic across all rows in content_items.ai_keywords / user_tags:

API RouteRPC
DELETE /api/tagsdelete_tag
POST /api/tags/renamerename_tag
POST /api/tags/mergemerge_tags
POST /api/tags/bulk-deletebulk_delete_tags
POST /api/tags/bulk-mergebulk_merge_tags

Targets in TagRenameBodySchema, TagMergeBodySchema, and TagBulkMergeBodySchema pass through normaliseTag before reaching the RPC. This applies the proper-noun allowlist (ISO 27001, GDPR, etc.) and plural canonicalisation (pluralize@8) at the write boundary, so the stored form is canonical regardless of admin input casing.


Access note: The /coverage page is editor+admin only since S189 P1-11 (viewers redirected to /browse). API routes remain auth-only without defence-in-depth role checks.

Trigger: GET /api/coverage/gaps Owner: app/api/coverage/gaps/route.ts

Check cache → Fetch coverage matrix + targets + guide coverage + templates in parallel → Build taxonomy gaps → Build template gaps → Build guide gaps → Score all gaps → Filter/paginate → Cache result → Return
  1. Cache check — 60-second in-memory cache keyed by sorted query parameters. Returns cached result if still valid.
  2. Parallel data fetch
    • get_coverage_matrix (taxonomy item counts)
    • coverage_targets (admin-set thresholds)
    • get_guide_coverage (guide section status)
    • listAvailableTemplates (template list from DB)
  3. Build taxonomy gaps — Empty subtopics (item_count = 0) from coverage matrix. Scored higher if domain has zero items overall, or if domain has unmet coverage targets.
  4. Build template gaps — For each template, fetches requirements and content items. Computes coverage via computeTemplateCoverage(). Extracts requirements with coverage_status: 'gap'. Mandatory requirements scored higher.
  5. Build guide gaps — Empty sections (content_count = 0) and stale sections (stale_count > 0, fresh_count = 0) from guide coverage. Required sections scored higher than optional.
  6. Score and aggregate — All gaps via scoreGap() from lib/coverage/gap-scoring.ts. Assigned priority tier: critical, high, medium, low. Filtered by source/priority/domain. Sorted by priority_score desc. Paginated.
SourceFactorEffect on Score
TaxonomyDomain has zero itemsSignificant boost
TaxonomyDomain has unmet coverage targetModerate boost
TemplateRequirement is mandatoryModerate boost
GuideSection is requiredModerate boost
GuideSection is stale (not empty)Lower than empty sections

Trigger: GET /api/coverage Owner: app/api/coverage/route.ts

Fetch coverage matrix + summary in parallel → Return combined result
RPCPurpose
get_coverage_matrixDomain × subtopic item count grid
get_coverage_summaryAggregated coverage statistics

The p_layer parameter optionally filters the matrix to a specific content layer.


Trigger: GET /api/coverage/guides Owner: app/api/coverage/guides/route.ts

Call get_guide_coverage RPC → Group rows by guide → Compute per-guide section stats → Build summary → Return
Content CountFresh CountStale CountDerived Status
0empty
> 0> 0populated
> 00> 0stale
> 000populated
MetricComputation
total_sectionsCount of all sections
populated_sectionsSections with non-empty status
required_sectionsSections where is_required = true
populated_requiredRequired sections that are populated
fresh_sectionsSections with fresh_count > 0
stale_sectionsSections with stale_count > 0 and fresh_count = 0
MetricComputation
fully_populatedGuides where populated_sections = total_sections
emptyGuides where populated_sections = 0
partially_populatedAll others

Workflow 14: Template Coverage Computation

Section titled “Workflow 14: Template Coverage Computation”

Trigger: GET /api/coverage/templates Owner: app/api/coverage/templates/route.ts

Validate template_name param → Fetch requirements + content items in parallel → Compute coverage → Return sections with requirements and coverage status

Key Functions (all in lib/templates/template-coverage.ts)

Section titled “Key Functions (all in lib/templates/template-coverage.ts)”
FunctionPurpose
listAvailableTemplatesQuery template_requirements for distinct templates
fetchTemplateRequirementsGet all requirements for a template (with version)
fetchContentForMatchingGet content items with fields needed for matching
computeTemplateCoverageMatch requirements against content, produce coverage report
computeGapSummaryAggregate gap counts across multiple templates
StatusMeaning
coveredRequirement fully met by one or more content items
partialRequirement partially met (some but not all criteria)
gapNo matching content found for this requirement

Trigger: User types in tag input field Owner: app/api/tags/suggest/route.ts

Validate prefix + type params → Call suggest_tags RPC → Return up to 10 matching tags ordered by frequency

Rate limited at 60/min per user to handle rapid typing.


Trigger: Admin CRUD operations on layers Owner: app/api/layers/ routes

Validate (LayerCreateSchema) → Auto-assign display_order → Insert into layer_vocabulary → Return 201
Look up layer key → Count content_items with this layer → If count > 0: return 409 with count → If count = 0: delete layer → Return 204

The key field is immutable after creation. Content items reference layers by key string (not UUID), so changing the key would orphan existing assignments.

API routes that accept a layer key now validate against live layer_vocabulary rows via fetchActiveLayerKeys() from lib/validation/layer-schemas.ts:

RoutePurpose
POST /api/guides/[slug]/sectionsGuide section creation
PATCH /api/guides/[slug]/sections/[sectionId]Guide section update
PATCH /api/items/[id]/metadataContent item metadata update

If fetchActiveLayerKeys() throws (DB unavailable or no active layers), the route returns 503 Service Unavailable. MCP tool layer validation was loosened accordingly to accept any string, delegating validation to the API layer.


Trigger: bun run sync:taxonomy (manual, after taxonomy changes) Owner: package.json script

generate-classification-prompt-taxonomy.ts → generate-taxonomy-snapshot.ts → sync-plugin-taxonomy.ts → build:plugin
  1. scripts/generate-classification-prompt-taxonomy.ts — Reads taxonomy from DB; regenerates the taxonomy section of lib/ai/skills/classification.md (the classification prompt single source of truth, v4.5).
  2. scripts/generate-taxonomy-snapshot.ts — Writes scripts/tests/fixtures/taxonomy_snapshot.json consumed by the Python pipeline. The snapshot is the Python pipeline’s view of the canonical taxonomy.
  3. scripts/sync-plugin-taxonomy.ts — Reads taxonomy_domains / taxonomy_subtopics from DB; rewrites the plugin classification skill, search-strategy skill, and settings template (.claude/plugins/knowledge-hub/1.0.0/...).
  4. bun run build:plugin — Regenerates lib/mcp/plugin-bundle.ts (the committed base64 plugin ZIP).

Two facts the script cannot derive from the DB:

  • Key signals — short summary phrases per domain (currently embedded in the classification prompt template; require manual updates if domain semantics shift).
  • Guide definitions — guide rows + guide_sections live in DB but the initial seed of guides for a new client is a manual product decision.

TaxonomyDriftBanner (in components/settings/) compares the live DB taxonomy hash against the prompt’s taxonomy hash and warns the admin if the two diverge. The banner does NOT block operation — admins must remember to run bun run sync:taxonomy after taxonomy edits.


Workflow 18: Tag Morphology Drift Flag Bulk Insert

Section titled “Workflow 18: Tag Morphology Drift Flag Bulk Insert”

Trigger: POST /api/admin/tag-morphology/flags (called by scripts/eval-tag-morphology-adoption.ts) Owner: app/api/admin/tag-morphology/flags/route.ts

Validate (TagMorphologyFlagsBulkInsertSchema) → Upsert each flag (UNIQUE on stored_tag + proposed_canonical) → Return count

The eval script iterates the corpus, applies normaliseTag() to every stored tag in content_items.ai_keywords, and posts each disagreement as a flag. UNIQUE constraint makes re-runs idempotent — usage_count and affected_content_ids are updated, but already-decided flags are left alone.


ProcessTriggerRoute/ModulePurpose
Entity extractionContent creation/updatePart of classification pipelineExtract entities + relationships into entity_mentions + entity_relationships
Holder metadata derivationClassification pipelinelib/ai/classify.tsSet metadata.holder on certification mentions (TS+Python parity)
ISO family type overrideClassification pipelinelib/ai/classify.ts Step 15bForce six ISO families to entity_type='certification'
Coverage matrix computationOn-demand/api/coverageDomain-subtopic item count grid
Unified gap aggregationOn-demand (60s cache)/api/coverage/gapsScored gaps from taxonomy, templates, guides
Guide coverage computationOn-demand/api/coverage/guidesSection-level content counts and freshness
Template coverage computationOn-demand/api/coverage/templatesRequirement-level coverage matching
Reverse bridge propagationEntity metadata update/api/entities/[name]/metadataExpiry date propagation to content items
Tag suggestionUser typing/api/tags/suggest (RPC)Prefix-based tag autocomplete
Tag morphology evalManual cron-stylescripts/eval-tag-morphology-adoption.tsSurfaces drift between stored and library-canonical tag forms
External SystemDirectionProtocolPurpose
SupabaseRead/WriteREST + RPCPrimary data store; RPCs for atomic mutations and aggregation
Claude APIRequestHTTPEntity extraction during content classification
MCP ClientsRead+WriteMCPList/get/create/update guides; entity relationships; certification status; coverage gaps
Target Feature AreaMechanism
Content ManagementClassification populates primary_domain/primary_subtopic on content items; entity extraction runs during creation pipeline; layer assignment via content_items.layer referencing layer_vocabulary.key
Quality GovernanceCoverage targets inform governance review prioritisation; guide section freshness feeds into quality assessment
SearchTaxonomy domains/subtopics used as search filters; entity relationships queryable via MCP
Completing Forms (Procurement)Template coverage maps form requirements to KB content; certification status reports serve form responses
Administrationcontent_owner_id (S205 WP-A) integrates with knowledge organisation by tracking ownership of curated structural items (guides, taxonomy entries) for accountability

Valid values for provenance on taxonomy_domains and taxonomy_subtopics:

ProvenanceMeaning
baselineSeeded during initial setup
clientManually created by admin
recommendedAI-suggested (e.g. from classification pipeline)
baseline → (edit) → baseline (provenance unchanged, fields updated)
recommended → (accept) → recommended (provenance unchanged, accepted_at set, is_active=true)
recommended → (reject) → recommended (provenance unchanged, is_active=false)
client → (deactivate) → client (is_active=false)
client → (reactivate) → client (is_active=true)

The provenance field does not change after creation. Only is_active, accepted_at, and editable fields change. The accepted_at timestamp indicates when a recommended item was promoted into active use.


Entity types resolve via COALESCE semantics:

effective_type = COALESCE(entity_type_override, entity_type)
entity_typeentity_type_overrideeffective_typeScenario
organisationNULLorganisationNo override applied
personorganisationorganisationAdmin corrected mistyped entity
certificationNULLcertificationCorrect extraction
standardcertificationcertificationISO-family override (Workflow 9)

The has_type_conflict flag is set when types_seen contains more than one distinct type across all mentions for a canonical name.


Tag Morphology Drift Flag — State Machine

Section titled “Tag Morphology Drift Flag — State Machine”

Valid values for decision on tag_morphology_drift_flags:

DecisionMeaning
pendingAwaiting admin/editor disposition (default on insert)
acceptLibrary output is correct; backfill proposed_canonical
add_overridePreserve current stored form; library should not transform this
dismissLibrary agrees, or noise — no action
pending → (admin/editor patches /api/admin/tag-morphology/flags/[id]) → accept | add_override | dismiss
(no transitions back from terminal states; re-running the eval inserts new rows for new disagreements but does not reset existing decisions)

After accept is set, the admin runs scripts/apply-tag-morphology-backfill.ts to perform the actual UPDATE on content_items.ai_keywords. The triage UI itself does not mutate tags.


  1. Entity extraction is coupled to the content classification pipeline — there is no standalone entity extraction endpoint
  2. Coverage gap computation fetches all template requirements and content items on each uncached request, which can be expensive for large knowledge bases
  3. The taxonomy sync chain (bun run sync:taxonomy) is manual and must be run after any taxonomy changes; key signals + guide seeds are manual outside the chain
  4. Entity merge is one-way — there is no unmerge operation
  5. Tag operations are atomic but have no audit trail of who performed them (separate from the tag morphology triage flow which DOES record decided_by)
  6. Guide section matching is filter-based (content_type_filter, subtopic_filter, expected_layer) which can produce broad matches when filters are NULL
  7. Holder synonym fallback is conservative — it intentionally rejects valid complies_with/evidences claims when the source is not a known organisation, to prevent garbage rels. Some legitimate cert claims are missed in exchange for higher precision.
  8. Tag morphology drift backfill is out-of-band — the API records the disposition, but scripts/apply-tag-morphology-backfill.ts must be run manually against accepted flag IDs to apply the actual UPDATE.