The id-133 semantic-linter gate

The id-133 semantic-linter gate

Mechanic 4 of 5. The gate that decides whether a concept (or an extraction) may be written — closed vocabularies HARD-reject, open dimensions SOFT-WARN — and the single register that both sides share. Source of truth: scripts/cocoindex_pipeline/producer/validator.py (the concept-write gate, BI-13); scripts/cocoindex_pipeline/extraction.py (the extraction gate); id-133 specs/id-133-ontology-three-layer-pass/TECH.md; DR-020 (two-tier admission).

---

Two gates, one register

The same closed entity/relationship vocabulary gates two different write paths (id-133 TECH.md, “Gates”):

1. id-131 extraction writes — the Pydantic gate in extraction.py (the EntityMentionExtraction.entity_type / RelationshipExtraction.relationship Literals).
2. id-132 concept writes — the concept-frontmatter validator, validator.py:check_concept (BI-13), run before every declare_file.

Neither imports the other (collection-safety: extraction.py eagerly imports cocoindex), so the vocabularies are hand-mirrored and parity-guarded. The concept-write copy (validator.py:182-212) mirrors extraction.py:426-439 / 471-482, itself parity-guarded against lib/validation/schemas.ts:VALID_ENTITY_TYPES and the TS ExtractedRelationship union. A drift in any copy fails a parity test at build/CI.

The concept-write gate (BI-13)

check_concept (validator.py:496-524) runs every check and returns the full list of violations — it is deliberately not fail-fast, so the Pass-1/Pass-2 agent loop can surface every problem in one soft-error turn for model self-correction. validate_concept (:526-547) wraps it and raises ConceptValidationError (carrying all violations) at the gate boundary. No concept is written or published unless it passes.

What it checks:

Check	Function	Rule
Required keys	check_required_keys (:324)	type/title/description/timestamp/tags present; the four string fields non-empty. resource: is NOT hard-required (BI-12: required “where one exists”).
Concept type	check_type_membership (:343)	type ∈ {topic, product, company, certification, case_study} — the closed BI-4 set (ALLOWED_CONCEPT_TYPES, :110-112). metric/playbook are tags, not types.
Resource scheme	check_resource_scheme (:371)	when present, resource: matches one of the two canonical:// forms the producer emits (per-row anchor OR q_a_pairs query).
No stray pointer	check_no_stray_pointer (:426)	no Canonical uuid / canonical:// uri in any field other than resource:, nor in the body outside # Citations (BI-10).
Entity/relation ontology	lint_entity_relation_mentions (:458)	any supplied entity/relationship mention must be in the closed 12-entity / 10-relation ontology.

HARD-reject vs SOFT-WARN

This is the load-bearing enforcement line, and the id-133 pass was explicit that it must not move (id-133 TECH.md, “Enforcement-semantics invariant”):

• HARD-reject (raises) for the closed enums: concept type, content_type (trimmed), form_type/form_format, entity_type, relationship. A violation raises — the concept fails the write, the extraction row is refused. Example on the extraction side: _validate_content_type (extraction.py:518-529) reads the taxonomy snapshot and raises on an out-of-taxonomy value. On the concept side: validate_concept raises ConceptValidationError; the TS mirror parseConceptFrontmatter calls .parse() (not .safeParse()) so it throws a ZodError (concept-schema.ts:119-122).
• SOFT-WARN (never raises) for the open dimensions: primary_domain/primary_subtopic/secondary_classifications. The _surface_out_of_taxonomy_classification model-validator (extraction.py:531-580) bumps a counter and logs, but always returns self, never raises. An out-of-taxonomy domain is recorded, not rejected — these dimensions are client-extensible.

The rule of thumb: closed vocabulary → HARD-reject; open dimension → SOFT-WARN. The id-133 pass changed what is gated and where the extraction stamp points, not this split.

The open tag vocabulary is not a rejection list

RECOGNISED_FACET_TAGS (validator.py:141-143 — metric, dataset, playbook, reference, policy, capability) names the facets the producer treats as first-class, but check_concept does not reject a tag for being absent from it — a concept may still carry arbitrary short domain tags (BI-12, the tags: list is open). One alias is folded: methodology → playbook (FACET_TAG_ALIASES, :151; canonical_facet_tag, :154), so a bid-outcome methodology facet lands on disk as playbook.

The augmentation guard (detection only)

detect_citation_shrink (validator.py:666-687) compares a concept’s prior committed # Citations against a new draft and returns any citations the new draft drops — the “augment, not replace” guard. It is the single shared detection implementation; it does not itself refuse a write. Two call sites enforce on top of it: the Pass-2 write gate ({132.9}) and the git-sync 3-way reconcile ({132.12}). This exists because a Pass-2 web-enrichment must not silently shrink a concept’s record-grounded provenance.

Crucially the comparison is format-normalised: both sides are reduced to their citation TARGETS via citation_target (_citation_entries → _ordered_citation_entries → parse_citation_entry; validator.py:580-585, resource_uri.py:258-264) before the set-difference, so the legacy bare-path form and the new SPEC §8 numbered-link form of the SAME citation never falsely read as a shrink. The trailer itself is deterministically re-emitted at write time by normalise_citations_section (validator.py:635-663) — detection compares meaning, not surface text.

Where this sits in admission (DR-020)

Corpus admission is a two-tier gate. id-133 owns only the semantic tier — the linter over extraction and concept writes described here. The document admission gate (which bytes may enter at all) is {131.24}, a separate concern. Do not conflate the semantic gate (this page) with document admission.