S436 — Corpus-Admission Policy: does the ontology gate what joins the corpus?

Task: frame the owner’s fundamental question — do we gate what joins the corpus, or can anything be added? — against the ratified OKF direction and the live ingest entry points. Read-only research; this artefact is the only write. Date: 2026-07-02.

Owner’s framing (verbatim intent): “The folder-drop UI was created to enable users to add documents to the corpus, but stepping back, we need to establish the most fundamental part of the system… helping SMBs to start getting their data and documentation in order, to benefit from working with AI. We don’t wish to store clients’ raw documents, but we DO need to decide if we gate what joins the corpus or whether anything can be added — we created the ontology as the starting point for this.”

Source docs status: BOTH phase-b docs LOCATED and READ IN FULL (not reconstructed) in the private docs-site:

${KH_PRIVATE_DOCS_DIR}/src/content/docs/themes/canonical-pipeline/phase-0-investigation/phase-b-prerequisite-1-onthology-pipeline.md (667 lines, S234 audit)
${KH_PRIVATE_DOCS_DIR}/.../phase-b-prerequisite-1-onthology-pipeline-feedback-investigation.md (945 lines, S235 Wave 1)

A. What the phase-b docs established — the ontology’s intended role as corpus-admission foundation

The two phase-b docs mapped every piece of KH data onto Jessica Talisman’s six-layer ontology pipeline (Controlled Vocabulary → Metadata Standards → Taxonomy → Thesaurus → Ontology → Knowledge Graph). Two findings are the whole point for this decision:

The ontology is the platform’s foundation, and it was half-built. KH scattered its controlled vocabularies across ~29 places (DB enums, tables, TypeScript tuples, Zod schemas) with no single register that lists every allowed term, defines it, and says whether it is core-platform or client-defined (doc-1 §3.1 “vocabulary scatter”). KH also jumped straight from Layer-1 enums to Layer-5 app schema, skipping the formal Metadata/Taxonomy/Thesaurus products (doc-1 §3.2 “layer-skip”). The doc’s thesis, quoting Talisman: “Introducing logic becomes nearly impossible when underlying data itself is not logically structured.” The ontology register is the artefact that makes the data logically structured — the starting point the owner is referring to.
The core-vs-client dimension is where admission lives. Doc-2 §6 classified every vocabulary and entity as Core (ships with the platform, cannot be deleted), Client-defined (empty by default, clients add their own), or Hybrid (ships with a baseline that clients can extend), using a provenance enum (core / client / recommended) already proven on taxonomy_domains. This is precisely the “what may legitimately join” question: an admission policy is nothing more than a rule that says these types/shapes are allowed in, these are not.

The critical update since those docs (May 2026): the June OKF work (okf-record-model-v3.md, okf-v3-owner-feedback.md, ratified round-4) supersedes content_items with a three-layer model — L-raw (the client’s raw files, client-owned, never stored by us) → L-records (our DB: source_documents, q_a_pairs, reference_items, chunks, entity graph) → L-concepts (the distilled OKF knowledge bundle, client-owned git repo). Against that model:

The corpus is already ratified as GATED. okf-v3-owner-feedback.md §B (verbatim): “The corpus is GATED. Original + ongoing intent: gate the source_documents that get ingested so (i) they are authoritative sources, and (ii) we help businesses prepare/organise their data to create and upload more authoritative sources. The ontology pipeline was ‘phase 0’ of enabling this capability.” The lineage synthesis confirms GATE-KEEP was ratified over ingest-all-then-sort. So “anything can be added” is already off the table; the open question is the shape of the gate.
The ontology is explicitly named as the gate mechanism. okf-v3 §H: “Ontology as semantic linter (allowed_types / allowed_relations kept in sync; governance for what gets added to the OKF directory AND what gets extracted).” This is exactly what id-133 is building right now (in-progress): a 34-entity-type (12) + 35-relationship (10) register with per-value core-vs-client provenance, plus an OKF concept-frontmatter validator, “gating BOTH Task-B extraction AND Task-A concept writes” with HARD-reject vs SOFT-WARN semantics.

Live ingest entry points (code-verified): (1) corpus walk over the mounted gated dir COCOINDEX_SOURCE_PATH — L-raw; (2) feed-URL publish → nudge /walk → cocoindex → reference_items (lib/intelligence/pipeline.ts); (3) folder-drop upload UI → /api/ingest/folder-drop → lib/upload/folder-drop.ts → POST {COCOINDEX_WORKER_URL}/stage then /walk — this is the broken path; (4) app-side upload /api/upload — auth-gated, writes Supabase Storage + a DB row, never touches cocoindex /stage//walk; (5) manual form upload (/api/procurement/.../templates) — app-side, cocoindex-free (DR-014); (6) manual create /api/items + MCP create — born-digital. So there are two would-be corpus on-ramps for a user’s own documents: the broken cocoindex folder-drop (Path B), and the working app-side upload (Path A).

B. Corpus-admission options, honestly compared

The gate has two tiers, and it matters to keep them apart:

Document-admission tier — which source documents are allowed to become part of L-raw / the corpus at all (who may upload, authoritative-source criteria, provenance/dedup). Today this is operator curation only (“we define the source-doc structure + Q&A format for the first client”).
Semantic-admission tier — once a document is in, what may legitimately be extracted from it and written (allowed entity types/relations into L-records; allowed concept shapes into L-concepts). This is the id-133 ontology linter — already in build.

The three options below differ only on the document-admission tier. The semantic tier (id-133) is common to all of them.

Option 1 — Operator-gated only (strictest GATE-KEEP reading). No self-serve upload at all; the corpus is curated exclusively by us/admin arranging the mounted corpus dir. Fit with SMB mission: poor — it serves “authoritative sources” but abandons the owner’s explicit half-(ii), “help businesses… upload more authoritative sources.” Fit with ratified GATE-KEEP: maximal. Folder-drop: retire it.
Option 2 — Privileged-user upload THROUGH an admission gate (recommended). Admin/editor users can add documents, but every upload passes a gate: a document-admission check (authoritative-source criteria + provenance) at the app entry point, and then the id-133 ontology linter on what gets extracted/written. Fit with SMB mission: strongest — it is the literal realisation of okf-v3 §B’s two-part intent (“authoritative sources” and “help businesses upload more of them”). Fit with ratified GATE-KEEP + Pass-2 authoritative-corpus discipline: full — a gate is curation, not prohibition; Pass-2 enrichment already constrains itself to the gated authoritative corpus. Folder-drop: re-route it through the app-side gated path (Path A), retire the broken /stage leg.
Option 3 — Open upload (anything joins). Any file a user drops becomes corpus and gets AI-extracted. Fit with SMB mission: superficially “easy on-ramp” but it destroys the core value (“high-quality, structured, authoritative data”) and burns extraction tokens on noise. Fit with ratified GATE-KEEP: directly contradicts it. Folder-drop: keep as-is (fix /stage, no gate) — but this is a corpus-injection vector (s436-infra Q2). Not viable.

C. Folder-drop Path B disposition under each option

Path B is a confirmed live break, not cosmetic (s436-infra-truth-and-posture.md Q2): the folder-drop UI POSTs {COCOINDEX_WORKER_URL}/stage from Vercel, but /stage is deliberately not routed by Traefik (Inv-13, kept loopback-only because it accepts arbitrary bytes straight into the corpus dir — a corpus-injection seam). So the stage-leg 404s at the edge and Path B cannot complete in the deployed split topology.

Under Option 1: Retire Path B (and the whole folder-drop UI). Corpus changes go through operator curation of the mounted dir. Simplest; abandons self-serve.
Under Option 2 (recommended): Retire the Path B /stage transport; re-route folder-drop onto the app-side gated path (/api/upload shape — auth-gated, no /stage dependency). The upload lands a source_document (not the eliminated content_items), passes the document-admission check, and is extracted under the id-133 linter. /stage stays loopback-only, test-fixture only, never public.
Under Option 3: Fix the /stage transport and add nothing in front — i.e. expose a public bearer-only staging route. Rejected: reintroduces the corpus-injection vector the loopback design exists to prevent.

Note: Path A (/api/upload) today writes a content_items row, and content_items is being eliminated by id-131. So “re-route via Path A” is not literally free — Path A must be reworked to land source_documents under the OKF model. That rework is in-flight anyway (id-131 / DR-012 source-doc route re-point), so Option 2 rides an existing workstream rather than opening a new one.

D. Recommendation

Adopt Option 2 — privileged-user upload through a two-tier admission gate. In plain terms: let approved client users add documents, but nothing joins the corpus unmediated — every upload is checked for being an authoritative source, and everything the AI extracts from it is validated against the ontology. This is the only option that honours both halves of the mission the owner stated: it keeps the data high-quality and authoritative (curation discipline, already ratified), and it gives SMBs the on-ramp to “get their data in order” by contributing more authoritative sources — which okf-v3 §B names as the original intent. Option 1 quietly drops that on-ramp; Option 3 quietly drops the quality bar. Option 2 is also the smallest departure from what is already being built (id-133 is the gate’s semantic tier) and already broken (folder-drop needs fixing regardless).

What it implies for id-133 scope. id-133’s semantic linter (allowed_types/allowed_relations register + concept-frontmatter validator, HARD-reject/SOFT-WARN) is the admission gate — but only its semantic tier: it governs what may be extracted and written, not which documents may be uploaded. So: yes, the semantic linter is the corpus-admission gate for extracted knowledge, and id-133 does not need to expand to cover document admission. The document-admission tier (who may upload + authoritative-source criteria at the entry point) is a distinct, small piece of scope that belongs with the app-side upload path / id-131 source_documents rework, not with id-133’s ontology register. Recommend recording this explicitly so id-133 is not over-scoped: id-133 owns the semantic gate; a new thin slice (or an id-131 sub-item) owns the document gate + the folder-drop re-route. The one thing id-133 should confirm it carries is that the register’s per-value provenance (core/client) is what lets a client legitimately widen the allowed set later — that is the mechanism by which “admission policy” becomes client-configurable without a code change.

What it implies for the deployment-architecture doc’s corpus section. Describe the corpus as gated at two tiers: (a) document admission at the app-side upload entry (authoritative-source curation; privileged users only), and (b) semantic admission via the ontology linter during extraction into L-records and concept writes into L-concepts. State plainly that folder-drop Path B is retired in favour of the app-side gated on-ramp, closing the latent /stage break (s436-infra Q2), and that /stage remains loopback-only test-fixture plumbing that is never publicly routed. This also aligns the doc with the already-true forms story (DR-014): forms are app-side end-to-end; under Option 2 the corpus document on-ramp becomes app-side and gated in the same spirit.