DR-143: An unpublished source_document may not back or be cited by a concept

DR-143 — An unpublished source_document may not back or be cited by a concept

Context

Found by {427.9} while building the corpus census, and re-verified independently by the Coordinator before the ruling. Two of the producer’s source_documents reads — _SQL_SOURCE_DOCUMENTS_BY_IDS and _SQL_SOURCE_DOCUMENTS_BY_FILENAME_PATTERNS in scripts/cocoindex_pipeline/sources/l_records.py — carry no publication_status predicate, while every q_a_pairs read filters to published at eleven measured sites. Four of the six built-in grains use those reads.

The consequence is that a concept can today be drafted from, and cite, a document that has never passed the knowledge-admission gate DR-025 makes authoritative (“record promotion + dedup/confidence; concept linter + publish gate — human review is the launch posture”).

{427.9} filtered its own coverage queries to published, so the census arithmetic is internally sound. That left the census and the read disagreeing about which documents a concept may be built from — the census counting a published-only corpus while the read draws from a wider one.

Decision

Filter the reads. Both pattern-matched source_documents queries gain publication_status = 'published'. An unpublished document is not a corpus unit and may neither back a concept nor appear in its sources[].

The reads are the outlier, not the census: TECH §2.1’s ratified residual anti-join already defines the corpus as published-only, and the census was written to that definition.

Alternatives Considered

Widen the corpus definition so unpublished rows count as units, and change the census and anti-joins to match. Rejected: it would oblige {427.10}’s residual grain to mint concepts for exactly the records DR-025’s gate has deliberately withheld, inverting the gate’s purpose while appearing to improve coverage.

Defer behind the pre-launch bundle rebaseline. Rejected, and the reasoning is worth preserving because it separates this from its sibling question. A full rebaseline dissolves {427.8}’s WQ-1 — no shipped body survives to carry a wrong-grain grounding. It does not dissolve this one: an unfiltered read pulls unpublished documents into a freshly rebaselined bundle exactly as readily as into the old one. This is a code fix or nothing.

Consequences

This changes what a bundle contains, which is why it is a decision and not a patch. A concept currently backed by an unpublished document loses that backing, and some may lose their only backing — falling either out of the bundle or to {427.10}’s residual grain.

The measurement comes before the filter. Executed by tasks/id-427.md {427.15}: how many source_documents rows are unpublished, how many concepts are backed by at least one, and how many would lose their only backing. That third number decides whether this ships quietly or needs a migration note.

One assumption in this ruling is untested and must be confirmed in the doing: that publication_status on source_documents IS the DR-025 gate, rather than a different lifecycle sharing the column name. DR-025 describes a binding gate and a knowledge-admission gate as two distinct things. If they differ, {427.15} escalates rather than proceeding.

{427.9} deliberately kept the pair queries’ unfiltered inner document subquery on the ground that “it is the pair that must be counted, on its own publication status”. {427.15} must assert that reasoning still holds after this change, or record that it does not.

considered is unchanged (the census already filtered); routed may fall and unrouted may rise. {427.10}’s “zero unrouted” acceptance must still hold — if it does not, that is a finding about the residual grain, not grounds to weaken this filter.

Amendment (S550) — the untested assumption was measured FALSE; the ruling stands on the other ground

The ruling above flags one untested assumption: “that publication_status on source_documents IS the DR-025 gate, rather than a different lifecycle sharing the column name”, and instructs {427.15} to escalate rather than proceed if they differ. They differ. It escalated. The filter shipped anyway, on a different ground, and this amendment records why — so the false ground is not re-derived later.

What was measured (S550, read-only senior review + independent Coordinator re-check). source_documents.publication_status is neither of DR-025’s two gates. It is the content_items publication-approval lifecycle, ported to source_documents by a column-placement rule — 20260707210000_..._content_items_residue.sql:29 says “mirroring content_items 1:1” — whose carrier spec (publication-approval-gate-spec.md, S220, pre-reframe) is archived and no longer in the repo. lib/governance/publication-transitions.ts still declares itself the source of truth for content_items.publication_status. Both of DR-025’s gates have named homes elsewhere (reference/deployment-architecture.md §6): the binding gate is retention_class/origin_type, and knowledge admission is record promotion plus the id-133 ontology linter, whose ratified reference usage is lib/q-a-pairs/promote-corpus.ts (id-408).

Two independent checks of the load-bearing evidence:

• q_a_search does not reference source_documents at all (pg_get_functiondef, executed). The column’s own COMMENT justifies it as “inline-hot read-path lifecycle field (q_a_search filters publication_status on every query)” — a requirement that is measurably a q_a_pairs requirement, not this column’s.
• reference/entity-glossary.md:18 states for the sibling evidence entity that “R2’s sign-off gate governs records … a reference item has no publication_status and no promotion step … That is correct, not incomplete.” (Weighed honestly: the same line also calls reference items and source_documents “separate first-class entities discriminated by lifecycle”, so this is an argument by analogy, not ratified identity. The rest of the chain does not depend on it.)

Therefore this clause of the Decision above is WRONG as written and must not be cited: “a document that has never passed the knowledge-admission gate”. An unpublished source_document has not failed DR-025’s admission gate, because that gate does not live on this column.

The ruling itself stands, on the ground it did not need to borrow: a read and the anti-join that complements it must agree about what the corpus is, or {427.10}’s “zero unrouted” acceptance is computed against two different corpora. TECH §2.1’s ratified residual anti-join and _SQL_CENSUS_CORPUS_TOTALS both define the corpus as publication_status = 'published'; the reads were the outlier. That is sufficient and is independent of what the column means. Under the S550 owner rulings (DR-145) it is also independently justified: the column is ratified as a document-level review gate, and a document that has not passed its own review should not back a concept.

Why this amendment exists at all. Left standing, the false ground becomes the register’s authority for “document publication IS the knowledge-admission gate” — the citation a future wave would use to build a document-level approval queue in front of extraction, reacquiring DR-020’s collision problem. Same shape, and the same remedy, as this register’s S548 amendment to DR-027.

Amendment (S550) — executing this ruling required filtering the version fingerprints too

Not anticipated by the ruling, and it introduced a live regression before it was caught by an adversarial audit of the executing commit.

source_documents has no updated_at trigger (verified against pg_trigger: only trg_coerce_empty_classification_to_null and trg_record_lifecycle_mint_source_document), and no application path sets it on a publication change. Before this ruling the reads were unfiltered and so were the four *_VERSION content-fingerprint queries — a matched pair: a status flip changed neither what a concept read nor its content_version.

Filtering the reads alone broke that pairing. Publishing a document then changed what a concept’s read returned without moving its fingerprint, so cocoindex never re-drafted it — a published document silently failed to enter the concept it belongs to, which is this ruling’s own requirement failing in the mirror direction. Reproduced on the company and certification grains before the fix.

So the ruling’s scope is wider than “the two reads”: every query that decides what a concept is built from, enumerates its keys, or fingerprints its content must share one corpus definition. Executed as four reads/enumerations plus four *_VERSION queries (predicate in the ON clause for the LEFT JOINs, preserving outer-join semantics), and the coverage query’s inner document subquery — which resolves {427.17}’s carried UNDECIDABLE: it did not overcount before, and would have begun overcounting the moment the read was filtered.

The general shape, worth carrying: filtering a read is not a local change when something else derives a cache key from the same rows.