Skip to content

PRODUCT — {434.2} Mentions are declared once, on the resolved canonical

PRODUCT — {434.2} Hoist entity resolution before declaration

Section titled “PRODUCT — {434.2} Hoist entity resolution before declaration”

Task: id-434. Artefact: {434.2} PRODUCT (of {434.1} → {434.2} → {434.3}). Date: 11/08/2026 (S555). Reads: RESEARCH.md {434.1}; the ratified ruling record — DR-147, DR-148, and the full D1–D11 set (id-434 Progress, S554; board persisted at reports/s554-entity-architecture-review.html). Executes: DR-140 clause 1, DR-147, DR-148, DR-105, DR-135, DR-093, DR-036 (D10). Interlocks: id-433 (landed baseline), id-435 (holds on this), id-449 (gates D11 — not this task), id-450, id-451, id-452, id-420.


An entity mention is a provenance claim: this document, read on this run, mentions this thing. The product is only as good as a reader’s ability to trust that claim across time — across re-runs, corpus growth, deploys, and human corrections.

Today that trust is structurally unavailable. The per-document component declares mention rows keyed on a pre-resolution name, and Stage-5 then UPDATEs and DELETEs those same rows — mutating a component of the table’s natural unique key on rows the engine’s tracking record still describes under their old values. Four mechanisms exist solely to absorb the resulting collisions, and the engine’s record and the database disagree by construction after every resolving run. The worst case is not exotic: an LMDB volume reset on deploy pushes the whole corpus through the mutation pass at once (RESEARCH §6).

The defect is the divergence, not the collisions (DR-140’s rejected alternative: adding a resolved_canonical column stops collisions and leaves the divergence intact). After this task:

A mention row is declared exactly once, keyed on the resolved canonical, by a component the engine tracks — so the engine’s record and the database are the same state, and what the database says is what the pipeline meant.

Everything else in this spec is a consequence of that sentence.


Statements a reader must be able to make about the pipeline after this task. Each names its requirement and that requirement’s current source (DR-139 discipline).

Mentions are declared after resolution, keyed on the resolved canonical, by a phase-2 component. No code path updates or deletes an engine-declared mention row post-hoc. Two surface forms of one entity in one document is the normal case — a dict collapse at resolution time, not a collision to absorb (upstream’s stated shape, RESEARCH §2.2).

Requirement source: DR-140 clause 1 (verbatim: “phase one declares documents and collects names; phase two resolves, then declares mentions keyed on the resolved canonical”); ruling D1.

PI-2 — Identity and the natural key agree

Section titled “PI-2 — Identity and the natural key agree”

The surrogate id is uuid5(ns, "em:{sd_id}:{resolved}:{type}") — a function of the same facts as the natural key (canonical_name, entity_type, source_document_id), so the primary key and the natural key cannot disagree. No backfill: pre-launch mention rows are mock-tier; they are dropped and re-walked (DR-093).

Requirement source: DR-147 clause 1 (ruling D2-A); RESEARCH OQ-3 resolved.

PI-3 — The stability rule: groups grow, they are never renamed

Section titled “PI-3 — The stability rule: groups grow, they are never renamed”

The resolved canonical is a function of the accumulated decision recordentity_pair_resolutions, curation pins, and previously established canonicals seeded into is_existing_canonical under ExistingCanonicalPolicy.PINNED — never of the current run’s name set alone. A re-run over a grown corpus may add members to a group but never rename one. A rename is a curation act; a resolution-behaviour change that would rename a group must surface as a curation proposal, not as pipeline output.

Requirement source: DR-147 clause 2 (ruling D3-A); the failure mode it forecloses is RESEARCH §6’s “identity churn indistinguishable from data loss”. Probe (stated AC): add a document introducing a longer spelling of an existing entity (KhPairResolver prefers longer on a cache miss), re-run — the group’s canonical is unchanged.

PI-4 — Human curation survives every run

Section titled “PI-4 — Human curation survives every run”

A curator-pinned mention is re-declared verbatim on every run — canonical, entity name, confidence, snippet, metadata and op_id byte-preserved, no re-stamp under full_reprocess — and on a natural-key clash the pin wins and the fresh candidate is dropped, with cocoindex.ingest.curation_pin_won_natural_key logged. The carry-forward pattern relocates with the declare site into phase 2, and pins additionally seed is_existing_canonical, so a pin is honoured at resolution time as well as declare time.

Requirement source: DR-105; DR-147 clause 3 (ruling D1 A+C — guardrail 2 ratified as “the pattern survives, probe-verified”, not “the lines are immovable”). Verification standard: the probe, never inspection — pin → re-run → byte-identical row; the clash log still fires.

PI-5 — The engine’s record and the database agree at end of run

Section titled “PI-5 — The engine’s record and the database agree at end of run”

cocoindex show <app> --target-states diffed against the entity_mentions table is empty after any run. Today the diff is non-empty by construction after any Stage-5 UPDATE; that this diff closes is the task’s central AC.

Requirement source: id-434 AC 3; DR-140’s stated point (“the two states stop diverging”).

PI-6 — The collision machinery is removed, not disabled

Section titled “PI-6 — The collision machinery is removed, not disabled”

All four named mechanisms go — _select_prior_op_key_holders, the cross-op widened DELETE with curation re-assertion, the bl-225 collision-collapse block, the Step-6 UPDATE loop — and both adjacent mechanisms go with them (ruling D4, requirements and current sources named on the board, so neither is UNDECIDABLE):

  • _select_existing_canonical_roster + the ID-81.9 self-subtraction — its real requirement (cross-run stability) is re-served by PI-3’s seeding; the mechanism is not.
  • the curation-pin write-back exclusion — its threat, the mutation pass, ceases to exist.

Requirement source: id-434 AC 2; ruling D4; DR-139 (an invariant authored by the task that built the mechanism is the weakest possible authority).

PI-7 — Every fresh pair decision is durably recorded

Section titled “PI-7 — Every fresh pair decision is durably recorded”

The phase-2 resolve component is not memoised. entity_pair_resolutions is the authoritative durable record of resolution decisions: it survives an LMDB reset, and it is the only archaeology of the real extractor. A memo hit that silently skips pair-cache writes is the rejected alternative, by name.

Requirement source: DR-147 clause 4 (ruling D5); RESEARCH OQ-5 resolved. The accepted cost is carried verbatim: cache rows keyed on formerly alias-applied names go unreachable post-id-433; the one-off re-ask is accepted and the key-space is not repaired.

PI-8 — Resolution failure is loud, and containment stays per-item in phase 1

Section titled “PI-8 — Resolution failure is loud, and containment stays per-item in phase 1”

A phase-2 resolution failure reds the whole run. Per-item containment (bound_ingest_file’s per-document error routing) remains a phase-1 property; no flow-scope analogue is invented to swallow phase-2 errors.

Requirement source: ruling D8 (RESEARCH OQ-7 resolved); DR-047’s “silent success is forbidden” posture.

PI-9 — The evidence chain is computed where the evidence is

Section titled “PI-9 — The evidence chain is computed where the evidence is”

The DR-135 context_snippet is computed in phase 1, where per-document content_text exists, and carried in the transfer type — never recomputed in phase 2. A mention that cannot be anchored is refused, the refusal is logged per mention and is not an item failure, and a curator-pinned row is never subject to the rule. All three DR-135 properties survive the relocation.

Requirement source: DR-135; S554 board A2 (four live consumers of the snippet named); RESEARCH §1.6, §5.3.

PI-10 — Ownership follows the retention class; this task builds the walked shape

Section titled “PI-10 — Ownership follows the retention class; this task builds the walked shape”

Engine-owned declarative mention rows are the correct ownership for keep-and-watch and live-connected sources — the source persists, so reconciliation is right. For ingest-once sources, mentions must cross the promotion boundary so extracted knowledge outlives the source; for external-referenced, no walk, no mentions from bytes. This task designs and builds the walked shape only, and must leave the ingest-once seam named, not built — the phase-2 component may not foreclose a promote step moving mention rows out of engine ownership.

Requirement source: DR-148 (ruling D6-revised); executed as verify-and-extend by id-450 (id-138 is archived with the contract built — {138.11}/{138.16} — and is not reopened). Today’s corpus is all keep-and-watch, so this task is not blocked on id-450.

PI-11 — Mention confidence is telemetry, never trust

Section titled “PI-11 — Mention confidence is telemetry, never trust”

entity_mentions.confidence survives as extraction telemetry only. It must never surface as a bundle trust signal — the v0.2 trust vocabulary is the verified event list (distinct from the retiring OKF concept-level confidence, id-428).

Requirement source: S554 transfer-floor ruling; id-428’s trust-slot migration.

Same inputs always yield the same id, at every grain: document ids, mention ids, component paths. Re-running never duplicates. This is an explicit AC, not an emergent property.

Requirement source: S554 podcast-example adoption (ii); it is also what makes PI-2 + PI-3 testable at all.

PI-13 — Pin reads match the ratified predicate

Section titled “PI-13 — Pin reads match the ratified predicate”

Every pipeline-side pin read matches DR-105’s effective-type predicate — COALESCE(entity_type_override, entity_type) — not the base column. The base-column reads at stage_5.py:129/203 and flow.py:3021 were ruled a live gap (D7), aligned in {434.3}; the mechanisms among them that PI-6 removes take their reads with them.

Requirement source: DR-105 (“no consumer may re-derive its own pin predicate”); ruling D7 (RESEARCH OQ-2 resolved).

PI-14 — Entity embeddings are persisted in the single embeddings home

Section titled “PI-14 — Entity embeddings are persisted in the single embeddings home”

The entity embeddings computed for resolution are persisted under DR-036 (record_embeddings; no permanent inline-vector exceptions) inside this chain — plumbing, cost and substrate. What they rank is not decided here: the ranking-term decision is id-452, gated on a real-tier run, with id-432 the interim entity consumer.

Requirement source: ruling D10-revised; DR-036.


Ruled, not open (D6 + podcast adoptions; ownership granularity — RESEARCH OQ-6 — closed):

PhaseComponent(s)OwnsEmits
1 — per documentthe existing memoised per-file component (mount_each)document-scoped target states, as todaythe transfer record, per mention candidate
2a — resolveper-entity_type mounted resolution subcomponents (not memoised)no target states — decisions land in entity_pair_resolutionsresolved canonical per name
2b — declareone mention-declare component for the corpusevery walked entity_mentions rowrows keyed per PI-2, pins honoured per PI-4

The transfer type floor (phase 1 → 2, ruled): at minimum (source_document_id, entity_type, entity_name, per_doc_key, snippet, confidence, spans). The snippet is PI-9’s carried evidence; the floor is a floor — {434.3} finalises the type, it does not shrink it.

Per-type subcomponents buy concurrency and finer failure attribution; they do not soften PI-8 — any subcomponent failure still reds the run. The podcast example was read from current docs, not the pin: the shape is adopted here; {434.3} verifies every API surface it implies against the installed 1.0.18 pin before relying on it.


ClaimBacked by
”This document mentions this entity — here is the passage.”PI-9: an admissible snippet, computed from the document’s own text, or the row does not exist
”This id will still name this mention next run.”PI-2 + PI-3 + PI-12: identity is a function of stable facts, and the canonical cannot be renamed by a re-run
”The group grew because the corpus grew.”PI-3: growth is the only pipeline-initiated change; renames are curation acts
”My correction stuck.”PI-4: the pin survives every run, byte-identical, and wins clashes
”What the DB holds is what the engine did.”PI-5: the target-state diff is empty
”If resolution broke, the run says so.”PI-8: fail-loud; no partially-resolved corpus presented as complete

What the pipeline must never do: rename a group as a side effect of a re-run; present a mention with no anchored evidence; report a run green when resolution failed; let engine reconciliation touch a promoted (post-boundary) mention row once id-450 lands the ingest-once seam.


The referee is a real-tier supervised extraction over the vendored Platform corpus (scripts/cocoindex_pipeline/fixtures/platform-corpus/). Mock tier is CI determinism only — it bypasses the extraction prompt and can referee nothing about product quality (standing owner posture, S554: “build the product, not passing tests”).

The acceptance probes, all on instruments that already exist (RESEARCH §6):

  1. Record-vs-DB diffcocoindex show --target-states vs the table: empty (PI-5).
  2. Unchanged-corpus re-run — nothing moves (xmin + target-states snapshot; run once memo-warm and once memo-cold, so “no work” and “idempotent work” are distinguished).
  3. One-changed-document re-run — the set of rows whose xmin advanced is exactly the changed document’s mentions, asserted at row level, not component level.
  4. Add-a-longer-spelling probe — PI-3’s stated AC: the group grows; the canonical does not change.
  5. Pin probe — PI-4’s standard: pin → re-run → byte-identical row; curation_pin_won_natural_key still fires on a clash.
  6. The bl-225 fixture inverts — both forms of CYE 14001 in one document is the normal case; the corpus-manifest constraint forcing the pair across two documents becomes unnecessary and is retired with the mechanisms.

items_processed is not evidence for any of these — its counter sits inside the memoised body (guardrail 1).


In scope: the two-phase reshape (§3); removal of all six mechanisms (PI-6); the identity change and drop-and-re-walk (PI-2); pin relocation + resolution-time seeding (PI-4); the D7 pin-read alignment (PI-13); entity-embedding persistence under DR-036 (PI-14); the acceptance probes (§5).

Out of scope, with its owner:

  • Inv-7 (stage-5-op-id-memo, the nightly’s last red) — a separate live bug; both hypotheses untested; the reshape may subsume it but promises nothing (guardrail 1).
  • The entity_type vocabulary opening (D11) — direction is to open it, gated on id-449; the DB CHECK, the extractor Literal and the TS union do not move before that spike’s verdict. This spec assumes the closed set.
  • The entity grain family and the silent standard/certification zero — id-451 (D9-revised, straight to target, no transitional type-set). The zero persists until id-451 lands, by owner acceptance; nothing here quick-fixes it.
  • The ingest-once promotion mechanics — id-450 (PI-10 names the seam only).
  • The embedding ranking term — id-452, gated on a real-tier run.
  • Relationship endpoints from resolved mentions — id-435. This task leaves the seam: whichever phase-2 component owns the resolved mention key-space is where id-435 reads. er_target sits in the same per-document ownership position em_target does today; its reshape is id-435’s, not this task’s.

None UNDECIDABLE. Every open question in {434.1} (OQ-1–OQ-8) was ruled on the S554 board (D1–D11); the rulings are recorded in id-434’s Progress entry and bound above. Two measurements are deliberately deferred, with owners:

  • Whether mention rows sit inside id-138’s survives-the-engine contract today is presumed-not, not measured — id-450’s first AC.
  • Whether opening the entity_type vocabulary improves real-tier extraction — id-449’s measurement; D11 waits on it.