TECH — {434.3} Two phases, one declare site, and the retirement of the collision machinery

TECH — {434.3} Hoist entity resolution before declaration

Task: id-434. Artefact: {434.3} TECH. Date: 11/08/2026 (S555). Reads: RESEARCH.md {434.1}, PRODUCT.md {434.2}; the D1–D11 record (id-434 Progress, S554), DR-147, DR-148. Paths: scripts/cocoindex_pipeline/flow.py, stage_5.py, entity_embedder.py, pair_resolver.py. All line numbers are main e24ee6199 (post-id-433 strip — they differ from {434.1}, which was written pre-strip).

Pin verification note. The S554 podcast-example adoptions were read from current upstream docs — shape guidance, not pin-verified API. §6 records what this spec verified against the installed 1.0.18 pin before relying on it. Everything structural below rests on a verified surface.

---

1. Architecture — the one structural move

Today the per-file component declares entity_mentions rows keyed on the per-document canonical, and _run_stage_5_resolution — a plain await in app_main’s frame (flow.py:4561), not a component, invisible to the engine — mutates those rows post-hoc.

Target shape (D1/D2/D6 + podcast adoptions, PRODUCT §3):

app_main
├─ phase 1  ingest_file (per file, memo=True — EXISTING, loses em declares)
│             declares: sd / qa / cc / er target states, as today
│             returns:  list[EntityMentionCandidate]     ← NEW return value
├─ phase 2a resolve_entities_for_type (per entity_type, NOT memoised — NEW)
│             owns no target states; writes entity_pair_resolutions via KhPairResolver;
│             reads/writes the record_embeddings entity-name cache (D10)
│             returns:  ResolvedEntities
└─ phase 2b declare_entity_mentions (ONE component for the corpus — NEW)
              owns every walked entity_mentions row, keyed on the resolved canonical;
              honours pins (relocated DR-105 carry-forward)

stage_5.py is deleted whole (§3). entity_embedder.py and pair_resolver.py survive as phase-2a collaborators.

---

2. Decisions

2.1 The fan-out returns values: mount_each → per-file use_mount + gather

mount_each returns a readiness handle only — no return values (verified at the pin: _internal/api.py, “a handle that can be used to wait until all processing units are ready”). Phase 2 needs the per-file candidate lists, so the fan-out moves to the upstream docs_to_knowledge_graph shape:

results = await asyncio.gather(*(
    coco.use_mount(
        coco.component_subpath("ingest_file", key),   # same subpath root as today
        bound_ingest_file, item, qa_target, sd_target, cc_target, er_target, re_target,
    )
    for key, item in source.items()
))

• Why not a side-channel collector: a memo hit skips the component body, so a collector would receive nothing for every unchanged file and phase 2 would resolve an empty name set on exactly the runs where stability matters most. use_mount replays the memoised return value on a hit — the only correct channel.
• component_subpath("ingest_file", key) reproduces mount_each’s ingest_file/<key> component paths (verified: component_subpath(*key_parts); mount_each builds child_path.concat(key) under the same subpath), so component identity and cleanup semantics are unchanged.
• Per-item containment is preserved (PRODUCT PI-8): bound_ingest_file’s swallow-and-log wrapper (flow.py:4370-4373) keeps returning None on a contained item failure; the gather filters Nones. em_target drops out of its parameter list (threading sites: flow.py:1885, 2054, 2068, 2121, 2135, 4384).
• Live-mode consequence, stated: mount_each auto-handles LiveMapView/ LiveMapFeed; a use_mount loop does not. The pipeline runs supervised catch-up walks (/walk), not live mode — no current behaviour is lost. If a live lane ever arrives, phase 1’s fan-out is the seam to revisit.

2.2 The transfer type

@dataclass(frozen=True)
class EntityMentionCandidate:
    source_document_id: uuid.UUID
    entity_type: str
    entity_name: str          # surface form, verbatim
    per_doc_key: str          # canonicalise_entity_name(entity_name) — post-id-433 key
    context_snippet: str      # DR-135: computed in phase 1, carried, never recomputed
    confidence: float
    source_span_start: int | None
    source_span_end: int | None

This is the ruled floor exactly (S554; PRODUCT §3). Phase 1 keeps the DR-135 admission check where content_text lives: _admissible_context_snippet (flow.py:2598-2606) runs before a candidate is emitted, and an unanchored mention is refused there — a refused candidate never crosses the transfer. The refusal log and its not-an-item-failure semantics are unchanged.

2.3 Phase 2a — per-type resolve subcomponents

One mounted child per entity_type present in the candidate set (podcast adoption (i)):

resolved = dict(await asyncio.gather(*(
    coco.use_mount(
        coco.component_subpath("resolve_entities", entity_type),
        _resolve_type_group, entity_type, sorted(names), pinned_by_type[entity_type],
    )
    for entity_type, names in names_by_type.items()
)))

• Plain @coco.fn — never memo=True (D5 / DR-147 clause 4; memo defaults to False at the pin, verified). Every fresh pair decision writes entity_pair_resolutions; the pair cache is the durable record and the memo is deliberately not a second one.
• The body calls resolve_entities (pin-verified signature) with:
  • embedder= the D10 cache-backed KhEntityEmbedder (§2.6);
  • resolve_pair=KhPairResolver(db_pool, op_id, entity_type) — unchanged (pair_resolver.py);
  • is_existing_canonical= membership in the seed set for this type (below);
  • existing_policy=ExistingCanonicalPolicy.PINNED (the pin-verified default).
• The seed set implements DR-147 clause 2 (groups grow, never rename), replacing the roster mechanism (D4). For type T:
  1. established canonicals — SELECT DISTINCT canonical_name FROM entity_mentions WHERE entity_type = $T (prior runs’ groups; the declared-key axis);
  2. pinned canonicals with effective type T (D1-C + D7): WHERE (metadata->>'curation_pinned') = 'true' AND COALESCE(entity_type_override, entity_type) = $T. No self-subtraction is needed: phase 2a runs before this run declares anything, so the table holds only prior-run state — the ID-81.9 workaround was an artefact of resolving after declaring, and it retires with the roster.
• Names fed to resolution are the per_doc_key values (the one post-id-433 key space, DR-140 clause 2 unchanged). Resolution stays partitioned by extracted entity_type — the S551 cross-type divergence is id-451/id-449 ground, not this task’s (PRODUCT §6).

2.4 Phase 2b — the declare component

One component, mounted after all 2a children return, receiving em_target and the full candidate + resolution + pin state:

1. Row build. For each candidate: canonical_name = resolved[entity_type].canonical_of(per_doc_key); id = uuid5(_KH_PIPELINE_DOC_NS, f"em:{source_document_id}:{canonical_name}:{entity_type}") (D2; same namespace constant, flow.py:1847).
2. Collapse, not collision. Candidates are grouped by (source_document_id, canonical_name, entity_type); the survivor is the deterministic pick max(group, key=lambda c: (c.confidence, c.entity_name)) — the bl-225 survivor rule as a pure function at build time, replacing a DELETE. Losers are logged (cocoindex.ingest.mention_collapsed), mirroring today’s honesty rule.
3. Pin carry-forward, relocated whole (D1-A; DR-147 clause 3). The two-pass declare plan at flow.py:2545-2677 ports structurally intact, now corpus-wide:
   • one pin fetch for the corpus (replacing per-doc _fetch_curation_pinned_mentions, flow.py:2949), predicate per §2.5;
   • a candidate row whose id matches a pinned id re-declares the stored row verbatim (canonical, entity_name, confidence, snippet, metadata, op_id — no re-stamp under full_reprocess);
   • unconsumed pins are re-declared anyway, and on a natural-key clash the pin wins and the candidate is dropped, with curation_pin_won_natural_key logged — keyed per (source_document_id, canonical_name, entity_type) now that the plan spans documents;
   • the best-effort pin-read failure mode (curation_pin_read_failed, flow.py:2540-2557) ports with it. The probe (pin → re-run → byte-identical row; clash log fires) remains the verification standard — pattern-survives-probe-verified, per the ratified guardrail.
4. _bump("postgres_upsert") per declared row moves here (from flow.py:2677).

Ownership note (upstream discipline): after this change exactly one component declares into em_target. Phase 1 keeps er_target per-document — that table’s reshape is id-435’s (§2.8).

2.5 D7 — the one surviving pin read matches DR-105

Post-reshape the pipeline has one pin-matching read: the phase-2b/2a pin fetch. It matches on (metadata->>'curation_pinned') = 'true' and the effective type COALESCE(entity_type_override, entity_type) (column verified on entity_mentions, squash baseline :2574). The three base-type reads the D7 ruling flagged (stage_5.py:113/187, flow.py:2974) all retire or relocate into this one predicate — no consumer re-derives it (DR-105).

2.6 D10 — entity-name embeddings persist in record_embeddings

KhEntityEmbedder (entity_embedder.py:67) becomes a read-through cache over the DR-036 single home:

• Row shape: owner_kind='entity_name' (new CHECK value — one migration, the DROP/ADD-constraint pattern of 20260712066000_id145_form_question_embedding_owner_kind.sql); owner_id = uuid5(_KH_PIPELINE_DOC_NS, f"entity_name:{name}") (the no-FK owner-kind precedent: concept already has no DB row); model = ENTITY_EMBEDDING_MODEL; embedding fits the existing vector(1024) column exactly (ENTITY_EMBEDDING_DIMENSIONS = 1024, entity_embedder.py:63-64).
• Semantics: cache, not engine target state. SELECT on hit; on miss embed then INSERT … ON CONFLICT (owner_kind, owner_id, model) DO NOTHING — the entity_pair_resolutions write pattern (pair_resolver.py), chosen for the same reason: the substrate must survive corpus changes and LMDB resets, which an engine-declared row would not (a name leaving the corpus would reconcile its vector away and defeat “only new names embed”).
• The embedder gains a db_pool constructor argument (phase 2a passes the same env-scope DB_CTX pool Stage-5 used). A cache-read fault degrades to a live embed + log, never a run failure.
• A per-entity_name partial HNSW index is not added now — the ranking term is id-452’s decision, gated on a real-tier run; this is plumbing + cost + substrate only.

2.7 Failure route — D8 wiring

Phase 2a and 2b awaits sit inside app_main’s existing try, re-wrapped exactly as today’s call site is (raise _EntityResolutionStageError(str(exc)) from exc, flow.py:4567) so _classify_stage_exception keeps attributing to entity_resolution_failed. Any phase-2 failure — including a single per-type subcomponent — reds the whole run. Per-type mounting buys failure attribution (the component path names the type), never containment. The id-415 walk-phase timers replace stage5_resolution with entity_resolution + entity_declare phases at the same observation points in app_main’s frame; the cocoindex.stage_5.resolved log event becomes cocoindex.entity_resolution.resolved with the same payload.

2.8 The id-435 seam

Phase 2b’s input state — candidates + resolved[entity_type] (ResolvedEntities: canonical_of / canonicals / groups, pin-verified) — is the resolved mention key-space id-435 will read. er_target declares stay per-document on the interim DR-140-clause-3 key (flow.py:2686-2689 comment); id-435 relocates them behind its own admission gate. Nothing in this task forecloses that: the transfer type and 2a return values are plain data id-435 can consume.

Likewise the ingest-once seam (DR-148 / id-450): phase 2b declares all walked mentions today (the corpus is all keep-and-watch). The promote step that will move ingest-once mentions out of engine ownership operates on rows after declare — nothing in the declare path assumes engine ownership is permanent. The seam is this: id-450’s promote must (a) exclude promoted rows from phase 2b’s pin fetch and seed reads only if it moves them out of entity_mentions, or (b) mark them so phase 2b stops re-declaring them. That decision is id-450’s; phase 2b keeps both open by reading its inputs from queries id-450 can scope.

---

3. Removal inventory (at e24ee6199)

Item	Site	Disposition
stage_5.py — whole module	548 lines	Deleted. Contains all four named mechanisms and both adjacent ones (D4): _select_run_entity_mentions (:80), _select_existing_canonical_roster (:118) + ID-81.9 self-subtraction, _select_prior_op_key_holders (:156), and _run_stage_5_resolution (:202-548) with the bl-225 collapse, the cross-op widened DELETE (:486-497) and the Step-6 UPDATE loop
Stage-5 import + call site	flow.py:153, flow.py:4530-4590	Import goes; call site replaced by phase-2 mounts (§2.3/2.4); _EntityResolutionStageError re-wrap kept (§2.7)
Per-doc em declare block incl. pin carry-forward	flow.py:2545-2677	Relocated into phase 2b (§2.4) — pattern intact, probe-verified
_fetch_curation_pinned_mentions	flow.py:2949	Relocated to the corpus-wide fetch with the D7 predicate (§2.5)
em_target threading through per-file components	flow.py:1885, 2054, 2068, 2121, 2135, 4384	Parameter removed; em_target passes to phase 2b only
Roster/prior-holder pin reads on base type	stage_5.py:113/187, flow.py:2974	Retire with their mechanisms / superseded by §2.5
Corpus-manifest Inv-14 cross-document constraint	docs/reference/testing/corpus-manifest.json:470	Retired: both forms in one document becomes the normal case (PRODUCT §5 probe 6) — fixture layout freed
Stage-5-bound tests	scripts/tests/test_cocoindex_stage_5_resolution.py, test_cocoindex_stage_5_crossrun_integration.py, test_cocoindex_stage_5_crossrun_guard.py, plus stage_5 references in conftest.py, test_cocoindex_curation_pinning.py, test_cocoindex_flow_live_ingest.py, test_cocoindex_flow_failure_mode.py, test_cocoindex_qa_dedup_proposer.py, test_disposable_pg_guard.py	Swept with the module: behaviour-level assertions (pin survival, stability, agreement) re-land against the phase-2 shape; mechanism-level assertions (roster, prior-holders, collapse DELETEs) die with their mechanisms — a test asserting a retired concept is not carried (DR-104/S515 discipline)

A surviving mechanism not on this table has no home in this spec: per id-434 AC 2, name its requirement and current source or the verdict is UNDECIDABLE and the owner rules.

---

4. Migration and the first run

Pre-launch posture (DR-093): correct structure, delete bad rows, no backfill.

1. One migration: add entity_name to the record_embeddings owner-kind CHECK (§2.6). No entity_mentions schema change — PK, natural key and columns are unchanged; only the derivation of id and canonical_name values changes.
2. Wipe mention rows: DELETE FROM entity_mentions (mock-tier, every row; pins on mock-tier rows drop with them — there is no pin worth preserving before the first real-tier walk).
3. Wipe the engine store (/cocoindex-state/lmdb) — the DR-146 precedent: the declare-site ownership moved, so the old per-file components’ em target-state records must not survive to be reconciled against the new shape. Cost: full memo loss and a full re-extract of the corpus — accepted; the phase-1 code change bumps every ingest_file fingerprint anyway (logic_tracking='full').
4. Re-walk supervised over the vendored Platform corpus — real tier for the AC referee run (PRODUCT §5), mock tier for CI determinism.
5. entity_pair_resolutions is untouched throughout — it is the archaeology and the stability substrate (D5); the unreachable alias-keyed rows stay as ruled.

---

5. Verification — probes to implementation

PRODUCT §5’s six probes, instrumented:

#	Probe	Instrument
1	Record-vs-DB diff empty	cocoindex show <app> --target-states (pin-verified CLI) diffed against SELECT id, canonical_name, entity_type, source_document_id FROM entity_mentions
2	Unchanged-corpus re-run	snapshot --target-states + table xmin before/after; run memo-warm and memo-cold
3	One-changed-doc re-run	xmin delta set == the changed document’s mentions, row level
4	Longer-spelling stability (DR-147 probe)	add doc with longer spelling; assert group canonical unchanged, membership grown
5	Pin probe	pin → re-run → byte-identical row; curation_pin_won_natural_key fires on a manufactured clash
6	bl-225 inversion	both CYE 14001 forms in ONE document → two rows collapse to one declare, zero errors

Stable-ID discipline (PRODUCT PI-12) is asserted inside probes 2–4: every id observed twice is byte-identical. items_processed appears in no probe (guardrail 1).

---

6. What was verified against the 1.0.18 pin (vs adopted as shape)

Pin-verified this pass (installed package, signatures inspected):

• resolve_entities(entities, *, embedder, resolve_pair, is_existing_canonical=None, existing_policy=PINNED, on_resolution=None, max_distance=0.3, top_n=5); ExistingCanonicalPolicy = {PINNED, PREFERRED}; ResolvedEntities.canonical_of/ canonicals/groups/to_dict — everything §2.3 relies on, including PINNED as default.
• @coco.fn memo defaults False; memo_key, version, logic_tracking='full' params exist as assumed.
• mount_each returns a readiness-only handle (source read) — the finding that forces §2.1.
• component_subpath(*key_parts) multi-part form — what makes the ingest_file/<key> path reproduction exact.
• use_mount returns the child’s value; mount returns a handle (_internal/api.py:231/355).

Adopted as shape, with no further pin surface to verify: the three-phase per-type-resolution layout itself (podcast example) — its mechanics reduce entirely to the verified primitives above; nothing else from the example’s API is used.

on_resolution(ResolutionEvent) is noted as available for phase-2a telemetry; this spec does not require it.

---

7. Not covered here, with owners

The D11 vocabulary opening (id-449 — the CHECK does not move); the entity grain family and the standard/certification silent zero (id-451); the ingest-once promotion mechanics beyond the seam statement in §2.8 (id-450); relationship endpoints (id-435); the embedding ranking term and any new HNSW index (id-452); Inv-7 (separate live bug — both hypotheses remain untested and this reshape promises nothing about it).