Stage-5 cross-run canonical stability — RESEARCH (ID-81.1)

Stage-5 cross-run canonical stability — RESEARCH (ID-81.1)

Spec slug: ID-81-canonical-stability Subtask: ID-81.1 (RESEARCH, precedes {81.2} PRODUCT → {81.3} TECH → {81.4} PLAN per Q-PLANNER-2 fresh-per-Subtask discipline) Parent Task: ID-81 — Stage-5 cross-run canonical stability (is_existing_canonical seeding) Author: task-planner (opus-4-8 [1m], thinking: max, isolation=worktree) on cmux-worker-subo-id81-27a24fe2. Spec-only — investigate and decide; does NOT write production code or run the staging pipeline (parent’s shared resource).

---

§1. Context

Stage-5’s post-pass _run_stage_5_resolution (scripts/cocoindex_pipeline/stage_5.py:107-307) resolves each run’s entity_mentions in op_id-scoped isolation (Inv-5): it reads only WHERE op_id = $1 (_select_run_entity_mentions, stage_5.py:100-105), so it cannot see canonicals a PRIOR run pinned. The same real-world entity can therefore land under different canonical_names across separately-ingested runs, degrading KG join quality. CocoIndex’s resolve_entities ships is_existing_canonical: Callable[[str],bool]|None + ExistingCanonicalPolicy (PINNED / PREFERRED) for exactly this seam. This RESEARCH decides how to seed already-existing canonicals into the resolver so new mentions chain under them, WITHOUT regressing Inv-5, the bl-225 intra-run collapse, or Inv-10/Inv-20.

Code-intelligence orientation (per .gitnexus/CLAUDE.md Always Do). The output of this Subtask is a .md design doc, so orientation is light (not a code-edit gate). Two tools were run against the indexed knowledge-hub graph:

• gitnexus_query({query: "entity resolution canonical stage 5", repo: "knowledge-hub"}) — returned "processes": [] (NO execution flows surface; Stage-5 is an out-of-band app_main continuation, not part of an indexed reactive flow), and a definitions list of the Stage-5 symbols: _run_stage_5_resolution, _preload_entity_aliases, _select_run_entity_mentions (all scripts/cocoindex_pipeline/stage_5.py), _FlowStageCounter / _classify_stage_exception / _empty_stage_counts (flow.py), plus the three real-body regression tests in test_cocoindex_stage_5_resolution.py. No further symbols surface beyond those read for this RESEARCH.
• gitnexus_context({name: "_run_stage_5_resolution", repo: "knowledge-hub"}) — verdict surface: incoming.calls = flow.py:app_main + the three regression tests; outgoing.calls = _preload_entity_aliases, _select_run_entity_mentions, _FlowStageCounter.increment; processes: [] (no execution-flow participation indexed). Caller count = 1 production caller (app_main). This bounds the blast radius of any seeding change to _run_stage_5_resolution + its two private readers — a LOW-fanout, single-production-caller surface. (gitnexus has no per-symbol risk verdict in context; the single-production-caller fact is the orientation evidence.)

These outputs are cited verbatim, not paraphrased, so the Checker can verify the orientation step ran.

---

§2. (a) Authoritative existing-canonical SOURCE

Question. What op-agnostic read supplies “already-existing canonicals”? Options: (i) canonical entity_mentions.canonical_name across all op_ids incl. prior runs; (ii) legacy entity_aliases; (iii) app-side NULL-op_id rows; (iv) a UNION. And what workspace + entity_type scoping applies?

Schema reality (verified against the squashed migration + the live UPSERT schema)

• entity_mentions columns (supabase/migrations/20260416102457_pre_squash_reconciliation.sql:3612-3626 + the ID-53 op_id add at 20260528122543:19-22): id, content_item_id, entity_type, entity_name, canonical_name, confidence numeric(3,2) NULL DEFAULT 1.0, context_snippet, created_at, entity_type_override, normalisation_version, metadata jsonb, op_id uuid NULL. NO embedding column. NO workspace_id column.
• Load-bearing UNIQUE: entity_mentions_canonical_name_entity_type_content_item_id_key UNIQUE (canonical_name, entity_type, content_item_id) (...:4363) — this is the constraint bl-225 collapses against.
• Indexes relevant to an op-agnostic read: idx_entity_mentions_canonical btree (canonical_name, entity_type) (...:4799) and idx_entity_mentions_op_id btree (op_id) WHERE op_id IS NOT NULL (20260528122543:21-22). The (canonical_name, entity_type) composite index directly supports an op-agnostic SELECT DISTINCT canonical_name ... WHERE entity_type = $1 — it is an index-only-friendly prefix-scan on entity_type as the second column (Postgres can use it as a skip/filter; a covering scan on the leading canonical_name ordered set is available). So the read is cheap and already indexed; no new index is required for the unscoped-by-workspace variant.
• entity_aliases (...:3598-3606): id, alias varchar, canonical varchar, category varchar DEFAULT 'client' CHECK in (client, generic), is_active bool DEFAULT true, created_at. UNIQUE (alias) (...:4353); partial index idx_entity_aliases_active (alias) WHERE is_active = true (...:4795). This is the ADMIN-CURATED surface; Stage-5 ALREADY preloads it via _preload_entity_aliases (stage_5.py:66-79) and applies alias_map.get(canonical, canonical) BEFORE resolve_entities (stage_5.py:145-154). It is an alias→canonical REWRITE map, NOT a roster of existing canonicals.

Workspace-scoping reality — the decisive finding

entity_mentions has no direct workspace linkage. Workspace association is doubly indirect AND not populated at ingest time:

• entity_mentions.content_item_id → content_items.id; but content_items has NO workspace_id column (confirmed: the content_items CREATE TABLE block carries no workspace column; the M2M junction content_item_workspaces (content_item_id, workspace_id, assigned_at, id) at ...: (junction table) is the only link).
• The pipeline DELIBERATELY does NOT populate that junction during ingest. flow.py:1841-1853 (ID-69 BI-1) states verbatim: “everything ABOVE (content_items / source_documents / content_chunks / q_a_extractions / entity_mentions) is the CANONICAL, workspace-AGNOSTIC record layer. content_items has no workspace_id (ID-69 BI-1), and the content_item_workspaces M2M junction is DELIBERATELY not populated here: association is ID-69’s job (operator-side in v1 …)”.
• Consequence: at Stage-5 time, the run’s freshly-written entity_mentions rows have NO workspace edge at all (the junction is filled later, operator-side). Any “scope existing canonicals to this workspace” read would have to join through an unpopulated junction and would return EMPTY for the in-flight run’s content — the workspace scope is unavailable by construction at Stage-5 time.

This directly conditions Inv-21 (“single-workspace scope at v1”). ID-53 PRODUCT Inv-21 says Stage-5 “is naturally honoured because the run’s rows come from one workspace” — but the schema shows that is an ingest-batch property (one manifest → one logical workspace’s files), NOT a queryable workspace_id filter. There is no column to scope on.

DECISION (a)

SOURCE = canonical entity_mentions.canonical_name, op-agnostic (DISTINCT across ALL op_ids including NULL-op_id app-side rows and prior-run rows), scoped by entity_type only — NOT by workspace. Concretely, per entity_type batch already present in stage_5.py:186, seed from:

SELECT DISTINCT canonical_name
FROM public.entity_mentions
WHERE entity_type = $1

Rationale:

1. Single source, not a UNION. Option (iv) UNION with entity_aliases is REJECTED for the existing-canonical roster: entity_aliases is already applied as a rewrite upstream (stage_5.py:145-154, Inv-10). Its canonical values are, by construction, already present in entity_mentions.canonical_name for any entity that has ever been ingested-and-aliased; and an admin alias canonical that has NEVER been ingested is not a real “existing canonical” to chain under (chaining a new mention under a never-materialised name produces a canonical with no rows — useless for KG joins). The alias map remains where it is (pre-resolve_entities rewrite); it is NOT folded into the seed roster.
2. Include NULL-op_id rows (option iii folds into option i). NULL-op_id rows are app-side writes (classifyContent, Admin curation). They are legitimate existing canonicals an operator has materialised. Inv-5 ALREADY permits READING them as resolution input (“MAY be READ … but MUST NOT be UPDATED”). A DISTINCT canonical_name read is op-agnostic, so it naturally subsumes (iii). The Inv-5 write-side protection is preserved by-construction (see §5.(d)).
3. entity_type scoping = YES (free, mandatory). The batch loop is already per-entity_type (stage_5.py:186-195); the KhPairResolver cache is entity_type-keyed (P-OQ3). Seeding ‘Cisco’ as organisation must not pollute ‘Cisco’ as technology. WHERE entity_type = $1 is index-backed by idx_entity_mentions_canonical (canonical_name, entity_type).
4. Workspace scoping = NO (not achievable, and not needed at v1). Per the schema finding above, there is no workspace_id to scope on at Stage-5 time, and the canonical record layer is explicitly workspace-agnostic (ID-69 BI-1). Cross-tenant merging is OUT of v1 (Inv-21), but the v1 corpus is effectively single-tenant (UK SMB procurement) so an unscoped roster does not violate Inv-21 in practice. This is flagged for {81.2} PRODUCT to ratify as an explicit invariant (“existing-canonical seeding is workspace-agnostic at v1; revisit when content_item_workspaces is ingest-populated” — i.e. when ID-69 ingest-side association lands). See OQ-81-WORKSPACE-SCOPE in §7.

Self-membership caveat for {81.3} TECH. The seed roster (DISTINCT canonical_name WHERE entity_type=$1) will, for a re-ingest run, include canonicals THIS run also produced (its own rows are in entity_mentions by Stage-5 time). That is harmless for PINNED seeding (a name being both “existing” and “in this run’s name set” is just is_existing=True for that name — it is pinned, never demoted), but TECH must confirm the is_existing_canonical callback is a set membership test over this roster, not an identity test that could mis-flag.

⚠ RESEARCH-ERROR — corrected S306 (bl-223 / {81.9}). Do not reuse the “harmless” judgment above. Self-membership was proven FATAL to cross-run chaining, not harmless. Because the roster read is op-AGNOSTIC (DISTINCT canonical_name WHERE entity_type=$1), it includes the run’s OWN in-flight canonicals (their rows are in entity_mentions by Stage-5 time). Each in-flight name therefore self-matches and self-pins (is_existing=True against itself) BEFORE it can chain to a PRIOR-run canonical — so PINNED seeding was inert cross-run: run-2 names never chained under run-1 canonicals, and the S304 build shipped GREEN-but-WRONG on exactly this. The {81.9} fix subtracts the run’s own names_by_type[entity_type] from the seed roster before the merge (one line, internal to _run_stage_5_resolution), leaving ONLY foreign-op existing canonicals in the roster. The set-membership-vs-identity point still stands but was NOT the operative risk — op-agnostic self-membership was. Lesson: when a read is op-agnostic, always reason explicitly about the in-flight op’s OWN rows appearing in the result set.

---

§3. (b) PINNED vs PREFERRED

Source read in full: /Users/liamj/Library/Python/3.14/lib/python/site-packages/cocoindex/ops/entity_resolution/__init__.py (also empirically imported — see §Verification). Exact semantics confirmed at the line level:

PINNED (ExistingCanonicalPolicy.PINNED, the default)

• pass_1 / pass_2 split (__init__.py:262-264): pass_1 = [i for i in entity_map.values() if i.is_existing]; pass_2 = [... if not i.is_existing].
• pass_1 seeding (:269-279): each existing entity is added to the dedup map as its OWN canonical (dedup[info.name] = None), added to the faiss index FIRST (_add_to_index), and emits ResolutionEvent(seeded=True) — no resolver call, no candidate search.
• pass_2 (:281-337): each NON-existing name searches candidates (which now INCLUDE the seeded existings, because they were indexed first), then resolve_pair is consulted.
• _new_wins under PINNED (:350-353): if matched_info.is_existing: return False — when the matched candidate is an existing canonical, NEW never wins → the new name chains UNDER the existing (dedup[info.name] = matched, :324). If the match is NOT existing, falls through to decision.canonical == CanonicalSide.NEW.
• Two existings are never compared — both are in pass_1, neither runs the resolver against the other; each stays its own canonical. PINNED therefore guarantees existing canonicals are STABLE: a prior run’s pinned canonical can never be demoted by a later run’s mention.

PREFERRED (ExistingCanonicalPolicy.PREFERRED)

• pass_1 empty (:265-267): pass_1 = []; ALL names (incl. existing) go through pass_2 and the resolver IS consulted for every one.
• _new_wins under PREFERRED (:355-359): existing status only breaks ties — if entity_info.is_existing and not matched_info.is_existing: return True; if matched_info.is_existing and not entity_info.is_existing: return False; otherwise falls through to decision.canonical == CanonicalSide.NEW. So an existing CAN be repointed/demoted if the resolver (via decision.canonical) and the tie-break both favour the new side in an existing-vs-existing or new-vs-new comparison — i.e. PREFERRED does NOT guarantee prior-run canonical stability; it only prefers existings on a tie.

The product axis

The choice is genuinely a stability-vs-correctness trade:

• PINNED = prior-run canonical names are immutable cross-run (a name that was canonical in run 1 stays canonical forever; new mentions always chain under it). Maximises cross-run join stability — which is the LITERAL goal of ID-81. Cost: if a prior run pinned a worse canonical (e.g. the short “eir 2004” before the long form was ever seen), PINNED freezes the worse name and the better cross-doc name chains under it. The KhPairResolver’s “longer name wins” preference (pair_resolver.py:139-141) is OVERRIDDEN by PINNED for any matched existing.
• PREFERRED = the resolver re-litigates every name each run; existings only win ties. Better eventual canonical quality, but a canonical can FLIP between runs (run 2 picks a different winner than run 1) — which is exactly the instability ID-81 exists to kill, and which also costs an LLM/cache resolver call for every existing name (pass_1 is empty → all existings hit pass_2; see §4 cost).

RECOMMENDATION (b): PINNED — but flag the canonical-quality trade as a product call

Lean: PINNED. It is the policy that directly delivers ID-81’s stated goal (cross-run stability: “new mentions chain under [prior canonicals]”). The PINNED _new_wins contract (:350-353) is a hard guarantee that a prior run’s canonical is never demoted — that IS the invariant ID-81 wants. PINNED is also the cocoindex default, and it is strictly CHEAPER (existings skip the resolver entirely — §4).

However, the residual trade is a product-owner call and I will NOT silently bury it. PINNED freezes whatever canonical a prior run happened to pin first, even if a later run would pick a better (longer/more-disambiguating) name per the KhPairResolver preference. Whether “stable but possibly-suboptimal canonical” beats “higher-quality but cross-run-mutable canonical” is a value judgement about the KG product surface, not a code fact. → OPEN QUESTION OQ-81-POLICY (escalate to orchestrator/parent). The recommended default for {81.2} PRODUCT dispatch is PINNED; if Liam wants canonical-quality to win over stability, PREFERRED is the alternative (with the cost + flip caveats above). The first-pin-wins suboptimality is mitigable independently of policy by a one-time admin entity_aliases rewrite (the existing curation surface), so PINNED’s downside is recoverable operator-side.

---

§4. (c)+(e) faiss-seeding COST at corpus scale, and embedding-REUSE feasibility

CRITICAL fact, confirmed by reading the source AND by empirical import (§Verification). resolve_entities embeds every name in sorted(set(entities)) internally:

entity_list = sorted(set(entities))                       # __init__.py:201
raw_embeddings = await _asyncio.gather(
    *(embedder.embed(name) for name in entity_list)       # :205-207  — embeds ALL, no exceptions
)

There is NO precomputed-embedding parameter in the public signature (empirically: resolve_entities(entities, *, embedder, resolve_pair, is_existing_canonical=None, existing_policy=PINNED, on_resolution=None, max_distance=0.3, top_n=5) — §Verification). Existings are embedded in the SAME _asyncio.gather as new names; is_existing only changes the pass_1/pass_2 split (:262-264), NOT whether the name is embedded. And entity_mentions has NO embedding column (§2). So the only embedding-reuse levers are exactly the two the brief names: (i) a custom embedder that consults a stored cache, or (ii) prefiltering which existings get seeded.

The cost seam, quantified

If the seed roster is “ALL existing canonicals for entity_type” (DECISION (a) unbounded), then per run, per entity_type:

• Embeddings: O(D_existing + D_new) calls to KhEntityEmbedder.embed where D_existing = distinct existing canonicals of that type across the WHOLE corpus and D_new = distinct new per-doc canonicals in this run. KhEntityEmbedder is a thin LiteLLM pass-through to OpenAI text-embedding-3-large dim=1024 with no caching (entity_embedder.py:81-106). Each call is a network round-trip. As the corpus grows, D_existing grows monotonically and UNBOUNDED → every run re-embeds the entire historical canonical roster for every type. This is the O(N) seam: a corpus with, say, 5,000 distinct organisation canonicals re-embeds 5,000 names on EVERY run regardless of how few new orgs that run introduced. At text-embedding-3-large latency + token cost, this dominates Stage-5 wall-clock and spend within a few hundred runs.
• faiss IndexFlatIP build: index.add per name (:233), exact inner-product index, O(D_existing + D_new) adds + O(top_n · D_new) searches. faiss-cpu IndexFlatIP build/search over a few thousand 1024-d vectors is sub-second (CPU exact search is fine into the low tens of thousands) — faiss is NOT the bottleneck; embedding is. The IndexFlatIP cost is real but second-order to the embedding round-trips.

Bounding levers (assessed)

1. Per-entity_type batch — ALREADY PRESENT (stage_5.py:186-195). Caps each resolve_entities call to one type’s roster. Keeps faiss indices small and stops cross-type pollution. Necessary but insufficient alone (a single type’s historical roster still grows unbounded).
2. Workspace scope — UNAVAILABLE (§2: no workspace_id at Stage-5 time). Cannot be used as a bound. Noted so {81.3} TECH does not attempt it.
3. Candidate-prefilter (seed only existings PLAUSIBLY matching this run’s names) — RECOMMENDED PRIMARY BOUND. Instead of seeding the entire historical roster, seed only existing canonicals that could conceivably match a name in THIS run. Cheap prefilters available without embeddings:
   • Exact / case-fold prefilter: seed existings whose canonical_name exactly equals (or case-folds to) one of this run’s per-doc canonicals. This is the high-value case — it catches “same string ingested in a prior run” with ZERO extra embeddings (the existing is already in the run’s name set, so it is embedded anyway). But this alone misses near-matches (“ISO 27001” vs “ISO27001”).
   • Trigram / pg_trgm similarity prefilter: WHERE entity_type=$1 AND canonical_name % ANY($run_names) (or a similarity() threshold) returns only existings lexically close to a run name. This bounds D_existing to “existings plausibly near this run’s content” — typically a tiny fraction of the corpus. Cost caveat for TECH: there is no pg_trgm GIN index on entity_mentions.canonical_name today (only the btree idx_entity_mentions_canonical); a %-filter would seq-scan unless TECH adds a gin (canonical_name gin_trgm_ops) index. That index is a 1-migration add and is the recommended cost-control investment. Extension prerequisite for TECH: the gin_trgm_ops operator class AND the % operator both require the pg_trgm extension — {81.3} TECH MUST verify it is installed (SELECT * FROM pg_extension WHERE extname = 'pg_trgm') before specifying the index migration; if absent, the migration MUST include CREATE EXTENSION IF NOT EXISTS pg_trgm (DDL via CLI only per CLAUDE.md). The lexical prefilter can MISS embedding-space near-matches that are not lexically close (rare for entity NAMES, which are short and domain-specific — the embedder’s value is mostly catching casing/punctuation/abbreviation variants that ARE lexically close). Accept the small recall loss at v1; document it.
4. Embedding REUSE — assess both sub-options:
   • (i) Custom caching embedder. Wrap KhEntityEmbedder so embed(name) first checks a persistent name→vector cache (a new table, e.g. entity_name_embeddings (name, entity_type, embedding vector(1024)), or a process-local LRU). cocoindex calls embedder.embed(name) per name regardless, so the cache must live INSIDE the embedder — cocoindex offers no hook to skip embedding a name. Feasibility: HIGH but with a real cost. It turns the per-run embedding cost from O(D_existing+D_new) network calls into O(D_new + cache_misses) calls + O(D_existing+D_new) cache lookups. The persistent-cache variant needs a new migration + a vector column (the very thing entity_mentions lacks) + cache-invalidation thinking (canonical names are stable strings, so invalidation is near-trivial: the cache key IS the immutable name+type, never stale). The process-local LRU variant is free of migration but only helps WITHIN a run (existings re-embedded once per run, not per name occurrence) — limited value since sorted(set(...)) already dedups within a run.
   • (ii) Prefilter (lever 3). Reduces the SET of existings embedded at all, so it reduces both the embedding cost AND the faiss cost, with no new vector column. It is strictly simpler than a persistent embedding cache and attacks the same O(N) seam.

DECISION (c)+(e): bound by candidate-prefilter (primary); persistent embedding cache is a deferrable second-order optimisation

Primary bound = candidate-prefilter (lever 3): seed only existing canonicals lexically plausible against this run’s per-doc canonical names, per entity_type. Recommended concrete shape for {81.3} TECH: a pg_trgm-backed SELECT DISTINCT canonical_name FROM entity_mentions WHERE entity_type=$1 AND canonical_name % ANY($run_names::text[]) (plus a gin_trgm_ops index migration), UNION’d with the exact case-fold matches of the run’s names. This collapses the unbounded historical roster to “existings near this run” and reuses the run’s own embeddings for the exact-match case for free.

Embedding-REUSE verdict (e): Reuse via the cocoindex public API alone is NOT POSSIBLE (no precomputed-embedding param; embeds all of sorted(set(entities)); no embedding column on entity_mentions — all empirically confirmed). The two achievable routes are (i) a custom caching embedder and (ii) prefiltering. Prefiltering (route ii) is RECOMMENDED as the v1 bound because it attacks both the embedding AND faiss cost with a single migration and no vector-cache lifecycle. A persistent caching embedder (route i) is DEFERRED — it is a clean second-order optimisation if telemetry shows the prefiltered roster is still large enough that re-embedding it dominates; recommend {81.2} PRODUCT carry it as an explicit non-goal-for-v1 / future-optimisation note rather than building it speculatively. The brief’s premise is confirmed: “embedding reuse is only achievable via (i) or (ii)” — and (ii) subsumes the win of (i) for the dominant exact-match case at far lower complexity.

---

§5. (d) Idempotency / determinism interaction with the bl-225 intra-run collapse, and Inv-5

bl-225 recap (verified stage_5.py:197-307). After resolution, Step 5 groups the run’s rows by the POST-resolution natural key (content_item_id, entity_type, resolved), picks a deterministic highest-confidence survivor per group (min(members, key=lambda m: (-(conf if conf is not None else -1.0), id)), :246-248), DELETEs the losers (:251-253), and Step 6 issues DELETE-first-then-UPDATE in one transaction, op_id-scoped on BOTH the DELETE (WHERE ... AND op_id = $2, :268-273) and the UPDATE (WHERE id = $2 AND op_id = $3, :279-286).

Interaction 1 — Seeding MUST NOT reintroduce intra-run collisions

Seeding changes only the names SET fed to resolve_entities and the is_existing flag — it does NOT change Step 5’s grouping/collapse logic. The bl-225 collapse operates on name_pairs (the run’s op_id-scoped rows), grouping by the POST-resolution resolved = canonical_of(alias_applied_canonical). Seeding can only CHANGE which resolved value a run name maps to (it may now chain under a seeded existing instead of staying per-doc-local). That is exactly the kind of “two distinct per-doc canonicals in one doc now resolve to one value” case bl-225 was BUILT to absorb — the collapse already DELETE-survivor-dedups it. So seeding cannot reintroduce an UNHANDLED collision: any new same-doc collapse that seeding induces flows through the identical bl-225 survivor logic. Two confirmations TECH must preserve as regression coverage:

• The existing real-body tests (test_collapse_no_update_needed, test_collapse_with_survivor_update, test_zero_confidence_survivor_not_treated_as_missing) stub the resolver chain at the source modules (_stub_resolver_chain, test_...:266-293) and assert the collapse. {81.3}/{81.4} must ADD a test where a seeded existing is the resolution target and two same-doc run rows collapse onto it — proving the collapse still fires with seeding live.
• Determinism of the seed roster. resolve_entities does sorted(set(entities)) (:201), so the seed order into faiss is deterministic IFF the roster is a stable set. The roster comes from SELECT DISTINCT canonical_name ... whose result-set ORDER is unspecified by SQL, but it is fed through sorted(set(...)), so order is normalised. TECH must ensure the seed set is passed as part of the same entities iterable (so sorted(set(...)) covers it), NOT added out-of-band — otherwise the pass_1 index-add order could vary run-to-run. Under PINNED, pass_1 iterates entity_map.values() whose construction order follows sorted(set(entities)), so determinism holds as long as seeds are members of entities.

Interaction 2 — Inv-5 (NEVER write rows outside the in-flight op_id, even while READING prior-run canonicals)

This is the load-bearing safety argument, and it holds by construction with the DECISION (a) source. Trace:

• The seed roster (DISTINCT canonical_name WHERE entity_type=$1, op-AGNOSTIC) is a READ of strings only. It produces names, never row identities.
• Those seeded names enter resolve_entities purely as members of the entities name set and as is_existing=True flags. resolve_entities returns a ResolvedEntities dedup MAP over NAMES — it has no concept of entity_mentions rows or op_ids.
• Insertion point (explicit for TECH): the seed roster MUST be merged into names_by_type[entity_type] AFTER the name_pairs → names_by_type fan-out (stage_5.py:182) and BEFORE the resolve_entities call (stage_5.py:186). name_pairs itself is NEVER extended with seed rows — only the per-type NAME SET fed to resolve_entities grows. This is what keeps the by-construction Inv-5 safety argument intact: the write-back iterates name_pairs (untouched, op_id-scoped), while only the resolution INPUT (names_by_type) carries the op-agnostic seeds.
• The write-back (stage_5.py:197-307) iterates ONLY name_pairs, which are built ONLY from _select_run_entity_mentions(op_id) (stage_5.py:134, WHERE op_id = $1). Seeded existing canonicals from OTHER ops are in the names set but are NEVER in name_pairs → they are never grouped, never collapsed, never UPDATE’d, never DELETE’d. The Step 6 DELETE and UPDATE are both additionally guarded AND op_id = $current (:270, :282).
• Therefore a prior-run or NULL-op_id canonical can be READ as a chaining target but its ROW is physically unreachable by any write statement in the pass. Inv-5 is preserved by-construction; seeding does not weaken it. This is the exact property the brief asked to verify, and it holds: the op_id scope lives on the WRITE side (name_pairs + WHERE op_id), entirely independent of the op-agnostic READ side (the seed roster).

Interaction 3 — Cross-run determinism (idempotency)

PINNED seeding INCREASES idempotency: re-running Stage-5 (e.g. full_reprocess) re-reads the same existing roster, re-pins the same canonicals, and the KhPairResolver determinism cache (Inv-14, entity_pair_resolutions) replays prior pair decisions. The only new determinism dependency is the seed roster’s stability — addressed in Interaction 1 (sorted(set(...)) normalises order). Under PREFERRED the cross-run mapping could flip (§3), which is WHY PINNED is recommended for an idempotency-sensitive surface.

DECISION (d)

• bl-225 collapse is PRESERVED and SUFFICIENT — seeding-induced same-doc collisions flow through the existing survivor/DELETE-first logic unchanged; {81.3}/{81.4} add a seeded-target collapse regression test.
• Inv-5 is PRESERVED BY-CONSTRUCTION — op-agnostic READ of the seed roster + op_id-scoped WRITE via name_pairs are independent; seeded foreign-op canonicals are unreachable by any DELETE/UPDATE. No new safeguard needed beyond keeping the write path scoped exactly as today.
• Seed-set determinism — TECH MUST feed seeds as members of the entities iterable so sorted(set(...)) normalises pass_1 order.

---

§6. Constraints / non-regression

Constraint	Source	How seeding respects it
Inv-5 — UPDATE only op_id = current; prior-run + NULL-op_id rows READ-only	ID-53 PRODUCT.md §Area C; stage_5.py:100-105,268-286	Op-agnostic READ for seed roster; write-back iterates op_id-scoped name_pairs only; DELETE/UPDATE both AND op_id=$current. Preserved by-construction (§5 Interaction 2).
bl-225 intra-run collapse — group by (content_item_id, entity_type, resolved), highest-confidence survivor, DELETE losers, DELETE-first	stage_5.py:197-307; test_cocoindex_stage_5_resolution.py	Seeding only changes resolved targets; collapse logic untouched and absorbs any seeding-induced same-doc collision (§5 Interaction 1). New regression test for seeded-target collapse required.
Inv-10 — legacy entity_aliases preloaded + applied BEFORE resolve_entities	stage_5.py:66-79,145-154	Alias map stays a pre-resolve_entities REWRITE; NOT folded into the seed roster (§2 DECISION rationale 1). Seed reads post-alias canonical strings, so the roster is consistent with the alias-applied namespace.
Inv-20 — unresolved mentions retain per-document canonical; no UPDATE	ID-53 PRODUCT.md Inv-20; stage_5.py:228-235	Seeding adds chaining TARGETS; a name with no candidate within max_distance=0.3 still canonical_of(name)==name → stays per-doc. Inv-20 holds; only the SET of possible matches grows.
Inv-21 — single-workspace scope at v1; cross-tenant merge OUT	ID-53 PRODUCT.md Inv-21	Workspace scope is UNAVAILABLE at Stage-5 (no workspace_id; junction unpopulated — §2). Unscoped roster is acceptable on a v1 single-tenant corpus but must be ratified explicitly — OQ-81-WORKSPACE-SCOPE.
Façade gap — _coco_api.py re-exports resolve_entities/ResolvedEntities/PairDecision but NOT ExistingCanonicalPolicy	_coco_api.py:55-88 (confirmed)	Wiring prerequisite for {81.3} TECH: add ExistingCanonicalPolicy to _SYMBOL_SOURCES + __all__ + the TYPE_CHECKING block. (is_existing_canonical is a plain Callable arg — no symbol re-export needed; only the policy enum needs façade wiring.)
CLAUDE.md — DDL via CLI only; UK English; no silent failures; PYTHONUNBUFFERED=1 for Python bg output; Python pipeline tests python3 -m pytest scripts/tests/	CLAUDE.md	Any new index (gin_trgm_ops) or cache table is a CLI migration. The bl-225 collapse already wraps writes in a transaction whose exceptions propagate (no swallow). Seeding adds no silent-failure surface.

Inv-10/Inv-20 from .ast-dataflow/CLAUDE.md (Inv-10 propagation discipline) note: that file’s “Inv-10” is the sub-agent tool-discipline directive, distinct from ID-53 PRODUCT’s Inv-10 (alias preload). This RESEARCH is a spec-only .md deliverable; the code-intelligence orientation block (§1) satisfies the orientation expectation; {81.3} TECH / {81.4} PLAN briefs that touch _run_stage_5_resolution MUST carry the gitnexus impact-analysis tool-discipline instruction (Inv-2 Planner duty) because that symbol is the single seam being modified.

---

§7. Open Questions (escalate to orchestrator → parent)

ID	OQ	Why it is a product/parent call (not a code fact)	Recommended default for {81.2} PRODUCT
OQ-81-POLICY	PINNED vs PREFERRED ExistingCanonicalPolicy?	A stability-vs-canonical-quality value judgement about the KG product surface. PINNED freezes first-pinned canonical (max cross-run stability, ID-81’s stated goal) but can lock a suboptimal name; PREFERRED re-litigates quality each run but allows cross-run canonical flips (the instability ID-81 exists to kill) + costs a resolver call per existing.	PINNED. Directly delivers ID-81’s stated goal; cheaper; suboptimal-canonical downside is recoverable via the existing admin entity_aliases rewrite.
OQ-81-WORKSPACE-SCOPE	Is workspace-agnostic existing-canonical seeding acceptable at v1?	The schema makes workspace scoping UNAVAILABLE at Stage-5 (no workspace_id; content_item_workspaces junction is ingest-empty per ID-69 BI-1). Whether an unscoped roster is acceptable depends on the v1 single-tenant assumption (Inv-21) — a product framing, not a code constraint.	Workspace-agnostic at v1; revisit when ID-69 ingest-side workspace association lands. v1 corpus is effectively single-tenant; cross-tenant merge already OUT (Inv-21).

Both are surfaced in this return message for the orchestrator’s OQ channel.

---

§8. Recommendation summary (for {81.2} PRODUCT to build invariants from)

1. (a) SOURCE — Existing-canonical roster = SELECT DISTINCT canonical_name FROM public.entity_mentions WHERE entity_type = $1 (op-AGNOSTIC, includes NULL-op_id + prior-run rows), scoped by entity_type only, NOT workspace (unavailable). NOT a UNION with entity_aliases (that stays a pre-resolve_entities rewrite, Inv-10). The is_existing_canonical callback is a set-membership test over this roster.
2. (b) POLICY — PINNED (default; max cross-run stability; cheaper). Canonical-quality trade flagged as OQ-81-POLICY for parent.
3. (c)+(e) COST / REUSE — Embedding reuse via the public API is impossible (no precomputed-embedding param; embeds all of sorted(set(entities)); no embedding column). Bound the O(N) seam with a candidate-prefilter (pg_trgm % ANY($run_names) + exact case-fold matches, per entity_type) backed by a NEW gin_trgm_ops index migration (TECH must first verify pg_trgm is installed — SELECT * FROM pg_extension WHERE extname='pg_trgm' — and include CREATE EXTENSION IF NOT EXISTS pg_trgm in the migration if absent); this attacks both embedding + faiss cost with no vector-cache lifecycle. A persistent caching embedder is DEFERRED as a future optimisation (PRODUCT non-goal-for-v1 note). Per-entity_type batching already bounds cross-type pollution.
4. (d) IDEMPOTENCY / Inv-5 / bl-225 — Inv-5 preserved BY-CONSTRUCTION (op-agnostic READ + op_id-scoped WRITE via name_pairs are independent; seeded foreign-op canonicals are unreachable by DELETE/UPDATE). bl-225 collapse preserved and sufficient (absorbs seeding-induced same-doc collisions unchanged). Feed seeds as members of the entities iterable so sorted(set(...)) normalises pass_1 order (determinism). New regression test: seeded-target same-doc collapse.
5. Wiring prerequisite — _coco_api.py must add ExistingCanonicalPolicy (façade gap confirmed). The seam to add seeding is exactly the resolve_entities(...) call at stage_5.py:186-195 (add is_existing_canonical= + existing_policy=).
6. Effort signal for {81.4} PLAN — Touches: stage_5.py (new seed-roster reader + seeding wiring), _coco_api.py (1-line façade add), 1 CLI migration (gin_trgm_ops index), Python tests. Single-migration, single-production-caller surface (gitnexus: 1 caller). Estimate ~2-4h; PLAN likely warranted if PRODUCT adds the prefilter + workspace-scope invariants as distinct slices, otherwise borderline-skippable.

---

§Verification (OQ-3 / Q-EX2 pre-ratification empirical import-and-call check)

Cited external symbols: cocoindex.ops.entity_resolution.{resolve_entities, ExistingCanonicalPolicy, ResolvedEntities, PairDecision, CanonicalSide, ResolutionEvent}.

• Date: 03/06/2026.
• Pinned version: cocoindex[postgres]==1.0.3 (requirements.txt); faiss-cpu==1.14.2. Runtime cocoindex.__version__ reported 1.0.3 — pin matches installed.
• Check run: python3 -c "from cocoindex.ops.entity_resolution import resolve_entities, ExistingCanonicalPolicy, ResolvedEntities, PairDecision, CanonicalSide, ResolutionEvent; ..." (import + inspect.signature + enum-member enumeration).
• Results:
  • cocoindex.ops.entity_resolution.resolve_entities — PRESENT. Signature verbatim: (entities, *, embedder, resolve_pair, is_existing_canonical: Callable[[str], bool] | None = None, existing_policy: ExistingCanonicalPolicy = ExistingCanonicalPolicy.PINNED, on_resolution=None, max_distance=0.3, top_n=5) -> ResolvedEntities. Matches the brief’s grounded fact 1 exactly; default existing_policy is PINNED. No precomputed_embeddings / embeddings parameter exists (confirms §4 embedding-reuse-impossible finding).
  • ExistingCanonicalPolicy — PRESENT. Members ['pinned', 'preferred'].
  • CanonicalSide — PRESENT. Members ['new', 'matched'].
  • ResolutionEvent — PRESENT. Fields ['entity', 'canonical', 'candidates', 'decision', 'repointed', 'seeded'] (the seeded flag the PINNED pass_1 emits, per brief fact 1).
  • ResolvedEntities / PairDecision — PRESENT (importable).
• Façade-gap check: _coco_api.py:55-74 _SYMBOL_SOURCES re-exports resolve_entities, ResolvedEntities, PairDecision but NOT ExistingCanonicalPolicy (nor CanonicalSide / ResolutionEvent) — confirmed ABSENT from the façade; a wiring prerequisite for {81.3} TECH (NOT a drift defect — the symbol exists upstream, it is merely not yet re-exported).
• Verdict: PRESENT — no ABSENT / SIGNATURE_DRIFT / BEHAVIOUR_DRIFT. All cited symbols exist at the pinned version with the signatures the brief asserted. Source-read of __init__.py and runtime introspection agree. Spec is safe to return for ratification.

---

End of RESEARCH.md. ID-81.1 design investigation: 5 questions answered (a-e) with DECISIONs; 2 Open Questions escalated (OQ-81-POLICY, OQ-81-WORKSPACE-SCOPE); Inv-5 preservation proven by-construction; bl-225 collapse preserved; embedding-reuse verdict = prefilter-not-cache; empirical cocoindex==1.0.3 verification PRESENT.