Stage-5 entity-resolution — PRODUCT (ID-53.2)

Spec slug: stage-5-entity-resolution Subtask: ID-53.2 (PRODUCT, precedes {53.3} TECH → {53.4} PLAN per Q-PLANNER-2 fresh-per-Subtask discipline) Parent Task: ID-53 — Canonical pipeline Stage-5 entity-resolution — spec rescope + op_id migration (S273 deferred from ID-49.5) Critical-path position: The second half of the canonical-pipeline §1 critical-path risk (“unresolved entities”). Stage-4 embedding LANDED via ID-49.2; until Stage-5 lands the corpus serves vector search but cannot serve deduplicated entity queries. Gates Mempalace T12 KG integration (UNPROMOTED) and conditions the v1 product behaviour for cross-document entity dedup. Author: task-planner (opus-4-7, thinking: max, isolation=none) on cmux-worker-subo-id-53-f63aba0a from parent tip d807471f. FRESH INSTANCE — NOT the {53.1} RESEARCH author (Q-PLANNER-2).

§1. Context

Stage-5 entity-resolution was nominally in scope at v1 from S265 onwards (docs/specs/id-28-cocoindex-flow-scaffolding/TECH.md header — OQ-C OVERRIDDEN by Liam: “Stage-5 entity resolution IS in scope at v1; uses ops.entity_resolution.resolve_entities; faiss to be PINNED in requirements.txt; new subtask ID-28.29”). The S265 TECH §P-2 sketch wrote resolved_entities = await resolve_entities(entity_mentions) inside per-item ingest_file. S273 empirical reality (docs/research/s273-canonical-pipeline-finals/id49-final.yaml lines 113–116) proved this incompatible: cocoindex 1.0.3’s resolve_entities is collection-level (one IndexFlatIP over sorted(set(entities)) for the entire input), with no per-item / streaming API. ID-49.5 was deferred per Liam OQ-1 Option C — write spec first; ID-53 is that spec rescope, folding in the S273 OQ-2 ruling that entity_mentions needs an op_id column independent of the architecture choice.

The ratified RESEARCH (docs/specs/id-53-stage-5-entity-resolution/RESEARCH.md, §R-RECOMMENDATION) recommends Option B — a net-new flow-scope post-fan-out resolution stage — anchored on 8 mitigations covering op_id scoping, legacy alias coexistence, PairResolver determinism, substrate reuse, T12 sequencing, faiss-cpu pinning, and the entity_mentions op_id migration. This PRODUCT.md ratifies Option B and turns the 8 mitigations into testable behaviour invariants the Checker can gate against.

The Stage-5 user-facing surface is:

Pipeline-driven entity_mentions consumers — MCP tools (lib/mcp/tools/entities.ts:212–214, lib/mcp/tools/dashboard.ts:358,410, lib/mcp/resources.ts:526), app routes (app/api/certifications/route.ts:137–139, app/api/items/[id]/route.ts:582, app/api/cron/freshness-transitions/route.ts:752–754), dashboard widgets (lib/dashboard.ts:344–348, lib/mcp/formatters/dashboard.ts:37). They read canonical_name as the cross-document-stable identity for an entity. (RESEARCH §R4.2.)
Pipeline-run observability consumers — operators reading pipeline_runs.result.stage_counts["entity_resolution"], structured logs at error_class='entity_resolution_failed', and the flow-end webhook payload.
Re-ingest consumers — the pipeline’s own memo / full_reprocess semantics determine when a row’s canonical_name is re-stamped vs preserved across runs.
App-side entity-curation consumers — the Admin merge / split / type-override / metadata-edit surfaces (app/api/entities/merge/, app/api/entities/split/, app/api/entities/[canonical_name]/type/, app/api/entities/[canonical_name]/metadata/) and the classifyContent delete-before-insert path (lib/ai/classify.ts:1543–1546). These coexist with Stage-5 writes through op_id scoping.
Downstream pipeline consumers — Mempalace T12 KG integration (UNPROMOTED) and ID-127 entity merge/split KG-side (backlog) — both sequenced strictly after Stage-5 stable.

Source-of-truth documents (read in full to ratify these invariants):

docs/specs/id-53-stage-5-entity-resolution/RESEARCH.md — full ratified RESEARCH (§R1 cocoindex capability probes; §R2 schema reality; §R3 architecture A vs B; §R4 platform implications; §R5 faiss pin; §R6 op_id migration; §R7 cross-Task refs; §R-RECOMMENDATION + 6 OQs).
docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md §1 + §2.5 + §6 — critical-path framing; T12 sequencing under the canonical-pipeline umbrella.
docs/specs/id-28-cocoindex-flow-scaffolding/PRODUCT.md — peer PRODUCT (invariant-numbering convention; structure; OQ-tracking shape); especially Inv-3 (“Six-stage topology observable per document”), Inv-11 (memo-respecting op_id semantic), Inv-25 (failed pipeline runs roll up).
docs/specs/id-28-cocoindex-flow-scaffolding/TECH.md header — S265 OQ-C override that placed Stage-5 in scope at v1.
CLAUDE.md — DDL via CLI only; UK English; semantic tokens (not relevant — Stage-5 is server-side); getAuthorisedClient() discriminated-union (not relevant — server-side pipeline); sb() / tryQuery() (Python pipeline does not consume — note for TECH).

§2. Behaviour invariants

Each numbered invariant is testable in isolation. The Checker compares against acceptance criteria per-invariant during ratification. Behaviour-only — implementation choices land in {53.3} TECH.md. UK English throughout. Cross-references use [file:line] or [§Rx.y] against the ratified RESEARCH. Where an invariant depends on a Pydantic / DB / cocoindex symbol verified empirically in RESEARCH §R1, the citation is direct; PRODUCT introduces no new external-library claims.

Area A — Architecture decision (Option B ratification)

Inv-1. Stage-5 runs as a flow-scope post-fan-out resolution stage (Option B). The canonical pipeline executes Stage-5 entity-resolution as a second write phase that runs AFTER the per-item mount_each fan-out has settled — specifically, after await handle.ready() returns and before the flow-end webhook emits — the Stage-5 insertion window is bounded by scripts/cocoindex_pipeline/flow.py:1069 (handle.ready() attach-point) and flow.py:1135 (_emit_pipeline_run_webhook); Stage-5 code is inserted between these lines at implementation time (current HEAD has no Stage-5 code in this range — lines 1083–1084 carry the “deferred” comment). Stage-5 is NOT executed inside the per-item ingest_file body. The ingest_file body writes per-document canonical_name values (deterministic per-doc normalisation) at declare_row time; Stage-5 issues UPDATE entity_mentions SET canonical_name = ... for rows that resolve to a different cross-document canonical. Verifiable: a passing run is observable to write all per-item entity_mentions rows BEFORE any Stage-5 UPDATE statement fires; and an external observer can verify the post-pass UPDATE phase ran by reading pipeline_runs.result.stage_counts["entity_resolution"] > 0 (or = 0 when no UPDATEs were needed). Cross-ref: RESEARCH §R3.2 (Option B shape); §R-RECOMMENDATION (recommendation 1); §R1.5 (mount_each(...).handle.ready() as legitimate attach-point); §R1.4 (no per-item API empirically ABSENT — Option A’s per-doc-only canonicalisation cannot deliver cross-document dedup at v1 using cocoindex 1.0.3’s collection-level resolve_entities).

Inv-2. Stage-5 is a second-phase WRITE that breaks the managed_by=USER row-only contract — DELIBERATELY and circumscribed. The managed_by=USER contract (cocoindex’s declare_row per-item write pattern) is preserved for the per-item phase (ingest_file’s declare_row writes remain the sole INSERT mechanism). The Stage-5 UPDATE pass is a NEW kind of pipeline-owned write — issued from app_main AFTER mount_each.handle.ready() returns — and is explicitly NOT framed as a cocoindex declare_row write. KH owns this “second-phase canonicalisation” as a deliberate platform-level architectural choice, NOT as a temporary workaround. Verifiable: the v1 architecture documentation (docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md once amended in the TECH-driven ledger pass) names the row+update hybrid as the stable pattern. Cross-ref: RESEARCH §R3.2 (UPDATE pass design); §R-RECOMMENDATION (recommendation 1); OQ-53-CONTRACT-BREAK (RESEARCH leaning ratified — see §5 P-OQ1 below).

Area B — canonical_name freshness contract

Inv-3. canonical_name freshness on successful run. After a pipeline run terminates with pipeline_runs.status='completed', every entity_mentions row whose op_id matches that run’s pipeline_runs.op_id MUST reflect the latest cross-document resolution Stage-5 produced for the documents in that run — OR the per-document default when no cross-document match was found within max_distance=0.3 (the cocoindex resolve_entities default per RESEARCH §R1.2). A successful run is defined as: pipeline_runs.status='completed' AND pipeline_runs.result.stage_counts["entity_resolution"] is present (zero when the run produced zero entity mentions; positive integer when one or more canonical_name UPDATEs were issued; the per-document phase wrote rows but Stage-5 produced no UPDATEs is the = 0 case). Verifiable: ingest a corpus containing “ISO 27001” in doc A and “ISO27001” in doc B in the SAME run; on pipeline_runs.status='completed', both rows’ canonical_name resolve to the same cross-document canonical (the specific canonical chosen by resolve_entities is implementation-determined; the invariant is that they MATCH). Cross-ref: RESEARCH §R1.2 (max_distance=0.3 default); §R3.2 (UPDATE phase scope); peer PRODUCT.md Inv-3 (six-stage topology — Stage-5 is observable per document).

Inv-4. Per-document default canonical_name MUST be written BEFORE Stage-5 runs. The per-item ingest_file body computes a deterministic per-document canonical (via a KH-owned canonicalise_entity_name(name, type) pure function — lowercase + strip + ASCII-fold + entity_type-aware normalisation, mirroring scripts/kb_pipeline/classify.py:1293 resolve_entity_alias semantics transposed to a memoised @coco.fn) and writes it as the canonical_name at declare_row time. Stage-5 UPDATES this value when cross-document resolution maps to a different canonical; Stage-5 NEVER inserts a row that the per-item phase did not write. Verifiable: a run whose Stage-5 phase fails mid-pass leaves every per-item row in place with the per-document default canonical (Inv-12). Cross-ref: RESEARCH §R3.1 (per-doc canonicalisation shape); §R3.2 (the per-item write happens first; UPDATE phase second).

Area C — op_id scoping invariant (collision avoidance)

Inv-5. Stage-5 UPDATEs ONLY rows whose op_id equals the current run’s op_id. The Stage-5 UPDATE phase MUST scope its UPDATE statements to WHERE op_id = $current_run_op_id (the pipeline_runs.op_id for the in-flight run). Rows in entity_mentions with NULL op_id (rows written by pre-pipeline app-side paths; rows from runs prior to the op_id migration) MAY be READ by Stage-5 as cross-document resolution INPUTS, but MUST NOT be UPDATED. Rows whose op_id belongs to an earlier completed pipeline run MUST NOT be UPDATED by the current run. Verifiable: ingest a corpus across two pipeline runs (op_id A in run 1, op_id B in run 2); after run 2 completes, all rows with op_id = A retain their run-1 canonical_name regardless of what run 2’s resolve_entities produced. Cross-ref: RESEARCH §R4.1 (collision mitigation path 2 — RECOMMENDED — op_id-scoped UPDATEs); §R-RECOMMENDATION (recommendation 1).

Inv-6. op_id column exists on entity_mentions and round-trips to pipeline_runs. The entity_mentions table has an op_id uuid NULL column (added via the CLI migration per RESEARCH §R6.2 verbatim), populated by ingest_file’s declare_row for every row Stage-5 / the per-item phase writes. Given any op_id value extracted from an entity_mentions row, an audit-forensics consumer can resolve back to the originating pipeline_runs row via a single PK SELECT (SELECT * FROM pipeline_runs WHERE op_id = $value returns exactly one row). Verifiable: ingest a file via a single pipeline run; query an entity_mentions row the run produced; read its op_id; SELECT pipeline_runs WHERE op_id = <value> returns exactly one row whose op_id matches. Cross-ref: RESEARCH §R6 (migration spec); peer PRODUCT.md Inv-11 + Inv-12 (T8 op_id pattern this mirrors); empirically equivalent to the T8 pattern at supabase/migrations/20260521203414_t8_op_id_propagation.sql for content_items / q_a_extractions / source_documents.

Inv-7. Memo-respecting op_id semantic on entity_mentions. “Produced or updated” in Inv-6 is exact in the peer PRODUCT.md Inv-11 sense: op_id records the run that LAST MATERIALLY produced or changed the row, NOT the most recent run that merely scanned it. Because the per-item ingest_file is memoised (@coco.fn(memo=True)), an unchanged re-ingest of a source file does NOT re-stamp the per-item rows’ op_id. A full_reprocess run re-runs every function and re-stamps every row with the current run’s op_id. Stage-5’s UPDATE phase ONLY re-stamps op_id when it issues an UPDATE — i.e. when cross-document resolution produces a different canonical for the row; Stage-5 MUST NOT re-stamp op_id on rows whose canonical_name it does not change. Verifiable: ingest a file at run 1 (op_id A); the resulting entity_mentions rows have op_id A. Re-ingest the same file unchanged at run 2 (op_id B); the rows still show op_id A (memo SKIP). Trigger a full_reprocess at run 3 (op_id C); the rows show op_id C. Cross-ref: RESEARCH §R1.7 (memo + UPDATE interplay); peer PRODUCT.md Inv-11 (memo-respecting op_id, RATIFIED OQ-A); §R3.2 (post-pass idempotency surface).

Area D — App-side coexistence (collision avoidance with five write paths)

Inv-8. Stage-5 does not collide with the classifyContent delete-before-insert path. The lib/ai/classify.ts:1543–1546 path (DELETE FROM entity_mentions WHERE content_item_id = $1, then re-INSERT) is invoked from classifyContent (re-classify; governance publish-from-draft via lib/mcp/tools/governance.ts:472–499). Stage-5’s op_id scoping (Inv-5) means: app-side INSERTs from classifyContent create rows with NULL op_id (no pipeline run in flight); Stage-5 NEVER UPDATEs these rows. Conversely, app-side DELETEs from classifyContent MAY delete rows the pipeline wrote (a pre-existing app-side behaviour out of Stage-5 scope); when this happens the deletion is intentional and Stage-5 does not regenerate the row in the same run. Verifiable: while a pipeline run is in flight, an admin invocation of classifyContent on a content_item completes successfully without Stage-5 overwriting the new INSERTs (because their op_id is NULL). Cross-ref: RESEARCH §R4.1 (collision scenario — classifyContent path); §R-RECOMMENDATION (op_id-scoped UPDATEs mitigation).

Inv-9. Stage-5 does not collide with the Admin entity-curation surfaces. The Admin merge (app/api/entities/merge/route.ts:48–52 → atomic merge_entities RPC), split (app/api/entities/split/route.ts:51,78 — UPDATE canonical_name), type-override (app/api/entities/[canonical_name]/type/route.ts:48 — UPDATE entity_type_override), and metadata-edit (app/api/entities/[canonical_name]/metadata/route.ts:57,74,94 — UPDATE metadata) paths operate on rows from arbitrary historical runs. Stage-5’s op_id scoping (Inv-5) ensures: rows updated by Admin actions on prior runs (rows with older op_id) are NEVER overwritten by a later Stage-5 pass on a different run; rows whose op_id matches the in-flight run (the narrow race window) inherit the same race semantics the per-item declare_row already has — same race window, same surface, not expanded. Verifiable: an admin merge operation on rows from op_id = A (a completed run) is NEVER reverted by a subsequent run op_id = B. Cross-ref: RESEARCH §R4.1 (five distinct app-side write paths; op_id-scoped mitigation reasoning).

Inv-10. Legacy entity_aliases map preloads BEFORE resolve_entities runs. Stage-5 loads the legacy alias map from entity_aliases (active rows: is_active = true) at the start of its post-pass, and applies the alias map BEFORE invoking cocoindex.ops.entity_resolution.resolve_entities. The cross-document canonicalisation outputs are therefore CONSISTENT with what app-side callers see when they call resolveAlias(canonical_name) from lib/entities/entity-aliases.ts:92. Verifiable: when entity_aliases contains 'iso-27001' → 'ISO 27001 (UK)' as an active row, a pipeline run that resolves a corpus entity to 'iso-27001' Stage-5 writes 'ISO 27001 (UK)' as the final canonical, matching what a direct app-side query through resolveAlias would return. Cross-ref: RESEARCH §R4.4 (entity_aliases coexistence); §R-RECOMMENDATION (recommendation 2).

Area E — Observability invariant

Inv-11. stage_counts["entity_resolution"] reflects the count of rows whose canonical_name Stage-5 modified. pipeline_runs.result.stage_counts["entity_resolution"] is a count of entity_mentions rows whose canonical_name value Stage-5 CHANGED in this run (a delta count, NOT the count of input rows). When Stage-5 ran but produced zero changes (every per-document canonical was already cross-document-correct), the value is 0. When Stage-5 was not reached (run failed before await handle.ready() returned), the value is 0 (initial value from _empty_stage_counts() at scripts/cocoindex_pipeline/flow.py:382–395). Verifiable: ingest a corpus of N entity mentions where M of them resolve to a different cross-document canonical; after the run completes, stage_counts["entity_resolution"] == M. PRODUCT elevation: RESEARCH §R3.2 sketched a once-per-pass increment; PRODUCT specifies per-row delta semantics for parity with the existing per-row embedding stage counter (scripts/cocoindex_pipeline/flow.py:915 — stage_counter.increment("embedding") fires once per produced vector). The per-row surface is more useful to downstream consumers (they see “M rows changed” vs only “Stage-5 ran”) and is the established platform pattern; TECH calls flow_stage_counter.increment("entity_resolution") once per UPDATE-eligible row inside the bulk-update loop, not once for the pass. Cross-ref: RESEARCH §R2.7 (bind_stage_counter / _empty_stage_counts substrate verified reusable, scripts/cocoindex_pipeline/flow.py:382 slot present); peer PRODUCT.md Inv-17 (per-stage counters); scripts/cocoindex_pipeline/flow.py:915 (per-row embedding counter — the platform pattern).

Inv-12. Failure semantics — Stage-5 failure is non-destructive to the per-item write phase. When resolve_entities raises (or any Stage-5 substep raises — alias-preload, PairResolver invocation, UPDATE statement, etc.), the run records pipeline_runs.status='failed' with error_class='entity_resolution_failed' (the error class is already declared at scripts/cocoindex_pipeline/flow.py:162 (_PIPELINE_ERROR_CLASSES tuple) with the "entity_resolution_failed" literal at flow.py:168), AND the per-document canonical_name values written by the per-item phase REMAIN in place on the rows. Stage-5 failure does NOT delete or null-out per-item writes. Verifiable: inject a deterministic failure into Stage-5 (e.g. mock resolve_entities to raise); after the run terminates as status='failed', query entity_mentions for the rows the per-item phase wrote — they retain their per-document canonical_name and their op_id value (Inv-7). Cross-ref: RESEARCH §R3.2 (failure-recovery semantics — partial Stage-5 failure leaves per-doc canonicals frozen); peer PRODUCT.md Inv-25 (failed pipeline runs roll up to pipeline_runs.status='failed'); §R2.7 (entity_resolution_failed substrate already declared).

Inv-13. Flow-end webhook surfaces Stage-5 outcome. The _emit_pipeline_run_webhook call at scripts/cocoindex_pipeline/flow.py:1135 payload includes stage_counts["entity_resolution"] (per Inv-11) and, when status='failed' with error_class='entity_resolution_failed', the stage-error log emitted via _emit_stage_error_log(stage="entity_resolution", ...) per the existing _classify_stage_exception substrate (scripts/cocoindex_pipeline/flow.py:278). Verifiable: capture the flow-end webhook payload from a successful run with cross-document dedup — the payload carries stage_counts.entity_resolution > 0; capture the payload from a deliberately-failed run — the structured log carries stage='entity_resolution', error_class='entity_resolution_failed'. Cross-ref: peer PRODUCT.md Inv-26 (structured log emission per failed invocation); RESEARCH §R3.2 (observability surface — substrate reuse).

Area F — PairResolver determinism (idempotency)

Inv-14. PairResolver decisions persist across runs via a determinism cache. Stage-5 uses a KH-owned LLM-backed PairResolver callback (per the cocoindex.ops.entity_resolution.resolve_entities API contract — RESEARCH §R1.2) to tier-break ambiguous near-matches. PairResolver decisions are cached by (name_a, name_b) to a persistent store, so re-running Stage-5 on the same corpus + the same alias-preloaded canonical names produces the SAME canonical mapping. The cache hits replay prior LLM decisions without re-invoking the LLM. (The concrete store — extend entity_aliases with a provenance='cocoindex_pair_resolver' value, OR a new entity_pair_resolutions table, OR in-memory only — is a TECH-level decision per P-OQ3 below; PRODUCT requires only that the cache exists and is persistent across runs.) Verifiable: ingest a corpus that triggers PairResolver decisions in run 1; re-ingest the same corpus unchanged in run 2 (memo would SKIP per-item rewrites, so this test uses a full_reprocess to force Stage-5 to re-evaluate); the canonical mapping in run 2 matches run 1’s mapping byte-for-byte. Cross-ref: RESEARCH §R3.2 (PairResolver idempotency concern — LLM non-determinism); §R-RECOMMENDATION (recommendation 3 — determinism cache); OQ-53-PAIR-RESOLVER-CACHE.

Area G — Schema parity invariants (RESEARCH-surfaced)

Inv-15. Pydantic mention_confidence and DB confidence are explicitly reconciled. The Pydantic EntityMentionExtraction.mention_confidence field (scripts/cocoindex_pipeline/extraction.py:207, float with Field(ge=0.0, le=1.0)) and the DB entity_mentions.confidence column (numeric(3,2) NULL DEFAULT 1.0 with CHECK (confidence >= 0 AND confidence <= 1)) are mapped at the declare_row site inside ingest_file (specifically: "confidence": mention.mention_confidence at row-construction time). The Pydantic field name is NOT renamed (the more disambiguating mention_confidence is preserved as the LLM-output contract). Verifiable: pipeline-produced entity_mentions.confidence values equal the LLM’s mention_confidence floats clamped to the [0.0, 1.0] range; no Pydantic schema change is required by Stage-5. Cross-ref: RESEARCH §R2.4 (naming drift FINDING; recommendation defaults to map-at-declare-row).

Inv-16. source_span_start / source_span_end stash in metadata jsonb. The Pydantic EntityMentionExtraction.source_span_start / .source_span_end fields (extraction.py:205–206) are persisted to the entity_mentions.metadata jsonb column at declare_row time as {"source_span_start": <int>, "source_span_end": <int>}. No DB migration adds dedicated columns. (RESEARCH lean ratified — the OQ leans keep-as-is, but PRODUCT chooses jsonb stash to preserve the data; the alternative — drop on declare_row — loses information the S272 Q-EX2 contract preserves.) Verifiable: pipeline-produced rows expose metadata->>'source_span_start' and metadata->>'source_span_end' matching the LLM’s extracted spans. Cross-ref: RESEARCH §R2.5 (three options surfaced); RESEARCH §R2.5 recommendation defaults to (2) — stash in metadata jsonb.

Inv-17. context_snippet is computed inside ingest_file via a Python port of extractEntityContext. The per-item phase computes context_snippet for each entity_mentions row via a Python port of lib/ai/classify.ts:1611 extractEntityContext(plainText, e.name) (the function is a deterministic string operation on content_text + entity span — RESEARCH lean OQ-53-CONTEXT-SNIPPET-COMPUTE leans port). The column is NOT left NULL; downstream MCP / API / dashboard consumers (app/api/certifications/route.ts:137–139, lib/dashboard.ts:344–348, etc.) see the column populated for every Stage-5-produced row. Verifiable: pipeline-produced entity_mentions.context_snippet matches the Python-port output of extract_entity_context(content_text, entity_name) for the same inputs. Cross-ref: RESEARCH §R2.6 (context_snippet finding); §R-RECOMMENDATION (recommendation 5 substrate — already reusable for the error-class layer; the port is the only new code).

Area H — Substrate / dependency invariants

Inv-18. faiss-cpu==1.14.2 is pinned in requirements.txt at implementation start. The requirements.txt carries faiss-cpu==1.14.2 exactly (matching the cocoindex spike-verified pin per RESEARCH §R5.1). The Cloud Run sidecar build (cloudrun/cloudbuild-cocoindex.yaml) installs faiss-cpu in the pre-warm step so the cold-start budget is not blown by a runtime pip-install. Verifiable: pip show faiss-cpu in the deployed sidecar returns version 1.14.2; grep faiss-cpu requirements.txt returns faiss-cpu==1.14.2. Cross-ref: RESEARCH §R5 (pin assessment + image-budget evaluation); §R-RECOMMENDATION (recommendation 7); OQ-53-FAISS-PIN.

Inv-19. T12 (Mempalace KG integration) is HARD-GATED on Stage-5 stable. Mempalace T12 (docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md §3 row labelled T12) stays UNPROMOTED in the umbrella until: (a) ID-53 reaches done status; AND (b) a tier-1 observation window of one full re-ingest cycle without Stage-5 regressions has passed. Once both conditions hold, the Workflow Curator promotes T12. The T12 spec inherits Stage-5’s op_id-scoped semantics (Inv-5) when it lands. Verifiable: at v1, docs/reference/task-list.json does NOT contain a T12 / Mempalace-KG-integration task record; the canonical-pipeline-sequencing doc §3 still lists T12 as UNPROMOTED. Cross-ref: RESEARCH §R4.3 (T12 KG coupling); §R-RECOMMENDATION (recommendation 6); OQ-53-T12-SEQUENCING.

[S299 correction] T12 has since been promoted to ID-60 (docs/reference/task-list.json, status spec_needed) and ID-53 is now done — so gate (a) is SATISFIED and the original verifiable clause (“task-list does NOT contain a T12 record”) no longer holds. The invariant’s intent stands: ID-60 stays spec_needed and does NOT enter implementation until gate (b) — one clean re-ingest observation window without Stage-5 regressions — is met. Updated verifiable: ID-60 exists but remains pre-implementation until the observation window passes. Cross-ref: docs/themes/canonical-pipeline/reference/v1-completion-sequence.md Lane C (ID-60 after-cutover gate).

Area I — Resolution outcomes (corner cases)

Inv-20. Unresolved mentions retain the per-document canonical. When resolve_entities.canonical_of(name) returns None for a per-document canonical (no cross-document match within max_distance=0.3), the row’s canonical_name STAYS at the per-document value. Stage-5 issues NO UPDATE for that row; op_id is preserved per Inv-7. (RESEARCH lean OQ-53-PRE-EXTRACTION-LATE-DROP ratified — option (a): keep the row, accept “this mention stays per-doc local” as v1 norm.) Verifiable: ingest a corpus containing a unique entity name (e.g. an obscure organisation that has no near-matches anywhere); the resulting row’s canonical_name equals the per-document default and stage_counts["entity_resolution"] does NOT count this row in its delta. Cross-ref: RESEARCH OQ-53-PRE-EXTRACTION-LATE-DROP (option (a) leaning); §R1.3 (ResolvedEntities.canonical_of returning None semantics).

Inv-21. Cross-tenant entity merging is OUT of v1 scope. Stage-5 resolution is scoped to entities within a single workspace boundary at v1; cross-workspace entity merging is explicitly OUT (§3 below). Verifiable: when the corpus contains “ISO 27001” in workspace W1 and “ISO27001” in workspace W2, the two rows DO NOT resolve to a shared canonical at v1 — each workspace’s Stage-5 pass operates on its own entity_mentions subset. (Implementation detail for TECH: this likely means Stage-5 either scopes by workspace_id filter on the rows it reads, or runs once per workspace per pipeline invocation; PRODUCT requires only the v1 single-workspace scoping.) Cross-ref: §3 Out of scope below.

§3. Out of scope

These items are explicitly NOT specified at the PRODUCT level — they are either (a) implementation choices that belong in {53.3} TECH.md, (b) deferred to a downstream task, (c) owned by a sibling spec, or (d) a deliberate v1 scope decision.

Item	Status	Owner
Mempalace T12 KG integration (`entity_mentions` temporal + provenance wrappers)	Hard-gated on Stage-5 stable per Inv-19	T12 (UNPROMOTED until ID-53 done + tier-1 window)
App-side governance review of merged / split entities (admin curation UI flow)	Separate workflow; PRODUCT only invariants the coexistence guarantee (Inv-8, Inv-9)	`app/api/entities/{merge,split,...}` existing surfaces
Backfill of historic `entity_mentions` rows with NULL `op_id`	Out of v1 scope; the migration is forward-only. NULL-op_id rows are READ for resolution input (Inv-5) but never UPDATED	Future maintenance task; not ID-53
Cross-workspace / cross-tenant entity merging	Explicit v1 scope decision (Inv-21)	Future spec post-v1 if surfaced
ID-127 entity merge/split (KG-side; backlog)	RELATED but kept in backlog per RESEARCH §R7.4	Backlog ID-127; consider folding into T12 when promoted
Cocoindex-side App.on_complete / on_flow_complete callback	NOT BUILT — empirically ABSENT (RESEARCH §R1.6); Stage-5 attaches via in-line `app_main` continuation instead	Cocoindex upstream (no roadmap commitment)
Specific PairResolver LLM prompt + model choice	TECH.md `{53.3}` decision	T8 TECH (Inv-14 — Pydantic / prompt shape)
Specific embedder for `resolve_entities` (KH-owned, distinct from Stage-4 LiteLLMEmbedder)	TECH.md `{53.3}` decision	T8 TECH (Inv-3 / Inv-14)
Specific `entity_pair_resolutions` table schema (if chosen)	TECH.md `{53.3}` decision per P-OQ3 below	T8 TECH (Inv-14 cache substrate)
`requirements.txt` `faiss-cpu` exact pin alternatives (range pin vs exact pin)	RESEARCH lean ratified — exact `==1.14.2` (Inv-18)	TECH validates at impl time
Per-MIME extractor invariants (Docling / pullmd)	Already ratified upstream	`docs/specs/id-28-cocoindex-flow-scaffolding/{PRODUCT,TECH}.md`
`entity_aliases` table migration (rename / restructure)	Out of v1 scope; Stage-5 reads existing shape (Inv-10)	Future task
MCP / API freshness invalidation surface (TanStack Query / cache invalidation on `pipeline.complete` webhook)	Recommended mitigation (RESEARCH §R4.2 path 1) but app-side concern; PRODUCT only invariants the data-layer semantics	App-side / future task
Stage-5 audit_log emission shape	Inherits the peer T8 PRODUCT Inv-13 + P-OQ1 deferral pattern; v1 substrate is structured-log shipping (Inv-13 above)	Peer T8 PRODUCT + RLS-PATTERN v1.1
Cross-Task PLAN-level Subtask decomposition	`{53.4}` PLAN scope	Fresh Planner per Q-PLANNER-2

§4. Success criteria and validation framing

Stage-5 ratification at the Checker layer compares each Behaviour invariant (§2) against test artefacts produced by the Executor. The expected test surface (for TECH.md {53.3} and PLAN.md {53.4} to elaborate):

Integration tests under __tests__/integration/ or scripts/tests/ (Python pipeline tests): full canonical-pipeline runs that exercise cross-document dedup (Inv-3, Inv-5, Inv-11), op_id round-trip (Inv-6, Inv-7), Stage-5 failure-mode (Inv-12), unresolved-mention persistence (Inv-20), legacy alias preload (Inv-10), schema parity (Inv-15, Inv-16, Inv-17).
Unit tests for the per-document canonicalisation function (Inv-4): deterministic per-doc normalisation; idempotent under identical inputs.
Migration verification test: the entity_mentions op_id ALTER applies clean on staging, the partial index lands, the COMMENT is recorded; re-applying the migration is a no-op (Inv-6).
Test-philosophy alignment (per docs/reference/test-philosophy.md): tests verify real behaviour (cross-document dedup happens; canonical_name values match; op_id round-trips) not implementation details (specific embedder, specific PairResolver model, specific UPDATE-batch chunk size).

Per the canonical-pipeline-sequencing doc §1 critical-path framing, Stage-5 closing the unresolved-entities gap is what makes the corpus a “usable golden source” for entity queries. The acceptance bar is therefore: a cross-document-dedup ingest test PASSES (RESEARCH §R-RECOMMENDATION cited mitigation stack works end-to-end), AND no app-side write path silently overrides Stage-5 (Inv-8, Inv-9 verified by deliberate concurrent-write tests).

§5. Open Questions

Each P-OQ here is a behaviour-level uncertainty surfaced during PRODUCT.md authoring. The RESEARCH leans are ratified into invariants above where the evidence is unambiguous; the items below either (a) defer a concrete TECH-level shape (P-OQ3), (b) reserve a re-evaluation gate (P-OQ1), or (c) flag a known-narrow race window for TECH validation (P-OQ2). None blocks PRODUCT ratification; all carry recommended defaults applicable at {53.3} TECH dispatch.

ID	OQ	Inv where surfaced	Recommended default for TECH dispatch	Cost of deferral
P-OQ1 (RATIFIED-PERMANENT lean from RESEARCH)	Is the `managed_by=USER` row-only → row+update contract break (Inv-2) PERMANENT or TEMPORARY (revisit when cocoindex publishes a cross-doc reactive write API)?	Inv-2	PERMANENT. KH owns the row+update hybrid as the stable architectural pattern. Document explicitly in `docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md` §2 once Stage-5 lands. No re-evaluation TODO. The platform implications (RESEARCH §R4) are worked through; locking B with the mitigation stack is the honest stance. RESEARCH OQ-53-CONTRACT-BREAK leans (a).	LOW — if cocoindex eventually ships such an API (no roadmap as of 28/05/2026), revisit via spec amendment then; until then no value in carrying a re-evaluation tag.
P-OQ2	What is the narrow race window between an admin RPC (merge / split / type-override / metadata-edit) executed against rows from the in-flight run’s op_id and Stage-5’s UPDATE pass on the same rows?	Inv-9	Document + accept the same-window race. The race window is identical in shape and surface area to what the per-item `declare_row` already has against the same admin RPCs (RESEARCH §R4.1 — “this still leaves a narrow race window … but it’s the same window the per-item `declare_row` already has, so it doesn’t expand the failure surface”). TECH.md confirms via a deliberate concurrent-write test scenario that ratifies the existing race semantics for the post-pass surface. No new locking, no new transactional wrapping, no advisory-lock infrastructure — acceptable v1 risk.	LOW — same window already accepted at v1 for the per-item phase; not a Stage-5-specific surface.
P-OQ3	Where does the `PairResolver` determinism cache (Inv-14) live?	Inv-14	NEW `entity_pair_resolutions` table — RESEARCH §R-RECOMMENDATION recommendation 3 + OQ-53-PAIR-RESOLVER-CACHE option (b). Shape (TECH commits): `(name_a text, name_b text, decision text, model_version text, created_at timestamptz, PRIMARY KEY (name_a, name_b))`. Clean separation from the legacy admin-curated `entity_aliases` table (RESEARCH §R4.4) means PairResolver pollution does not contaminate the admin surface. Migration cost: 1 CLI migration (CREATE TABLE + indices + COMMENT). Alternative (extend `entity_aliases` with `provenance='cocoindex_pair_resolver'`) is REJECTED — pollutes the legacy alias-curation surface with machine-generated rows. Alternative (in-memory only) is REJECTED — loses cross-run idempotency, contradicting Inv-14.	LOW — the table shape is constrained; the migration mirrors the entity_mentions op_id migration’s CLI pattern.

§6. Glossary

New terms introduced in this PRODUCT.md (terms inherited from the peer cocoindex-flow-scaffolding/PRODUCT.md and RESEARCH.md are not restated):

Term	Definition
Per-document canonical	The deterministic `canonical_name` value produced by the per-item `ingest_file` phase via a pure-function `canonicalise_entity_name(name, type)` — lowercase + strip + ASCII-fold + entity_type-aware normalisation. Inv-4.
Cross-document canonical	The `canonical_name` value produced by the Stage-5 post-pass via `cocoindex.ops.entity_resolution.resolve_entities` over the run’s per-document canonicals (with legacy `entity_aliases` preloaded per Inv-10). Inv-3, Inv-10.
Stage-5 post-pass	The flow-scope UPDATE phase running between `await handle.ready()` and the flow-end webhook emit, scoped to op_id-matching rows. Inv-1, Inv-5.
PairResolver	The KH-owned LLM-backed callback supplied to `resolve_entities` for tier-breaking ambiguous near-matches. Inv-14.
Determinism cache	The persistent store backing PairResolver decisions, keyed by `(name_a, name_b)`. Concrete shape per P-OQ3. Inv-14.
op_id round-trip	The audit-forensics property that any `entity_mentions.op_id` value resolves back to its `pipeline_runs` row via a single PK SELECT. Inv-6. Inherits the peer T8 PRODUCT Inv-12 semantics for `entity_mentions`.
Memo-respecting op_id (Stage-5)	The semantic that `entity_mentions.op_id` records the run that LAST MATERIALLY changed the row; unchanged re-ingests do NOT re-stamp `op_id`; `full_reprocess` DOES; Stage-5 UPDATEs do (because the row IS being materially changed). Inv-7. Inherits peer PRODUCT.md Inv-11 (RATIFIED OQ-A).

§7. Coverage matrix

Required area (from S274 dispatch brief)	PRODUCT.md invariants	Status
Architecture invariant — Option B as flow-scope post-fan-out at `mount_each.handle.ready()` boundary	Inv-1, Inv-2 (2 invariants)	Covered
canonical_name freshness contract	Inv-3, Inv-4 (2 invariants)	Covered
op_id scoping invariant + migration	Inv-5, Inv-6, Inv-7 (3 invariants)	Covered
App-side coexistence invariant (5 paths)	Inv-8, Inv-9, Inv-10 (3 invariants — `classifyContent`, Admin curation, legacy `entity_aliases`)	Covered
Observability invariant — `stage_counts` + webhook + structured log	Inv-11, Inv-13 (2 invariants)	Covered
Failure semantics — non-destructive to per-item writes	Inv-12 (1 invariant)	Covered
PairResolver determinism cache	Inv-14 (1 invariant + P-OQ3)	Covered
Schema parity (3 RESEARCH-surfaced gaps)	Inv-15, Inv-16, Inv-17 (3 invariants)	Covered
Substrate / dependency invariants — faiss pin + T12 sequencing	Inv-18, Inv-19 (2 invariants)	Covered
Resolution outcomes — unresolved mentions + single-workspace scoping	Inv-20, Inv-21 (2 invariants)	Covered

Total: 21 numbered Behaviour invariants distributed across 9 areas (A through I). 3 P-OQs left open for TECH ratification (P-OQ1 contract-break permanence; P-OQ2 narrow race window documentation; P-OQ3 PairResolver cache concrete shape). 6 RESEARCH OQs ratified into invariants where evidence is unambiguous: OQ-53-FAISS-PIN → Inv-18; OQ-53-T12-SEQUENCING → Inv-19; OQ-53-PRE-EXTRACTION-LATE-DROP → Inv-20; OQ-53-CONTEXT-SNIPPET-COMPUTE → Inv-17. (OQ-53-CONTRACT-BREAK → P-OQ1; OQ-53-PAIR-RESOLVER-CACHE → P-OQ3.)

§8. References

§8.1 Primary sources (read in full for this PRODUCT)

docs/specs/id-53-stage-5-entity-resolution/RESEARCH.md — ratified RESEARCH (Checker PASS_WITH_NOTES at commit 369e77be; nit-fixed; 686 lines + 4 nit-fix lines).
docs/specs/id-28-cocoindex-flow-scaffolding/PRODUCT.md — peer PRODUCT (invariant-numbering convention, structural mirroring, Inv-3 / Inv-11 / Inv-17 / Inv-25 cross-link).
docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md §1 (critical-path framing) + §2.5 (why not the obvious cocoindex ops — faiss-cpu add) + §3 (T12 UNPROMOTED) + §6 (backlog folds — ID-127, ID-158).
docs/reference/task-list.json ID-53 record (status spec_needed, priority high, dependencies []).
scripts/cocoindex_pipeline/flow.py (specifically line 168 _PIPELINE_ERROR_CLASSES, line 278 _emit_stage_error_log, line 382 _empty_stage_counts, line 864 extract_entity_mentions discard site, lines 1060–1069 mount_each.handle.ready() attach point, line 1135 _emit_pipeline_run_webhook).
supabase/migrations/20260521203414_t8_op_id_propagation.sql — peer migration that ID-53’s op_id migration mirrors (T8 pattern for content_items / q_a_extractions / source_documents).
lib/entities/entity-aliases.ts:59,92 — legacy entity_aliases map shape (Inv-10).
CLAUDE.md — DDL via CLI only; UK English; canonical-pipeline gotchas.

§8.2 Cross-spec relations

Predecessor: docs/specs/id-53-stage-5-entity-resolution/RESEARCH.md — ratified S274 (commit 369e77be).
Successor (fresh Planner per Q-PLANNER-2): docs/specs/id-53-stage-5-entity-resolution/TECH.md {53.3} — implements one Proposed change per PRODUCT invariant, with the 3 P-OQ defaults applied (or amended at Liam ratification gate). TECH writes the op_id migration SQL verbatim per RESEARCH §R6.2.
Decomposition (conditional, after TECH): docs/specs/id-53-stage-5-entity-resolution/PLAN.md {53.4} — decomposes ratified PRODUCT + TECH pair into TM-shape implementation Subtasks {53.5+} if the estimated effort exceeds the 2h single-slice threshold (RESEARCH §R3.3 estimate: 5–7h impl). Effort estimate suggests PLAN is warranted.
Sibling specs (already ratified, cross-linked): docs/specs/id-28-cocoindex-flow-scaffolding/{PRODUCT,TECH}.md (S265 ratification; T8 substrate Stage-5 inherits); docs/specs/id-36-cocoindex-extraction-contract/PRODUCT.md (Q-EX2 contract; Stage-5 inherits EntityMentionExtraction shape).
Downstream consumers (UNPROMOTED, sequenced after Stage-5 stable): T12 (Mempalace KG integration); ID-127 (entity merge/split KG-side).
Cross-Task refs (independent): ID-54 (Path-A q_a_extractions lossy fix — independent per RESEARCH §R7.1); T10 (procurement-question-matching — independent per RESEARCH §R7.2).

End of PRODUCT.md. S274 ID-53.2 ratification gate: 21 Behaviour invariants; Option B architecture (Inv-1) ratified with mitigation stack distributed across Inv-5 (op_id scoping), Inv-8/9 (app-side coexistence), Inv-10 (legacy alias preload), Inv-14 (PairResolver determinism cache), Inv-18 (faiss-cpu pin), Inv-19 (T12 sequencing gate), Inv-6/7 (op_id migration + memo-respecting semantic); 3 P-OQs deferred to {53.3} TECH. Re-grounded substrate verified against scripts/cocoindex_pipeline/flow.py HEAD; no ABSENT / SIGNATURE_DRIFT findings introduced beyond RESEARCH §R1’s empirical evidence.