ID-101 PLAN — cocoindex entity-relationship + holder-rule extraction parity

ID-101 — cocoindex entity-relationship + holder-rule extraction parity (PLAN)

Spec chain: {101.1} RESEARCH (folded into Task description) → {101.2} PRODUCT (done) → {101.3} TECH (done) → {101.4} PLAN (this document) → implementation Subtasks {101.5}–{101.9}. Artefact kind: {N.4} PLAN. Decomposition only — no new behaviour or mechanism is introduced here; this maps the ratified TECH PCs to ordered, sibling-only implementation Subtasks. Invariant numbers (Inv-1 … Inv-16) refer to PRODUCT.md §Behavior; PC numbers (PC-1 … PC-7) refer to TECH.md §Proposed changes. Fresh-Planner note (Q-PLANNER-2 / B4): authored by a fresh Planner instance, not the PRODUCT or TECH author. Both predecessor specs were read in full; the code locations the TECH spec cites were re-verified against the working tree before decomposition (see §Verified code locations).

Overview

The cocoindex ingest path must capture directed/typed relationship triples (the 10-type vocabulary) and self/supplier certification-holder attribution at parity with the legacy TS classifyContent path, before the ID-45 full-corpus re-ingest. This is extraction-only: no schema DDL (the entity_relationships table + 10-type CHECK + NULLS NOT DISTINCT unique tuple already exist), no consumption surface (ID-60/ID-71 out of scope).

TECH.md decomposes the work into seven Proposed changes across four Python files plus two test surfaces. This PLAN groups those PCs into five vertical slices, ordered by dependency, each leaving the pipeline in a working state. The headline risk (TECH R1 — the two canonicalisers diverge today) is front-loaded into slice 1 as a hard gate so cross-path triples cannot silently fork post-re-ingest.

Architecture decisions (inherited from TECH — restated for the Executor)

• Mechanism (Inv-1): a new @coco.fn(memo=True) relationship extractor calling the Anthropic SDK directly + Pydantic TypeAdapter validation, mirroring the three shipped ID-94 extractors (extract_classification, extract_qa_form, extract_entity_mentions). NOT cocoindex.ExtractByLlm / cocoindex.LlmSpec — both empirically ABSENT in cocoindex[postgres]==1.0.7 (re-confirmed; see §Verification).
• Canonicaliser port (PC-3 — the R1 seam): a dedicated canonicalise_for_relationship(name) reproducing the TS resolveAlias(canonicalise(x)).toLowerCase() chain step-for-step. Do NOT reuse canonicalise_entity_name — it is the ISO-only per-mention canonicaliser and conflating the two reintroduces the divergence. v1 ports BASELINE_ALIASES; the DB alias load is a documented follow-up gated on oracle evidence.
• No DDL: er_target mounts with managed_by=ManagedBy.USER (rows only). The table + constraints already exist.
• Best-effort / non-blocking (Inv-7, Inv-15): the relationship write + holder stamp run inside the same flow-scope try/except that already guards ingest_file; a failure for one document logs (not silently swallows) and does not abort that document’s other declares or the batch.
• Idempotency (Inv-6): deterministic uuid5 PK seeded on the natural triple tuple, mirroring the entity_mentions natural-key PK (em:{rel_path}:{canonical}:{type}).

Code-intelligence orientation (Inv 2)

Per .gitnexus/CLAUDE.md “Always Do” + .ast-dataflow/CLAUDE.md (ast-dataflow / ts-morph does not cover the Python pipeline — Python surface mapped with grep).

gitnexus_query({query: 'entity relationship extraction holder', repo: 'knowledge-hub'}) → processes: [], process_symbols: []. No named execution flow indexes the cocoindex relationship/holder surface (it does not exist yet) — greenfield write tail on the existing ingest_file component. The definitions block confirmed the legacy oracle + downstream-consumer symbols verbatim:

• Function:lib/ai/classify.ts:deriveHolderMetadata (lines 524–595, module Entities) — the function the Python holder-stamp must reproduce.
• Interface:lib/ai/classify.ts:ExtractedRelationship (lines 652–666) — the 10-member relationship union (the relationship_type parity contract).
• Function:scripts/eval-holder-rule-ts.ts:fetchHoldsRelationships (lines 385–401), runEvaluation (lines 694–1081) — the existing holder-rule oracle to extend for the cross-path eval (PC-6 lane 3).
• Downstream consumers that must not be able to tell which path wrote a row (Inv-16): lib/mcp/formatters/entities.ts:EntityRelationship (22–28), formatEntitySummary (40–95); app/api/certifications/route.ts:GET (74–319); scripts/quality-gate.ts:entity_relationship_coverage (772–848), audit_required_relationships (1472–1534).

gitnexus_context({name: 'deriveHolderMetadata', repo: 'knowledge-hub'}) → verdict-level context: 1 direct caller (classifyContent); outgoing calls to lib/entities/entity-aliases.ts:resolveAlias and lib/entities/entity-dedup.ts:canonicalise; processes: []. This is the exact TS function (lines 524–595) the Python derive_holder_metadata port mirrors.

gitnexus_context({name: 'canonicalise', repo: 'knowledge-hub'}) (per TECH) → 6 direct callers, HIGH blast-radius on the TS side — but this Task does NOT modify canonicalise; it ports its behaviour into Python. The TS function is the oracle, not a modification target (PRODUCT §Non-goals). Executors porting it run gitnexus_impact only on any Python symbol they edit; the TS canonicaliser is read-only.

Verified code locations (re-checked against the working tree, 09/06/2026)

The Executor should not need to re-read the full specs — these are the load-bearing locations, confirmed present:

Symbol / site	Location (verified)
Legacy relationship writer	lib/ai/classify.ts:1785-1819 — resolveAlias(canonicalise(x)).toLowerCase() both endpoints, confidence: 1.0, upsert(..., { onConflict: 'source_entity,relationship_type,target_entity,source_item_id', ignoreDuplicates: true }) in try/catch
deriveHolderMetadata	lib/ai/classify.ts:524-595; HOLDS_SYNONYMS (complies_with, evidences) at :522; Pass-2 builds certTargets/orgSources from row.canonical_name then compares vs resolveAlias(canonicalise(rel.x))… — the R2 mismatch is visible here
ExtractedRelationship 10-type union	lib/ai/classify.ts:652-666
TS canonicalise (12-step)	lib/entities/entity-dedup.ts:114 (ABBREVIATIONS map :9, slugToProperCase :69, titleCase :83)
TS resolveAlias + baseline	lib/entities/entity-aliases.ts:92 (resolveAlias), BASELINE_ALIASES :16 (DB-backed cache fallback)
Prompt source (port verbatim)	lib/ai/skills/classification.md:698-778 (§Relationship Extraction + §Holder Disambiguation)
Holder-rule oracle	scripts/eval-holder-rule-ts.ts — fetchHoldsRelationships :385, runEvaluation :694, NEXT_PUBLIC_CLIENT_ID guard :756
Python per-doc canonicaliser (do NOT reuse for rels)	scripts/cocoindex_pipeline/canonicalisation.py:22 — canonicalise_entity_name(name, entity_type), NFKD + lowercase + ISO-only
Shipped @coco.fn extractors (precedent)	scripts/cocoindex_pipeline/extraction.py — extract_classification :988, extract_qa_form :1016, extract_entity_mentions :1042; module-level TypeAdapters :773-779; _MAX_TOKENS_* :796-798; ANTHROPIC_MODEL = "claude-opus-4-6" :70; _ExtractionCore :225; EntityMentionExtraction :363; TestNoFictionalApiSurvives survey at :1233 (test file)
Prompt constants home	scripts/cocoindex_pipeline/prompts.py — CLASSIFICATION_PROMPT :48, Q_A_FORM_PROMPT :87, ENTITY_MENTION_PROMPT :139
ENTITY_MENTIONS_SCHEMA	scripts/cocoindex_pipeline/flow.py:1267
ingest_file signature	flow.py:1581 — positional targets ci, qa, sd, em, ft, ftf, cc=None, then keyword-only * block. Insert er_target after em_target, before ft_target
_KH_PIPELINE_DOC_NS (uuid5 seed)	flow.py:1548; content_item_id mint :1839/:1952
em dedup + declare loop	flow.py:2161 (_em_dedup), :2173-2200 (declare loop; metadata = { source_span_start, source_span_end } at :2195-2198)
mount_table_target for em_target	flow.py:3026-3031 (managed_by=ManagedBy.USER) — add er_target block beside it
coco.mount_each content call	flow.py:3154-3166 — positional targets ci, qa, sd, em, ft, ftf, cc; add er_target after em_target
Write-path test surface	scripts/tests/test_cocoindex_flow_write_path.py — TestIngestFileWritePath :150, TestMountEachArityContract :537 (the _faithful_mount_each(flow.ingest_file, feed, ci, qa, sd, em, None, None) call at :605 — arity must grow), test_ingest_file_signature_matches_mount_each_extra_args :637, TestStablePrimaryKeysAcrossRuns :698, TestNoFictionalApiSurvives :1233

Guard-test caveat (carried from CLAUDE.md): CLAUDE.md names pipeline-parity.test.ts as a guard that runs on every test, but no such file resolves in the current tree (__tests__/ search returns only unrelated *-parity.test.ts files). mcp-fixture-sync.test.ts does exist at __tests__/mcp/mcp-fixture-sync.test.ts. Slices 6 and 7 instruct the Executor to run bun run test after their changes and update any guard fixture that trips — but must not assume pipeline-parity.test.ts exists; verify-if-present, do not fabricate.

Dependency graph

{101.5} Canonicaliser port + golden test (PC-3, PC-6 lane1)  ─┐  [R1 hard gate, no deps]
                                                              │
{101.6} Relationship extractor + prompt (PC-1, PC-2)  ────────┤  [no deps, parallel with .5]
                                                              │
                          ┌───────────────────────────────────┘
                          ▼
{101.7} Write target + declare-row loop + arity tests (PC-4, PC-7)   [deps .5, .6]
                          │
                          ▼
{101.8} Holder-rule port + stamp wiring + holder unit parity (PC-5, PC-6 lane2)  [deps .5, .7]
                          │
                          ▼
{101.9} Cross-path parity eval extension (PC-6 lane3)   [deps .6, .7, .8]

Bottom-up: the canonicaliser (slice 5) is foundational — every later slice canonicalises through it. The extractor (slice 6) is independent of the canonicaliser (it produces raw triples; canonicalisation happens at the write site), so 5 and 6 are parallelisable. The write loop (7) needs both. Holder stamping (8) needs the canonicaliser + the em declare loop that slice 7 edits (sequenced after 7 to avoid a merge collision on the declare loop). The cross-path eval (9) needs both paths wired.

Task list

Phase 1: Foundations (parallelisable)

{101.5} — Cross-language canonicaliser port + golden test (PC-3, PC-6 lane 1)

The R1 hard gate. Add canonicalise_for_relationship(name: str) -> str to scripts/cocoindex_pipeline/canonicalisation.py reproducing the TS resolveAlias(canonicalise(name)).toLowerCase() chain step-for-step (12-step canonicalise body + BASELINE_ALIASES port + .lower()), called with no entity-type arg so the type-aware plural branch stays inert (matching the legacy relationship-writer call shape). Do NOT modify canonicalise_entity_name. Build the shared cross-language golden fixture scripts/tests/fixtures/canonicalisation_parity.json of [{raw, expected}] pairs covering every branch (slug, ISO variants, Cyber Essentials, WCAG, Ltd→Limited, abbreviation-map, multi-word title-case, baseline-alias entries) and the two assertions reading it: a Python pytest (canonicalise_for_relationship(raw) == expected) and a TS Vitest (resolveAlias(canonicalise(raw)).toLowerCase() === expected). This fixture is the gate that must pass before ID-45 re-ingest.

Acceptance: Python + TS read the same fixture and agree byte-for-byte on every pair including divergence cases (Acme Ltd → acme limited, WCAG, abbreviation-map). Existing canonicalise_entity_name callers unaffected.

{101.6} — Relationship extractor @coco.fn + Pydantic model + ported prompt (PC-1, PC-2)

Add RelationshipExtraction(_ExtractionCore) (source/target min_length=1, relationship: Literal[...] enumerating exactly the 10 types — the Inv-4 enforcement; no op_id/content_items_id per the bl-220 stamp-free convention) and module-level _relationships_adapter: TypeAdapter[list[RelationshipExtraction]] beside the existing three (:773-779) in scripts/cocoindex_pipeline/extraction.py. Add @coco.fn(memo=True) async def extract_relationships(content_text: str) -> list[RelationshipExtraction] as a verbatim structural copy of extract_entity_mentions (anthropic.AsyncAnthropic(), _anthropic_retry → _anthropic_message, _guard_not_truncated, _strip_code_fence, _relationships_adapter.validate_json), with _MAX_TOKENS_RELATIONSHIPS = 16384. Port RELATIONSHIP_PROMPT into scripts/cocoindex_pipeline/prompts.py verbatim from lib/ai/skills/classification.md:698-778 (10-type table + full §Holder Disambiguation: trigger phrases, disclaimer-paragraph rule, supplier-attribution example) — JSON-array output of { source, relationship, target }, “if none found return []” (Inv-8), UK English, no fences. No ExtractByLlm/LlmSpec (Inv-1).

Acceptance: invalid predicate → ValidationError; empty doc → []; RELATIONSHIP_PROMPT contains the verbatim trigger-phrase + disclaimer rules; no ExtractByLlm/LlmSpec reference (extend TestNoFictionalApiSurvives).

Checkpoint: Foundations

• {101.5} golden test passes both languages (the R1 gate is green).
• {101.6} extractor + prompt land; TestNoFictionalApiSurvives extended and green.
• bun run test + python3 -m pytest scripts/tests/ (from worktree CWD) clean.

Phase 2: Write path

{101.7} — entity_relationships write target + declare-row loop + arity tests (PC-4, PC-7) — deps {101.5}, {101.6}

In flow.py: add ENTITY_RELATIONSHIPS_SCHEMA beside ENTITY_MENTIONS_SCHEMA (:1267) — id, source_entity, relationship_type, target_entity, source_item_id, confidence, op_id; primary_key=("id",). Mount er_target with managed_by=ManagedBy.USER beside the em_target block (:3026). Thread er_target through the ingest_file signature (after em_target, before ft_target :1581), the bound_ingest_file/_ingest_file_impl chain, and the coco.mount_each content call (:3154). Add a if relationships:-guarded declare-row loop after the em loop (:2200): canonicalise both endpoints via canonicalise_for_relationship ({101.5}), defensively drop out-of-10-set predicates (skip+log, no crash — Inv-4), de-dup identical tuples within the doc into a dict keyed (source_c, relationship_type, target_c), mint deterministic PK uuid5(_KH_PIPELINE_DOC_NS, f"er:{rel_path}:{source_c}:{relationship_type}:{target_c}") (Inv-6), declare { ..., source_item_id: content_item_id, confidence: 1.0, op_id }, _bump("postgres_upsert") per row. Update the arity guards in test_cocoindex_flow_write_path.py (TestMountEachArityContract :605 feed, the signature test :637) and add TestIngestFileRelationshipWritePath (payload shape, canonical endpoints, confidence=1.0, deterministic PK, zero-rows-on-empty, idempotent across two runs).

Acceptance: declare-row payload matches legacy column set with confidence=1.0 and canonical endpoints; empty extractor → zero declares; two ingest runs → identical PKs / no duplicate rows; arity tests pass with the new positional arg.

{101.8} — Holder-rule port + stamp wiring + holder-unit parity (PC-5, PC-6 lane 2) — deps {101.5}, {101.7}

Port deriveHolderMetadata (classify.ts:524-595) into a new scripts/cocoindex_pipeline/holder_rule.py as derive_holder_metadata(mentions, relationships, client_org_lower) -> dict returning a holder_by_mention_id map. Pass-1 canonical holds (holds_by_target[target_c] = source_c via canonicalise_for_relationship, last-wins). Pass-2 S196 synonym fallback (complies_with/evidences, cert target, client-org-or-extracted-org source, holds wins on tie). R2 bridge (load-bearing): build the cert_targets/org_sources membership sets by applying canonicalise_for_relationship to each mention’s raw entity_name — NOT the stored canonicalise_entity_name canonical_name — so both sides compare in relationship-canonical space; document this in the docstring. Stamp only certification mentions (Inv-14): {holder:'self'} when source == client_org_lower, else {holder:'supplier', supplier_name}; leave untouched mentions with no signal (Inv-10 — never default to 'self'). Source client_org_lower from a required pipeline client-org env knob, fail-fast if unset (R4 — mirror the eval-holder-rule-ts.ts:756 guard). Wire into the em declare loop (:2173-2199) by merging holder keys into the existing metadata dict: metadata = { "source_span_start": ..., "source_span_end": ..., **holder_md } — preserve the span keys. Add test_holder_rule_parity.py covering self / supplier-disclaimer / S196-synonym / untouched-not-self (Inv-10) / non-cert (Inv-14) / R2-divergence cases. Run inside the same best-effort try/except (Inv-15: stamp failure logged, em declare still happens with span-only metadata).

Acceptance: holder-unit parity test green across all six cases; em metadata still carries span keys (holder keys merged, not overwritten); no-signal cert has no holder key; non-cert never stamped; unset client-org env → fail-fast (logged, not silent).

Checkpoint: Write path

• Both paths write entity_relationships rows with identical shape (Inv-16).
• Holder metadata stamped at parity; Inv-10 untouched-not-self holds.
• python3 -m pytest scripts/tests/ (worktree CWD) + bun run test clean.

Phase 3: Empirical confidence gate

{101.9} — Cross-path parity eval extension (PC-6 lane 3) — deps {101.6}, {101.7}, {101.8}

Extend scripts/eval-holder-rule-ts.ts: add a --path=cocoindex|legacy|both mode + a triple-set comparator. Reuse the existing fetchHoldsRelationships/fetchEntityMentions/ four-threshold machinery against a small pinned fixture corpus (residual + positive-control items the script already knows). Run each fixture document through BOTH paths and compare on set-equality of {source_entity, relationship_type, target_entity} after canonicalisation (Inv-2, order-tolerant) + exact holder-state match per cert (Inv-9). Non-determinism handling: compare on the canonicalised triple set; treat a one-path-only triple as a parity miss only if it recurs across N≥3 runs; pin model + prompt. Keep --mode=run legacy behaviour intact (it remains the TS oracle of record). Supports --dry-run first.

Acceptance: --path=both over the pinned corpus shows triple-set parity (after canonicalisation, N≥3) and per-cert holder-state match; the existing --mode=run legacy behaviour is unchanged.

Checkpoint: Complete

• All 16 PRODUCT invariants verified per the TECH §Testing-and-validation table.
• The PC-6 lane-1 golden test (R1 gate) is green — prerequisite for ID-45 re-ingest.
• bun run test + python3 -m pytest scripts/tests/ (worktree CWD) clean; ready for Checker review.

Risks and mitigations (carried from TECH §Risks)

Risk	Impact	Slice	Mitigation
R1 — canonicalisers diverge today (the central risk)	HIGH	{101.5}	Dedicated canonicalise_for_relationship port + shared cross-language golden fixture as the hard pre-re-ingest gate. Do NOT reuse canonicalise_entity_name.
R2 — em canonical_name vs rel-canonical mismatch in holder Pass-2 set membership	MEDIUM	{101.8}	Build cert/org sets by applying canonicalise_for_relationship to raw entity_name; lane-2 fixture where the two canonicalisers would otherwise disagree.
R3 — ingest_file arity change is a breaking signature change	MEDIUM	{101.7}	Update arity tests in the same slice; insert er_target after em_target, before the keyword-only *, to keep mount-each extra-args order stable.
R4 — client-org env unset → mass mis-attribution as supplier	LOW	{101.8}	Fail-fast guard in derive_holder_metadata; best-effort wrapper logs the raise rather than corrupting metadata.
R5 — prompt-cache invalidation churn	LOW	{101.6}	Port RELATIONSHIP_PROMPT once, verbatim, freeze it; lane-1 fixture covers canonicalisation not prompt text.

Sibling-only dependency check (§3.3 / A6 forcing function): PASS

Every Subtask dependency is intra-ID-101 (slice→slice within the same Task): {101.7} ← {101.5}, {101.6}; {101.8} ← {101.5}, {101.7}; {101.9} ← {101.6}, {101.7}, {101.8}. No cross-Task Subtask dependency surfaced. The ID-45 relationship is a Task-level ordering gate (the C1 pre-re-ingest gate; Task.dependencies — ID-45 should depend on ID-101), not a Subtask-level dependency — consistent with both predecessor specs. No escalation required.

25-Subtask soft ceiling (§3.4 / A7): 5 of 25 — well within cap

Five implementation Subtasks. No split signal.

Verification

Per the Q-EX2 pre-ratification empirical-verification forcing function:

• Date: 09/06/2026.
• Pinned versions: cocoindex[postgres]==1.0.7 (requirements.txt:54), anthropic==0.79.0 (requirements.txt:3) — re-confirmed.
• External symbols (cocoindex): cocoindex.ExtractByLlm, cocoindex.LlmSpec — carried from PRODUCT/TECH §Verification (both ABSENT on the pin). No PLAN-introduced external symbol; the decomposition reuses only the shipped @coco.fn + anthropic.AsyncAnthropic + Pydantic TypeAdapter pattern, all PRESENT and in-use on the pin.
• Internal code locations re-verified against the working tree (see §Verified code locations) — all cited line numbers confirmed present at decomposition time; the legacy relationship writer, deriveHolderMetadata (incl. the visible R2 mismatch), the shipped extractors + adapters, and the flow.py write/mount/arity sites all match the TECH citations. The ingest_file positional contract (ci, qa, sd, em, ft, ftf, cc) and the _faithful_mount_each(... ci, qa, sd, em, None, None) arity-test call were confirmed so the {101.7} arity-change brief is accurate.
• Guard-test note: pipeline-parity.test.ts (named in CLAUDE.md) does not resolve in the current tree; mcp-fixture-sync.test.ts exists at __tests__/mcp/. Slices instruct verify-if-present, not fabricate.