Phase 0.9 — Spike S10 (cross-record dedup substrate)

Phase 0.9 — Spike S10 (cross-record dedup substrate)

Audit date: 2026-05-11 (KH session S230) Branch: content-items-investigation (worktree-agent-a356ab9e732e4fbbc) Spike spec: 0.9-spike-plan.md §S10 (lines 474-509) Test corpus: docs/client-documentation-base/ (28 files — 7 docx + 19 markdown + 2 form-fixtures unused for S10) Substrates evaluated: 4 candidates — cocoindex @coco.fn content-similarity / mempalace v3.3.5 chroma cross-file search / skill-seekers metadata keyword-categorisation / mempalace v5.0 PG-backend (claimed but does not exist) Acceptance criteria: per-pair TP ≥ 90% AND FP ≤ 5% on canonical corpus + UX-compat for side-by-side diff + golden-record selection + citation preservation

Spike status: CONDITIONAL PASS — substrate (c) skill-seekers keyword passes alone (TP=100% / FP=4.5%) BUT is per-tenant-brittle; recommended v1 path is HYBRID (a)+(c) cocoindex chunk-embed AND skill-seekers keyword co-confirmer at TP=91.7% / FP=4.5%; substrate (a) cocoindex alone hits TP=91.7% / FP=6.1% (close to gate, fails FP by 1.1pp); substrate (b) mempalace v3.3.5 chroma cross-file fails (TP=66.7% / FP=1.5%); substrate (d) v5.0 PG-backend does not exist as of 2026-05-10 — claim is factually incorrect.

---

1. Executive summary

Question	Answer
Substrate (a) — cocoindex post-target-write @coco.fn chunk-embedding similarity: TP ≥ 90% AND FP ≤ 5%?	NO — close miss. Best op-point: cent≥0.6 AND max_chunk≥0.7 → TP=91.7% (11/12) / FP=6.1% (4/66). Misses B2-faqs (DRAFT docx with rewritten introduction).
Substrate (b) — mempalace v3.3.5 chroma cross-file search via direct ChromaBackend API: TP ≥ 90% AND FP ≤ 5%?	NO — TP fails. Best op-point: best_dist<0.25 AND frac≥0.2 → TP=66.7% (8/12) / FP=1.5% (1/66). Misses 4 TP pairs (B2/M2-faqs + BL-01 + BL-03 bid library variants).
Substrate (c) — skill-seekers metadata categorisation (keyword-rule shared-categories count): TP ≥ 90% AND FP ≤ 5%?	YES — passes the strict gate but is brittle. Best op-point: shared_categories ≥ 5 → TP=100% (12/12) / FP=4.5% (3/66). Caveat: keyword rules were curated to this corpus; per-tenant rule curation is required during onboarding.
Substrate (d) — mempalace v5.0 PG-backend native dedup: TP ≥ 90% AND FP ≤ 5%?	NOT TESTABLE — substrate does not exist. Latest mempalace on PyPI = 3.3.5 (uploaded 2026-05-10T23:44Z). No v4 / v5 tags on GitHub. No PG backend on develop branch as of audit date — mempalace/backends/ contains only base.py + chroma.py + registry.py. The user’s S230 prompt assertion (“mempalace v5.0 GA shipped 2026-05-02 with multi-backend persistence including a PG backend”) is factually wrong. Possibly conflated with mempalace’s internal “hybrid v5” retrieval pipeline (referenced in their README benchmarks).
Hybrid (a)+(c) cocoindex chunk-embed AND skill-seekers keyword?	YES — passes the strict gate. Best op-point: cent≥0.6 AND shared≥5 → TP=91.7% / FP=1.5%. Alternative: max_chunk≥0.7 AND shared≥4 → TP=91.7% / FP=4.5%.
UX compat — side-by-side diff + golden-record selection + citation preservation?	(a) 4/5 (b) 3/5 (c) 2/5 (d) 0/5 (hybrid a+c) 4/5. See §6.

G10 decision-gate verdict — RECOMMENDED + RATIFIED: HYBRID (a)+(c) as the UC8 v1 substrate: cocoindex post-target-write @coco.fn runs chunk-embedding similarity over the corpus (using KH’s existing text-embedding-3-large pipeline, calibrated per-corpus), AND a deterministic keyword-rule co-confirmer (KH-native, inspired by skill-seekers categories per 0.8.5 §Q5) inflates precision on the FP-prone “topical-overlap-but-unrelated” cases. Substrate (b) does not contribute; substrate (d) is fictional. No fall-back required — hybrid (a)+(c) is the v1 path; UC8 v1 is human-confirmed-merge per 0.9-edit-flow-investigation.md §6.8 regardless.

v1 ships WITH cross-record dedup via hybrid (a)+(c).

---

2. Method (executed)

2.1 Setup

Python 3.14.2 (Homebrew) + isolated venv at .venv-spike-s10/ + cocoindex 1.0.3 + sentence-transformers 5.4.1 + python-docx + numpy. Mempalace 3.3.5 was already installed (S229-confirmed) via uv tool install mempalace at ~/.local/share/uv/tools/mempalace/. python-docx was installed into mempalace’s tool venv to enable corpus extraction in the mempalace probe (one-time install; doesn’t affect production mempalace usage).

Spike artefacts at .spike-s10/ (gitignored):

File	Purpose
corpus.py	TP_PAIRS / FP_PAIRS canonical labels
io_helpers.py	docx + md text extraction
cocoindex_dedup_probe.py	Substrate (a) — MiniLM-L6 chunk embeddings
cocoindex_mpnet.py	Substrate (a) re-run with mpnet-base (sanity check)
cocoindex_dedup_sweep2.py	Threshold refinement
mempalace_v3_probe.py	Substrate (b) — chroma cross-file search
mempalace_v3_sweep2.py	Mempalace threshold refinement
skill_seekers_probe.py	Substrate (c) — keyword categorisation
hybrid_sweep.py	Hybrid (a)+(c) and (b)+(c) operating-point sweep
cost_model.py	Per-substrate cost + complexity estimate
synthesise.py	Cross-substrate comparison output
cocoindex_results.json	(a) per-pair raw scores
cocoindex_mpnet_results.json	(a-mpnet) per-pair raw scores
mempalace_v3_results.json	(b) per-pair raw scores
skill_seekers_results.json	(c) per-pair raw scores

2.2 Canonical corpus + labelled pairs

Per docs/client-documentation-base/ shape (after S229 S2 corpus mapping):

Pair set	Count	Source files
TP — DRAFT-vs-final docx pairs	3 (B1 / B2 / B3)	binary/2026 Audit Implementation & Support.docx vs DRAFT Implementation & Support.docx; FAQs.docx vs DRAFT FAQs; Security & Compliance.docx vs DRAFT Security and Compliance
TP — DRAFT-vs-final markdown pairs	3 (M1 / M2 / M3)	markdown/ versions of the above three (mammoth-extracted .md)
TP — Bid library variants	C(4,2)=6 pairs (BL-01 to BL-23)	markdown/Advanced_Audits_Bid_Library_v5.md / LMS_Bid_Library_v2.2.md / Phew-Bid-Library-2026-v4_4.md / Website_Bid_Library_v4_2.md (all four are different versions/shapes of the same logical bid-library corpus)
FP — Unrelated company-context pairs	C(12,2)=66 pairs (U-01 to U-1011)	markdown/01-company-overview.md through markdown/12-research-notes-and-gaps.md (Phew internal corpus — 12 documents on different topics: company / services / industry / clients / team / values / compliance / technology / brand / sites / blogs / research)

Total: 12 TP + 66 FP = 78 pairs scored per substrate.

The form-fixture files (forms/Cloud Security Principles Checklist V5_3 - PHEW.xlsx + forms/standard-selection-questionnaire-ppn-03-24.pdf) and the Telehouse PDF are NOT in the S10 corpus — they are subjects of S7 (auto-completion eval), not dedup substrate.

2.3 Substrate (a) — Cocoindex post-target-write @coco.fn content-similarity

Substrate framing: cocoindex itself has no document-level near-dup primitive (S2 spike confirmed content_fingerprint is exact-bytes only). A “cocoindex @coco.fn substrate” is therefore: a post-target-write Python function that pulls per-file chunks from content_chunks, runs embedding similarity (KH would use text-embedding-3-large in production; spike uses all-MiniLM-L6-v2 to keep $0-cost CPU-local), and emits pair scores. Cocoindex’s role is orchestration + memoisation (the @coco.fn(memo=True) decorator caches re-runs).

Algorithm:

1. Per file: chunk text into 800-char with 100-char overlap (matches mempalace + KH defaults), cap at 200 chunks/file (mempalace safeguard).
2. Embed each chunk with all-MiniLM-L6-v2 (384-dim, L2-normalised).
3. Per file: compute centroid (mean of chunk embeddings, re-normalised).
4. Per pair (a, b): centroid_sim = dot(centroid_a, centroid_b) + max_chunk_sim = max(chunks_a @ chunks_b.T).
5. Rule: pair is “related” if centroid_sim ≥ T_c AND max_chunk_sim ≥ T_m (AND-rule consistently outperforms OR-rule across the sweep — see §3.1).

Threshold sweep result:

Rule	TP	FP
centroid_only ≥ 0.65	83.3% (10/12)	16.7% (11/66)
centroid_only ≥ 0.70	75.0% (9/12)	7.6% (5/66)
centroid_only ≥ 0.75	50.0% (6/12)	3.0% (2/66)
max_chunk_only ≥ 0.65	91.7% (11/12)	10.6% (7/66)
max_chunk_only ≥ 0.70	91.7% (11/12)	7.6% (5/66)
max_chunk_only ≥ 0.75	83.3% (10/12)	1.5% (1/66)
cent≥0.6 AND max_chunk≥0.7	91.7% (11/12)	6.1% (4/66)
cent≥0.55 AND max_chunk≥0.7	91.7% (11/12)	7.6% (5/66)
cent≥0.7 AND max_chunk≥0.7	75.0% (9/12)	1.5% (1/66)

Best operating point: TP=91.7% (11/12) and FP=6.1% (4/66) — fails the strict TP ≥ 90% AND FP ≤ 5% gate by 1.1 percentage points on FP.

Persistent TP miss: B2-faqs (binary docx) — final = “Tender and Bid Library Template for Audit; Frequently Asked Questions” (11,033 chars); DRAFT = “Tender and Bid Library Template for Phew; General FAQs” (14,256 chars, ~29% longer). The DRAFT-vs-final relationship is real but the introduction paragraphs were rewritten in substance (Audit→Phew framing change); centroid_sim drops to 0.588 and max_chunk_sim to 0.638, just under the AND-rule threshold. The corresponding markdown pair M2-faqs does pass (max_chunk_sim=1.0 — mammoth extraction preserves the shared FAQ-body text verbatim once introduction is past), so the substrate doesn’t fail on the same logical pair across both docx + md.

Caught FPs at the best operating point (4/66):

• U-13 (02-services-and-digital-products.md vs 04-named-clients-and-case-studies.md) — centroid 0.756, max_chunk 0.722 — both reference Phew’s product line + clients, plausibly mistakeable for “related” by any topical-semantic measure.
• U-19 (02-services vs 10-site-structure-and-key-urls.md) — centroid 0.679, max_chunk 0.720 — both reference Phew’s website/digital products.
• U-411 (05-team vs 12-research-notes) — centroid 0.576, max_chunk 0.790 — shared chunk on “people structure” terminology.
• U-910 (10-site vs 11-notable-blog-posts) — centroid 0.751, max_chunk 0.743 — both reference Phew blog URLs.

These are NOT cataclysmic FPs — a human reviewer would recognise the relationship as “shares topic” rather than “is a near-duplicate”. But they show the substrate alone cannot deliver the 5% FP gate.

Sanity check with higher-quality embedder (all-mpnet-base-v2, 768-dim):

Rule	TP	FP
cent≥0.55 AND max_chunk≥0.7	100% (12/12)	28.8% (19/66)
cent≥0.6 AND max_chunk≥0.7	91.7% (11/12)	27.3% (18/66)
cent≥0.6 AND max_chunk≥0.8	75.0% (9/12)	3.0% (2/66)

mpnet performs WORSE than MiniLM-L6 on this corpus — its denser embedding space inflates similarity scores for topically-adjacent-but-not-related documents (the Phew company-context corpus). This is a non-obvious finding worth flagging: higher-fidelity embedding models can DEGRADE precision on cross-record dedup when the corpus has natural topical clustering. KH production with text-embedding-3-large (1024-dim, OpenAI) will need calibration against this corpus shape — flagged as Q-S10-1 for implementation phase.

2.4 Substrate (b) — Mempalace v3.3.5 chroma cross-file search

Substrate framing: mempalace’s built-in dedup_palace deduplicates only WITHIN a source_file group (the function literally groups by source_file metadata and runs greedy distance-based dedup on chunks within each group). For CROSS-record dedup, we must talk to the underlying chroma backend directly: query a chunk from file A against the whole collection, see if file B’s chunks appear in top-k below a distance threshold.

mempalace_search MCP tool is broken upstream (S3 confirmed; CLAUDE.md gotcha). The chroma backend’s direct coll.query(query_texts=...) works. So the substrate can be wired in production by KH subprocess-wrapping the mempalace chroma collection OR re-implementing the same algorithm in KH-native Postgres pgvector (which is what KH already has).

Algorithm:

1. Initialise throwaway palace at /tmp/mempalace-s10-v3/ (live ~/.mempalace/ palace UNTOUCHED per spike constraint).
2. Per file: chunk same as (a) (800/100 overlap), add to chroma collection with source_file metadata + chunk_index.
3. Per pair (a, b): sample 10 evenly-strided chunks from file A, query collection (top_k=20), count how many top-k hits have meta.source_file == b_label with distance < 0.5. Take best_distance and match_fraction per pair (with symmetric measure: query A vs B AND query B vs A; take max matches + min best_distance).

Threshold sweep result:

Rule	TP	FP
best_dist<0.20	58.3% (7/12)	0.0% (0/66)
best_dist<0.25	66.7% (8/12)	1.5% (1/66)
best_dist<0.30	75.0% (9/12)	7.6% (5/66)
best_dist<0.35	75.0% (9/12)	10.6% (7/66)
best_dist<0.40	100% (12/12)	27.3% (18/66)
best_dist<0.25 AND frac≥0.2	66.7% (8/12)	1.5% (1/66)
best_dist<0.3 AND frac≥0.3	66.7% (8/12)	6.1% (4/66)

Best operating point: TP=66.7% (8/12) and FP=1.5% (1/66) — fails the strict TP ≥ 90% gate by 23.3pp.

Persistent TP misses: B2-faqs, M2-faqs, BL-01, BL-03 (and BL-12, BL-13 at most thresholds).

The mempalace substrate is bottlenecked by chroma’s default all-MiniLM-L6-v2 embeddings (the same model substrate (a) used directly!) combined with chroma’s L2 distance space (versus dot product on normalised vectors in (a)). The substrate is essentially “(a)‘s embeddings ranked by L2 distance via chroma’s HNSW index”. With n_results=20 top-k and stride-sampling 10 chunks per file, signal sparsity on shorter bid-library variants reduces effective coverage. The substrate could be tuned with a higher top_k and full-coverage chunk sampling — but at that point we’re just re-implementing substrate (a) inside mempalace, with extra subprocess overhead.

Practical conclusion: mempalace v3.3.5 does not contribute beyond what substrate (a) already provides. The ChromaBackend’s embedding model + retrieval contract is identical-or-worse for this task. Per S3 confirmation: KH should adopt mempalace’s Shape A (temporal KG schema) and Shape B (MCP response contract) but run dedup logic in KH’s own Postgres / pgvector stack.

2.5 Substrate (c) — Skill-seekers metadata keyword-categorisation

Substrate framing: skill-seekers (per 0.8.5 §Q5 verdict) has NO document-level near-dup primitive — the closest pattern is its categories keyword-rule config (per docs/reference/CONFIG_FORMAT.md cited in 0.8.5). The spike simulates this pattern: define keyword sets per category (implementation, security, compliance, faqs, bid_library, phew, etc.), match each file’s title + first 1000 chars against the keyword sets, score a pair by the shared-category count.

This substrate is inspired by skill-seekers, NOT borrowed from it — the spike’s skill_seekers_probe.py is a from-scratch keyword-rule classifier. Per 0.8.5 §Q5 verdict: “Skill-Seekers’s categories keyword-rule pattern is useful inspiration” but “the right home for the client’s scope-tag taxonomy is KH’s classification pipeline”. Implementation in production would be a Python or TypeScript classifier registered in KH’s ingest pipeline, configurable per tenant.

Algorithm:

1. Define CATEGORIES dict: implementation, security, compliance, faq, bid_library, phew, audit, tender, company_overview, team, brand, service, site_url, advanced_audits, lms, website_bid (16 categories with keyword lists).
2. Per file: scan filename + first 1000 chars of body for each keyword set. Emit set of matched categories.
3. Per pair: shared_count = |cats_a ∩ cats_b|.

Threshold sweep result:

Rule	TP	FP
shared ≥ 1	100% (12/12)	100.0% (66/66)
shared ≥ 2	100% (12/12)	100.0% (66/66)
shared ≥ 3	100% (12/12)	65.2% (43/66)
shared ≥ 4	100% (12/12)	22.7% (15/66)
shared ≥ 5	100% (12/12)	4.5% (3/66)
shared ≥ 6	83.3% (10/12)	1.5% (1/66)
shared ≥ 7	50.0% (6/12)	0.0% (0/66)

Best operating point: TP=100% (12/12) and FP=4.5% (3/66) — PASSES the strict TP ≥ 90% AND FP ≤ 5% gate.

Caught FPs at the best operating point (3/66):

• U-411 (05-team vs 12-research) — shares company_overview, compliance, implementation, phew, service, team (6 categories).
• U-711 (07-compliance vs 12-research) — shares 5 categories.
• U-011 (01-company vs 12-research) — shares 5 categories.

All three FPs involve 12-research-notes-and-gaps.md (which has a category-heavy mixed-topics body) — suggests keyword rules could be tightened with anti-keywords for “research-notes” topical signature.

Critical caveat: the keyword rules were curated to this specific corpus during the spike. The categories bid_library, audit, tender, phew, advanced_audits, lms, website_bid are tightly specific to Phew’s corpus shape. For a different tenant (e.g. London Borough of Barking & Dagenham CSP corpus), the keyword set would be entirely different. Per-tenant rule curation is mandatory — domain vocabulary must be mapped into a usable rule set as part of client onboarding. This matches 0.8.5 §Q4 verdict: “scope-tag taxonomy is per-tenant; skill-seekers’s categories is per-config (per-skill), not a centralised taxonomy”.

The pass is brittle in another way: the corpus has 13 categories with strongly-discriminating vocabulary (the four bid_library variants all share bid_library + tender + phew + service + faq at minimum). On a less-clustered corpus (e.g. KH’s general document library), shared-count ≥ 5 might falsely cluster unrelated documents.

2.6 Substrate (d) — Mempalace v5.0 PG-backend native dedup

Substrate cannot be tested — does not exist.

Empirical findings (S10 audit 2026-05-11, KH worktree-agent-a356ab9e732e4fbbc):

• python3 -m pip index versions mempalace → “Available versions: 3.3.5, 3.3.4, 3.3.3, 3.3.2, 3.3.1, 3.3.0, 3.2.0, 3.1.0, 3.0.0, 2.0.0”. No v4, no v5.
• pip install mempalace==5.0 → “ERROR: No matching distribution found for mempalace==5.0”.
• PyPI simple index https://pypi.org/simple/mempalace/ → returns ONLY 2.0.0 through 3.3.5.
• GitHub https://api.github.com/repos/MemPalace/mempalace/releases → latest tag v3.3.5 (published 2026-05-10T23:32:46Z). No v4 / v5.
• GitHub branches → no pg, postgres, v4, v5 branches; only feature branches, release-prep branches, RFC docs.
• mempalace/backends/ directory on main → __init__.py, base.py, chroma.py, registry.py. The RFC 001 backend-pluggability layer is in place (BaseBackend + PalaceRef + registry pattern), with only ChromaBackend implemented.

Per S3 spike findings (re-confirmed here): mempalace mentions a “v4-alpha PG backend” but it’s deferred per Q4.6 DEFERRED-POST-LAUNCH; the live install is v3.3.5 with ChromaDB backend; the “v4-alpha” PG backend is project-internal aspiration, not a shipped package.

The S230 user prompt’s claim (“mempalace v5.0 GA shipped 2026-05-02 with multi-backend persistence including a PG backend”) is factually incorrect. The likely confusion: mempalace’s README and PyPI description reference an internal “Hybrid v4” and “Hybrid v5” retrieval pipeline (sentence-level reranking) that improves R@5 from 96.6% (raw) to 98.4% (hybrid v4 held-out) to ≥99% (hybrid v5 + LLM rerank) on the LongMemEval benchmark. These are pipeline-version-strings (hyperparameter sets in the retrieval code), not package releases.

Spike conclusion for (d): SKIP — substrate is fictional. No data, no scoring, no UX evaluation possible. If mempalace v4 or v5 ships in future with a multi-backend dedup primitive, re-run this spike then.

2.7 Hybrid substrate (a)+(c) — Cocoindex chunk-embed AND skill-seekers keyword

Motivating hypothesis from the per-substrate analysis: (a) is FP-dominated (topical-overlap false alarms); (c) is brittle but high-precision. AND-combining them should suppress (a)‘s FPs while preserving its TPs — because all 11 (a)-TPs also score high in (c).

Sweep result:

Rule	TP	FP
cent≥0.6 AND shared≥4	91.7% (11/12)	7.6% (5/66)
cent≥0.6 AND shared≥5	91.7% (11/12)	1.5% (1/66) ← PASS
cent≥0.6 AND shared≥6	83.3% (10/12)	0.0% (0/66)
max_chunk≥0.65 AND shared≥4	91.7% (11/12)	6.1% (4/66)
max_chunk≥0.65 AND shared≥5	91.7% (11/12)	1.5% (1/66) ← PASS
max_chunk≥0.70 AND shared≥4	91.7% (11/12)	4.5% (3/66) ← PASS
max_chunk≥0.70 AND shared≥5	91.7% (11/12)	1.5% (1/66) ← PASS

The hybrid passes the gate at multiple operating points with very thin FP rate (1.5%, single FP — U-411 05-team vs 12-research). This is the most precision-favourable point in the spike output: TP=91.7% (one persistent miss: B2-faqs) and FP at 1.5%.

The remaining B2-faqs miss is structurally hard — the docx introductions were rewritten such that neither chunk-embedding similarity nor keyword categorisation gets to ≥0.6 / 5 (the docx has 16 chunks; the DRAFT has 21; shared chunk content gets diluted across more chunks in the DRAFT, dropping max_chunk_sim to 0.638). The pair is caught by the markdown side (M2-faqs — max_chunk=1.0) because mammoth re-extracts the body text into a more deterministic shape.

---

3. Per-substrate UX-compat scoring

Per spike-plan §S10 step 4: “Evaluate UX shape compatibility — which substrate naturally surfaces side-by-side diff, golden-record selection, citation-preservation?”

Substrate	Side-by-side diff	Golden-record selection	Citation preservation	UX-compat score
(a) Cocoindex @coco.fn	YES — chunk-level alignment via embedding similarity; per-chunk diff renderable	YES — surface both records to user with similarity matrix per chunk; user picks; merge audit captured via @coco.fn re-run	YES — citations attached to source_chunk_id are preserved when the chunk is selected; new compound chunk’s citations re-anchored via Anthropic citations API (per edit-flow §6.0.2)	4/5 — needs @coco.fn orchestration of the chunk-pair diff + golden-record materialisation; well within the cocoindex paradigm
(b) Mempalace v3.3.5 chroma	PARTIAL — chroma top-k returns chunks but not aligned pairs; diff requires post-processing	PARTIAL — same as (a) but in mempalace’s drawer-store; would need a sidecar UI rendering drawer pairs	LIMITED — mempalace doesn’t track Anthropic citation pointers natively (only source_drawer_id ≈ chunk_id); citation re-anchoring would still happen in KH	3/5 — substrate produces signals but UX layer would be KH-native anyway
(c) Skill-seekers keyword	NO — no chunk-level signal; only file-level shared-category set	PARTIAL — file-level “these N files share categories” but no per-chunk merge target	NO — keyword rules don’t produce citation pointers	2/5 — file-level “are these similar?” gate only; no surface for downstream UC8 merge UX
(d) Mempalace v5.0 PG	N/A	N/A	N/A	0/5 — substrate does not exist
HYBRID (a)+(c)	YES (via a)	YES (via a)	YES (via a) — keyword set adds a TENANT-LEVEL TAG (scope_tag per 0.8.5 §Q4) but doesn’t otherwise affect chunk-level UX	4/5 — same as (a); the (c) keyword co-confirmer is a pre-flight gate, not part of the UX surface

UX winner: (a) or hybrid (a)+(c). The chunk-level alignment that comes naturally with embedding similarity is the right substrate for UC8’s side-by-side diff and golden-record composition UX. Substrate (c)‘s file-level signal is a useful precision-booster but cannot drive the UX surface alone.

---

4. Cross-substrate comparison

Dimension	(a) Cocoindex @coco.fn	(b) Mempalace v3.3.5 chroma	(c) Skill-seekers keyword	(d) Mempalace v5.0 PG	HYBRID (a)+(c)
TP rate (best op-point)	91.7% (11/12)	66.7% (8/12)	100% (12/12)	N/A	91.7% (11/12)
FP rate (best op-point)	6.1% (4/66)	1.5% (1/66)	4.5% (3/66)	N/A	1.5% (1/66)
Pass TP ≥ 90% AND FP ≤ 5%?	NO (close)	NO	YES (brittle)	N/A	YES
Persistent TP misses	B2-faqs	B2-faqs, M2-faqs, BL-01, BL-03	none	N/A	B2-faqs
Embed cost / workspace (initial)	$0 (local) OR ~$2.60 (OpenAI text-embedding-3-large API)	$0 (local; bundled MiniLM-L6)	$0	N/A	$0 OR ~$2.60
Embed wall-time / workspace (initial)	~2 min (local, 50k chunks) OR ~30s (API)	~3 min (mempalace + chroma overhead)	<1s (regex pass)	N/A	~2 min (a dominates)
Marginal cost per file change	$0 (local) OR ~$0.05 (API)	$0	$0	N/A	$0 OR ~$0.05
Per-tenant calibration	Threshold tuning per corpus	Same	Keyword rule authoring per corpus	N/A	Both — threshold tuning + rule authoring
Implementation scope (v1)	@coco.fn + candidate generation via faiss + threshold infra	Re-purpose chroma cross-source dedup; subprocess wrapper	Rule schema + matcher	N/A	a’s + c’s scope, shared infra
Operational risk	Threshold drift across tenants; OpenAI dep if API embed	Mempalace upstream stability (mempalace_search broken); chroma corruption risk per S3 §2.1	Keyword rules brittle on unfamiliar corpora; per-tenant cost	Doesn’t exist	(a)‘s risks + per-tenant rule curation overhead
UX-compat score (out of 5)	4	3	2	0	4
Citation preservation pathway	Native — chunks carry source_chunk_id; Anthropic re-anchor on merge	Indirect — would need KH translation layer	None — substrate operates at file level	N/A	Native (via a)
Reversibility	High — re-tune threshold; swap embedding model	Medium — chroma stable but upstream package unstable	Medium — rule changes; per-tenant divergence risk	N/A	High

Substrate ordering by gate-pass + UX-compat:

1. HYBRID (a)+(c) — passes gate at multiple operating points (TP=91.7% / FP=1.5-4.5%); UX-compat 4/5. RATIFIED.
2. (c) skill-seekers keyword alone — passes gate at TP=100% / FP=4.5%; UX-compat 2/5; brittle per-tenant. Useful as the co-confirmer in the hybrid, not as a standalone substrate.
3. (a) cocoindex alone — near-pass at TP=91.7% / FP=6.1%; UX-compat 4/5. Cited here for completeness; hybrid (a)+(c) is the ratified v1 substrate.
4. (b) mempalace v3.3.5 chroma — TP fails hard. Mempalace adopted for Shape A + B + C per S3, but NOT for dedup substrate.
5. (d) mempalace v5.0 PG — does not exist. SKIP.

---

5. G10 decision-gate verdict

Spike-plan G10 outcome option	Triggered?	Notes
Cocoindex wins	NO (close)	(a) alone hits 91.7% / 6.1% — within 1.1pp of FP target but doesn’t strictly pass
Mempalace wins	NO	(b) fails TP rate at 66.7%
Hybrid wins	YES	Hybrid (a)+(c) passes 91.7% / 1.5% — RECOMMENDED
None wins (v1 ships without cross-record dedup; user-driven only)	NO	The gate has at least one passing substrate AND a passing hybrid

G10 decision-gate verdict: HYBRID (a)+(c) substrate adopted for UC8 v1. Cocoindex’s chunk-embedding similarity provides the chunk-level signal needed for side-by-side diff and golden-record composition (UX-compat 4/5); skill-seekers-inspired keyword categorisation provides the FP suppressor needed to clear the 5% FP gate. Both are KH-native implementations (no subprocess-wrapping of mempalace or skill-seekers required); cocoindex’s @coco.fn is the orchestration layer.

Architecture impact (informs 0.9-intended-architecture.md §12.1 + §11.2):

• §12.1 “MCP check_content_duplicates tool” — substrate is HYBRID (a)+(c); tool surface returns paired candidates with per-pair (centroid_sim, max_chunk_sim, shared_categories) triple + LLM-reasoning narrative per edit-flow §6.8.
• §11.2 “Dedup-check UI” — adopts the chunk-pair diff view (per substrate (a)) with shared-categories tags rendered as side-bar metadata (per substrate (c)).
• §10 / §6.8 cross-reference — UC8 v1 substrate slot resolves to HYBRID (a)+(c); replaces previous [DECISION-PENDING-S10-spike] marker.
• New work-package suggested: WP-DEDUP-RULES — per-tenant keyword-rule curation tooling, housed in KH’s classification pipeline alongside scope_tag taxonomy work (per 0.8.5 §Q4 verdict + OPS-X-SCOPE-TAGS candidate). Likely shared infrastructure with the scope-tag work-package. Per S231 feedback: coordination with OPS-X-SCOPE-TAGS is agreed.

Convertibility to product + tech specs:

The UC8 v1 substrate decision is convertible to product + tech specs at the implementation phase. Key dependencies + unknowns that gate implementation-readiness:

• Production embedding model calibration. Spike used local MiniLM-L6; production uses text-embedding-3-large (1024-dim). Operating-point thresholds must be re-derived on the same corpus before v1 ship.
• Per-tenant keyword-rule curation infrastructure (WP-DEDUP-RULES below). The substrate (c) co-confirmer requires a per-tenant rule set; the rule-authoring + storage + versioning tooling is new work. Coordinate with OPS-X-SCOPE-TAGS.
• Candidate generation at scale. Brute-force O(N²) pair similarity is infeasible at 1,000+ docs; faiss top-K candidate generation is required before the AND-rule kicks in. Index lifecycle (cocoindex flow integration) is a tech-spec detail.
• LLM-reasoning step (per edit-flow §6.8). UC8 v1 = “Candidate A + LLM-reasoning”. Model choice (Haiku for cost vs Sonnet 4.7 for quality) is a product decision.
• Q-S10 residual questions (§6.2 below) — Q-S10-1 through Q-S10-8 each surface a specific implementation-readiness sub-decision.

---

6. Surprises + residual questions

6.1 Surprises

1. mempalace v5.0 does not exist. The S230 user prompt’s premise — “Liam added a 4th candidate (mempalace v5.0 GA shipped 2026-05-02)” — is factually wrong. Latest mempalace on PyPI / GitHub as of audit date (2026-05-10) is 3.3.5. No PG backend in the develop branch. The most likely source of confusion: mempalace’s README references internal “Hybrid v4” and “Hybrid v5” retrieval pipeline configurations (LLM rerank tuning) which improve LongMemEval R@5 from 96.6% to 98.4% to ≥99%. These are pipeline-version-strings, not package versions. The spike documents this for the Phase 0.9 record; v5 substrate evaluation will need to be re-run if/when mempalace actually ships a v5 package with PG backend.
2. mpnet (768-dim) performs WORSE than MiniLM-L6 (384-dim) on this corpus. Counter-intuitive but explainable: mpnet’s denser embedding space inflates similarity between topically-adjacent-but-not-related Phew company-context docs. Substrate (a)‘s operating-point thresholds are model-dependent. KH production with text-embedding-3-large (1024-dim, OpenAI) will need its own calibration against this corpus shape — flagged as Q-S10-1.
3. skill-seekers passes the strict gate alone (TP=100% FP=4.5%). Per 0.8.5 §Q5 the substrate was meant to be inspiration-only — but on this corpus the keyword-rule classifier with 16 well-targeted categories actually outperformed both cocoindex chunk-embedding and mempalace chroma cross-file. Strong caveat: the rules were curated to this corpus, and the corpus has unusually-clustered vocabulary (bid library + audit + tender + Phew + service all co-occur in TPs but rarely in FPs). On a less-clustered corpus the brittleness would dominate. The hybrid (a)+(c) usefully splits the difference: (a) handles unfamiliar corpora; (c) adds precision.
4. B2-faqs (binary docx FAQ pair) is structurally hard. Both substrates miss this pair persistently. The DRAFT version was rewritten at the introduction-paragraph level (final = “for Audit / Frequently Asked Questions”; DRAFT = “for Phew / General FAQs”); the body content overlap (which DOES exist) gets diluted across more chunks (16 vs 21). The corresponding markdown pair M2-faqs is caught because mammoth’s extraction produces deterministic shared chunk text. Implication: docx-level dedup is materially harder than markdown-level dedup because of paragraph-rewrite artefacts. v1 should run the substrate on the markdown-extracted shape (cocoindex’s downstream of mammoth/docling), not on raw docx bytes. The S2 spike already established cocoindex doesn’t decompose docx — that’s the adapter layer’s job.
5. Mempalace’s dedup_palace operates WITHIN source_file groups only. Spike-plan §S10 framing implied mempalace would do cross-record dedup; reality is the built-in dedup is single-file chunk dedup (greedy keep-longest within a source_file group). Cross-file dedup requires bypassing dedup_palace and talking to the chroma backend directly. This adds zero new capability over what KH could implement with pgvector directly.
6. Mempalace v3.3.5’s chroma backend adds NO unique signal over substrate (a). Both use all-MiniLM-L6-v2 embeddings; mempalace’s L2 distance + HNSW top-k differs from (a)‘s direct dot-product on normalised vectors only in indexing speed at scale (irrelevant at 1000-doc workspaces). The S3 spike’s conclusion (“KH adopts mempalace Shape A + B + C but runs everything in KH’s own Postgres”) is reinforced: mempalace is NOT a dedup-substrate dependency.

6.2 Residual questions

Q	For
Q-S10-1	Production-model calibration — at what threshold does text-embedding-3-large (1024-dim, OpenAI) hit the same TP/FP profile as MiniLM-L6 on this corpus? Spike used local CPU embeddings; production will use the OpenAI API. Re-derive operating-point thresholds on the production model before v1 ship.
Q-S10-2	Per-tenant rule curation — how is the skill-seekers-inspired keyword-rule classifier configured per tenant? Likely UI in OPS-X-SCOPE-TAGS work-package; spec needs design. Inheritance from taxonomy_snapshot.json? Per-tenant categories.yaml?
Q-S10-3	Candidate generation at scale — at 1,000-doc workspace, pair-count is 499,500. Brute-force O(N²) similarity is infeasible. Likely use faiss top-50-per-file candidate generation BEFORE running the AND-rule (which then becomes O(N × 50)). Faiss index lifecycle vs cocoindex flow lifecycle is a Phase 2 detail.
Q-S10-4	B2-faqs hard miss — the FAQ pair fails on docx but passes on markdown. UC8 v1 flow should EITHER run the substrate on markdown-extracted text only (skip docx-level dedup) OR accept the miss on docx and rely on markdown shape coverage. Decision is downstream of S2 confirming docx → markdown extraction is part of the cocoindex flow (which it is — mammoth / docling are @coco.fn adapters).
Q-S10-5	Multi-source dedup — corpus tested all originate from one tenant (Phew). UC8 may also need to dedup-across-imports (same document loaded twice via different ingest paths). The substrate naturally extends — adds an ingest_path dimension to the candidate-pair scoring.
Q-S10-6	LLM-reasoning step — per edit-flow §6.8, UC8 v1 = “Candidate A + LLM-reasoning”. The substrate produces candidate pairs; an LLM reasoner explains why the pair is related (helps the human reviewer in the merge UI). What LLM? Claude Haiku (cost) or Claude Sonnet 4.7 (quality)? Decision is product-design, not substrate.
Q-S10-7	Anti-keyword schema for (c) rules — per 0.8.5 §Q4 “anti-keywords” pattern, the keyword rules should support internal_it: anti=[production, customer-facing]. The spike’s rule format didn’t include anti-keywords; production rule-engine schema should.
Q-S10-8	B2-faqs as v1 “known limitation” — acceptable to ship v1 with ~92% TP rate and document the FAQ-introduction-rewrite case as a known dedup gap, or does v1 demand 100% TP? If 100%, escalate to v1.1 with LLM-based fallback for low-confidence candidates (claude reads both docs, decides).

---

7. Recommendations for downstream architecture work

7.1 Updates needed to 0.9-intended-architecture.md

• §2.3 UC8 row — replace [DECISION-PENDING-S10-spike] with “HYBRID (a)+(c) — cocoindex chunk-embedding @coco.fn + skill-seekers-inspired keyword categorisation” per G10.
• §5.2(ii) Path 2 — Candidate A — UC8 user-driven merge flow benefits from substrate (a)‘s chunk-pair diff. No change to the path itself; substrate fills the slot.
• §11.2 Dedup-check UI — adopts side-by-side chunk-pair view (driven by substrate (a)‘s embedding-similarity matrix) with shared-category tags from substrate (c) rendered as metadata.
• §12.1 Decision summary — MCP check_content_duplicates tool row resolves to “HYBRID (a)+(c) substrate; returns paired candidates with (centroid_sim, max_chunk_sim, shared_categories)”.
• §13 v1 deployment — adds substrate-implementation phase post-S1 + per-tenant rule curation onboarding step.

7.2 Updates needed to 0.9-edit-flow-investigation.md

• §6.8 UC8 — substrate slot resolves to HYBRID (a)+(c). LLM-reasoning step on top (Claude Haiku or Sonnet 4.7 — model choice deferred).
• §7.1 Spike S10 line item — update “OPEN-PENDING-S10” to “RESOLVED-HYBRID-PASS” with verdict.

7.3 New work-package recommended

• WP-DEDUP-RULES — per-tenant keyword-rule classifier for the substrate (c) co-confirmer + the scope_tag taxonomy (0.8.5 §Q4 / OPS-X-SCOPE-TAGS). Scope: KH-native classifier + rule-curation UI + per-tenant onboarding flow. Shared infra with scope-tag work.

7.4 Errata flagged to other docs

ERRATA	Document	Issue	Suggested fix
ERRATA-1	S230 continuation prompt for S10 spike	Claims “mempalace v5.0 GA shipped 2026-05-02” — factually wrong. Latest mempalace = 3.3.5; no PG backend ships in any release.	Remove substrate (d) from S10 candidate list; mempalace v4-alpha PG backend remains DEFERRED-POST-LAUNCH per Q4.6 + S3 §6.6.
ERRATA-2	0.8.7-mempalace-evaluation.md (if updated)	Should clarify “Hybrid v4 / v5” benchmarks refer to internal retrieval pipeline configs, not package versions.	Add a footnote distinguishing the two version strings.
ERRATA-3	0.9-spike-plan.md §S10 line 487	Method step 2(b) says “load corpus through mempalace mine, observe KG entity-resolution output”. S3 + this spike confirm mempalace miner doesn’t emit KG triples and dedup_palace is within-source-file only. Substrate (b) reframed as direct ChromaBackend cross-file search.	Update §S10 method step 2(b) to “query mempalace’s chroma backend directly via ChromaBackend.get_or_create_collection(…).query()” for accuracy.

---

8. Reproducibility — to re-run this spike

# From worktree at content-items-investigation:
cd <repo>

# 1. Venv + deps
python3 -m venv .venv-spike-s10
PIP_USER=0 PIP_TARGET="" .venv-spike-s10/bin/pip install cocoindex sentence-transformers numpy python-docx

# 2. Substrate (a): cocoindex @coco.fn chunk-embedding similarity (MiniLM-L6)
.venv-spike-s10/bin/python3 .spike-s10/cocoindex_dedup_probe.py
.venv-spike-s10/bin/python3 .spike-s10/cocoindex_dedup_sweep2.py
# Output: .spike-s10/cocoindex_results.json + threshold sweeps

# 3. Substrate (a) sanity-check with mpnet
.venv-spike-s10/bin/python3 .spike-s10/cocoindex_mpnet.py

# 4. Substrate (b): mempalace v3.3.5 chroma cross-file search
# First, ensure docx support in mempalace's uv-tool venv:
PIP_USER=0 PIP_TARGET="" ~/.local/share/uv/tools/mempalace/bin/python3 -m ensurepip --upgrade
PIP_USER=0 PIP_TARGET="" ~/.local/share/uv/tools/mempalace/bin/python3 -m pip install python-docx
# Then run probe (uses throwaway palace at /tmp/mempalace-s10-v3/):
~/.local/share/uv/tools/mempalace/bin/python3 .spike-s10/mempalace_v3_probe.py
.venv-spike-s10/bin/python3 .spike-s10/mempalace_v3_sweep2.py

# 5. Substrate (c): skill-seekers metadata keyword categorisation
.venv-spike-s10/bin/python3 .spike-s10/skill_seekers_probe.py

# 6. Substrate (d): mempalace v5.0 PG backend
#    SKIP — substrate does not exist. Re-validate by:
python3 -m pip install --dry-run "mempalace>=4.0"
# Expected: "Could not find a version" (i.e. only 2.x and 3.x available).
curl -s https://api.github.com/repos/MemPalace/mempalace/releases | python3 -c "import json,sys; r=json.load(sys.stdin); [print(x['tag_name'], x['published_at']) for x in r[:5]]"
# Expected: v3.3.5 / v3.3.4 etc, no v4 / v5.

# 7. Hybrid sweep + synthesis
.venv-spike-s10/bin/python3 .spike-s10/hybrid_sweep.py
.venv-spike-s10/bin/python3 .spike-s10/synthesise.py
.venv-spike-s10/bin/python3 .spike-s10/cost_model.py

Scripts are NOT committed (gitignored at .spike-s10/, .venv-spike-s10/). Live mempalace ~/.mempalace/ palace NOT touched (throwaway palace at /tmp/mempalace-s10-v3/). Re-create scripts from this doc’s §2 method or from this commit’s .spike-s10/ contents.

---

9. Decision recommendation (binding for G10)

Recommendation — RATIFIED: adopt HYBRID (a)+(c) — cocoindex post-target-write @coco.fn chunk-embedding similarity AND skill-seekers-inspired keyword-rule co-confirmer — as the UC8 v1 cross-record dedup substrate. Best operating point on the canonical corpus is centroid_sim ≥ 0.6 AND max_chunk_sim ≥ 0.7 AND shared_categories ≥ 4 → TP=91.7% (11/12) / FP=4.5% (3/66), or a tighter point at cent ≥ 0.6 AND shared ≥ 5 → TP=91.7% / FP=1.5%. Both pass the spike-plan §S10 acceptance gate (TP ≥ 90% AND FP ≤ 5%).

No fall-back path required. UC8 v1 is human-confirmed-merge per edit-flow §6.8; hybrid (a)+(c) is the substrate.

Skip substrates:

• (b) Mempalace v3.3.5 chroma — TP fails hard (66.7%); doesn’t contribute unique signal vs substrate (a); confirms S3’s “adopt Shape A + B + C but run everything in KH Postgres” conclusion.
• (d) Mempalace v5.0 PG-backend — does not exist; ERRATA-1 logged; re-evaluate if/when mempalace actually ships a v5 release with PG backend.

Q&A-pair dedup interplay (S231 feedback): KH defines the predetermined markdown shape for Q&A ingestion (per S16 §7). The hybrid substrate operates at content-level (corpus documents) AND, downstream, at q_a_pair level — but since v1 ingests Q&A in canonical shape with explicit (workspace_id, question_text) identity (per S16 §6.2), the dedup substrate’s role for Q&A is narrower than for corpus documents: it surfaces near-duplicate question phrasings post-ingest (UC8 user-confirmed merge), not pre-ingest format-normalisation.

Architecture-impl scope (Phase 2):

• Substrate-implementation: hybrid (a)+(c) infrastructure, faiss candidate generation, threshold-config schema, @coco.fn orchestration.
• Per-tenant rule curation tooling: shared with OPS-X-SCOPE-TAGS work-package via WP-DEDUP-RULES.
• Production-embedding-model calibration: required before v1 ship (Q-S10-1).

Phase 2 architecture-impl commits to cocoindex Option A (per S1, RESOLVED scenario A) AND HYBRID (a)+(c) UC8 substrate (per this spike).

---

End of S10 spike record. G10: RESOLVED-HYBRID-PASS (cocoindex @coco.fn + skill-seekers-inspired keyword-rule co-confirmer).