Phase 0.9 — Spike S2 (cocoindex external-folder source binding)

Phase 0.9 — Spike S2 (cocoindex external-folder source binding)

Audit date: 2026-05-10 (KH session S229) Branch: content-items-investigation (worktree-agent-a09d486bb914c8c9c) Spike spec: 0.9-spike-plan.md §2 S2 (lines 112-180) Test corpus: docs/client-documentation-base/ (35 files; 8 binary + 25 markdown + 2 forms) Cocoindex version installed: 1.0.3 (PyPI; published shortly before the spike) Python version: 3.14.2 (Homebrew); venv at .venv-spike-s2/ (worktree-local; not committed)

Spike status: PASSED — localfs source binding + content-fingerprint Δ-engine works end-to-end on canonical corpus. All five UC4.4 change kinds (add / edit / delete / rename / binary-replace) correctly classified by the engine. DRAFT-vs-final dedup is NOT a built-in cocoindex primitive — must be a custom layer feeding S10. No SharePoint / Notion / Dropbox / Box connector in 1.0.3; v1 connector list = localfs only.

---

1. Executive summary

Question	Answer
Does localfs.walk_dir(...) handle our client content shape?	Yes — 35 files across binary/, markdown/, forms/ enumerated correctly with recursive=True. DOCX, PDF, XLSX, MD, JPG all opaque-blob-pass-through (extraction is downstream adapter work, not cocoindex’s job).
Does it classify add/edit/delete/rename/binary-replace per UC4.4?	Add: yes (1 invocation). Edit: yes (1 invocation). Delete: yes (cache purged, 0 invocations). Rename: yes — but as add+delete pair at the key level (no native rename tracking). Binary-replace: yes (1 invocation; presents identically to edit at key granularity).
What polling cadence + webhook strategy per connector?	localfs: native fs-watch (FSEvents/inotify) via watchfiles 1.1.1 with default 1600ms debounce; polling fallback at 300ms when forced. Google Drive: poll-only (no webhooks). Amazon S3: poll-only. Kafka: native streaming. OCI Object Storage: event-streaming via OCI Streams. No SharePoint / Notion / Dropbox / Box connector exists in 1.0.3.
What’s the v1 connector list?	localfs ONLY for filesystem-sourcing v1. SharePoint defers to v1.1 (custom shim required — either build a @coco.fn wrapper on MS Graph polling, OR sync SharePoint -> local folder out-of-band and use localfs). Postgres-as-target is available for KH’s downstream content_items table binding.
Does cocoindex detect DRAFT-vs-final near-duplicates?	No. content_fingerprint is exact-bytes (128-bit deterministic hash); all DRAFT-vs-final pairs in corpus have fully-distinct fingerprints (sizes also differ by ~2-3x). The ops.entity_resolution module is embedding+faiss-based but applies to ENTITY STRINGS not file-level near-duplicates. S10 substrate must extend cocoindex with a custom @coco.fn (e.g. content-embedding similarity over chunk extracts) or look outside (mempalace KG entity-resolution / skill-seekers metadata).

G2 decision-gate verdict: v1 connector list = localfs only; UC10 polling cadence = native fs-watch (live=True) for localfs, fall back to scheduled App.update() re-invocation (cron) for remote sources (Google Drive, S3, OCI). SharePoint v1.1 (out-of-scope for v1).

---

2. Method (executed)

2.1 Setup

Cocoindex installed in a worktree-local venv via:

python3 -m venv .venv-spike-s2
PIP_USER=0 PIP_TARGET="" .venv-spike-s2/bin/pip install cocoindex

(The PIP_USER=0 PIP_TARGET="" reset was needed because the worktree’s parent shell exports default --user install flags that conflict with venvs. Recorded for reproducibility.)

Cocoindex install hits PyPI; required dangerouslyDisableSandbox: true per CLAUDE.md sandbox notes. Subsequent Rust-engine startup (LMDB ops-DB open) also requires sandbox bypass — recorded as a gotcha (§4.4 below).

Cocoindex version installed: 1.0.3. Per 0.8.2-cocoindex-evaluation.md line 30, version available at the time of the evaluation was pre-1.0.0; the 1.0.3 API has evolved materially from the names used in the §S2 spec — see §3 below for the corrected API surface.

2.2 Corpus enumeration (baseline snapshot)

Script: .spike-s2/observe_walk.py (worktree-local, not committed). Walks the corpus recursively, prints (relative_path, size, content_fingerprint_hex) per file.

.venv-spike-s2/bin/python3 .spike-s2/observe_walk.py docs/client-documentation-base/

Result: 35 files enumerated across:

• binary/ (8 files): 4 final docx + 3 DRAFT docx + 1 Telehouse PDF
• markdown/ (25 files including markdown/assets/17746269306138.jpg): 13 numbered .md + 4 final audit .md + 3 DRAFT audit .md + 4 bid library .md + 1 image asset
• forms/ (2 files): 1 Cloud Security checklist .xlsx + 1 Standard Selection Questionnaire .pdf

Recursive walk traversed markdown/assets/ correctly; no symlinks or .DS_Store filtering applied (would need a FilePathMatcher).

2.3 UC4.4 change-classification simulation (against live engine)

Script: .spike-s2/uc4_live_observe.py. Defines a @cocoindex.fn(memo=True) observer, mounts the localfs walker via cocoindex.mount_each, and runs App.update() three times:

• Run A: cold cache — expect all files invoked (35)
• Run B: warm cache, no changes — expect 0 invocations (full memo hit)
• Run C: warm cache after applying UC4.4 mutations — expect exactly 4 invocations (1 add + 1 edit + 1 rename-target + 1 binary-replace; delete should produce 0 invocations as the engine purges target state)

Mutations applied between Run B and Run C (against an isolated .spike-s2/corpus-uc4/ clone — canonical baseline at docs/client-documentation-base/ was never touched):

UC4.4 kind	Mutation
Add	Created markdown/99-add.md with new content
Edit	Appended \\n\\nUC4.4 edit appended.\\n to markdown/01-company-overview.md
Delete	Removed markdown/12-research-notes-and-gaps.md
Rename	Renamed markdown/05-team-structure-and-key-people.md -> markdown/05-people-and-team.md (same content)
Binary-replace	Copied binary/DRAFT 2026 Phew - Tender and Bid Library - Implementation & Support .docx over binary/2026 Audit - Tender and Bid Library Template - Implementation & Support.docx (same filename, DRAFT content)

Observed result (captured stderr, verbatim):

======================================================================
# RUN A (cold cache):
  invocations: 35
======================================================================
# RUN B (warm cache, no changes):
  invocations: 0
======================================================================
# RUN C (warm cache, after UC4.4):
# Expected invocations:
#   - 1 add (99-add.md)
#   - 1 edit (01-company-overview.md)
#   - 1 rename target (05-people-and-team.md) - new key, same content
#   - 1 binary-replace (Implementation & Support.docx) - same key, new content
#   - 0 for delete (engine should purge cache entry)
#   = 4 invocations expected
  invocations: 4
    - 05-people-and-team.md  size=1488  fp=82a3d592
    - 99-add.md  size=28  fp=43ce08e5
    - 01-company-overview.md  size=1446  fp=1a966e8b
    - 2026 Audit - Tender and Bid Library Template - Implementation & Support.docx  size=160683  fp=923d943d
======================================================================

Engine matched the prediction exactly — 4 invocations in Run C. Delete produces 0 invocations (cocoindex purges the cache entry as the input key disappears from the source). Rename’s deleted source-side key (the old name 05-team-structure-and-key-people.md) presents as a delete from the engine’s view; the new key (05-people-and-team.md) presents as an add. There is no native rename-tracking primitive — recoverable post-hoc via fingerprint-equality cross-reference, see §4.2.

2.4 Out-of-engine diff harness (for reference)

Script .spike-s2/observe_walk.py --diff does fingerprint-equality cross-referencing across two snapshots. Output for the same UC4.4 mutations:

# unchanged: 31 / added: 2 / edited: 2 / deleted: 2 / renames-by-fp: 1
  ADD     markdown/05-people-and-team.md (size=1488, fp=82a3d592...)
  ADD     markdown/99-add-test.md (size=63, fp=c7869f69...)
  EDIT    binary/2026 Audit - Tender and Bid Library Template - Implementation & Support.docx (size 68493->160683, fp 8f517ddd->923d943d)
  EDIT    markdown/01-company-overview.md (size 1423->1477, fp 93999163->b3f18cf9)
  DELETE  markdown/05-team-structure-and-key-people.md (was size=1488, fp=82a3d592...)
  DELETE  markdown/12-research-notes-and-gaps.md (was size=2261, fp=873bb4d4...)
  RENAME  markdown/05-team-structure-and-key-people.md -> markdown/05-people-and-team.md (fp preserved)

The 1 rename was correctly identified post-hoc by matching fp=82a3d592... across the add+delete pair. Implication: if KH wants rename-tracking (e.g. preserve identity across SharePoint move, preserve content_items.id under rename), KH must implement this logic in the adapter layer — cocoindex won’t do it natively.

2.5 DRAFT-vs-final fingerprint observation

Script: .spike-s2/probe_draft_vs_final.py. Walks binary/ + markdown/, groups by topic-derived key (e.g. "implementation & support"), prints fingerprint per file.

## topic: 'implementation & support'  (4 files)
  identical_fingerprint = False
    [final] 'binary/.../Implementation & Support.docx'         size=68493   fp=8f517ddd...
    [DRAFT] 'binary/DRAFT ... Implementation & Support .docx'  size=160683  fp=923d943d...
    [final] 'markdown/.../Implementation & Support.md'         size=44106   fp=31cdd19e...
    [DRAFT] 'markdown/DRAFT ... Implementation & Support .md'  size=80306   fp=4e61bb5a...

Every DRAFT-vs-final pair has fully distinct fingerprints. Sizes differ by ~1.8-2.5x (drafts are roughly twice the byte-count of finals — consistent with track-changes / longer comment payloads in the DRAFT docx files).

Implication for S10 (cross-record dedup substrate): cocoindex content_fingerprint is exact-bytes and cannot natively flag DRAFT-vs-final near-duplicates. S10 evaluation must look at:

1. Within cocoindex via custom @coco.fn: chunk extract -> embedding -> faiss similarity comparison (the same building blocks as ops.entity_resolution.resolve_entities but applied at chunk-level not entity-level). This would be a substantial custom layer.
2. Outside cocoindex via mempalace KG: mempalace’s drawer + KG primitives include configurable entity resolution that may handle near-dup at document level. Untested.
3. Outside cocoindex via skill-seekers metadata: unknown whether skill-seekers has a near-dup primitive applicable to documents (vs codebase elements).

S10 should evaluate all three. Cocoindex does NOT solve UC8 cross-record dedup out of the box.

2.6 Live-watch (UC10 polling cadence) inspection

walk_dir(..., live=True) returns a _LiveDirItems that calls watchfiles.awatch(...) internally. From source inspection (.venv-spike-s2/.../localfs/_source.py):

• Native fs-watch on macOS (FSEvents), Linux (inotify), Windows (ReadDirectoryChangesW) via Rust notify crate (vendored by watchfiles 1.1.1).
• Three watchfiles.Change kinds: added, modified, deleted. No native rename event — appears as deleted+added.
• Default debounce=1600ms (coalesces bursts of changes). Optional force_polling=True + poll_delay_ms=300 (default) fallback.
• On directory move, watchfiles may not decompose into individual file events; cocoindex’s _LiveDirItems triggers subscriber.update_all() rescan.

UC10 cadence per connector (cocoindex 1.0.3 inventory):

Connector	Native change-detection	Polling fallback	Webhook	UC10 cadence recommendation
localfs	FSEvents/inotify via watchfiles (live=True)	300ms forced-polling	n/a	live=True; debounce 1600ms
postgres (source)	LISTEN/NOTIFY (DB-side trigger)	yes	n/a	LISTEN-based; KH staging branch may need wal-level config
kafka (source)	streaming consumer	n/a	n/a	streaming (continuous)
amazon_s3	none — list-based only	required	not natively wired; you’d configure SNS+SQS externally and read via a custom @coco.fn	scheduled App.update() (every N minutes); webhook gap
google_drive	none — list-based only	required	not natively wired; Drive API supports push notifications but cocoindex 1.0.3 doesn’t bind to them	scheduled App.update()
oci_object_storage	event-stream via OCI Streaming	yes (initial scan + tail)	OCI Streams subscription	live-stream where available
qdrant / lancedb / turbopuffer / neo4j / falkordb / surrealdb / doris / sqlite	target-only (no source change-detection)	n/a	n/a	n/a

SharePoint: does NOT appear in cocoindex/connectors/ directory in 1.0.3. No native binding. v1.1 candidate paths:

1. Build a @coco.fn source wrapper on MS Graph delta queries. ~3-5 days; uses Graph’s delta token cursor for change-detection. Needs OAuth app registration.
2. Sync SharePoint -> local NAS mount out-of-band; localfs the mount. Existing tools (rclone, Microsoft’s SharePoint sync client, OneDrive Files-On-Demand). ~1-2 days plumbing.
3. Defer SharePoint to v1.1 entirely. Localfs covers single-user / Phew-internal corpus; SharePoint becomes a sales-enabler later.

Recommendation: option (2) for v1.1 (out-of-band sync), option (1) only if Microsoft is a top-3 customer requirement.

2.7 Connector inventory (authoritative for cocoindex 1.0.3)

From find .venv-spike-s2/lib/python3.14/site-packages/cocoindex/connectors -maxdepth 1 -type d:

Sources + dual-role: localfs, postgres, kafka, amazon_s3, google_drive, oci_object_storage, sqlite

Targets only: qdrant, lancedb, turbopuffer, neo4j, falkordb, surrealdb, doris

Total: 14 connectors. Matches the list claimed in 0.8.2-cocoindex-evaluation.md line 51 exactly. No connectors added; none removed.

Confirmed absences vs §S2 v1 wishlist:

Wishlist	Status in 1.0.3
localfs	Present
SharePoint	Absent — v1.1 candidate
Notion	Absent
Google Drive	Present (poll-only)
Dropbox	Absent
Box	Absent

---

3. API drift findings — 0.8.2-cocoindex-evaluation.md lines 38, 46, 51, 142-143, 227

The §S2 spec specifies these API names (per 0.8.2-cocoindex-evaluation.md):

• Line 38: RecursiveSplitter(2000, 500).split(md) — chunking transform
• Line 46: localfs.walk_dir(src).items() — source binding
• Line 51: connector inventory (“localfs, amazon_s3, google_drive, oci_object_storage, kafka, postgres, sqlite, doris, lancedb, qdrant, turbopuffer, neo4j, falkordb, surrealdb”)
• Line 142-143: localfs.walk_dir + docling.convert(file) for binaries
• Line 227: localfs.walk_dir for markdown

Drift vs cocoindex 1.0.3 (installed):

API as documented in §S2	Actual in 1.0.3	Drift severity
from cocoindex.sources import localfs	from cocoindex.connectors.localfs import walk_dir, ...	MEDIUM — module renamed sources -> connectors (and added separate _target.py); functional surface identical
from cocoindex.transforms import RecursiveSplitter	from cocoindex.ops.text import RecursiveSplitter	MEDIUM — same name, different module path
from cocoindex.targets import postgres	from cocoindex.connectors.postgres import ...	MEDIUM — module renamed targets -> connectors.postgres._target
@cocoindex.flow_def(name=...)	No such decorator. Use cocoindex.App(name, main_fn) with an inner async def main_fn containing await cocoindex.mount_each(...). The flow is now an App-with-main-fn instead of a @flow_def-decorated function.	HIGH — App-based programmer model is new; rewrite required
cocoindex update CLI	cocoindex CLI still present (.venv-spike-s2/bin/cocoindex); also App.update() async / update_blocking() sync; App.update(live=True) enables fs-watch	LOW — CLI still present, library surface expanded
localfs.walk_dir(src).items()	walk_dir(path, recursive=False, live=False, path_matcher=None) — note recursive=False is the default; must pass recursive=True to traverse subdirs	LOW (gotcha) — needs explicit recursive=True
Ops DB = SQLite	LMDB (configurable via Settings(db_path=...) or COCOINDEX_DB env var); SQLite-backend dropped	MEDIUM — backend swap; operational impact for Cloud Run job state persistence (LMDB single-process semantics affect concurrency; see §4.5)
@coco.fn(memo=True)	@cocoindex.fn(memo=True, memo_key=..., batching=False, version=None, logic_tracking='full', deps=None, ...)	LOW — same shape, expanded params
coco.mount_each(fn, items_iter, target)	cocoindex.mount_each(fn, items_iter) — the target-mount is a separate mount_target call OR target is implicit via @coco.fn return type; mount_each signature is (*pos_args, **kwargs) (polymorphic)	MEDIUM — semantics shifted; needs verification when wiring postgres target binding in S1

Implication for S1 (which gates Phase 2): S1 must update its spec to:

1. Use cocoindex.App("name", main_fn) programmer model, not @flow_def
2. Use cocoindex.connectors.localfs.walk_dir(..., recursive=True) not cocoindex.sources.localfs.walk_dir
3. Use cocoindex.ops.text.RecursiveSplitter not cocoindex.transforms.RecursiveSplitter
4. Plan for LMDB persistence (Cloud Run job state file)
5. Verify postgres.mount_table_target(...) shape vs current cocoindex.connectors.postgres._target.* — the actual function name has likely changed too; spec needs API-name re-discovery as its day-1 task

0.8.2-cocoindex-evaluation.md flagged but not corrected here (flagged below as ERRATA-1).

---

4. Per-spike-question findings

4.1 Initial-ingest behaviour observations

• File enumeration: correct across 35 files spanning 3 subfolders. recursive=True mandatory.
• Per-file content-hash: cocoindex computes a 128-bit fingerprint via File.content_fingerprint() (cached after first call). Algorithm is internal Rust-side; not md5 / not sha256. Deterministic; 1-byte change avalanches.
• Binary files: opaque-blob-pass-through. File.read() returns raw bytes for docx/pdf/xlsx; cocoindex itself does NO extraction. DOCX/PDF/XLSX extraction is downstream @coco.fn work (e.g. wrap docling, KH’s existing mammoth / unpdf adapters, or python-docx via scripts/docx_utils.py).
• FileMetadata: mtime + size only. No modified-by / ACL / extended attributes. Path is the only stable identity.

4.2 UC4.4 change-classification confirmation

Change kind	Cocoindex behaviour	KH-adapter need
Add	New key; engine invokes the @coco.fn once. Cache populated.	None — works as-is.
Edit	Same key, new fingerprint; engine invokes the @coco.fn once. Cache updated.	None — works as-is.
Delete	Key removed; engine purges target state and does NOT invoke the @coco.fn.	If KH wants soft-delete with archival, the adapter must intercept the “no longer in source” signal and write a content_items.archived_at rather than deleting. Engine-level: App.update() with the deleted key absent will trigger downstream target deletion. Investigation point for S1: does postgres.mount_table_target perform hard DELETE or soft DELETE? Likely hard; KH may need a target wrapper.
Rename	Old key disappears (delete semantics); new key appears (add semantics). Engine sees as add+delete pair.	KH must add rename-detection in the adapter layer if KH wants to preserve content_items.id across rename. Approach: post-update, compare old + new keys’ fingerprints; if match found, issue UPDATE on the existing row rather than DELETE-then-INSERT. Cost: ~1 day of work; needs hooks into cocoindex’s per-row diff output. Alternative: accept that rename = re-ingestion (the user-visible content remains accessible via the new URL; old row is orphaned then archived).
Binary-replace	Same key, new fingerprint. Engine invokes the @coco.fn once. Cache updated. Indistinguishable from edit at engine level.	This is the S188-feedback friction. Detecting “user re-uploaded a different doc with same filename” vs “user edited the file in-place” requires the UI to surface the difference at upload time, OR the adapter to inspect the diff and apply heuristics (size-ratio threshold, content-shape change). Cocoindex’s job ends at “tell you the bytes changed”; the product/UI/UX work is downstream.

4.3 DRAFT-vs-final dedup observation

Confirmed (§2.5 above) that exact-bytes fingerprinting does NOT collapse DRAFT-vs-final pairs. S10 substrate decision is genuinely open — cocoindex does not solve UC8 out-of-the-box. S10 evaluation must compare:

1. Custom cocoindex @coco.fn pipeline: chunk -> embedding -> faiss similarity. Building blocks exist in cocoindex.ops.entity_resolution, but they apply to entity strings, not whole files. ~3-5 days work to adapt.
2. mempalace KG entity resolution at document granularity. Untested.
3. skill-seekers document metadata dedup primitive. Unknown if exists.

S10 should run all three on the same corpus and score; cocoindex’s content_fingerprint is the BASELINE (exact-match dedup only).

4.4 UC10 polling cadence per connector

See §2.6 table. Critical findings:

• localfs: native fs-watch via FSEvents/inotify. live=True mode delegates to watchfiles.awatch(...) with 1600ms default debounce. Best-in-class change detection for local-mounted folders.
• SharePoint: no native connector. Recommended v1.1 path is out-of-band sync (rclone / OneDrive Files-On-Demand) to a local folder mount + localfs binding.
• Google Drive: poll-only. List API + per-file mtime comparison. Cadence is controlled by how often the operator schedules App.update(). For KH this would be a Cloud Run job on cron; recommend 15-60min cadence for v1.
• Postgres: LISTEN/NOTIFY available; needs staging Supabase branch’s wal-level config check before relying on. Spike S1 covers this.

4.5 Operational concerns (uncovered during spike)

• Sandbox bypass required for engine startup. cocoindex.App(...).update() opens an LMDB ops-DB file. Macos sandbox profile rejects this (Operation not permitted (os error 1)). Production Cloud Run unaffected; dev-loop friction for sub-agents running in sandboxed harness. Workaround: dangerouslyDisableSandbox: true on the Python invocation.
• LMDB single-process semantics. LMDB is single-writer; multi-process concurrent writes are blocked. For KH’s planned multi-worker Cloud Run topology, the ops-DB needs partitioning (one LMDB per worker shard) OR the engine must run as a singleton orchestrator with workers as @coco.fn runners only. Architecture revision needed if multi-worker concurrent ingest is mandatory for v1. Confirm with S1 + a separate “cocoindex concurrency” probe (not in current spike-plan; recommend adding as S14).
• watchfiles ignores node_modules, .git, __pycache__ etc. by default (per DefaultFilter in watchfiles.main). If KH’s connected folders ever contain these names as legitimate user content, they’d silently disappear from live-watch updates. Configure a custom watch_filter if relevant.
• No native rename tracking. §4.2 above.

---

5. Decision gate G2 outcome

G2: v1 connector list + UC10 polling cadence

Sub-decision	Verdict
v1 connector list	localfs only. SharePoint, Notion, Dropbox, Box absent from 1.0.3. Google Drive + S3 are usable but poll-only — defer to v1.1 unless a specific client mandates them.
UC10 polling cadence for localfs	Native fs-watch via walk_dir(live=True) — sub-second change detection with 1600ms debounce.
UC10 polling cadence for any future remote source	Scheduled App.update() at 15-60min cadence via Cloud Run cron; webhook integration would be a custom @coco.fn wrapper.
SharePoint v1 vs v1.1	v1.1. Recommended path: out-of-band sync to local mount (rclone / OneDrive Files-On-Demand); fall back to MS Graph delta-query @coco.fn wrapper if mount-based approach blocked by client IT policy.
Architecture impact	v1 architecture must NOT assume multi-source ingest. The “many connectors” claim from 0.8.2-cocoindex-evaluation.md line 51 is technically correct but operationally limited to localfs + Postgres for KH v1.

G2 status: RESOLVED. Phase 2 architecture commits to localfs-only for filesystem-sourcing in v1.

---

6. Surprises and residual questions

6.1 Surprises

1. API has drifted substantially since 0.8.2-cocoindex-evaluation.md was written. Modules renamed (sources -> connectors); programmer model changed (@flow_def -> App + main_fn); ops-DB swapped (SQLite -> LMDB). S1 needs API-rediscovery as its first task. (See ERRATA-1.)
2. recursive=True is NOT the default for walk_dir. A naive walk_dir(path).items() only enumerates the root level. Easy to miss; would have caused a silent 0-file ingest in production if not caught.
3. No native rename tracking. Cocoindex treats rename as add+delete at the key level. KH’s 0.9-edit-flow-investigation.md UC10 §4.4 assumes “renamed docs preserve identity”; this requires KH-side adapter work.
4. No SharePoint connector despite 0.8.2-cocoindex-evaluation.md not explicitly listing it as absent — the absence was implicit in the connector inventory but the spike-plan’s §S2 v1-wishlist still included it as “if available”. It isn’t.
5. LMDB single-process write-lock. Not explicitly flagged in any prior doc. Major implication for the planned multi-worker Cloud Run topology in 0.9-intended-architecture.md. Needs a new spike (S14 — cocoindex concurrency) or architecture revision to single-orchestrator + worker-fan-out.
6. watchfiles.DefaultFilter ignores node_modules and friends by default. Quiet gotcha for clients who happen to name folders __pycache__ etc.

6.2 Residual questions

Q	For
Q-S2-1	Does cocoindex.connectors.postgres._target.* accept a pre-existing 70-col schema with FKs + triggers? Gates S1 (G1 decision). Spike S2 didn’t touch the Postgres target binding (out of scope per spike-plan; S1’s question).
Q-S2-2	Cocoindex Rust engine’s per-row failure isolation under DOCX extraction errors? When mammoth blows up on a corrupt docx, does the engine continue with the other 34 files or halt? Defer to S1 / S8.
Q-S2-3	Multi-process concurrency model. Add new spike S14 (cocoindex concurrency) or fold into S1.
Q-S2-4	DRAFT-vs-final dedup substrate — which of {cocoindex custom @coco.fn / mempalace KG / skill-seekers metadata} wins? S10’s job.
Q-S2-5	LMDB persistence model for Cloud Run jobs (ephemeral container filesystem). Either persist LMDB to a Cloud Storage volume (latency hit) or accept cold-cache on every job (full re-fingerprint of corpus). Tradeoff analysis needed in Phase 2 architecture-impl.
Q-S2-6	Webhook wiring for Google Drive / S3 / SharePoint (if any becomes a v1 requirement). Each needs a custom @coco.fn source with its own change-detection pattern. ~3-5 days per connector.
Q-S2-7	When mount_each mounts a live=True source, does the engine run continuously (process holds open the fs-watcher) OR does it terminate when the items iterator drains? Important for Cloud Run job lifecycle. Defer to operational spike before Phase 2 ship.

---

7. Recommendations for downstream spikes

7.1 S1 (cocoindex schema-coupling, gating G1)

Update S1 spec to:

1. Import paths corrected per §3: cocoindex.connectors.localfs.walk_dir, cocoindex.ops.text.RecursiveSplitter, cocoindex.connectors.postgres.*.
2. Programmer model: cocoindex.App("name", main_fn) + inner async def main_fn + cocoindex.mount_each(...).
3. LMDB ops-DB path via Settings(db_path=...) or COCOINDEX_DB env var.
4. Add a Q-S2-1 sub-task: confirm the EXACT postgres target binding API in 1.0.3 before attempting the mount. The names in §S1 (postgres.mount_table_target, postgres.from_env) likely no longer exist — re-discover from .venv-spike-s2/lib/python3.14/site-packages/cocoindex/connectors/postgres/_target.py first.

7.2 S10 (cross-record dedup substrate)

S10 input is now concrete: DRAFT-vs-final pairs from docs/client-documentation-base/ exhibit ~1.8-2.5x size differences with fully-distinct fingerprints. Embedding-similarity at chunk granularity is the most likely successful primitive. S10 must score:

• Substrate A: custom @coco.fn pipeline using cocoindex.ops.entity_resolution building blocks adapted to chunk-level. Estimated effort 3-5 days. Predicted recall: high (embedding semantics); precision dependent on threshold tuning.
• Substrate B: mempalace KG document-level dedup. Untested.
• Substrate C: skill-seekers document metadata dedup. Likely doesn’t exist.

Recommend running Substrate A first; if recall > 0.85 on the DRAFT-vs-final pairs and false-positive rate < 0.1 on unrelated audit-doc topics, accept Substrate A and skip B/C.

7.3 S14 (new spike — cocoindex concurrency)

Add to spike plan. Question: Does LMDB single-writer constrain KH’s planned Cloud Run multi-worker topology? If yes, what’s the rework cost?

Method:

1. Spin up 2 Python processes pointing at the same LMDB. Run App.update() concurrently. Observe.
2. If lock contention: model the engine as a singleton orchestrator + worker-pool downstream.
3. Estimate Phase 2 architecture-impl cost delta of singleton orchestrator vs the originally-planned multi-worker shape.

Owner: foreground. Budget: 0.5 day. Sequencing: parallel with S1.

7.4 Updates needed to 0.9-intended-architecture.md

• §13 architecture must specify localfs-only v1; SharePoint v1.1.
• §13 must call out LMDB single-writer; either resolve via S14 or commit to singleton orchestrator topology.
• §13 must specify rename-handling adapter strategy (or accept rename = re-ingest, with the user-visible cost called out).

(Flagged here; NOT modified in this spike per task constraints.)

---

8. Errata flagged to other docs (not modified per task constraints)

ERRATA	Document	Issue	Suggested fix
ERRATA-1	0.8.2-cocoindex-evaluation.md lines 38, 46, 51, 142-143, 227	API names have drifted in cocoindex 1.0.3. cocoindex.sources is now cocoindex.connectors; cocoindex.transforms is now cocoindex.ops.text; cocoindex.targets is now cocoindex.connectors.*._target.*; @flow_def decorator removed in favour of App(name, main_fn) programmer model; SQLite ops-DB replaced with LMDB. Connector list (line 51) remains accurate.	Add a “1.0.3 API correction” note at the head of 0.8.2-cocoindex-evaluation.md, OR replace the example block at lines 35-47 with corrected code that imports from current modules.
ERRATA-2	0.9-spike-plan.md §S2 line 116	”v1 connector list” question is partially answered by spike: localfs only for v1. Plan to update spike-plan §S2 success-criteria to reflect this once Liam ratifies.	Update §2 S2 success-criteria + decision-gate text to “All 5 file types work via localfs; SharePoint deferred to v1.1” once confirmed.
ERRATA-3	0.9-spike-plan.md §S1 line 60 (“API names verified against 0.8.2-cocoindex-evaluation.md lines 38, 46, 51, 142-143, 227 - use localfs.walk_dir source + RecursiveSplitter transform + mount_each target binding”)	API names cited in S1 are for an older cocoindex version; first action of S1 must be re-discovery (not “verified”).	Rephrase to “API names from 0.8.2-cocoindex-evaluation.md lines 38-227 are PRE-1.0.0; first S1 task is to verify against installed 1.0.3 and update example block.”
ERRATA-4	0.9-edit-flow-investigation.md line 444 (“binary replacements extract + update sidecars”)	Implicit assumption that the engine distinguishes binary-replace from edit. Cocoindex cannot — both present as “key unchanged, content changed”.	Note in UC10 §4.4 that distinguishing binary-replace from edit requires UI-level user input at upload OR heuristic in the adapter.

---

9. Reproducibility — to re-run this spike

# From worktree or main:
cd <repo>

# 1. Venv
python3 -m venv .venv-spike-s2
PIP_USER=0 PIP_TARGET="" .venv-spike-s2/bin/pip install cocoindex

# 2. Verify install
.venv-spike-s2/bin/python3 -c "import cocoindex; print(cocoindex.__version__)"
# Expected: 1.0.3 (or later)

# 3. Snapshot canonical corpus
.venv-spike-s2/bin/python3 .spike-s2/observe_walk.py docs/client-documentation-base/

# 4. UC4.4 live test (sandbox-bypass required for LMDB)
mkdir -p .spike-s2/coco-db-uc4
cp -R docs/client-documentation-base/ .spike-s2/corpus-uc4
COCOINDEX_DB=$(pwd)/.spike-s2/coco-db-uc4 .venv-spike-s2/bin/python3 .spike-s2/uc4_live_observe.py
# Expected: Run A=35 invocations, Run B=0, Run C=4

# 5. DRAFT-vs-final probe
.venv-spike-s2/bin/python3 .spike-s2/probe_draft_vs_final.py
# Expected: all topic-paired DRAFT/final fingerprints distinct

Scripts are NOT committed (gitignored at .spike-s2/, .venv-spike-s2/). Re-create from this doc’s §2 method or from the inline code snippets above; the canonical corpus at docs/client-documentation-base/ is the only persistent artefact required.

---

10. Decision recommendation (binding for G2)

Recommendation: adopt localfs as the v1 source connector; rely on live=True fs-watch for UC10 change detection on locally-mounted folders. Defer SharePoint and all other remote sources to v1.1. Plan for a singleton-orchestrator + worker-pool Cloud Run topology pending S14 (cocoindex concurrency) outcome, OR accept that multi-worker writes are out of scope until LMDB is replaced by a multi-writer ops-DB upstream.

Phase 2 architecture-impl can commit to cocoindex IF AND ONLY IF G1 (Scenario A from S1) also resolves favourably. G2 alone does not block; localfs-only is sufficient for v1.

---

End of S2 spike record. G2: RESOLVED (localfs-only for v1; live=True fs-watch for UC10).