SPIKE-27.9 — Sharing the primary .gitnexus/ index into cmux worker worktrees

SPIKE-27.9 — Sharing the primary .gitnexus/ index into cmux worker worktrees

Research spike. Investigation only — no cmux script, ledger, or .gitnexus/ modification. Gates {27.8} PART B (the index-seeding mechanism choice). Code-intelligence tool-discipline is exempt for this subtask: it is .md/research only and touches no source symbols. UK English, no emoji.

Headline ruling

Native worktree.sparsePaths + worktree.symlinkDirectories: NO-GO for cmux’s raw-git worktree add path. Both settings are Claude-Code-managed-only and have no effect on the externally-created worktrees that launch-worker.sh deliberately uses.

Chosen {27.8} PART B mechanism: A+B hybrid — symlink the parent’s .gitnexus/lbug + .gitnexus/meta.json into the worker worktree’s own .gitnexus/ (preserving the worktree’s tracked .gitnexus/CLAUDE.md), then gitnexus index <worktree-path> to register it. Register-only (Option A alone) was empirically proven insufficient: gitnexus index requires a real .gitnexus/ folder at the target path and refuses to register a path that has none.

Read-consistency during re-index: ACCEPTABLE with a documented caveat. The shared index is read-consistent because gitnexus analyze swaps the lbug via atomic rename(2) (the inode changes), so a reader sees either the complete old snapshot or the complete new one — never a torn/partial file, and never silent bad data. The one residual risk is a rare, loud, transient reader crash (SIGSEGV, rc=139) if a read open()s exactly as the rename swaps the inode (observed ~1–2% of reads that overlapped a swap; 0% silent corruption). This is retry-safe and does not block GO.

---

1. Why native sparsePaths/symlinkDirectories cannot be used (given facts)

The following were verified against code.claude.com docs earlier this session and are cited here as established, not re-researched:

• worktree.symlinkDirectories applies only at Claude-Code-managed worktree creation (claude --worktree, EnterWorktree, Agent isolation:"worktree"). It has no effect on an externally-created git worktree add.
• worktree.sparsePaths is likewise managed-only; no effect on external git worktree add.
• .worktreeinclude is also managed-only (not honoured for raw git worktree add).
• cmux’s launch-worker.sh deliberately uses raw git worktree add to bypass Anthropic worktree-isolation structural caching and to keep durable attachable terminals.

Consequence (reasoned): because the worktrees that need the shared index are created by raw git worktree add, they never pass through the managed-creation code path that honours symlinkDirectories/sparsePaths/.worktreeinclude. The only way to obtain native seeding would be to abandon the raw-git worktree add path — which would forfeit the two properties cmux deliberately chose it for. Native seeding is therefore not a viable mechanism for cmux. The realistic mechanism is a manual step inside launch-worker.sh, evaluated empirically below.

---

2. Storage-engine finding

The primary index at /Users/liamj/Documents/development/knowledge-hub/.gitnexus/ is:

Item	Finding
Layout	lbug (single file, not a directory) + meta.json + .gitignore + tracked CLAUDE.md
lbug size	279,547,904 bytes (~270 MB); dir du -sh = 272M
File magic	First 4 bytes 4c 42 55 47 = ASCII LBUG → bespoke single-file DB
Engine	ladybugdb (confirmed by meta.json capabilities.graph.provider: "ladybugdb", plus ladybugdb-fts and ladybugdb-vector)
Pre-alloc	A 4-file throwaway repo still produced a ~17 MB lbug — the engine pre-allocates pages

Read-consistency implication of the engine: the decisive property is not the engine’s internal MVCC but how gitnexus analyze publishes a new index. Empirically (§3) the lbug inode changes on every reanalyze, which means analyze writes a fresh file and atomically renames it over the existing path rather than mutating in place. On a single filesystem rename(2) is atomic, so:

• the pathname /.../.gitnexus/lbug always resolves to a complete lbug (old inode before the rename, new inode after);
• a reader that has already open()ed the old inode keeps reading the complete old snapshot until it closes it (Unix unlink-after-open semantics);
• there is no window in which a partially-written file is visible at the path.

A symlink pointing at the pathname (not the inode) therefore transparently follows the swap and always resolves to a complete index.

---

3. Read-consistency test (TESTED, not reasoned)

Method (all on throwaway /tmp/gnx-spike-* repos — the real primary was never reanalysed):

1. git init a throwaway repo with a handful of multi-function JS files; gitnexus analyze it to produce a 17 MB lbug.
2. git worktree add a second checkout (mimics cmux’s raw worktree path) and symlink the primary’s lbug + meta.json into <worktree>/.gitnexus/, with a worktree-local real CLAUDE.md.
3. Run a read loop (gitnexus status then gitnexus query) against the symlinked worktree while concurrently re-running gitnexus analyze on the throwaway primary (each commit changed the source so the lbug was genuinely rewritten).

Observations:

• Atomic snapshot flip. Across a 21-read status loop spanning one reanalyze, reads 1–20 saw the complete old snapshot (commit 049133e, indexed 18:30:10); read 21 flipped cleanly to the complete new snapshot (commit 2090c28, indexed 18:30:57). No read saw a mixture. The stale flag at read 21 is expected and correct — it is a git-HEAD-vs-index comparison, not corruption.
• Inode swap confirmed. lbug inode was 244669830 before and a different inode after each reanalyze → atomic-rename publish, not in-place mutation.
• Symlink follows transparently. After reanalyze, the worktree’s meta.json symlink reported the new lastCommit/nodes with no relinking; the worktree-local CLAUDE.md (47 bytes) was fully preserved.
• No silent corruption — ever. Across all stress runs (~106 reads that overlapped a reanalyze swap), 0 returned rc=0 with malformed/empty data. A read either returned complete valid JSON or crashed loudly.
• Rare transient crash at the swap instant. 2 of ~106 overlapping reads crashed with rc=139 (SIGSEGV) — a heavier query (which opens and traverses graph pages) caught the inode mid-swap. Approx 1–2% crash rate when a read coincides with the rename; far lower in practice because most reads occur with no reanalyze in flight. The crash is loud, immediate, and resolved by a retry.

Verdict: the shared mutable index is read-consistent (atomic snapshot semantics, no torn reads, no silent bad data). The only failure mode is a rare transient reader crash at the rename instant, which is retry-safe.

---

4. Option A vs Option B (TESTED)

Mechanism	Test	Result
Option A — register-only (gitnexus index <worktree> with no local .gitnexus)	On a bare throwaway worktree	FAILS. gitnexus index prints No .gitnexus/ folder found at: <path>/.gitnexus … Run gitnexus analyze; subsequent gitnexus status reports Repository not indexed.
Option B — symlink lbug+meta.json, then register	On a symlinked throwaway worktree	WORKS. gitnexus status reports up-to-date; gitnexus query -r <worktree-path> returns real processes from the shared index; the worktree-local CLAUDE.md is preserved.

Why A alone fails (root cause from the registry): ~/.gitnexus/registry.json records each repo with an explicit storagePath = <path>/.gitnexus, and gitnexus index <path> resolves the index by reading <path>/.gitnexus/meta.json. It does not dedupe by remoteUrl/name to point a new path at an existing index. So a worktree with no .gitnexus/ of its own has nothing for the registry to point at — a local lbug (real or symlinked) is mandatory. Hence the seeding mechanism must be the A+B hybrid: provide a local lbug via symlink, then register.

Note on repo-name disambiguation: the registered name derives from meta.json’s repo-path-derived name, so multiple worktrees sharing one meta.json register under the same display name (primary in the test). MCP/CLI calls disambiguate by absolute path (-r <worktree-path>). For cmux, in-worktree status/MCP calls resolve by the worktree CWD, so this is not a blocker — but meta.json’s repoPath will point at the parent, not the worktree (cosmetic).

---

5. What {27.8} PART B should implement

In launch-worker.sh, after the raw git worktree add and before the worker Claude starts, add an idempotent, non-fatal seeding step:

1. Resolve the parent index: PARENT_GNX="<main-repo>/.gitnexus". If $PARENT_GNX/lbug or $PARENT_GNX/meta.json is missing, skip silently (worker simply gets the existing “stale (never)” behaviour — no regression).
2. mkdir -p "$WT/.gitnexus".
3. Symlink the two mutable artefacts, idempotently (replace any existing symlink, never clobber a real file):
   • ln -sfn "$PARENT_GNX/lbug" "$WT/.gitnexus/lbug"
   • ln -sfn "$PARENT_GNX/meta.json" "$WT/.gitnexus/meta.json" Use absolute targets (the test used absolute symlinks and they survived reanalyze).
4. Do not symlink CLAUDE.md — the worktree already has its own tracked .gitnexus/CLAUDE.md (and .ast-dataflow/CLAUDE.md); leave it in place. (gitnexus index may also drop a tiny .gitnexus/.gitignore containing * — harmless.)
5. Register: gitnexus index "$WT". Wrap so a non-zero exit is logged but non-fatal (gitnexus index "$WT" || echo "[gitnexus] seed skipped (non-fatal)"). Registration is idempotent — re-running on an already-registered path just refreshes the entry.

Idempotency: steps 3 and 5 are safe to re-run on every worker launch (ln -sfn is idempotent; gitnexus index re-registers cleanly).

Non-fatal guarantee: every step degrades to the current behaviour on failure; never fail the worker launch over index seeding. The worst case is the worker falls back to running its own gitnexus analyze later, exactly as today.

Read-consistency guidance to document for {27.8}: the shared index is read-consistent (atomic-rename publish). The only caveat is a rare transient reader crash if a query lands exactly during a parent reanalyze; gitnexus CLI/MCP calls are retry-safe, so no extra locking is needed. If a worker’s own toolchain runs gitnexus analyze it would replace the shared lbug (the symlink target = the parent’s file) — so {27.8} should either (a) instruct workers not to reanalyze (rely on the parent), or (b) accept that a worker reanalyze re-points the shared file (still atomic, but changes the parent’s index). Recommended: workers read-only; reanalysis stays a parent-orchestrator responsibility.

---

6. Cleanup confirmation

All throwaway artefacts removed and the real environment verified untouched:

• gitnexus remove -f on the two throwaway registry entries (primary, wt-symlink); wt-bare was never registered (Option A failed to register it).
• git worktree remove --force + git worktree prune on the throwaway worktrees.
• rm -rf /tmp/gnx-spike-* and all /tmp/gnx-*.log, /tmp/registry-snapshot-*.json.
• Final gitnexus list shows exactly one entry — knowledge-hub at the correct path, commit fffe0ae, 2481 files — i.e. the registry is back to its pre-spike state (in fact cleaner than the mid-spike snapshot).
• Real .gitnexus/ unchanged: still 272M, meta.json lastCommit still 1536f6a9… (never reanalysed). No throwaway worktrees under .claude/worktrees/.

---

Appendix — evidence (exact commands + key output)

A1. Index characterisation

$ ls -la /Users/liamj/Documents/development/knowledge-hub/.gitnexus/
-rw-r--r--  1 liamj staff       4471 CLAUDE.md
-rw-r--r--  1 liamj staff  279547904 lbug
-rw-r--r--  1 liamj staff        697 meta.json
$ du -sh .../.gitnexus/        -> 272M
$ xxd lbug | head -1           -> 00000000: 4c42 5547 ...  ("LBUG")
$ file lbug                    -> data
meta.json: "capabilities.graph.provider": "ladybugdb"  (+ ladybugdb-fts, ladybugdb-vector)

A2. gitnexus index semantics

$ gitnexus index --help
Register an existing .gitnexus/ folder into the global registry (no re-analysis needed)
  -f, --force      Register even if meta.json is missing (stats will be empty)
  --allow-non-git  Allow registering folders that are not Git repositories
$ cat ~/.gitnexus/registry.json   -> entries carry "storagePath": "<path>/.gitnexus"

A3. Option A (register-only, no local .gitnexus) — FAILS

$ gitnexus index /tmp/gnx-spike-*/wt-bare
  No .gitnexus/ folder found at: /tmp/gnx-spike-*/wt-bare/.gitnexus
  Run `gitnexus analyze` to build the index first.
$ (cd wt-bare && gitnexus status)
  Repository not indexed.  Run: gitnexus analyze

A4. Option B (symlink lbug+meta, then register) — WORKS

$ ln -s .../primary/.gitnexus/lbug      wt-symlink/.gitnexus/lbug
$ ln -s .../primary/.gitnexus/meta.json wt-symlink/.gitnexus/meta.json
$ echo "..." > wt-symlink/.gitnexus/CLAUDE.md      # worktree-local, NOT symlinked
$ gitnexus index .../wt-symlink   ->  Repository registered: wt-symlink (17 nodes ...)
$ (cd wt-symlink && gitnexus status)
  Indexed commit: 049133e | Current commit: 049133e | Status: up-to-date
$ gitnexus query "run" -r .../wt-symlink   -> returns proc_1_main / proc_0_main (valid)

A5. Read-consistency under concurrent reanalyze

# 21-read status loop spanning one reanalyze:
iter 1..20  -> up-to-date, commit 049133e, indexed 18:30:10  (old snapshot)
iter 21     -> commit 2090c28, indexed 18:30:57, "stale" flag (new snapshot, atomic flip)
# inode of lbug: 244669830 (before)  ->  244670687 (after)   => atomic rename publish
# symlinked meta.json after reanalyze: lastCommit 2090c28..., nodes 32  (follows swap)
# worktree-local CLAUDE.md (47 bytes) preserved

# Stress classification (query loop during reanalyze):
round set 1: TOTAL 25  OK 25  SEGV 0  BADDATA 0
round set 2 (12 rounds): TOTAL 60  OK 54  SEGV 1  BADDATA 0
earlier run: 1 SEGV (rc=139) at iter 4
=> aggregate ~2 SEGV / ~106 overlapping reads (~1-2%); 0 silent bad data ever.

A6. Cleanup

$ gitnexus remove -f .../primary       (removed)
$ gitnexus remove -f .../wt-symlink    (removed)
$ gitnexus remove -f .../wt-bare       ("Nothing to remove ... Available: knowledge-hub")
$ git worktree remove --force ...; git worktree prune
$ rm -rf /tmp/gnx-spike-*
$ gitnexus list
  Indexed Repositories (1)
  knowledge-hub  /Users/liamj/Documents/development/knowledge-hub  (commit fffe0ae, 2481 files)
$ du -sh .../knowledge-hub/.gitnexus  -> 272M  (unchanged)
  meta.json lastCommit: 1536f6a9...  (real primary never reanalysed)