RESEARCH.md — {27.5} cmux orchestrator-monitoring regression + script consolidation + worktree gitnexus-index inheritance

RESEARCH.md — {27.5} cmux orchestrator-monitoring regression + script consolidation + worktree gitnexus-index inheritance

Task ID-27 ({27.5} RESEARCH). Authored S280. UK English throughout. Scope: shell + config investigation (NOT TypeScript corpus). Empirical evidence gathered against the live main working tree on 29/05/2026.

---

0. Code-intelligence orientation

This Subtask touches shell scripts (.claude/skills/session-driver-cmux/scripts/*.sh) and git/cmux/gitnexus configuration only — none of which lives in the TypeScript corpus GitNexus indexes. Per the “Always Do” block in .gitnexus/CLAUDE.md, orientation was still run and is recorded verbatim:

• gitnexus query "cmux worker monitoring orchestrator events" initially FAILED:

  Error: Multiple repositories indexed. Specify which one with the "repo" parameter.
  Available: knowledge-hub (/Users/liamj/.gitnexus/repos/knowledge-hub),
             knowledge-hub (/Users/liamj/Documents/development/knowledge-hub)

  This failure is itself a load-bearing finding for thread 4 (see §5 — duplicate registry collision). Re-run with --repo /Users/liamj/Documents/development/knowledge-hub returned only Python pipeline worker-health functions (mark_worker_crashed, reset_worker_state, worker_is_healthy in scripts/cocoindex_pipeline/server.py) — unrelated to the cmux session-driver shell monitoring layer.

• Greenfield-for-shell disclaimer: the session-driver-cmux monitoring scripts are shell, not TS/Python symbols, so GitNexus and ast-dataflow do not index them. No existing indexed symbols match the cmux monitoring surface — greenfield from the code-intelligence tooling’s perspective. ccc search was not run because the entire surface is shell + on-disk config, which ccc (TS/Python-oriented) does not cover; evidence was instead gathered by direct file read + git history + live CLI probes.

---

1. Summary

Four threads, all rooted in absolute-path assumptions that break the moment the orchestrator shell’s CWD is not the canonical main working-tree root:

1. events.jsonl unreadability regression — every monitoring script derives its events base from git rev-parse --show-toplevel, which resolves to whichever working tree the orchestrator’s CWD is inside. When the orchestrator shell sits in (or has drifted into) a worktree, EVENTS_BASE points at a .claude/cmux-events/ that does not contain the worker session dirs → ls → no-such-file. This is a CWD-resolution regression, not a sandbox-allowlist or cmux-version regression.
2. fleet-watch.sh vs wait-for-fleet.sh — DIFFERENT files with DIFFERENT lineage and DIFFERENT contracts. fleet-watch.sh is a v2 evolution of the S262/S263 ad-hoc kh-fleet-watch.sh monitoring script; wait-for-fleet.sh is the canonical skill-family member. Ruling: co-locate fleet-watch.sh into the scripts family, renamed, as the durable smart-watcher — it is strictly richer than wait-for-fleet.sh and embodies the durable-monitoring fix.
3. Durable monitoring fix — replace bare stop-poll with the fleet-watch.sh smart-watcher model (multi-signal: session_end / final_report.* / OQ-heading growth / AskUserQuestion stall / stop pause / fleet-quiet), and harden the events-base resolution (thread 1). Event-driven (fswatch/inotifywait) is a later optional optimisation, not the primary fix.
4. GitNexus worktree index-inheritance — the index is NOT only the gitignored 272 MB .gitnexus/lbug; it is keyed in a global registry (gitnexus list) by absolute repo path. Worktrees live at a different absolute path, match no registered entry, and read “stale (never)”. Two seeding options are viable; the global-registry share (gitnexus index <worktree-path>) is recommended over symlink or per-worktree gitnexus analyze. A second, separate bug was found: the registry currently holds TWO knowledge-hub entries, which makes bare gitnexus query/MCP calls FAIL until disambiguated or de-duplicated.

---

2. Thread 1 — events.jsonl unreadability regression (ROOT CAUSE)

2.1 Symptom (from the brief)

Orchestrator shell cannot read .claude/cmux-events/<sid>/events.jsonl (ls → no-such-file; git -C → cannot-change) even with dangerouslyDisableSandbox, though git worktree list shows the worktree registered. converse.sh, wait-for-fleet.sh, and stop-worker.sh all poll that path.

2.2 Root cause — CWD-relative events-base resolution

Every monitoring script computes its events base identically:

PROJECT_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd -P)"
EVENTS_BASE="${KH_CMUX_EVENTS_DIR:-${PROJECT_ROOT}/.claude/cmux-events}"

(wait-for-fleet.sh:76-77, converse.sh:29-30, stop-worker.sh:93-94, send-prompt.sh:30-31.)

git rev-parse --show-toplevel returns the top of the working tree containing the current directory — NOT the canonical main checkout. Empirically verified on the live tree:

from main root: /Users/liamj/Documents/development/knowledge-hub
from worktree : /Users/liamj/Documents/development/knowledge-hub/.claude/worktrees/agent-a61d7700f6aa6af15

The workers’ events are written by launch-worker.sh to the main repo’s .claude/cmux-events/<sid>/ (launch-worker.sh:107 resolves EVENTS_BASE from the passed <base-dir> argument — typically . at launch time from the main root). But the monitoring scripts resolve EVENTS_BASE from the orchestrator’s runtime CWD at poll time. The instant the orchestrator shell’s CWD is inside a worktree, the two diverge:

• launch-worker.sh wrote to <main>/.claude/cmux-events/<sid>/events.jsonl.
• wait-for-fleet.sh / converse.sh / stop-worker.sh look in <worktree>/.claude/cmux-events/<sid>/events.jsonl — which does not exist.

Verified the worktree has no cmux-events subtree at all:

ls: <worktree>/.claude/cmux-events/: No such file or directory   # symptom = "no such file"

The “git -C → cannot-change” symptom is the same root cause from the other direction: when the events path is computed against a removed/renamed worktree root, the derived directory does not exist, so git -C <that-path> errors with “cannot change to the directory”. git worktree list still shows the worktree because the registration entry in .git/worktrees/ outlives a CWD/path mismatch.

2.3 Why “even with sandbox bypass” did not help

dangerouslyDisableSandbox removes the filesystem permission gate, but the path being listed genuinely does not exist — there is nothing for the sandbox to permit. The symptom is path-resolution, not permission, which is exactly why bypassing the sandbox changed nothing. (Confirmed separately that the sandbox allowlist in .claude/settings.local.json does not deny .claude/cmux-events/; its filesystem block is empty and autoAllowBashIfSandboxed: true. The events path was directly readable from the main root in this session: 666 lines of 879230c1-…/events.jsonl.)

2.4 Why it is a regression (“worked well previously”)

Two contributing drifts, both compatible with “worked before”:

• CWD drift via worktree adoption. Earlier sessions ran the orchestrator from the main root throughout, so --show-toplevel always returned the main root and the bug was latent. As the orchestrator-of-orchestrators pattern matured (S262→S279), the orchestrator increasingly operates with its CWD inside a worktree (or hops between them via git -C and cd), surfacing the latent path-resolution defect.
• The cd-guard PreToolUse hook (.claude/settings.json). A hook now BLOCKS cd /Users/liamj/Documents/development/knowledge-hub* (“causes commits to leak to the wrong branch; drop the cd prefix; use relative paths from your CWD”). This was added to protect commit-branch integrity, but it has a side effect: the orchestrator can no longer trivially cd back to the canonical main root to fix its CWD before polling, and is steered toward relative-path usage from whatever worktree CWD it currently holds — which is precisely the condition that breaks --show-toplevel-derived EVENTS_BASE. This hook is the most plausible “version change between the sessions where it worked and S279” trigger. (Note: this is a behavioural interaction, not a cmux binary version bump; no cmux-version evidence was found in the script git history — the last functional change to the family was 25a5fa1b S61-carryover, with FX-1/FX-3 resilience polish after.)

2.5 Recommended fix (thread 1)

Anchor EVENTS_BASE to the COMMON git dir, not the per-worktree toplevel. Replace the resolution preamble in all four scripts with a helper that resolves the canonical main root deterministically regardless of CWD:

# Resolve the MAIN working-tree root even when CWD is inside a linked worktree.
# --git-common-dir points at <main>/.git for every worktree; its parent is the
# canonical main root. Falls back to --show-toplevel then pwd.
resolve_project_root() {
  local common_dir
  common_dir="$(git rev-parse --git-common-dir 2>/dev/null)" || { pwd -P; return; }
  case "$common_dir" in
    /*) ;;                                   # absolute
    *) common_dir="$(pwd -P)/$common_dir" ;; # relative → absolutise
  esac
  ( cd "$(dirname "$common_dir")" && pwd -P )
}
PROJECT_ROOT="$(resolve_project_root)"
EVENTS_BASE="${KH_CMUX_EVENTS_DIR:-${PROJECT_ROOT}/.claude/cmux-events}"

git rev-parse --git-common-dir returns <main>/.git from inside ANY linked worktree (linked worktrees share the common dir), so dirname of it is always the main root. KH_CMUX_EVENTS_DIR remains the explicit override and continues to win — the most robust mitigation in the interim is for the orchestrator to always export KH_CMUX_EVENTS_DIR=<absolute-main>/.claude/cmux-events before any monitoring call, which the SKILL.md should document as the canonical pattern. (The <info> already shows read-turn.sh examples setting the env var; extend that discipline to all four scripts.)

This change touches wait-for-fleet.sh, converse.sh, stop-worker.sh, send-prompt.sh, and launch-worker.sh (for symmetry), and the new co-located smart-watcher (thread 2/3). Behaviour for the common case (orchestrator at main root) is unchanged.

---

3. Thread 2 — fleet-watch.sh vs wait-for-fleet.sh co-location ruling

3.1 They are different files with different lineage

	wait-for-fleet.sh	fleet-watch.sh
Location	.claude/skills/session-driver-cmux/scripts/	.claude/cmux-events/ (a runtime/gitignored dir)
Size / mode / date	4490 B, exec, 21 May	4711 B, non-exec, 26 May
Tracked?	Yes (skill family)	No — .claude/cmux-events/ is gitignored (.gitignore:39)
Lineage	Canonical skill primitive	v2 of S262/S263 kh-fleet-watch.sh (tracked ancestor at docs/continuation-prompts/s262-worker-reports/kh-fleet-watch.sh, commit 2ba53e70)
Contract	Blocks until any-of/all-of a session set emits stop; one signal	Smart multi-signal watcher: session_end, final_report.* in events dir, OQ-heading growth, AskUserQuestion stall (2-poll stable), stop pause (2-poll stable), fleet-quiet (no event growth) — trips and reports ALL actionable items
Inputs	session-id args	scans $EVENTS_BASE/*/meta.json; env-driven (IGNORE, SEEN_OQ, SEEN_FINAL, SEEN_SEND, INTERVAL, MAX_POLLS, QUIET_POLLS)
Path hygiene	resolves EVENTS_BASE via --show-toplevel (thread-1 affected)	hard-codes EVENTS_BASE=".claude/cmux-events" (relative) — even more CWD-fragile

Confirmed via diff: fleet-watch.sh is NOT a copy of wait-for-fleet.sh; it is a substantially rewritten v2 of the S262 monitoring script (the tracked ancestor is the v3/v4 kh-fleet-watch.sh periodic-sweep loop; fleet-watch.sh is the smarter poll-and-report rewrite). It was placed in .claude/cmux-events/ (a gitignored runtime dir) as a session scratch artefact (S274; cf. the co-located s274-fleet-manifest.json listing the five sub-orchestrators it watched), so it is currently UNTRACKED and would be lost on any .claude/cmux-events/ cleanup.

3.2 Ruling: CO-LOCATE (promote), do not leave it as scratch

fleet-watch.sh should be moved into .claude/skills/session-driver-cmux/scripts/, made executable, and tracked, because:

1. It is strictly richer than wait-for-fleet.sh and is the de facto durable-monitoring answer the orchestrator already reached for in S274 (thread 3 overlaps here).
2. Living in a gitignored runtime dir means it is invisible to the skill family, not version-controlled, and at risk of deletion by stop-worker.sh’s rm -rf $EVENTS_DIR or any events-dir sweep.
3. Co-location lets it inherit the thread-1 resolve_project_root fix (it currently hard-codes a relative EVENTS_BASE, which is even more fragile than the --show-toplevel form).

Recommended naming: keep wait-for-fleet.sh (simple block-on-stop primitive, still useful for race/first-to-finish) and add the smart watcher as watch-fleet.sh (or fleet-watch.sh retained) alongside it. The two are complementary: wait-for-fleet.sh answers “block until done”; watch-fleet.sh answers “wake me on the next actionable event across the fleet”. Both must be listed in SKILL.md’s script-summary table.

---

4. Thread 3 — durable monitoring fix recommendation

4.1 The model that already works: smart multi-signal poll (promote fleet-watch.sh)

The S279 manual workaround (cmux read-screen --workspace <ws> + git worktree list) was reached for because the events-path poll was broken (thread 1), not because polling events is wrong. With thread 1 fixed, fleet-watch.sh’s smart-watcher IS the durable fix: it watches the same events.jsonl stream but exits the parent’s wait on any of six actionable signals rather than only stop. This directly addresses the SKILL.md’s own documented S62E anti-pattern (polling session_end alone misses every mid-session stop).

Recommendation (primary): adopt watch-fleet.sh (promoted fleet-watch.sh) as the canonical orchestrator monitoring primitive, with the thread-1 path hardening applied. Document the canonical loop in SKILL.md: launch → send-prompt → watch-fleet.sh (re-arm on exit-2 timeout, act on exit-0 report) → stop-worker.

4.2 Event-driven completion signal (optional later optimisation)

A true event-driven signal (worker emits a sentinel the parent blocks on, e.g. via fswatch/inotifywait on the events dir, or the ID-43 decision-file channel) removes poll latency entirely. This is the cleaner long-term shape, but:

• It overlaps the ID-43 OQ-escalation channel (docs/specs/id-43-oq-escalation/), which already specifies a durable, file-based worker↔parent signalling channel with blocking/non-blocking semantics and a decision-polling cycle (PRODUCT §goals; the exact cadence is a TECH concern there). ID-27 must cross-reference, not absorb, ID-43: the OQ channel is for questions requiring a decision; the monitoring primitive is for lifecycle completion/attention signals. They share the events-dir transport but have distinct payloads.
• fswatch/inotifywait add a dependency (fswatch is not guaranteed on PATH; macOS default lacks inotifywait). The smart-poll model has zero new dependencies (jq + POSIX shell, already required).

Recommendation (secondary): keep the smart-poll watcher as the baseline; treat an fswatch-backed fast path as an opt-in enhancement gated on the binary being present, and align its sentinel-file convention with ID-43’s decision-file shape so the two channels do not diverge. Surfaced as OQ-3.

---

5. Thread 4 — GitNexus worktree index-inheritance (folded in)

5.1 The index is path-keyed in a GLOBAL REGISTRY, not just .gitnexus/lbug

The brief framed this as “the 272 MB .gitnexus/lbug index is gitignored/local, so every worktree reads stale (never)”. True, but the deeper mechanism is the global registry. gitnexus list shows the index registered by absolute repo path:

Indexed Repositories (2)
  knowledge-hub  (/Users/liamj/.gitnexus/repos/knowledge-hub)        Indexed 12/05/2026
  knowledge-hub  (/Users/liamj/Documents/development/knowledge-hub)  Indexed 29/05/2026

gitnexus status / gitnexus query resolve the active index by the current repo’s absolute path. A worktree at <main>/.claude/worktrees/<name> matches neither registered entry, so every gitnexus call inside a worktree reports “stale (never)” — even though the parent has a fresh 47k-symbol index. The gitignore (.gitignore:118-119 .gitnexus/** + !.gitnexus/CLAUDE.md) only governs whether the index files are committed; it is orthogonal to registry path-keying. Confirmed: the live worktree .gitnexus/ contains only CLAUDE.md (4471 B) — no lbug, no meta.json — so the worktree has no local index and is unregistered.

5.2 backlog-190 covers only HALF the problem

backlog-190 (from ID-23.13, S277 finding F3) tracks the .git/info/exclude piece: the per-worktree exclude carries a bare .gitnexus/ line that blocks git add .gitnexus/CLAUDE.md without -f. Confirmed on the live tree — the (shared, common-dir) .git/info/exclude contains:

7  .gitnexus/**
8  !.gitnexus/CLAUDE.md
9  .gitnexus/          ← bare line UNDOES the negation on line 8 (backlog-190's bug)

Because linked worktrees share $GIT_COMMON_DIR/info/exclude, this single bare line (.gitnexus/) re-ignores .gitnexus/CLAUDE.md for the main repo AND every worktree. The fix is to delete line 9 (and mirror the /** + negation form already present on 7-8). That is the tracking half — it makes CLAUDE.md committable. It does not seed the index into worktrees (the unsolved half this Subtask folds in).

5.3 Three seeding options evaluated

Option	How	Cost	Verdict
A. Global-registry share (gitnexus index <worktree-path>)	gitnexus index “registers an existing .gitnexus/ folder into the global registry (no re-analysis needed)”. After worktree creation, register the worktree path pointing at the parent’s index.	Cheap (registry entry only; no 272 MB copy, no re-analysis)	RECOMMENDED — matches the tool’s intended sharing mechanism; no disk blow-up
B. Symlink worktree .gitnexus/ → parent	ln -s <main>/.gitnexus <worktree>/.gitnexus at worktree creation	Cheap; but the worktree’s own .gitnexus/CLAUDE.md (the @imported directive) collides with the symlinked parent dir; and registry still keys by path	Fallback only; messy interaction with the tracked CLAUDE.md
C. Per-worktree gitnexus analyze	Run full analysis in each worktree at creation	Expensive (272 MB + minutes per worktree; worker churn is high)	REJECTED — prohibitive

Caveat for Option A — registry path semantics need confirming. gitnexus index’s own help says it registers “an existing .gitnexus/ folder”. It is not yet confirmed that it can register a worktree path that points at the parent’s .gitnexus/ (vs requiring a real .gitnexus/ at the worktree path). If it requires a local .gitnexus/, combine A with a symlink: ln -s <main>/.gitnexus/lbug (and meta.json) into the worktree’s .gitnexus/ (preserving the worktree’s own tracked CLAUDE.md), then gitnexus index <worktree-path>. This needs a one-shot empirical check before ratification (OQ-4).

5.4 SECOND, SEPARATE BUG: duplicate-registry collision breaks bare gitnexus calls

While probing, gitnexus query (and by extension the MCP gitnexus_* tools) FAILS with:

Error: Multiple repositories indexed. Specify which one with the "repo" parameter.
Available: knowledge-hub (/Users/liamj/.gitnexus/repos/knowledge-hub),
           knowledge-hub (/Users/liamj/Documents/development/knowledge-hub)

The registry holds TWO knowledge-hub entries (one at /Users/liamj/.gitnexus/repos/… indexed 12/05, one at the dev path indexed 29/05). Any code-intelligence call that does not pass --repo errors out. This is independent of the worktree-seeding problem and is arguably the more urgent gitnexus issue — it degrades code-intelligence for ALL agents, not just worktree workers. Fix: gitnexus remove "/Users/liamj/.gitnexus/repos/knowledge-hub" (the stale 12/05 entry; remove is idempotent and does not require being inside the repo). Surfaced as OQ-5.

5.5 Seeding recommendation (thread 4)

1. In worktree-creation tooling (launch-worker.sh, after git worktree add): delete the bare .gitnexus/ line from .git/info/exclude if present (closes backlog-190’s tracking half for the shared exclude), and run Option A (gitnexus index <worktree-path>, with the symlink-of-lbug fallback per §5.3 if the empirical check requires it).
2. De-duplicate the registry (§5.4) — one-shot gitnexus remove of the stale entry.
3. Keep this as a launch-worker.sh concern (per the brief: it is worktree-creation tooling), not a new global directive.

---

6. Recommendations for {27.6} PRODUCT / {27.7} TECH (if decomposed)

• Invariant set should cover: (a) events-base resolution is CWD-independent; (b) watch-fleet.sh is tracked, executable, in the scripts family, and listed in SKILL.md; (c) worktrees inherit a usable gitnexus index (status ≠ “stale (never)”); (d) .git/info/exclude no longer re-ignores .gitnexus/CLAUDE.md; (e) the registry has exactly one knowledge-hub entry.
• Cross-ref (do NOT absorb) ID-43 for the worker↔parent decision-file transport; the monitoring primitive shares the events-dir but carries lifecycle/attention payloads, not decisions.
• Decomposition signal: this is ~3-5 discrete shell/config edits (path-helper across 5 scripts; promote+track watcher; launch-worker seeding + exclude fix; registry de-dup). Likely > 2h and chain-dependent (the path helper underpins the watcher), so {27.4}-style PLAN decomposition is warranted. Sibling-only deps hold (all within ID-27).

---

7. OQs for Liam

1. OQ-1 (thread 1 fix surface). Fix events-base resolution via the --git-common-dir helper baked into all four scripts (default), AND/OR mandate the orchestrator always export KH_CMUX_EVENTS_DIR=<abs-main>/.claude/cmux-events before monitoring calls? Recommended default: both (helper as belt; env var as braces, documented in SKILL.md). Confirm you want the helper added to all five scripts (launch-worker.sh included for symmetry).

2. OQ-2 (cd-guard hook interaction). The .claude/settings.json PreToolUse cd-guard (“commits leak to the wrong branch”) is the most plausible regression trigger (§2.4) because it steers the orchestrator into worktree-relative CWD usage. Keep the guard as-is and fix the scripts around it (recommended), or relax the guard to permit cd <main-root> for read-only monitoring? Recommended default: keep the guard, fix the scripts (the guard protects commit integrity; the script fix is the right layer).

3. OQ-3 (durable-fix ambition). Ship the smart-poll watch-fleet.sh as the durable fix now (recommended), and defer the fswatch/inotifywait event-driven fast path to a later optional enhancement aligned with ID-43’s decision-file convention? Or invest in the event-driven path up front? Recommended default: smart-poll now, event-driven later (opt-in, dependency-gated).

4. OQ-4 (gitnexus seeding mechanism — needs empirical confirm). Can gitnexus index <worktree-path> register a worktree whose .gitnexus/ only contains the tracked CLAUDE.md (no local lbug), pointing at the parent’s index? If not, the fallback is symlinking the parent’s lbug/meta.json into the worktree .gitnexus/ then registering. Approve a one-shot empirical check at {27.7} TECH time to choose between “register-only” (Option A) and “symlink+register” (A+B hybrid)? Recommended default: yes, run the check; prefer register-only if it works.

5. OQ-5 (duplicate gitnexus registry — separate, arguably urgent). A stale duplicate knowledge-hub entry (/Users/liamj/.gitnexus/repos/knowledge-hub, indexed 12/05) currently makes bare gitnexus query/MCP calls FAIL for ALL agents until --repo is passed. Authorise gitnexus remove "/Users/liamj/.gitnexus/repos/knowledge-hub" as an immediate fix (independent of the worktree-seeding work)? Recommended default: yes, remove the stale entry now.

6. OQ-6 (backlog-190 fold-in vs handoff). backlog-190’s .git/info/exclude tracking fix is half of thread 4; the seeding is the other half. Fold backlog-190’s exclude-fix into ID-27’s worktree-creation changes (recommended — same file, launch-worker.sh), or keep backlog-190 as a separate low-priority item and have ID-27 only seed the index? Recommended default: fold in (it is one coherent worktree-creation change).