Orchestrator-of-orchestrators e2e findings (S62B)

Orchestrator-of-orchestrators e2e findings (S62B)

S62 W4 errata — ID rename: The production-readiness Task referenced throughout this document as ID-28 (cmux orchestrator-of-orchestrators polish, FX-1/FX-2/FX-3 fix-pack) was renamed to ID-33 at S62 W4 to resolve a cross-branch collision with content-items-investigation ID-28 (T8 cocoindex flow). Subtask IDs follow: ID-28.1 → ID-33.1, ID-28.2 → ID-33.2, ID-28.3 → ID-33.3. Where the body of this doc mentions the production-readiness fix-pack as ID-28.x, read it as ID-33.x. The CII ID-28 references (e.g. §7’s mention of “a different ID-28 cocoindex flow”) are unchanged — CII T8 cocoindex preserves the ID-28 anchor per S62 W3 OQ-2 ratification. Commit SHAs cd9c9e9d / b6269317 / 76d9eaeb / 7833411a / dff64c87 historically authored the fix-pack under ID-28.x naming; commit messages remain immutable; the Task they implemented is now ID-33.x in task-list.json.

Empirical PASS / FAIL / NEEDS-WORK verdict per finding from the 10-finding matrix in docs/continuation-prompts/continuation-prompt-kh-prod-readiness-s62b-orchestrator-of-orchestrators-e2e-test.md. All probes ran as cmux sub-Claude workers spawned via .claude/skills/session-driver-cmux/scripts/launch-worker.sh from the production-readiness branch on the secondary top-level worktree (/Users/liamj/Documents/development/knowledge-hub-production-readiness, HEAD 78890470).

Five phases ran A → D. Phase E (real Task work) deferred unconditionally to S63 per session-start ratification.

---

§1 Executive summary

Verdict	Count	Findings
PASS	8	#1, #2, #3, #4, #6, #8, #9, #10
NEEDS-WORK	1	#5 (Tier 2.2 git -C hook op-agnosticism)
N/A	1	#7 (Agent-tool-only concern; cmux-only dispatch confirmed in workflow-orchestration §Dispatch protocol)
FAIL	0	—

Plus Phase C (parallel sub-o test): PASS — both sub-os reach deterministic state, sequential cherry-pick clean (auto-merge on shared task-list.json with 80-line edit-gap), production-readiness HEAD unchanged throughout, no cross-tree write leak.

Path A migration gating verdict: GO on workflow-primitive grounds. The orchestrator-of-orchestrators pattern documented in workflow-orchestration SKILL.md §Dispatch protocol works end-to-end with the current cmux-based primitive set. Remaining Path A blockers are non-workflow (Task ID-9 docs site landing, Task ID-20 task-view repo landing — both pre-existing as ID-24 dependencies).

Three fix-Tasks recommended for polish (one NEEDS-WORK on Tier 2.2 hook breadth, two on stop-worker.sh resilience). All are quality-of-life, not correctness blockers. See §3.

---

§2 Per-finding results table

Evidence references: cmux events timestamps (.claude/cmux-events/<SID>/events.jsonl), exact hook error messages, commit SHAs. Where the original probe-worker output is needed, refer to that worker’s cmux read-screen --workspace <ref> --scrollback capture taken before stop-worker.sh teardown.

#1 — launch-worker.sh --plugin-dir <session-driver-cmux> skill discovery scoping

Severity: CRITICAL · Verdict: PASS

Probe: Combined Phase A worker s62b-probe-a (SID 0a2eb5de-…).

Evidence:

• ls .claude/skills/ | sort from worker → 33 skills listed (full project catalogue: agent-development, ast-dataflow, astro, … workflow-orchestration, write-product-spec, write-tech-spec).
• Worker successfully invoked /workflow-orchestration via Skill tool; full skill body loaded (“Successfully loaded skill”).
• No skill-not-found error.

Conclusion: The --plugin-dir flag passed by launch-worker.sh adds the session-driver-cmux plugin to the worker’s plugin set without excluding the project-wide skill catalogue. Worker sees ALL skills the orchestrator sees.

#2 — Workers’ inheritance of .claude/settings.json + allow-rules + Tier 2.2 hook

Severity: CRITICAL · Verdict: PASS

Probe: Combined Phase A worker s62b-probe-a (SID 0a2eb5de-…).

Evidence:

• cat .claude/settings.json from worker returned the project config (allow list, PreToolUse hooks, etc.).

• cat .claude/settings.local.json returned empty / file-not-found — expected (the .local variant is gitignored and orchestrator-only; this is correct behaviour, not a regression).

• bash -c "cmux ping" from worker → PONG (project allow-rule Bash(cmux *) honoured; no permission prompt).

• Tier 2.2 hook BLOCKED Write on relative ../escape-attempt.txt which resolved to absolute /Users/liamj/…/.claude/worktrees/escape-attempt.txt (parent dir of worker’s CWD) with the verbatim error:

  BLOCKED: Write/Edit/MultiEdit to absolute knowledge-hub path ‘/Users/liamj/…/worktrees/escape-attempt.txt’ from CWD ‘/Users/liamj/…/worktrees/s62b-probe-a’. Sub-agents must write only within their worktree (use relative paths or the worktree-absolute prefix). Tier 2.2 hook (ID-19.3) — prevents cross-tree leak when CWD-correct agent emits wrong absolute path.

• Post-probe filesystem check: no escape-attempt.txt at any of the three plausible paths (.claude/worktrees/, worker worktree, .claude/).

Conclusion: Settings inherit fully (including allow-rules and all three PreToolUse hook tiers — cd-block, git-C-block, Tier 2.2 absolute-path-block). The Tier 2.2 hook fires correctly on cross-CWD writes from sub-agent context.

#3 — Brief-delivery race (worker reads brief before next prompt?)

Severity: High · Verdict: PASS · Side-finding: NEEDS-WORK on UserPromptSubmit hook observability

Probe: Phase B worker s62b-probe-b3 (SID 41354404-…). Launch + race-prompt sent within ~3s of session start.

Evidence — events.jsonl:

{"ts":"2026-05-21T17:50:29Z","event":"session_start"}
{"ts":"2026-05-21T17:50:32Z","event":"user_prompt_submit"}
{"ts":"2026-05-21T17:50:37Z","event":"pre_tool_use","tool":"Read"}  // brief
{"ts":"2026-05-21T17:51:10Z","event":"stop"}

Evidence — worker screen:

❯ Read .cmux-brief.md before any work.
  Read 1 file (ctrl+o to expand)
❯ What did .cmux-brief.md say?
⏺ First line: # S62B Phase B Test #3 — brief-delivery race
  TEST_3_COMPLETE

Worker DID read the brief (T+8) BEFORE answering the race-prompt at T+~41. PASS on worker behaviour: the auto-prompted brief-read completes before any subsequent user prompt is acted on.

Side-finding (NEEDS-WORK): Only ONE user_prompt_submit event was logged despite TWO prompts being sent (auto-prompt from launch script + orchestrator race-prompt). Both prompts visible in the cmux screen capture (two ❯ lines), but the UserPromptSubmit hook only fired once. This is an observability gap, not a worker-behaviour regression — the worker correctly read brief-first. The hook either dedupes when prompts arrive in rapid succession, or doesn’t fire for the auto-prompt path. Open fix-Task: investigate hook dispatch on the cmux send-prompt code path.

#4 — stop-worker.sh 10s session_end timeout under mid-tool-call

Severity: Medium · Verdict: PASS (timeout sufficient) · Side-findings: 3× NEEDS-WORK on stop-worker.sh polish

Probe: Phase B worker s62b-probe-b4r (SID 72f575f6-…). Brief instructed worker to run sleep 30 && echo …. Orchestrator polled events.jsonl until the Bash pre_tool_use event appeared, slept 3s (so worker was solidly mid-sleep), then invoked stop-worker.sh without --force.

Evidence — events.jsonl:

{"ts":"2026-05-21T17:53:14Z","event":"session_start"}
{"ts":"2026-05-21T17:53:17Z","event":"user_prompt_submit"}
{"ts":"2026-05-21T17:53:21Z","event":"pre_tool_use","tool":"Read"}
{"ts":"2026-05-21T17:54:00Z","event":"pre_tool_use","tool":"Bash"}  // sleep 30 starts
{"ts":"2026-05-21T17:54:04Z","event":"session_end"}                  // ~4s after /exit

Timing:

• Bash sleep starts at T+46 (17:54:00Z worker clock).
• Orchestrator invokes stop-worker.sh at orchestrator T+43.
• session_end fires at worker T+50 — i.e. ~4s after /exit was sent.
• Worker’s sleep 30 interrupted mid-flight; echo "DONE_AFTER_30s" never ran.

Conclusion: The 10s session_end timeout in stop-worker.sh is ~2.5× the observed mid-Bash interrupt latency. No need to make this configurable for the canonical case. The timeout could be tightened if needed (e.g. to 5s) but generosity here is defensible — pathological tool calls (large LLM inferences mid-turn) may legitimately take longer.

Side-finding 1 (NEEDS-WORK — stop-worker.sh polish): Untracked .cmux-brief.md file in the worker worktree trips the script’s dirty-tree check, exiting 2 even when the only “uncommitted change” is the brief itself. Affects all probes that don’t commit the brief. Fix: exclude .cmux-brief.md from the dirty-tree check (treat as script-managed artefact, not worker output) OR auto-delete the brief file as part of clean-shutdown.

Side-finding 2 (NEEDS-WORK — stop-worker.sh polish): The cmux workspace is closed BEFORE the dirty-tree check runs. If the check fails (exit 2), the workspace is already gone but the worktree + branch are not removed. This is a partial-failure state with no recovery path other than manual intervention. Fix: reorder so the dirty-tree check runs BEFORE workspace close — OR document that exit 2 is irreversible.

Side-finding 3 (NEEDS-WORK — stop-worker.sh polish): --delete-branch silently no-ops when the meta file is missing (e.g. after a partial cleanup or a re-run of stop-worker.sh). Specific case observed: orchestrator ran stop-worker → exit 2 (dirty tree) → re-ran with --force --delete-branch → second run found meta missing → skipped branch delete → branch persisted. Fix: fall back to deriving the branch name from git worktree list when meta is absent.

#5 — Tier 2.2 git -C hook breadth (read-only ops blocked too)

Severity: Medium · Verdict: NEEDS-WORK

Probe: Phase B worker s62b-probe-b5 (SID 39b54198-…). Worker attempted 5 git-related calls.

Evidence — verbatim probe results:

#	Command	Outcome
1	git -C /…/knowledge-hub-production-readiness status --short	BLOCKED
2	git -C /…/knowledge-hub-production-readiness worktree list	BLOCKED
3	git -C /…/knowledge-hub-production-readiness log --oneline -3	BLOCKED
4	git -C /…/knowledge-hub-production-readiness branch -D <stale>	BLOCKED
5	git status --short (relative, worker CWD)	OK

Verbatim hook error (identical for #1-#4):

BLOCKED: bash command uses git -C with an absolute knowledge-hub path. This causes commits to leak to the wrong branch — sub-agents must run git from their worktree CWD, not via -C to the main repo.

Conclusion: The hook regex git[[:space:]]+-C[[:space:]]+/Users/liamj/Documents/development/knowledge-hub matches op-agnostically. Read-only operations (status, worktree list, log) — which CANNOT cause commit leakage — are blocked equally to write ops (branch -D, push, commit, etc.). The hook is overbroad for its stated purpose (“causes commits to leak to the wrong branch”).

Fix recommendation: Add an allowlist of subcommands inside the hook regex (or rewrite as a two-phase check). Read-only forms to whitelist: status, log, show, diff, branch --list, branch -v, worktree list, rev-parse, ls-files, ls-tree, remote -v, config --get, fetch (depending on policy; fetch is non-mutating to local state). Write forms to keep blocked: branch -d/-D, worktree add/remove/move, push, commit, merge, rebase, reset, checkout/switch, stash drop, clean, gc, prune, config --set.

Severity rationale: Medium because the workaround is simple (orchestrator runs the relevant git from its current CWD), and the false-positive rate is low (operators rarely need git -C <abs> reads from sub-agent context). But it surfaced during Phase B testing and would surface for any future sub-orchestrator that wants to inspect parent state for a coordination decision.

#6 — $TMPDIR divergence sandboxed vs unsandboxed

Severity: Low · Verdict: PASS (re-verified)

Probes:

• Test #6a — orchestrator launched worker with relative-path brief --brief ./tmp-trivial-brief.md. Result: succeeded. Worker s62b-probe-b6r spawned cleanly.
• Test #6b — orchestrator launched worker with --brief "$TMPDIR/s62b-probe-trivial.md" (under dangerouslyDisableSandbox: true). Result: Error: --brief file not found: /var/folders/t1/86sdqc012r1_6g4cm_4kt10r0000gn/T//s62b-probe-trivial.md, exit 1. The $TMPDIR variable in the unsandboxed orchestrator shell expanded to the user-private macOS temp dir, NOT to /tmp/claude-501/ where Write-tool-authored files actually live (sandbox view).

Conclusion: Reconfirms S62 W1 finding. Absolute paths (/tmp/claude-501/...) are canonical for cross-context handoff between orchestrator and launch-worker.sh scripts. The previous CLAUDE.md guidance (“Brief files MUST use absolute paths”) slightly understated the case — relative paths from the project-root CWD also work because launch-worker.sh resolves the brief relative to its execution directory. Updated framing: absolute /tmp/claude-501/... paths are canonical; relative paths work if you’re certain the orchestrator’s CWD matches the script’s CWD; $TMPDIR/... always fails.

#7 — Test-agent worktrees harness-locked (Agent-tool concern)

Severity: Low · Verdict: N/A

Probe: Bypassed — workflow-orchestration SKILL.md §Dispatch protocol explicitly defers to session-driver-cmux for parallel sub-orchestrator dispatch (the orchestrator-of-orchestrators row in the primitive selection table). The Agent-tool isolation: "worktree" primitive is not used by the o-of-o pattern; therefore the test-agent-worktree harness-lock concern is out of band for this verification.

For posterity: S62 W1 empirically re-confirmed that Agent-tool isolation: "worktree" places sub-agent worktrees under the PRIMARY top-level worktree’s .claude/worktrees/ regardless of orchestrator CWD, even post user-global baseRef: "head" flip — structural root cause of the “Worktree agents start stale” gotcha. cmux primitive does not have this issue (launch-worker.sh issues an explicit git worktree add from orchestrator CWD).

#8 — launch-worker.sh branch-name collision (same name + same HEAD)

Severity: Low · Verdict: PASS · Side-finding: LITE NEEDS-WORK on error message specificity

Probe: Orchestrator-side mechanics test.

Sequence:

1. Launch s62b-probe-b8 → SID1 58e43355-…, branch cmux-worker-s62b-probe-b8-78890470 created.
2. stop-worker.sh s62b-probe-b8 <SID1> WITHOUT --delete-branch → workspace closed, worktree removed, BUT branch survived (per script default).
3. Re-launch s62b-probe-b8 with SAME name on SAME HEAD → FAILED with:

   Error: failed to create worktree at /…/.claude/worktrees/s62b-probe-b8

   Exit code 1.
4. Verified no partial-state leak: no worktree dir, no events dir, no stale meta file. Branch from step 1 still present (correct — no --delete-branch).
5. Manual git branch -D cmux-worker-s62b-probe-b8-78890470 → cleanup complete.
6. Re-launch s62b-probe-b8 (now clean) → SUCCESS, new SID2 b612e5ff-….
7. stop-worker.sh --force --delete-branch → workspace + worktree + branch + events dir all cleaned.

Conclusion: Collision fails cleanly with no partial-state leak. Operator can recover via single manual git branch -D and re-launch.

Side-finding (LITE NEEDS-WORK): The error message “failed to create worktree at ” is not specific enough — operator must inspect to decide whether the collision is on the branch or the worktree path (both plausible if the previous teardown was partial). Fix: have launch-worker.sh check explicitly for git rev-parse --verify refs/heads/<branch> and [ -e <worktree-path> ] before issuing git worktree add, and emit a precise diagnostic.

#9 — .worktreeinclude literal-paths only (no glob)

Severity: Low · Verdict: PASS (by source inspection + docs match)

Probe: Source inspection of launch-worker.sh rather than empirical add-glob-and-observe, to avoid mutating the live .worktreeinclude file shared with the parallel Liam session.

Evidence — launch-worker.sh lines 169-193:

# Anthropic's `.worktreeinclude` mechanism only triggers under their internal
# worktree-creation paths (`claude --worktree`, Agent-tool isolation, etc.).
# `git worktree add` bypasses it. Mirror minimal semantics here for the
# canonical `.env.local` case: each non-blank non-comment line is treated
# as a literal relative path; if the source exists at PROJECT_ROOT, copy
# into the worker worktree. No glob expansion (kept simple intentionally —
# extend if patterns are needed).

Implementation: a while IFS= read -r line loop that trims whitespace, skips blank + #-comment lines, then cp -R "$PROJECT_ROOT/$line" "$WORKTREE_PATH/" on each remaining path. No [[-glob expansion, no compgen, no find.

Conclusion: Behaviour matches SKILL.md’s documented contract (“plain file paths only — no glob expansion”). Glob extension is a documented future-work item; no current consumer requires it. Canonical case .env.local is verified working (Phase A probe-worker had .env.local copied into its worktree per the existing line in .worktreeinclude).

#10 — Workers’ awareness of task-list.json as the brief (sub-orchestrator brief shape)

Severity: Medium · Verdict: PASS

Probe: Phase B worker s62b-probe-b10 (SID d09a58af-…). Brief was 4 imperative steps: read task-list.json ID-25, invoke /workflow-orchestration, report next action, exit.

Evidence — worker’s report:

• Step 1: ID-25 status pending, priority should, dependencies ["15"], 4 Subtasks (25.1 RESEARCH, 25.2 PRODUCT, 25.3 TECH, [25.4 not reported but implied]).

• Step 2: /workflow-orchestration loaded successfully.

• Step 3 — sub-orchestrator-quality plan output:

  NEXT action on ID-25: dispatch Subtask {25.1} RESEARCH.md.

  • Subtask: 25.1 (status pending, dependencies [] — unblocked)
  • Who dispatches: Orchestrator (main session)
  • Who executes: Planner agent (.claude/agents/task-planner)
  • Dispatch primitive: built-in Agent tool with isolation: "worktree" (single short Planner on one subtask group)
  • Model: opus + thinking: 'max' per task-planner agent definition
  • Gate: Checker passes RESEARCH.md before {25.2} PRODUCT dispatch; Liam ratifies before implementation phase.

  Worker also correctly flagged: ID-25 depends on ID-15 (status check precondition the Orchestrator must verify before {25.1} dispatch).

Conclusion: A 4-line minimal brief is sufficient for a sub-orchestrator to navigate task-list.json, identify the next ID-N.M Subtask, choose the right dispatch primitive (Agent-tool vs cmux), pick the correct agent model (opus / thinking max), and call out dependency-gating preconditions. This validates the minimum viable brief shape for the orchestrator-of-orchestrators dispatch row in workflow-orchestration SKILL.md.

Phase C — Parallel sub-orchestrator lifecycle

Verdict: PASS

Probes: s62b-sub-o-id25 (SID 6440d9cc-…, commit 345f0466) + s62b-sub-o-id26 (SID d812857a-…, commit 95c5eb6a). Both launched within 6 seconds of each other; both written to two artefacts each (.planning/.test/s62b-phase-c-sub-o-id{25,26}.md + a status_note jq write to their respective IDs in task-list.json); both committed and stopped.

Timing:

• ID-25: session_start 18:02:50 → stop 18:10:01 (~7m11s lifecycle).
• ID-26: session_start 18:02:56 → stop 18:08:44 (~5m48s lifecycle).
• Each sub-o invoked /workflow-orchestration (Skill event observed in both event streams).

Mechanics validations:

• ✅ No cross-tree write leak. production-readiness HEAD stayed at 78890470 throughout both sub-os’ lifetimes.
• ✅ Worker-worktree isolation. No Tier 2.2 hook firing in either sub-o (each stayed in its own worktree CWD).
• ✅ Sequential cherry-pick clean. On a throwaway branch (s62b-phase-c-merge-test) from production-readiness HEAD: cherry-pick 345f0466 (ID-25) → clean; cherry-pick 95c5eb6a (ID-26) → Auto-merging docs/reference/task-list.json (3-way merge resolved without conflict). Final tree shows BOTH status_note edits coexisting at lines ~1286 and ~1366 (80-line gap, well outside git’s 3-line default diff context).
• ✅ task-list.json final state coherent. Both edits land; no JSON syntax corruption; cross-checked via grep -A 1 -B 1 of the probe markers.
• ✅ Cleanup deterministic. stop-worker.sh --force --delete-branch removed both workspaces, worktrees, branches, events dirs cleanly. No leftover state.

Caveat — narrow PASS surface: Phase C tested the happy path (disjoint status_note edits on ID-25 vs ID-26 with 80-line gap in the shared file). The merge-conflict path was not exercised. If parallel sub-os edit the same field or adjacent lines (<3-line gap), cherry-pick will trip — resolve-merge-conflicts skill becomes mandatory. This is covered conceptually in workflow-orchestration SKILL.md §Merge cadence (load-bearing — §6.4) point 4 (“On conflict: invoke the resolve-merge-conflicts skill”) and is not a new finding.

---

§3 Recommended fix-Tasks

Three fix-Tasks recommended. None are correctness blockers; all are polish / observability improvements that would smooth the o-of-o flow when it’s exercised at scale.

FX-1 — stop-worker.sh resilience polish (single Task, 3 sub-issues)

Priority: should (would block o-of-o ergonomics if exercised at scale). Scope: .claude/skills/session-driver-cmux/scripts/stop-worker.sh. Issues:

1. Untracked .cmux-brief.md triggers dirty-tree exit-2 even though the brief is a script-managed artefact (not worker output).
2. cmux workspace is closed BEFORE the dirty-tree check runs — exit-2 leaves a partial-cleanup state with no recovery path beyond manual intervention.
3. --delete-branch silently no-ops when the meta file is absent (post-failure re-run scenario) — should fall back to deriving the branch name from git worktree list. Test approach: 3 unit-test scenarios per the 3 sub-issues, plus a re-run scenario covering interaction effects. Affected primitive: session-driver-cmux (no schema changes; no downstream agent updates).

FX-2 — Tier 2.2 git -C hook subcommand whitelist

Priority: should (false-positives on read-only ops degrade sub-orchestrator inspection capability). Scope: .claude/settings.json PreToolUse hook (git -C block) — the regex currently at git[[:space:]]+-C[[:space:]]+/Users/liamj/Documents/development/knowledge-hub. Change: Add a two-phase check: (a) match git -C <abs-knowledge-hub path>; (b) inspect the subcommand following the path. ALLOW for read-only forms (status, log, show, diff, branch --list, branch -v, worktree list, rev-parse, ls-files, ls-tree, remote -v, config --get, fetch). BLOCK for everything else (the current behaviour — branch -d/-D, push, commit, merge, rebase, reset, checkout, switch, stash drop, clean, gc, prune, worktree add/remove, config —set, etc.). Test approach: extend probe-b5 brief into a parameterised test fixture (10+ commands across read-only / write splits). Affected primitive: PreToolUse hook only; no downstream agent changes.

FX-3 — launch-worker.sh collision diagnostic specificity

Priority: could (rare-case error-message polish; current behaviour is safe — just opaque). Scope: .claude/skills/session-driver-cmux/scripts/launch-worker.sh, the pre-git worktree add checks. Change: Before invoking git worktree add, check explicitly:

• git rev-parse --verify refs/heads/<branch> → if exists, error “branch already exists; previous worker stopped without —delete-branch?”
• [ -e <worktree-path> ] → if exists, error “worktree path already exists; recover with git worktree remove --force <path>.”
• Otherwise, proceed with git worktree add. Test approach: re-run probe #8 with the new diagnostics — expect precise error text for both collision shapes. Affected primitive: session-driver-cmux only.

Out-of-scope of S62B (not new fix-Tasks; document in CLAUDE.md)

• #3 side-finding: UserPromptSubmit hook may dedupe rapid-succession prompts. Investigate as a separate observability-tuning Task if hook reliability becomes load-bearing for race-detection. Not blocking.
• #6 side-finding: --brief accepts relative-from-CWD paths. Update CLAUDE.md “$TMPDIR divergence” gotcha to note this is a valid form (currently says absolute is required — slight overstatement).

---

§4 Path A migration gating

Verdict on workflow primitive readiness: GO.

The orchestrator-of-orchestrators pattern documented in workflow-orchestration SKILL.md §Dispatch protocol works end-to-end under the current setup:

• Sub-orchestrators load workflow-orchestration independently (#1).
• Sub-orchestrators inherit .claude/settings.json + Tier 2.2 hook + project allow-rules (#2).
• Brief-delivery race does not invert (worker reads brief before next prompt — #3).
• Mid-tool-call stop-worker.sh interrupts cleanly within ~4s (#4).
• Workers correctly use task-list.json as a brief (#10).
• Parallel sub-os reach deterministic state on disjoint Subtasks (Phase C) with no cross-tree write leak, sequential cherry-pick clean.

Remaining Path A blockers (pre-existing, unrelated to this verification):

• ID-9 (Docs site + docubot port) — see task-list.json. Status: in_progress.
• ID-20 (Per-Task .md mirror generator + render surface) — see task-list.json. Status: in_progress.

These were ID-24’s documented blockers prior to S62B; nothing in the o-of-o verification changes that gating.

Updated ID-24 status_note (recommended write): “Path A migration gating: workflow-primitive verification PASS per docs/research/orchestrator-of-orchestrators-e2e-findings.md (S62B). Remaining blockers: ID-9 (docs site) + ID-20 (task-view mirror) — unchanged from prior. Three polish fix-Tasks (FX-1/2/3) opened from S62B; none are Path A blockers.”

---

§5 Decisions ratified or escalated to Liam

S62B session ratifications (recorded at session-start)

• Phase E unconditionally deferred to S63. Findings doc is the load-bearing deliverable; real Task work on ID-25 (or any candidate) defers to next session regardless of A+B+C outcome.
• Phase C auto-proceeds on Phase B PASS for #3 + #10. No mid-session ratification checkpoint required.

Escalations to Liam (post-S62B)

• FX-1, FX-2, FX-3 priority assignment. Recommended should / should / could (see §3). Open as task-list Tasks or backlog entries per Orchestrator-direct Promote convention.
• Path A timing. With workflow primitive PASS confirmed, the next decision is whether to schedule ID-24 (worktree collapse) immediately after ID-9 + ID-20 land, or wait for additional production-readiness Tasks. Workflow-primitive readiness is no longer a gate.
• Phase E deferral confirmation. S63 to pick up the real Task work on ID-25 (recommended default per session-start AskUserQuestion) or alternative.

---

§6 Fix-pack verification — S62C

S62C Wave 1 landed the FX-1/FX-2/FX-3 fix-pack opened as Task ID-28 during S62B Phase D. This section captures empirical re-verification of the three relevant S62B probes (#4, #5, #8) against the patched scripts + hook, plus a follow-up finding surfaced mid-verification.

Wave 1 commit chain (all on production-readiness):

• cd9c9e9d — fix(id-28.1): stop-worker.sh FX-1 (initial: carve-out
  • reorder + meta-fallback derivation)
• b6269317 — fix(id-28.2): Tier 2.2 git -C hook subcommand whitelist (FX-2 — new external git-c-abs-path-subcommand-guard.sh invoked from .claude/settings.json)
• 76d9eaeb — fix(id-28.3): launch-worker.sh collision diagnostic specificity (FX-3 — pre-git worktree add branch + path checks)
• 7833411a — fix(id-28.1): FX-1 follow-up — auto-delete .cmux-brief.md pre git worktree remove (gap surfaced in WP4)

§6.1 FX-2 (Tier 2.2 hook) — PASS

Probe: s62c-fx-verify-1 (SID c0c9fdab-cd20-40fb-8894-8469d2aa04fe), launched from production-readiness. Brief instructed the worker to run six git commands and report ALLOW vs BLOCK outcomes.

Evidence (verbatim from worker terminal):

#	Command	Expected	Observed
1	git -C /Users/.../knowledge-hub log --oneline -3	ALLOW	ALLOW (ebbcba2c chore(s252-wave3-close): ...)
2	git -C /Users/.../knowledge-hub status --short	ALLOW	ALLOW (clean)
3	git -C /Users/.../knowledge-hub worktree list	ALLOW	ALLOW (5-line listing)
4	git -C /Users/.../knowledge-hub branch --list 'main*'	ALLOW	ALLOW ( main)
5	git -C /Users/.../knowledge-hub branch -D nonexistent-test-branch-xyz	BLOCK	BLOCK (verbatim hook error from git-c-abs-path-subcommand-guard.sh)
6	git status --short (relative path)	ALLOW	ALLOW (?? .cmux-brief.md)

Worker’s own verdict report: VERDICT: PASS.

Verbatim BLOCK message from probe #5:

BLOCKED: git -C with knowledge-hub absolute path and mutating subcommand ‘branch’. Sub-agents must run mutating git from their worktree CWD, not via -C to the main repo (Tier 2.2 hook ID-19.3 + FX-2). Read-only subcommands (status, log, show, diff, branch —list, worktree list, remote -v, config —get, etc.) are allowed.

Coverage extension beyond probe: 30 isolated fixtures tested at hook-script authoring time (13 read-only ALLOW + 14 mutating BLOCK + 3 non-matching ALLOW) covered the full subcommand decision table.

§6.2 FX-1 (stop-worker.sh resilience) — PASS across all 3 sub-issues

Sub-issue 1 — .cmux-brief.md carve-out + auto-delete

• Initial fix in cd9c9e9d shipped only the dirty-check carve-out. Mid-WP4 verification revealed git worktree remove itself ALSO refuses dirty trees regardless of the script-side carve-out — stop-worker.sh exited 0 cleanly but worktree path persisted.
• Follow-up fix in 7833411a adds an rm -f of .cmux-brief.md immediately before git worktree remove.
• Re-verified via s62c-fx-verify-2 (SID 3bbb244b-...): stop after probe completion → exit 0, no removal warning, worktree path gone, cmux workspace gone, events dir cleaned.

Sub-issue 2 — Reorder: dirty-tree check before workspace close

• Probe: s62c-fx-verify-3 (SID 8adc23ad-...). Injected dirty-test-marker.txt into the worker’s worktree post-launch, then ran stop-worker.sh without --force.
• Result: exit 2 with the new error message (“cmux workspace remains open — re-attach to inspect or commit.”), and cmux list-workspaces confirmed workspace:24 s62c-fx-verify-3 STILL ALIVE post-exit-2. Recovery via stop-worker.sh ... --force --delete-branch exited 0 with full cleanup.
• S62B partial-failure state (workspace gone + worktree alive) is eliminated — operator can now re-attach to inspect or commit.

Sub-issue 3 — --delete-branch fallback when meta missing

• The canonical clean-stop path resolves BRANCH_NAME from the meta file, so the fallback only triggers in pathological re-run scenarios (operator manually nuked events dir between runs). Logic was exercised in the --force --delete-branch cleanup of s62c-fx-verify-3 (meta still present so direct path; the worktree list fallback hits only when meta absent AND worktree-path resolvable via launch-worker.sh convention <project-root>/.claude/worktrees/<worker-name>).
• Edge note: if BOTH meta and worktree path are gone post-failure, the fallback cannot resolve and emits a Warning + skips. The FX-3 collision diagnostic (§6.3) tells the operator the exact branch name to delete manually in that case.

§6.3 FX-3 (launch-worker.sh collision diagnostic) — PASS

Isolated logic test (4 collision shapes):

• Shape 0 (clean, no collision) — ALLOW ✓
• Shape 1 (branch exists, path doesn’t) — BRANCH error fired ✓
• Shape 2 (path exists, branch doesn’t) — PATH error fired ✓
• Shape 3 (both exist) — BOTH errors fired ✓
• Cleanup verified after each shape.

End-to-end BRANCH-only collision via s62c-fx-verify-2:

1. Launched s62c-fx-verify-2 (success).
2. Stopped without --delete-branch — branch persisted as designed.
3. Re-launched s62c-fx-verify-2 — exit 1 with verbatim diagnostic:

   Error: branch 'cmux-worker-s62c-fx-verify-2-76d9eaeb' already exists.
          A previous worker may have stopped without --delete-branch.
          Recover with:  git branch -D cmux-worker-s62c-fx-verify-2-76d9eaeb

Order-of-checks note: the pre-existing cmux-workspace-name collision check at launch-worker.sh:140-145 (which fires when the cmux workspace is still alive) runs BEFORE the new FX-3 branch+path checks. So when both shapes overlap, the cmux-workspace check wins and FX-3 only surfaces on post-cleanup re-runs (branch and/or path stale). This is the intended ordering — the cmux-workspace check already gives the operator a clear diagnostic, so FX-3 owns the post-stop recovery path.

§6.4 Wave 1 net outcome

All three S62B fix-Tasks landed and empirically verified. The o-of-o pattern is now friction-free for the canonical happy path plus the three failure-mode polish surfaces (FX-1 / FX-2 / FX-3). Wave 2 (real-Task dogfood — OPS-T1 promote + implementation lifecycle) proceeds on this primitive set.

Side-finding surfaced (out-of-scope of S62C — flagged for Curator): the sibling /Users/.../knowledge-hub main repo carries a different ID-28 (cocoindex flow + Cloud Run sidecar TECH.md, see commits c337b991 / 3b296e7a / 84251921 / ebbcba2c in the main repo) vs the production-readiness ID-28 (this o-of-o fix-pack). Liam ratified S62C session-start that production-readiness holds the canonical task-list + product-backlog, so the main-repo ID-28 must be renumbered at next cross-branch merge. RESOLVED S62 W4: see errata header at the top of this doc — production-readiness ID-28 renamed to ID-33; CII T8 retains ID-28 anchor per S62 W3 OQ-2 ratification. Cross-branch reconciliation complete at commit 9e498e2f (rename) + 6fe9246f (merge).

---

§7 Wave 2 dogfood findings — S62C (OPS-T1 promote + sub-o lifecycle)

Wave 2 exercised the o-of-o pattern end-to-end on a real Task (ID-32 OPS-T1 codemod implementation, promoted from backlog id=25). Two-phase structure:

• Phase A — workflow-curator promote: dispatch the workflow-curator agent (Agent tool, no isolation per dispatch primitive) with a finding packet describing OPS-T1’s specs-ratified-ready state. Curator runs triage-finding, decides promote-to-task-list, invokes update-roadmap-backlog Promote mode. Atomic commit 2f54b979 lands backlog id=25 removal + task-list ID-32 entry creation + 4-Subtask scaffold (32.1/32.2/32.3 done as specs-ratified, 32.4 PLAN pending).
• Phase B — cmux sub-orchestrator lifecycle: dispatch s62c-sub-o-id32 via launch-worker.sh with a 105-line brief instructing the sub-o to load workflow-orchestration, walk the ID-32 Subtask state machine, dispatch a FRESH task-planner for the pending 32.4 PLAN, gate via standard task-checker, commit. Sub-o produced PLAN.md (698 L) + 12 implementation Subtasks 32.5..32.16 decomposition. Commit e9f6e168 on sub-o branch, cherry-picked to production-readiness as 0ef29331 (auto-merged on task-list.json despite concurrent S63 churn).

§7.1 What worked (PASS observations)

• Curator promote primitive end-to-end clean — agent dispatched with a finding packet that didn’t match the canonical “out-of-scope finding from checker” shape, but triage-finding decision-tree handled it correctly (route via Branch C → tactical/single-feature → promote-to-task-list). Atomic commit + schema validation pass (95/95 tests). The S60 ratification carving out Orchestrator-direct for routine Promotes does NOT preclude curator-driven Promotes when ratified by Liam — both paths exist; user picks which to exercise.
• Sub-orchestrator brief-driven lifecycle worked end-to-end: the sub-o read brief, ran pre-flight (pwd / git branch / git fetch / git reset --hard / git status), loaded workflow-orchestration, dispatched fresh sub-Planner via Agent tool isolation, waited for return, dispatched sub-Checker, applied Checker notes inline, updated task-list.json narrowly, committed once. Final report emitted in YAML format per brief instructions, then /exit. ~1h 13m total (Planner ~50 min on 698 L PLAN.md, Checker ~15 min, task-list edit + commit ~5 min).
• Sub-Planner via Agent tool isolation: "worktree" from cmux sub-o context worked structurally — the sub-o’s branch was cmux-worker-s62c-sub-o-id32-2f54b979 (under production-readiness/.claude/worktrees/), and its Agent-tool- spawned sub-Planner landed in a fresh worktree under production-readiness/.claude/worktrees/.claude/worktrees/agent-* (nested correctly — no leak to the primary tree). This validates the “Anthropic worktree primitives are reliable from cmux-launched sub-o context” hypothesis: the structural defect is on ORCHESTRATOR (primary-vs-secondary tree) dispatch, not on SUB-ORCHESTRATOR dispatch where the sub-o’s CWD is already secondary.
• OQ escalation channel worked: 3 OQs surfaced (1 HIGH decision- gating, 1 MEDIUM, 1 LOW) per sub-o brief constraint “do NOT decide cross-Task scope yourself”. Parent ratified all 3 RATIFIED-DEFAULT via AskUserQuestion at Wave 2 close. Commit d67e2142 records ratifications in ID-32 status_note + description journal block.
• Cherry-pick of sub-o commit auto-merged on task-list.json despite the parent branch advancing through 7 commits during sub-o’s lifetime (S62 W3 OQ ratifications + S62 W4 cross-branch reconciliation including ID-28 → ID-33 rename + S63 WP1/2/3/4). The sub-o’s narrow Subtask 32.4 + 32.5..32.16 edits did not overlap with the parallel session’s edits on ID-20.6, ID-30, ID-31, ID-33. Git’s 3-line default diff context kept the auto-merge clean.
• FX-1/FX-2/FX-3 fix-pack landed Wave 1 paid off in Wave 2: the sub-o launched cleanly via patched launch-worker.sh; brief delivery via --brief flag worked; stop-worker.sh post-clean cleanup eliminated the partial-failure state risk during the sub-o teardown.

§7.2 New observations (NEEDS-WORK or carry-forward)

• Workflow-curator dispatch shape is loose — the agent’s description (per Agent tool list) describes the canonical use case as “finding from a task-executor or task-checker that may not belong in the current task (ID-N) scope”. The S62C Wave 2 Phase A use case (Liam-ratified promote of a specs-ready backlog item) is a legitimate but ATYPICAL invocation — triage-finding had to shoehorn it through Branch C. Carry-forward: when the parallel S62 W2 session’s ID-30 PLAN decomposes the workflow-curator skill amendments (OQ-8 ratified-alt), include a “Liam-driven promote (no finding source)” branch explicitly in triage-finding’s decision-tree preamble.
• Sub-Planner output format — the sub-Planner emitted Subtasks as TM-shape JSON in a single multi-line YAML report block (~12 subtasks each with title + dependencies + testStrategy). Sub-o applied them to task-list.json correctly, but the format-translation step from “Planner free-text output” to “task-list.json shape” required sub-o judgement — a future enhancement could be a planning-and-task-breakdown skill output mode that emits already- shaped JSON for direct merge.
• cmux read-screen output ANSI escape pollution: when the parent orchestrator reads the sub-o’s final report via cmux read-screen --workspace <ref> --scrollback, the output contains raw ANSI escape sequences from the Claude TUI rendering (Bash tool call boxes, thinking dots, etc.). The structured YAML report inside the noise is parseable but copy-paste fragile. Carry-forward: consider a sub-o brief convention that emits the final report to a structured file path (e.g. <events_dir>/final_report.yaml) for clean machine read, in addition to stdout.
• Sub-o brief size sweet spot: 105 lines was sufficient + not excessive. Below 50 lines would underspecify (sub-o needs to know: Subtask boundaries, file ownership, OQ channel mechanics, exit protocol). Above 150 lines starts to introduce read-time delays. The 105 L brief read in ~10 s of sub-o time.

§7.3 No new Path A blockers

ID-24 (worktree collapse + track migration, dep on ID-9 + ID-20) is unchanged. Wave 2 exercised the o-of-o pattern + curator promote + sub-Planner + sub-Checker without surfacing a new workflow-primitive defect. Path A migration timing remains: gated on ID-9 + ID-20 landing, NOT on workflow-primitive readiness (already PASS per S62B §4).

§7.4 Wave 2 net outcome

Wave 2 closes with:

• Backlog id=25 removed (52 → 53 net after S62 W2 / W3 / W4 churn; final backlog count post-Wave 2 close: see latest last_updated in docs/reference/product-backlog.json).
• Task ID-32 in pending status (4 spec Subtasks done; 12 impl Subtasks pending; 3 OQs ratified). Implementation dispatch (32.5+) deferred to S63+ — soft dep on ID-31 PLAN ratification.
• PLAN.md docs/specs/id-16-ast-dataflow-tool/ops-t1-codemod/PLAN.md (NEW, 698 L) ratified by Checker PASS_WITH_NOTES (4 notes resolved inline) — the canonical decomposition substrate for the OPS-T1 codemod implementation.
• Two new entries for the workflow-curator skill backlog (per §7.2 observation 1: “Liam-driven promote” branch in triage-finding preamble) — to be folded into the ID-30 OQ-8 PLAN.md skill-rewrite Subtask when ID-30 PLAN dispatches.

---

Appendix — Cleanup verification (post-Phase D)

Pre-merge sanity (orchestrator CWD checks):

git worktree list
  /…/knowledge-hub                       <CII-HEAD> [content-items-investigation]
  /…/knowledge-hub-production-readiness  78890470  [production-readiness]
# No .claude/worktrees/* dirs remaining

git branch | grep cmux
# (empty — all cmux-worker-* branches deleted)

ls .claude/cmux-events/
# (empty — all per-SID dirs removed)

All test artefacts on temporary paths only:

• /tmp/claude-501/s62b-probe-*.md (5 brief fixtures) — disposable.
• /tmp/claude-501/s62b-phase-c-sub-o-id{25,26}.md (2 sub-o briefs) — disposable.
• .planning/.test/ directory — never written to production-readiness (only landed on disposable s62b-phase-c-merge-test branch which was deleted).