Dummy-run: start-session skill — empirical evaluation

Dummy-run: `start-session` skill — empirical evaluation

Role: read-only evaluator. Walked the skill exactly as a fresh session orchestrator would, using the latest continuation prompt (continuation-prompt-ca-s471-id147-grounding-done.md) as input. No mutations performed; mutating steps recorded as “WOULD do”. STOP after Step 3.

Skill: /Users/liamj/Documents/development/canonical/.claude/skills/start-session/SKILL.md Inputs: s471 prompt (latest), s470 prompt (prior, for 2c carryover test). Session shape s471 declares: working dir = /Users/liamj/Documents/development/procurement; focus = ID-147 spec-chain + ID-145 item-2 re-key. Skill + ledger-cli live in canonical.

Cross-cutting noise observed on EVERY bun invocation: each ledger-cli call stdout is prefixed with a stray line — [branding] brandPrimaryColour vs light background is 2.85:1 (below WCAG 1.4.11 3:1 …). A hook is injecting a branding-contrast warning into ledger-cli stdout. Not fatal (JSON still parses on the next line) but it is pure noise on every single command and would confuse a fresh orchestrator / break naive jq piping.

Step 1 — Git hygiene (parallel)

(a) Commands run (read-only equivalents):

git -C …/canonical worktree list → 20 worktrees: main + procurement + re-brand-kick-off + 18 named agent worktrees under .claude/worktrees/agent-*.
git -C …/canonical branch --merged main | grep worktree → exit 1, ZERO matches. The “delete merged worktree branches” command would delete nothing.
git branch | grep worktree | wc -l → 18.
git status --short → M .gitignore only (clean).

WOULD do (not run): git worktree prune (clears only deleted dirs — the 18 named ones survive); the xargs … git branch -d deletes 0; per-worktree git worktree remove on the 18.

(b) Staleness: paths/verbs all still valid. (c) Ambiguity — REAL: the repo’s build is cherry-pick-to-track, never merge (stated in both prompts). So git branch --merged main NEVER catches agent worktree branches even when their commits are fully landed on track. The skill’s core cleanup command is a structural no-op for this repo’s actual workflow; 18 worktrees have accumulated precisely because of this. The skill’s own fallback prose (per-worktree git -C <wt> status + git worktree remove) is what actually works, but it’s framed as the exception (“If unmerged branches exist…”) when it’s the rule here. (d) Context cost: ~22 lines (~1.5KB). (e) Verdict: FIX. Invert the emphasis: lead with “cherry-pick-to-track means --merged won’t catch landed worktrees — reconcile each named worktree by origin-backed SHA, not merge status.” Give a concrete loop over .claude/worktrees/agent-* checking git -C <wt> status --porcelain + whether its branch SHA is reachable from origin/<track>. The --merged main line should be demoted or dropped.

Step 1b — GitNexus baseline

(a) Did NOT run npx gitnexus analyze (mutating/minutes). Both prompts say the index is stale (s470: last df703a4; s471: “GitNexus index stale (df703a4)”). WOULD: rebuild .gitnexus/lbug. (b) Staleness — pointer inconsistency: the skill says npx gitnexus analyze; CLAUDE.md + both continuation prompts say bun run gitnexus:analyze. Two different invocations for the same op. Also note: repo is indexed by both GitNexus and Memtrace/GitNexus MCP — the skill only refreshes one. (c) Ambiguity: “before a genuinely code-heavy wave” — s471’s session is spec-authoring (Planner writes PRODUCT/TECH), not code-heavy, so a faithful orchestrator would skip this step. Fine, but the skill puts a minutes-long step at position 1b that this session shouldn’t run. (d) Context cost: ~0 (background), minutes of wall-clock if run. (e) Verdict: FIX — reconcile the command to bun run gitnexus:analyze (match CLAUDE.md) and move the “skip unless code-heavy” qualifier to the front so it reads as conditional.

Step 2 preamble

(a) grep '^APP_URL' .env.local → APP_URL=https://kh.aisolutionhub.co.uk. Works. Anchor doc reference/platform-context.md exists (not re-read in full here; pointer valid). (e) Verdict: KEEP.

Step 2a — Memory recall

(a) Commands run:

mempalace_diary_read(agent_name: claude, last_n: 2) → MCP error -32002: Palace SQLite integrity check failed; refusing tool call until the palace is repaired. HARD FAIL.
mempalace_search("ID-147 … ID-145 …", limit 5) (no wing filter, per #1665) → same -32002 integrity-check failure. HARD FAIL.
Could not reach the CHECKPOINT-vs-narrative signal-quality judgement — the palace refuses ALL reads, so 0 diary rows retrievable (0 narrative, 0 checkpoint; the whole store is offline). (b) Staleness: tool names valid; the #1665 no-wing workaround the skill cites is real but irrelevant right now because the failure is upstream of vector search (SQLite integrity, not HNSW↔sqlite drift). The skill’s stated fallback (“lock-free mode=ro sqlite FTS read”) is a SessionStart-hook mechanism, not something the orchestrator can invoke as a tool mid-skill — so when the MCP path dies, the skill offers no actionable manual fallback. (c) Ambiguity: none in the instruction; the blocker is environmental. But a fresh orchestrator following 2a verbatim gets two hard errors and no guidance on “palace is down, proceed without memory.” (d) Context cost: ~2 error lines. (e) Verdict: FIX — add an explicit “if the palace errors (integrity/-32002), note it and proceed; do not block session start on memory recall.” Currently a dead MCP server silently strands the mandated first substantive step.

Step 2b — Task-list state inspection (slice reads)

(a) Commands run:

show task 145 → 5.8 KB, auto-degraded to summary (journals stubbed; notice lists escape hatches --full / --no-journals / --fields / get … details / journal). Subtask table 22/31; capability_theme: "2"; status_note = S470 arc-1-complete summary.
show task 147 → 8.0 KB, journals NOT stubbed (task < 48KB) → both <info added> blocks returned inline (~6KB of prose that largely duplicates what the s471 prompt already summarised: DR-063…068, grounding-doc content, bl-459/460). capability_theme: "2". Its sole subtask {147.1} is done. (b) Staleness: verbs valid; 48KB valve + escape hatches all functioning as documented. (c) Ambiguity — the owner’s question, answered empirically: how does an orchestrator know which field to query? It does not, from the skill. show task <id> returns the subtask TABLE (id/title/status) but stubs journals on large tasks (145) and dumps them inline on small ones (147) — so behaviour is size-dependent and you can’t predict which you’ll get. To hydrate a named subtask you then choose among get … details, get … status_note, journal <id.sub> with no rule mapping “I want this subtask’s current state” → one verb. In practice the field pointer came from the continuation prompt’s READ FIRST bullets (“journal 147.1”, “journal 145.23”), NOT from the skill. So 2b’s field-selection is only tractable when the prompt author pre-names the journal/field; when a prompt omits those pointers, 2b leaves the orchestrator guessing and over-fetching. (d) Context cost: 145 (5.8KB) + 147 (8KB) = ~14KB for two tasks, and for 145 you still need ≥1 more journal 145.23 / get 145.19 details call per named subtask (s471 names 7: 145.15/19/20/23/29/30/31) → easily +10–20KB to hydrate the wave. Much of 147’s 8KB was wasted (prompt already summarised it). (e) Verdict: FIX — add a one-line decision rule: “For a subtask named by the prompt, read the field the prompt points at; absent a pointer, use journal <id>.<sub> for narrative state, get <id>.<sub> details for the spec brief, get <id> status_note for task-level rollup. Skip show task <id> entirely when the prompt already summarises the task — go straight to the named journal.” This directly fixes the over-fetch the owner flagged.

Step 2c — Sandbox / allowlist carryover

(a) Findings:

s470 HAS a ## Sandbox / allowlist carryover heading (line 81): ledger-cli WRITE needs dangerouslyDisableSandbox; git cherry-pick/reset/push EPERM; next build Turbopack failure.
s471 (the latest = the actual input) has NO such heading. The word “allowlist” does not appear at all. The sandbox note is inlined into READ FIRST (line 23: “Ledger WRITE path needs dangerouslyDisableSandbox”). So the skill’s instruction “Read the prior handoff’s Sandbox / allowlist carryover section” fails against the newest prompt — that named section only exists in the older s470.
“apply any allowlist candidates” is NOT mechanically actionable. There is no allowlist file, no allowlist command, no .claude/settings.json mutation the skill points at. “Applying” has no referent. The carryover is purely informational (“be aware you’ll need dangerouslyDisableSandbox for X”). Empirically: no allowlist artefact exists to apply to. (b) Staleness: the section-name pointer is already stale vs the current prompt format. (c) Ambiguity: high — a reader is told to “apply allowlist candidates” with nothing to apply. (d) Context cost: ~7 lines (s470) / inline (s471). (e) Verdict: FIX — drop “apply any allowlist candidates” (no mechanism exists). Re-scope to: “scan the prompt (heading OR inline) for sandbox/dangerouslyDisableSandbox notes and carry them as known bypass commands.” Rename away from a section title that prompts no longer emit.

Step 2d — GitHub tooling

(a) No command run (advisory step: use gh-axi, not raw gh). Note: s470’s mechanical report recorded gh-axi failing with a TLS cert error at close — so the recommended tool is itself flaky, but that’s env, not skill. (b) Staleness: pointer valid. (e) Verdict: KEEP (advisory, cheap).

Step 2e — Owning-theme strategic context

(a) Command run: show roadmap 2 (task 145 & 147 both carry capability_theme: "2") → HARD FAIL: {"ok":false,"error":"ledger-read-failed","detail":"…/ledgers/product-roadmap.json: ENOENT: no such file or directory"}. Confirmed root cause: product-roadmap.json has been deleted/renamed to initiatives.json (the new file exists: 39KB, dated 2026-07-14, document_name: "Canonical Platform - Initiatives", schema = linked_tasks[] / linked_backlog[] + substrate_doc per Initiative). The CLI has NO show initiatives verb — show accepts only task|roadmap|backlog|retro (+ show umbrellas). So the new strategic file is unreachable via ledger-cli, and every task’s capability_theme back-link now resolves to a dead read. (b) Staleness — CRITICAL: entire step broken. show roadmap, and the mutation verbs update-roadmap / create-theme / --capability-theme, all still target the vanished product-roadmap.json. (c) Ambiguity: a fresh orchestrator runs show roadmap 2, gets ENOENT, and has no documented recovery — the skill still says “read the active Task’s capability_theme … via show roadmap”. (d) Context cost: 0 useful (error), but blocks the whole “why this Task matters” framing. (e) Verdict: FIX (urgent) — rewrite 2e for the initiatives model: resolve the owning Initiative by finding which linked_tasks[] contains the active task id (in initiatives.json), surface title + intent + substrate_doc. Requires either a new show initiatives CLI verb or a direct Read of initiatives.json (39KB — safe wholesale, unlike task-list.json). The capability_theme field on tasks is now an orphaned pointer with no reader.

Step 2f — Reconciliation sweep + umbrella resolution

(a) Commands run:

list task --status in_progress → 12 tasks (45, 46, 57, 71, 123, 127, 128, 132, 135, 143, 145, 147). Works. But only 145 & 147 are this session’s; the other 10 are unrelated in-flight tasks pulled in as “active.” Note 147 shows in_progress with subtasks 1/1 — its only subtask is done, the prompt calls it “spec-ready / grounding complete,” yet the task reads open. This is exactly the “closed work referenced as open” context-rot symptom the owner cited.
list task --status done --since 2026-07-11 → total 0. The S470/S471 waves gated many subtasks done, but no whole tasks flipped to done in-window (and archived done-tasks are CLI-invisible per DR-002). So the done-sweep — task-level granularity — catches nothing where the actual completion happened (subtask level).
2f step-4 umbrella resolution: show umbrellas → HARD FAIL: {"ok":false,"error":"ledger-parse-failed","detail":"…/umbrellas.json: JSON Parse error: Unable to parse JSON string"}. umbrellas.json is now 1.1KB and malformed (starts with a bare "umbrellas": — no opening {) — it is being dismantled as part of the umbrellas→initiatives migration. The skill’s documented fallback (direct Read of umbrellas.json) would return the same broken JSON. Both paths in step-4 are dead. (b) Staleness — CRITICAL for step-4: the umbrella abstraction is being removed and folded into initiatives.json (linked_tasks[] replaces task_ids[], substrate_doc carried over). The skill still instructs resolving “the owning umbrella … show umbrellas (S450 read verb).” (c) Ambiguity: --since <lastSessionDate> — which date? prior retro? handoff? s471 is dated 07-13, s470 07-12; I used 07-11 and got 0. Under-specified and, given task-level granularity, low-value. (d) Context cost: in_progress list ~2KB (12 rows, 10 irrelevant); done list ~0; umbrella fail ~0. (e) Verdict: main sweep KEEP-with-FIX; step-4 DROP/REWRITE. Main sweep: good “don’t re-flag” guard, but note it surfaces every in-flight task platform-wide, not this session’s — keep the “cross-check against prompt-named ids” line and make it primary. Consider a --since default (prior retro date) or drop the date arg. Step-4 umbrella resolution must be rewritten onto initiatives.json or deleted — it currently fails 100% and points at a deprecated abstraction.

Step 2g — Settled-state read-back (retros + decision register)

(a) Commands run:

list retro --recent 3 → S470/S469/S468. Works, BUT returns only {id,date,track} — not the unresolved_questions / workflow_improvements / failed_assumptions the skill says to “surface.” To get those you must fire show retro <id> ×3 (3 more calls). The skill’s → surface … arrow implies the list returns them; it doesn’t. (Also emits a duplicated {"warnings":…} line — stdout noise.)
Decision register: reference/decision-register.md exists (57KB). Skill says “read in-force entries … surface titles + one-line rulings; do NOT dump the whole file” — but the register is a markdown file with no ledger-cli slice path, so the only way to read it is a 57KB wholesale Read + self-filter. s471 references DR-063…068 as newest in-force. (b) Staleness: paths valid; list retro field-shape mismatch is the issue. (c) Ambiguity: “surface unresolved_questions…” not achievable from the given command. (d) Context cost: retro list ~0.5KB; decision register ~57KB if read wholesale — the single biggest context pull of the whole skill, with no slice mechanism. (e) Verdict: FIX — (i) change 2g to show retro <id> (or a list retro --fields flag) so the named fields are actually retrievable; (ii) give the decision register a slice path (grep for accepted/newest DR-NNN, or a CLI verb) so the orchestrator isn’t forced to ingest 57KB to obey “don’t dump the whole file.”

Step 3 — Review continuation prompt + confirm plan

(a) ls … | sort -V | tail -2 → correctly returns s470 + s471 (s471 latest). Read s471 fully (~90 lines). “Read any referenced tasks” = the 2b calls above (145 @5.8KB + 147 @8KB) + would need per-subtask journals for the 7 named 145 subtasks. (b) Staleness: ls | sort -V | tail -2 valid. (c) Ambiguity / cross-repo friction (the NOTE): s471 declares working dir = …/procurement, but start-session’s git hygiene targets canonical/.claude/worktrees and bun scripts/ledger-cli.ts only exists in canonical. The guard blocks git -C <main-repo> from the procurement worktree (s470 line 105 records this). The skill never acknowledges the cwd≠canonical case — it assumes single-repo cwd. A fresh orchestrator dropped in procurement per the prompt must mentally redirect every Step-1/Step-2 command to canonical. Real friction. (d) Context cost: s471 prompt ~4KB + the 2b task reads (~14KB+). (e) Verdict: KEEP-with-FIX — add a one-liner: “continuation prompts may set a working dir other than canonical (e.g. the procurement worktree); run all git-hygiene + ledger-cli commands against the canonical checkout regardless of the prompt’s stated cwd.”

Overall

The skill’s read/advisory scaffolding is sound, but three data-layer steps are outright broken by un-propagated infra changes, and two more are under-specified in exactly the way that produces the owner’s “context-rot” symptom:

BROKEN (hard-fail): 2a (mempalace palace integrity -32002), 2e (show roadmap → product- roadmap.json ENOENT, repurposed to initiatives.json with no CLI verb), 2f-step4 (show umbrellas → malformed JSON, abstraction being removed).
UNDER-SPECIFIED (drives over-fetch / rot): 2b (no field-selection rule; size-dependent journal behaviour), 2g (list retro doesn’t return the fields it says to surface; 57KB register with no slice).
STRUCTURALLY WRONG for this repo: Step 1 (--merged main never catches cherry-pick-to-track worktrees → 18 accumulated), Step 3 cross-repo cwd assumption.
DEAD INSTRUCTION: 2c “apply allowlist candidates” — no allowlist mechanism exists.
The “closed-as-open” rot is real and visible: task 147 = in_progress, subtasks 1/1 done, prompt says spec-ready. start-session currently has no step that reconciles a task whose subtasks are all done but whose status never flipped.