S49 Handoff Analysis — Post-Wave-2 Investigation

S49 Handoff Analysis — Post-Wave-2 Investigation

Date: 2026-05-18. Status: Active. Authored: S49 post-Wave-2, pre-handoff. Companion to: s49-open-resolutions.md (Wave 1 ID-0 work).

Purpose: Capture the investigative analysis produced at the end of S49 — covering (a) functional differences between create-skill and update-skill, (b) the surfaces-FIRST order-of-work argument that defers Wave 3 (ID-2/3/4) until the Roadmap/Backlog/Task-list schema migration is done, (c) the recommended S50 plan, and (d) a full carry-forwards audit across S46→S49.

Without this doc the analysis lives only in conversation memory and is lost on session boundary. The Wave 2 commits (9a1cbd5b…6c3a5791) implement the ID-N work but do not encode this analysis.

---

1. create-skill vs update-skill — functional differences

Both skills live under .claude/skills/. They serve different stages of skill lifecycle.

1.1 create-skill (29 KB, scripts + agents + eval-viewer)

Full develop-eval-iterate loop. Workflow:

1. Intent → interview — clarify skill purpose, scope, triggering surface.
2. Draft SKILL.md — frontmatter + structure per skill-authoring conventions.
3. Test prompts — author 5–10 prompts spanning the trigger surface.
4. Parallel subagent runs — with-skill vs baseline (no skill) on the same prompts, in parallel.
5. Grade — agents/grader.md scores each pair; agents/comparator.md runs blind A/B; agents/analyzer.md surfaces patterns.
6. Aggregate — scripts/aggregate_benchmark.py rolls runs into a CSV + HTML eval-viewer (qualitative tab + quantitative tab).
7. Iterate — adjust SKILL.md, re-run, repeat until quality plateau.

Scripts:

• run_loop.py — 5-iteration description-optimisation loop (60/40 train/test split, optimises the description: frontmatter for triggering accuracy).
• run_eval.py — single-shot eval run against a prompt set.
• aggregate_benchmark.py — CSV/HTML rollup.
• improve_description.py — focused description-only optimiser.
• package_skill.py — bundle skill for distribution.
• quick_validate.py — frontmatter + structure sanity check.

Agents:

• grader.md — pair scorer (with-skill vs baseline).
• comparator.md — blind A/B preference.
• analyzer.md — pattern detector across runs.

Best fit: skills with objectively verifiable outputs (file transforms, data extraction, code generation, schema validation). The grading loop needs something concrete to grade against.

1.2 update-skill (5 KB, no scripts)

Pure authoring guide. Covers:

• Frontmatter spec (name, description, body structure).
• Body best-practices (when-to-use, structure templates, examples).
• When-to-split rule for skills that have grown beyond their original scope.

No eval loop. No scripts. No agents. Lightweight by design.

Best fit:

• Light edits to existing skills (description refinement, example updates, structure tightening).
• Drafting simple new skills where the eval loop would be overkill (e.g. skills whose value is procedural guidance rather than verifiable output).

1.3 Wave 3 application (ID-2 + ID-4)

ID-N	Skill	Recommended tool	Eval loop?
ID-2	workflow-orchestration (NEW)	create-skill for body + run_loop for description-optimisation	Skip the output eval loop — orchestrator outputs are decisions, not verifiable artefacts. Description-optimisation IS worth running (triggering accuracy matters for an orchestrator skill loaded at session start).
ID-4	implement-subtask (NEW)	create-skill similarly	Trigger evals worth running; output evals (commits) too expensive to grade automatically.
ID-4	spec-driven-implementation (UPDATE per A2)	update-skill	Light edit pass — rewrite to be the trigger for new-task creation producing the {N.1}-{N.4} subtask chain. No eval loop.

Note: A1 ratification says NEW implement-subtask, not adapted implement-specs (see s49-open-resolutions.md §A1). This is a divergence from the S49 continuation prompt’s “adapt” wording — S50 brief must make this explicit so the agent doesn’t default to in-place editing of implement-specs.

---

2. Order-of-work — surfaces FIRST, skills SECOND

Premise tested: Should the new SDLC workflow surfaces (Task-list JSON, Roadmap restructure, Backlog alignment, Label-reversal) land before the Wave 3 skills/agents (ID-2 workflow-orchestration, ID-3 agents sweep, ID-4 implement-subtask), or after?

Conclusion: Surfaces FIRST. Wave 3 work depends materially on the new surfaces existing.

2.1 Skills/agents BLOCKED on surface migration

Wave 3 deliverable	Blocked because
workflow-orchestration skill body (ID-2)	References the Task-list JSON for phase routing, status transitions, finding destinations. Can’t author against a non-existent schema.
implement-subtask skill (ID-4 NEW)	Reads ONE subtask record from the Task list. Needs the schema fixed.
spec-driven-implementation skill (ID-4 UPDATE)	Writes a TM-shaped Task with {N.1}-{N.4} spec-authoring chain to the Task list. Needs the file + write API.
task-planner.md agent (ID-3 NEW)	Writes implementation subtasks {N.5+} to the Task list.
task-executor.md body (ID-3 UPDATE)	Reads its subtask brief; updates status pending → in-progress.

2.2 NOT blocked on surface migration

Wave 3 deliverable	Why unblocked
task-checker.md body (ID-3 UPDATE)	JSON output schema is defined in kh-sdlc-workflow.md §6.1 — decoupled from the surface schema. Can author body now.
workflow-curator.md agent (UPDATE)	Roadmap + Backlog already exist; update-roadmap-backlog is operational. Updates are terminology + ID-N + binary in-scope-ness rule (B10).
File-level renames (ID-3)	workflow-executor.md → task-executor.md, workflow-checker.md → task-checker.md. Pure rename, no body change required to land the rename.
File-level deletions (ID-3)	workflow-worker-minimal.md DELETE (concept invalidated per N4).

2.3 Surface migration scope (consolidated carry-forwards)

#	Item	Source
1	Task-list JSON creation — NEW file, schema, lifecycle, path	A6 adoption (TM shape) + ID-2/ID-4 dependency on the surface
2	Roadmap §3 restructure — phases as top-level Tasks	s48-feedback.md general comments
3	Backlog alignment to TM shape	S49 carry-forward (per A6 + general schema-consistency drive)
4	Label-reversal migration	N10
5	update-roadmap-backlog skill — append-only → CRUD extension	N7
6	triage-finding skill — ID-N terminology + binary in-scope-ness rule	B10

Categorisation: Items 1–4 are surface schema/data work. Items 5–6 are skill-update work that sits on the surfaces (and consumes them).

Spec implication: A single PRODUCT.md for items 1–4 (data surfaces). Items 5–6 are implementation-level changes to consuming skills and become Tasks in the breakdown that follows. Spec scope = surfaces only.

---

3. Recommended S50 plan

3.1 S50 Wave A — Spec the surface migration

• Single PRODUCT.md + TECH.md covering items 1–4 from §2.3.
• Decisions in scope:
  • Task list file path (likely docs/reference/task-list.json).
  • Schema vs TM (subtle KH-specific deviations: effort_estimate, owner, cross_doc_links, session_refs, commit_refs retention?).
  • Migration steps for items 2–4 (idempotency, render-pipeline impact, JSON ↔ MD generators if any).
  • Label-reversal mechanics (which file becomes “roadmap” and which becomes “backlog” — labels currently inverted per N10).

3.2 S50 Wave B — Land independent agent/file changes (parallel-safe with Wave A)

These have no surface dependency and can land in any order alongside the spec:

• DELETE workflow-worker-minimal.md — concept invalidated per N4.
• UPDATE workflow-curator.md — ID-N terminology, reference updates. No surface dependency.
• RENAME workflow-executor.md → task-executor.md (filename only; body update deferred to post-surface).
• RENAME workflow-checker.md → task-checker.md (filename only).
• BODY UPDATE task-checker.md — JSON output schema per workflow doc §6.1; two-variant body per B5 (standard + quality-review variants). Decoupled from surface.

3.3 S51–Sn — Surface migration implementation

Per the Wave A spec. Multi-WP, sequential merges. Likely 2–3 sessions depending on render-pipeline complexity and Roadmap §3 restructure scope.

3.4 S(n+1) — Wave 3 (ID-2 + ID-4 + remaining ID-3)

Now that the surfaces exist:

• ID-2 — workflow-orchestration SKILL.md authored against concrete Task-list schema.
• ID-3 remaining — NEW task-planner.md, BODY UPDATE of renamed task-executor.md, DELETE workflow-orchestrator.md (after the new skill body is in place).
• ID-4 — NEW implement-subtask skill, UPDATE spec-driven-implementation skill.

---

4. Carry-forwards audit (S46 → S49)

Inventory of every carry-forward observed across the four-session ratification arc. Cross-checked against s49-open-resolutions.md and the S49 continuation prompt — nothing visibly lost.

4.1 Surface migration cluster (blocks Wave 3)

• Roadmap + Backlog schema alignment to TM shape.
• Roadmap §3 restructure (phases as top-level tasks).
• Label-reversal migration (N10).
• update-roadmap-backlog CRUD extension (N7).
• triage-finding ID-N terminology + binary in-scope-ness rule (B10).
• Task-list creation (implied by A6 adoption).

4.2 Operational (independent)

• Subtask deferred state machine + journal entry pattern (A10) — uses mempalace for cross-session memory; <info added on …> blocks for intra-task journal. Operationalise when first orchestrated session runs.
• fix-errors skill KH-adaptation (drop Rust steps, add TS+Python). Trigger: when first Executor needs it post-Wave-3.

4.3 Track / session counter cleanup

• Worktree collapse (3–4 sessions, Liam-owned per B3).
• Global session counter rollout — drop kh-prod-readiness-sN prefix once tracks collapse, per B3.

4.4 Deferred long

• Workflow Auditor + Evaluator agents — defer ~10 orchestrated sessions hence (X5). Need evidence base to evaluate against.
• Empirical planning-and-task-breakdown decomposition runs (T3.x + T3.y) — blocked on updated main-track specs. Note: S50 will produce one such run as a side-effect of the surface-migration spec (empirical feedback opportunity).
• Astro + Starlight implementation — S47 WP3 spec ready, awaits scheduling.
• Docubot + 6 Warp docs skills port — after Astro + Starlight.
• Linear migration scoping — future-state; KH treats TM shape as stepping stone toward generic structured-task discipline.

4.5 Audit / hygiene (added by Wave 2 verification)

• Q-ID tag-citation discipline pass on kh-sdlc-workflow.md (A8/A9/X3 — substance correct, tags missing). Surfaced by ID-1 Checker as out-of-scope.
• code-simplification vs simplify skill-name verification — surfaced by ID-1 Checker as out-of-scope.

---

5. Open question resolution — consolidated vs split spec

Question asked at end of S49: Should the surface-migration spec be one consolidated PRODUCT.md (Task list + Roadmap §3 + Backlog align + Label-reversal as one feature) or split into 2–3 separate specs?

Decision (S49 → S50): ONE consolidated spec. Rationale:

• Schema decisions for items 1–4 are interlocking — Task-list shape constrains Roadmap section structure constrains Backlog item shape constrains the label-reversal direction.
• Splitting forces premature decoupling of decisions that must be made coherently.
• Implementation can still split into multiple Tasks/WPs from the same PRODUCT.md (the breakdown phase handles this).

Implication for write-product-spec invocation: one spec directory at specs/surface-migration/ containing one PRODUCT.md, one TECH.md, and one PLAN.md (from planning-and-task-breakdown).

---

6. Provenance

Source	Detail
Pre-Wave-2 commit baseline	7edf8428 (S48 close-out)
Wave 2 commits added in S49	9a1cbd5b (ID-0) · 086be9d4 (ID-5) · 28bdc829 (ID-6) · f9281d58 (ID-1 cherry-pick) · 6c3a5791 (ID-1 fix-pass)
Conversation source	S49 turns 9–10
Companion doc	s49-open-resolutions.md (Wave 1 ID-0 decisions)
Forward reference	S50 continuation prompt (to be authored in handoff)

End of S49 handoff analysis.