Workflow Evaluation — Feedback Dossier (S264)

Provenance: consolidated at the close of S264 (main-track orchestrator-of-orchestrators) from (a) the ID-32 cmux worker’s final_report.yaml (docs/continuation-prompts/s262-worker-reports/id32-final-report.yaml) and (b) Liam’s saved workflow notes. Seeds Task ID-48 (paired with ID-23). This is the INPUT to {48.1} RESEARCH — not the research output. Do not treat as ratified design.

1. Purpose

Form a workflow-evaluator capability and make targeted tweaks across the SDLC roles (Planner, Executor, Checker, Curator) and skills (workflow-orchestration, triage-finding, update-roadmap-backlog, start-session, session-driver-cmux). The core problem, stated plainly by Liam and proven twice in the codebase:

Blindly executing a spec is not a valuable workflow. A spec written against assumptions rather than the installed reality produced defects that green fixture/unit tests hid until the final acceptance gate — or that shipped silently.

2. ID-32 critical context — workflow lessons

From the OPS-T1 codemod work (id32 sub-orchestrator, S262–S263). The 32.16 acceptance gate (run against the REAL app/api corpus + real tests — what the unit/fixture suites never did) caught four defects every prior subtask’s green tests had hidden:

Blind spec execution (B4, the design defect). defineRoute was spec’d assuming a raw-payload contract, but 178/193 handlers return NextResponse inline (authFailureResponse→401 etc.). Mechanical wrapping double-wrapped → 1676/2075 route tests failed. The spec was followed without ever verifying it against the corpus it targeted. This drove the whole Option-4 re-scope.
Acceptance gate ran LAST. B1–B4 surfaced only at the final real-corpus gate. Per-slice fixture/unit tests were green the entire way. → Mandate: the real-corpus acceptance probe runs CONTINUOUSLY from the first slice, not as a terminal gate.
Vacuous AC. AC-9 = “no new lint errors” passed vacuously — ESLint is blind to no-undef on TS, so it never caught B3 (rewriter emitted defineRoute(...) without the z/schema imports → 130+ routes ReferenceError at load). → lint-delta ACs MUST be paired with a tsc/no-undef check or they pass without testing anything.
Local optimisation defeated the global goal. The 32.20 generator was instructed to “err permissive to protect AC-8” → 84 .loose() + 10 z.unknown() in the 37 generated schemas → the runtime defineRoute guarantee is FALSE confidence. A local AC-protection move silently defeated the global type-safety goal. (Owned + recorded by the worker.)
Prettier-non-canonical cherry-picks. 5 follow-up prettier commits were needed across the session. → candidate: a pre-commit prettier hook in agent worktrees, or Executors run bun run format pre-commit.
Doc-inventory drift (minor). documentation-inventory.md runbook count drifted (4→5); prose, no guard.

3. ID-28 echo — same root cause, independently

ID-28.20 exists because a subtask was closed by implementing a cocoindex API that does not exist: flow.py called content_text.bind_target(ci_target, op_id=…) under type: ignore — bind_target is not a DataSlice method in any cocoindex version. op_id columns write NULL; Stage-6 UPSERTs nothing. Identical failure mode to ID-32 B4: spec’d against an assumed API, implemented without validating reality, closed green. Two independent instances = a systemic workflow gap, not a one-off.

4. Liam’s notes (raw → categorised)

A. New role — workflow-evaluator

Build a means of evaluating the dev-workflow. Form: an agent (workflow-evaluator), a skill (evaluate-workflow), or a combination following the agent-development + create-skill pattern (agent body + companion skill).
Include efficiency assessment: token usage, duplication in the workflow process, etc.

B. Spec discipline (the core lesson)

Lots of spec-vs-scope issues, part-fixed. Evaluate recent session logs for more examples to identify the root cause. Blindly executing a spec is not valuable — ID-32 found this (see §2/§3).
Mandate which task types require a Checker and which don’t — e.g. should {N.1} RESEARCH subtasks be Checker-gated, or is surfacing any OQs sufficient?

C. Spec conventions

Initial task research must be saved as RESEARCH.md (rename docs-site-rebuild-research to sit alongside the PRODUCT/TECH/PLAN docs).
Spec directories should include the ID number in the folder, not just the name (e.g. docs/specs/id-48-workflow-evaluation/).

D. Hooks / config

Hook to block edits to agent or skill files unless the relevant skill was invoked (create-skill, update-skill, or agent-development).
Update Claude config to allow dangerouslyDisableSandbox: true for the known “sandbox blocks new .claude/skills/<dir>/ create” case — but pair it with the hook above so the allowance can’t be misused.

E. Orchestration ergonomics

Add an item to /workflow-orchestration or /start-session to create a harness Task (via the Claude Code task tools) — a simple, same-every-session task like “Continue working through Task List” — to silence the recurring TaskCreate reminders that fire throughout a standard-workflow session. (Ephemeral in-session tasks still use the harness task tools as normal.)

F. Tooling

Anthropic renamed /simplify → /code-review. Decide whether to integrate it into the workflow in place of the /code-simplification skill.

G. Knowledge-capture

Change the gotchas approach — a pointer file / memory / lesson-learned doc using the same schema as the other 3 ledger surfaces (task-list / roadmap / backlog). Liam’s guidance (captured S264): “Run a retro at the end of every session.” Before closing a session, ask what we learned, then save the answer across six categories: (1) bugs discovered, (2) failed assumptions, (3) architecture decisions, (4) rejected approaches, (5) workflow improvements, (6) unresolved questions. Each retro turns disposable session context into durable institutional memory that outlives any single session — stack enough and the memory compounds. Design target: a 4th ledger-style surface (JSON + per-record mirrors, same shape as task-list/roadmap/backlog), one retro record per session, plus the session-end retro habit wired into the workflow (handoff / workflow-orchestration / start-session). Reconcile with the Mempalace diary — it is the current partial implementation; {48.1} must decide: extend the diary, replace it with the JSON surface, or run both. Seed/example: docs/specs/id-48-workflow-evaluation/retro-S264.md (the first retro, written at S264 close as a working proof-of-concept).

5. Open design questions (for {48.1} RESEARCH to resolve)

workflow-evaluator form — agent / skill / combo? (lean: combo, per Liam.)
Checker mandate — per task-type matrix: which subtask kinds require a Checker vs OQ-surfacing-only? (RESEARCH? spec-authoring? mechanical dev-infra?)
/code-review integration — replace /code-simplification, or run both? What does the renamed command actually do now vs the old /simplify?
Retro/gotchas surface — per §4.G (CAPTURED): a session-end retro (6 categories → durable institutional memory). Design the 4th ledger-style surface (JSON + mirrors) + the session-end retro habit; reconcile with the Mempalace diary (extend / replace / both).
Efficiency metrics — what does the evaluator measure (token usage per role, duplicated reads, redundant dispatches, megaturn detection)?

6. Affected surfaces (tweak targets)

Surface	Likely tweak
Planner (agent + write-*-spec skills)	validate contract/API reality against installed code BEFORE spec’ing; don’t carry unresolved AC tensions into a spec
Executor (task-executor + implement-subtask)	escalate spec-vs-reality conflicts rather than blindly executing; run real-corpus probe continuously; `bun run format` pre-commit
Checker (task-checker)	non-vacuous AC enforcement (lint-delta + tsc); verify against real corpus not just fixtures; per-task-type checking mandate
Curator + triage-finding + update-roadmap-backlog	(efficiency; duplication audit)
Orchestration (workflow-orchestration) + start-session	harness-task seeding; real-corpus-gate-continuous mandate; checking-mandate matrix
session-driver-cmux	prettier pre-commit hook; skill-gated agent/skill edit hook
Conventions	RESEARCH.md naming; ID-prefixed spec dirs

7. Relationship to ID-23

ID-23 = Code-intelligence integration into SDLC workflow (gitnexus + ast-dataflow + ccc). Paired, not merged: the same next-session investigation terminal handles both. Overlap = both harden the SDLC workflow; ID-23 is the tooling axis, ID-48 is the role/process/evaluation axis. Cross-check for duplicated scope at {48.1}.

8. Suggested {48.1} RESEARCH approach

Review recent session logs (S256–S263 Mempalace diaries + the s262-worker-reports/* final_reports + Checker verdicts) for spec-vs-scope / blind-execution instances → build a root-cause taxonomy (not anecdotes).
Survey each role/skill file for where the anti-patterns enter and where a guardrail fits.
Decide the workflow-evaluator form + the efficiency-metric set.
Design the gotchas-capture surface (with Liam’s guidance).
Feed {48.2} PLAN — decompose into shippable tweaks (many are small, independent config/hook/skill edits; sequence the conventions + the evaluator role).