claude-reflect (v3.1.0) — evaluation for canonical dev-workflow

Read-only eval. Source: ~/.claude/plugins/cache/claude-reflect-marketplace/claude-reflect/3.1.0/ (mirror in .../marketplaces/...). Repo: github.com/bayramannakov/claude-reflect. Enabled in ~/.claude/settings.json (claude-reflect@claude-reflect-marketplace: true).

Verdict: SKIP for the target failure mode; if kept, disable the capture hook.

It is a generic “capture user corrections → propose CLAUDE.md/rules edits” tool. It cannot detect the failure shape we care about (agent concludes → owner nudges “search mempalace” → mempalace corroborates → agent reverses), because its detection is single-message and gated on English correction-opener regex. It also adds a second, ledger-unaware CLAUDE.md-writing surface that competes with mempalace + our retro pipeline.

1. Mechanics (what each piece does)

Two stages: capture (automatic hooks) and process (manual /reflect).

Hooks (hooks/hooks.json), all local Python:

UserPromptSubmit → capture_learning.py: regex-classifies each prompt via detect_patterns(). Matches → appends to a per-project queue. Skips prompts >500 chars (unless remember:), XML/JSON/tool-result content, and question/task-request shapes.
PreCompact → check_learnings.py, PostToolUse[Bash] → post_commit_reminder.py, SessionStart → session_start_reminder.py: reminder nags only; no writes to memory.

Commands (LLM-driven .md playbooks):

/reflect: loads queue, optional semantic validation, proposes edits, applies to memory files only through AskUserQuestion (13 gates) with --dry-run and timestamped backups. Human-in-the-loop; not silent. allowed-tools includes Edit/Write.
/reflect --scan-history [--days N]: greps ALL project session JSONL for correction patterns + tool rejections, semantic-validates, proposes. Default window 30 days.
/reflect-skills: reasons over sessions to propose NEW .claude/commands/*.md skills (generic skill files — not our ledger/spec Task lifecycle).
/view-queue, /skip-reflect: inspect / discard queue.

Reads: ~/.claude/projects/<encoded>/*.jsonl (session history) + existing CLAUDE.md/rules. Writes: per-project ~/.claude/projects/<encoded>/learnings-queue.json; on /reflect, CLAUDE.md / CLAUDE.local.md / .claude/rules/*.md / auto-memory — all human-gated.

Egress: capture + queue + scan are fully local (regex). The semantic layer (semantic_detector.py: semantic_analyze, validate_tool_errors, detect_contradictions) shells out to claude -p --model sonnet, which sends the message text to the Anthropic API — same trust boundary as Claude Code itself, but note each queue item = one nested sonnet call (cost + latency). No third-party servers.

2. Data availability — SETTINGS PREREQUISITE

~/.claude/settings.json has cleanupPeriodDays: 30 (not 99999). This project already has 126 sessions spanning 18 Jun–14 Jul (~26 days) — i.e., history is right at the 30-day truncation edge; anything older is already gone. No learnings-queue.json exists for this project yet (capture hook has produced nothing). Prereq if adopted: raise cleanupPeriodDays (owner decision — 99999 = sessions never auto-delete: disk + privacy trade-off).

3. Detection fit for the mempalace-nudge shape — NO

The canonical failure is a multi-turn epistemic shape. claude-reflect fails it twice over:

Wrong granularity. semantic_analyze judges ONE message in isolation (Message: "{text}"). It never sees that the agent had concluded the opposite, nor that mempalace corroborated the owner. It structurally cannot recognise an agent-wrong → nudge → corroboration → reversal arc.
Wrong trigger. Both capture and scan-history gate on correction-opener regex (^no,, ^don't, use X not Y, that's wrong, remember:, tool rejections). An owner nudge like “did you check mempalace?” or “search mempalace before concluding” matches none — and “did you check…?” is actively dropped by the ends-with-? false-positive filter. It would only fire if the owner happened to open with “no,” / “actually”.

The one thing it could capture is the nudge phrased as a standalone imperative (“always search mempalace before concluding”) — but that is exactly the case where our existing remember:/mempalace habit already suffices.

4. Overlap / composition

Retro pipeline (handoff Step 7 transcript-miner → retro ledger): direct overlap in “mine session JSONL for lessons,” but different sink. Retro miner → retro ledger (process retros, ledger-aware). claude-reflect → CLAUDE.md/rules (behavioural guardrails, ledger-blind). Running both mines the same transcripts twice with no shared taxonomy.
workflow-evaluator (docs-site): /reflect-skills overlaps its “propose workflow/skill improvements” role, but is generic (writes plain .claude/commands/*.md, unaware of our skill/agent conventions, sentinel-gated .claude/ edits, or the ledger). Its output would need manual reshaping to fit our surface.
mempalace / CLAUDE.md: adds a THIRD memory-write path. Net risk = drift and duplicated guardrails across mempalace, CLAUDE.md, and .claude/rules/.
Slot: one-time --scan-history now = low yield (see §5); per-handoff = redundant with the transcript-miner; cron = unwarranted.

5. Empirical sample (safe, read-only, no network, no writes)

Ran extract_session_learnings.py --corrections-only over the 6 most-recent real sessions (pure-local regex; the same first-pass /reflect --scan-history uses):

6 sessions → only 1 produced matches (15), the other 5 = 0.
The 15 “matches” are almost entirely one long planning brief being sliced because it contains “actually”/“we should” — i.e., a planning doc mis-flagged as corrections. The scan pattern has no length/false-positive filter (unlike live capture), so signal is sparse AND noisy; the downstream sonnet pass + human review exist precisely to burn that noise off.
Nothing resembling the mempalace-corroboration correction surfaced structurally.

Recommendation

Do not rely on it for the mempalace failure mode — it can’t see that shape. If the owner wants to keep it for generic remember: / “use X not Y” capture: (a) accept cleanupPeriodDays stays 30 or raise it deliberately; (b) consider disabling the UserPromptSubmit capture hook (env CLAUDE_REFLECT_REMINDER=false silences the nag; removing the hook stops queue accrual) and invoke /reflect --scan-history opportunistically instead, to avoid a standing second CLAUDE.md-writing agent competing with mempalace and the retro pipeline. The mempalace-nudge failure is better addressed by our own retro/recall discipline, not this plugin.