Status: Active. Owner: Liam (palace is local to the dev machine). Audience:
the operator repairing a drifted/corrupt MemPalace, or hardening it against the chronic
daily drift. Binding rulings: DR-009 (single-writer; recall read-only), DR-010
(the markdown decision-register is the system-of-record; MemPalace is a recall feed)
and DR-098 (the auto-mine daemon runs supervised; the chromadb 1.5.9 segfault risk is
accepted — supersedes DR-048’s disable mandate).
MemPalace stores drawers in a local ChromaDB palace at ~/.mempalace/palace
(chroma.sqlite3, ~1.5 GB; collection mempalace_drawers). It drifts or corrupts roughly
daily under multi-session use. This runbook covers the symptoms, the root cause, the
interim single-writer posture, the repair recipes, and the durable fix.
The daily drift is an upstream chromadb Rust HNSW thread-safety bug under concurrent
mempalace-mcp writers (multiple Claude sessions, each with its own MCP server, all
opening a ChromaDB writer on the same on-disk palace). ChromaDB’s PersistentClient is
single-writer/exclusive; concurrent writers race the HNSW index and corrupt it. This is the
finding behind DR-009.
Until the durable fix lands, every access path obeys single-writer discipline:
Reads → lock-free, never a writer. Open chroma.sqlite3 with
?mode=ro&immutable=1 (FTS5 over sqlite). immutable=1 takes no locks and never
blocks the live writer. Plain mode=ro fails when the palace is WAL-mode with no -wal
present (checkpointed) — it would need to create -shm; use immutable=1. This is what
the SessionStart recall hook (canonical:.claude/hooks/mempal-recall.sh) does.
Writes → exactly one writer. Route writes through a single process — the opt-in 3.5.0
daemon, or one MCP server — never two concurrently.
Subagents recall read-only, never write. Worktree subagents read mode=ro; they do
not open a writer.
Do NOT blanket sandbox-allow ~/.mempalace writes. A blanket write-allow re-enables
the concurrent-writer drift this posture prevents. Reads work with no allow via
mode=ro&immutable=1; only a deliberate, quiesced repair (below) writes, with the
allow granted just for that step.
Before ANY repair: quiesce + back up. Stop the daemon and every MCP writer (close
other Claude sessions) so the repair is the only writer. Confirm the backup:
cp ~/.mempalace/palace/chroma.sqlite3 ~/.mempalace/palace/chroma.sqlite3.bak-$(date +%Y%m%d-%H%M%S).
These steps mutate ~/.mempalace, so run them with the sandbox disabled.
launchd caveat (§5g): the daemon is supervised by
~/Library/LaunchAgents/com.mempalace.daemon.plist with KeepAlive — a plain
mempalace daemon stop gets auto-restarted within ~30s. To quiesce:
launchctl bootout gui/$(id -u)/com.mempalace.daemon (stops it AND holds it down).
Resume after the repair with
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.mempalace.daemon.plist.
The CLI repair aborts before it can fix this (it opens the corrupt FTS first — #1606), so
rebuild the FTS5 inverted index directly with sqlite (the S431 fix that cleared the
integrity failure):
Terminal window
1
DB="$HOME/.mempalace/palace/chroma.sqlite3"
2
# Rebuild the FTS5 inverted index from its content table.
3
sqlite3"$DB""INSERT INTO embedding_fulltext_search(embedding_fulltext_search) VALUES('rebuild');"
4
# Verify integrity, then a MATCH that previously aborted.
5
sqlite3"$DB""INSERT INTO embedding_fulltext_search(embedding_fulltext_search) VALUES('integrity-check');"
6
sqlite3"$DB""SELECT COUNT(*) FROM embedding_fulltext_search WHERE embedding_fulltext_search MATCH 'decision';"
FTS works without this; only restore the HNSW when semantic / wing-filtered search is
needed. Rebuild the HNSW from the sqlite source of truth, archiving the drifted segment:
The repairs above are reactive. The durable fix removes the recurrence. Status after S433
(2026-07-01) — all applied quiesced + backed up + sandbox-off, INLINE (not sub-agent):
~/.mempalace/config.json → hooks.daemon: true (verify: MempalaceConfig().hook_use_daemon
is True). The 3.5.0 daemon “serializes background mines, diary saves, and hook ingests
through a single process”; with this flag the SessionEnd save/mine runs INSIDE the daemon (the
single writer) instead of opening its own ChromaDB client — removing the frequent automated writer.
5b. MCP single-writer — UPSTREAM-BLOCKED in 3.5.0 (the residual driver)
Each Claude session’s mempalace-mcp opens its OWN ChromaDB writer: mcp_server.py always
builds a direct chroma backend (get_backend_for_palace); the backend registry is only
chroma/qdrant/sqlite_exact/pgvector (no daemon/remote backend); mempalace-mcp exposes no
daemon-client flag. So daemon + N live sessions = N+1 writers, unfixable by config. v3.5.0
is the LATEST release (checked git ls-remote — nothing routes the MCP through the daemon yet).
Interim: one active Claude session during heavy mining; track upstream MemPalace/mempalace for a
daemon-routed MCP (SSE/HTTP, #1646). 5a already removes the automated writer, so the residual is
narrower than it looks. Tracked on bl-391.
Stale as of 27/07/2026: “v3.5.0 is the LATEST release” was true at S433 (01/07) but is
not now — 3.6.0 shipped 17/07 and lands mempalace serve (full MCP surface over HTTP,
--read-only, bearer-token auth). That is the daemon-routed/remote MCP this section waits on,
so re-evaluate 5b against 3.6.0 before treating the single-writer residual as unfixable. §10
covers the upgrade.
Two wing systems: DIARY uses wing_{agent} (wing_claude, already stable); FILE-MINING derives
wing_{slug} from the cwd basename (post-#1675; older drawers used bare names). knowledge-hub ==
canonical (rebrand), so the identity duplicates were merged to wing_canonical — there is no
official merge command, so quiesced direct SQL:
UPDATE embedding_metadata SET string_value='wing_canonical' WHERE key='wing' AND string_value IN ('knowledge-hub','knowledge_hub','wing_hub'); then PRAGMA wal_checkpoint(TRUNCATE);
(30,765 drawers → wing_canonical = 30,924; PRAGMA quick_check = ok). The
knowledge-hub-<subtopic> wings (kpf/ui-ux/prod-readiness/…) are left as sub-scopes — recall is
wing-agnostic (FTS + no-wing search), and #1665 breaks wing-filtered search anyway, so
fragmentation doesn’t hurt recall. Going forward the cwd canonical derives wing_canonical.
Quiesced: find ~/.mempalace/locks -type f -delete (was 14,873) + rm -rf ~/.mempalace/palace/*.drift-* ~/.mempalace/palace/*.corrupt-* (was 105 dirs / 191 MB). Live
segment dirs untouched. There is no official lock-prune (mempalace palace only has set-embedder).
mempalace palace set-embedder recorded minilm/384 (was unknown → a latent mismatch-drift
hazard; sidecar mempalace_embedder.json). Embeddings are LOCAL ONNX only (minilm default /
embeddinggemma, both 384-dim) — no API embedder; changing the model would require
re-embedding the whole palace, so set-embedder must match the model the vectors were built with.
The mine’s closet step (miner.py build_closet_lines → closet_llm.py) reads LLM_ENDPOINT /
LLM_KEY / LLM_MODEL from env (no dotenv auto-load). With hooks.daemon: true the mine runs in
the daemon, so start the daemon via ~/.mempalace/start-daemon-with-llm.sh — it sources the
OpenRouter creds from ~/.gitnexus/config.json at runtime (single source of truth; never
duplicated into a dotfile, never printed) and exports them before mempalace daemon start.
MAX_OUTPUT_TOKENS = 1500; the gitnexus model is a reasoning model (z-ai/glm-5.2) — validated to
emit valid closet JSON in ~554 tokens, but closet_llm.py:166 does content.strip() with no
null-guard, so a model/budget returning content: null silently falls back to heuristic. Override
LLM_MODEL with a cheaper non-reasoning model if closets come back thin.
The §5f wrapper alone was not durable: the S433 daemon (started 01/07 00:18) was found
DEAD the next day while three mempalace-mcp writers ran — and with hooks.daemon: true
the SessionEnd hook silently falls back to spawning its own writer when the daemon is
unreachable (hooks_cli.py _daemon_available), so a dead daemon quietly reverts the
whole single-writer posture. Fix: ~/Library/LaunchAgents/com.mempalace.daemon.plist
runs start-daemon-with-llm.sh --foreground (the CLI’s documented
process-supervisor mode) with RunAtLoad + KeepAlive + 30s throttle; stdout/err land
in ~/.mempalace/daemon/launchd.{out,err}.log. The wrapper still sources the OpenRouter
creds from ~/.gitnexus/config.json at runtime, so LLM closet enrichment survives
restarts. Quiesce/resume via launchctl bootout / bootstrap (see §4 caveat) — a bare
mempalace daemon stop is auto-restarted by design.
All of the above are destructive ~/.mempalace mutations; run quiesced (single-writer via
launchctl bootout gui/$(id -u)/com.mempalace.daemon + no live mempalace-mcp), backed
up, sandbox-off. bl-391 tracks the residual (5b) + the deferred subtopic wings (5c).
HNSW health (read-only, no chromadb client):mempalace repair-status — sqlite vs HNSW
counts + verdict (OK / flush-lag / rebuild-needed). Prefer this over opening a search to
check drift. A large divergence labelled “within flush-lag tolerance” is not corruption.
FTS: the read-only count in §1 returns a number (no malformed).
HNSW search:mempalace search "<query>" (no wing filter — #1665) returns ranked
results without Error finding id.
Embedder identity:mempalace search no longer warns EmbedderIdentityUnknownWarning
(sidecar mempalace_embedder.json records minilm/384).
Single writer:ps aux | grep mempalace shows only mempalace.daemon serve (no stray
mempalace-mcp writers) after quiescing.
LLM wiring (5f): daemon started via start-daemon-with-llm.sh; a closet-shaped chat call
to $LLM_ENDPOINT/chat/completions returns non-null content (give reasoning models ≥1500
max_tokens).
SessionEnd mine:mempalace hook run --hook session-end --harness claude-code writes
end-to-end without error (the 3.5.0 CLI contract; pinned in S430).
Recall hook: a fresh session startup injects a bounded MemPalace digest (the
mempal-recall.sh SessionStart hook).
Daemon supervision (§5g):launchctl print gui/$(id -u)/com.mempalace.daemon
shows state = running, and mempalace daemon status reports the same PID after a
kill test (KeepAlive restarts it within ~30s).
Outcome: all steps ran clean. Wing fold: 19,803 knowledge-hub-* drawers →
wing_canonical (quick_check ok). Rebuild: repair --mode from-sqlite --archive-existing --yes (note the --yes — the staged command below omits it and the
run aborts on an interactive confirm otherwise) rebuilt a FRESH palace: drawers
104,786/104,786 and closets 52,957/52,957, divergence 0 — the closets UNKNOWN state is
gone. Embedder identity must be RE-RECORDED after a rebuild (the sidecar does not carry
over): mempalace palace set-embedder. Old palace + all four prior backups archived to
~/.mempalace/palace.pre-rebuild-20260702-160307/; the four .bak files were pruned
(~5.9 GB reclaimed) and the archived full palace (1.6 GB) kept as the single rollback —
prune it at the next clean window. Daemon resumed under launchd.
Motivation (02/07 probes): drawers HNSW is frozen at 50,169 vs 104,304 sqlite rows (flush
never catches up — semantic search covers only the older half), and the closets collection
(52,958 rows) has no flushed HNSW metadata at all (repair-status: status UNKNOWN,
“HNSW capacity unavailable”). Also folds the remaining knowledge-hub-* subtopic wings
(~19.8k drawers: -kpf/-ui-ux/-prod-readiness/-admin-dedup/-eval/-memory/
-content-architecture) into wing_canonical — the S433 identity-merge default, wing fold
BEFORE the rebuild so the fresh index reflects final metadata. Run sandbox-off, no live
Claude sessions that need MemPalace (quiescing severs their MCP tools until restart).
The 21/07 Intent-session rebuild FAILED at 37k/245k upserts (MineAlreadyRunning — a
SessionEnd hook writer grabbed the palace mid-rebuild; log
~/.mempalace/repair-20260721-235854.log). Root causes found 24/07: hooks.daemon had
reverted to false (§5a undone — hooks spawning their own writers again) and the
launchd job was left in the DISABLED list after the 21/07 quiesce (launchctl bootout
disables; bootstrap alone then fails with Input/output error 5 — you must
launchctl enable gui/$(id -u)/com.mempalace.daemon first). Both fixed; embedder identity
re-recorded (minilm/384). Drawers verified healthy without a rebuild (215,107 = 215,107,
divergence 0; semantic search clean). Residuals: closets = 0 rows (already 0 in the 21/07
archive; regenerates only via future daemon mines with LLM env), and five
palace.pre-rebuild-* archives (~6 GB each, 03/07–21/07) pending owner prune — the 21/07
one is the only copy of a 245,062-drawer snapshot (live holds 215,107 after dedupe).
Docs-site re-mine (manual, on demand — after major docs landings): unchanged files dedupe,
new/changed files get LLM closets (OpenRouter cost).
Corrected 27/07/2026: this section previously read “last full mine was 15/06/2026”. That
is stale — the docs-site was re-mined 24/07/2026 (newest filed_at2026-07-24T19:03:16; 109,232 docs-site drawers live, including 1,310 from
ledgers/retros/). Recent retros, decisions and continuation prompts are in the palace.
Verify before repeating the claim:
The §8 loss (June-era recall degraded: the failed 21/07 rebuild died ~37k rows in and
post-21/07 re-mining rebuilt July but never June) was repaired by merging the archives back
into the live palace. Quiesced per §4/§7 (bootout + pkill, full palace/ dir backup),
then a single-writer script upserted every archive drawer absent from live through
mempalace’s own backend (mempalace.palace.get_collection(...).upsert — palace embedder
minilm/384; chroma maintains FTS + HNSW incrementally, so no post-merge repair run was
needed). Newest-archive-wins (21/07 > 09/07 > 03/07), §5c wing-fold applied defensively,
content-derived drawer ids make the upsert idempotent; journal-file resumable; ~57 docs/s,
50 min for 169,823 drawers.
Outcome: live drawers 215,110 → 385,233 (= exactly the 170,123-id union of
archive-only drawers; the 03/07 archive contributed 0 unique ids — fully contained in the
newer two). repair-status: HNSW 385,233 / sqlite 385,233, divergence 0, OK. June-filed
drawers 35,761 → 96,914. Semantic + FTS recall of June sessions verified end-to-end.
Prune gate (every archive id present in live: 0 missing × 3 archives) passed, then all
three palace.pre-rebuild-* archives + the s458 FTS .bak were pruned (~11.8 GB
reclaimed). palace.pre-merge-20260724-171529/ (1.6 GB) kept as the single rollback —
prune at the next clean window. Daemon resumed (launchctl enable before bootstrap —
the §8 trap), hooks.daemon: true intact.
Gap composition (for the record): the lost drawers were conversation-transcript mines
(~115k, ~/.claude/projects/**), the 15/06 docs-site content mine (~52.6k), and ~1.4k
diary entries — no repo source code; intentional codebase deletions (e.g. the ledger
stack) were unrelated to the gap. Drawers whose source docs were since retired remain in
the palace as historical memory (DR-010: recall feed, not system-of-record).
Verification-methodology correction: the FTS table is
fts5(..., tokenize='trigram') — phrase checks must use literal substrings of the
document (≥3 chars, punctuation intact). Whitespace-normalised word-sequence phrases
silently fail (~2% self-match even on known-present docs) — the earlier “0% of sampled
phrases found” reads were that artifact. Corrected baseline: literal 40-char mid-doc
slices self-match at ~86%; merged docs matched at 89% post-merge (vs 41% pre-merge —
the true prior content gap, part-masked by overlapping chunks under other ids).
Post-merge FTS malformation (same evening): the 170k-row merge left the FTS5 trigram
index in the malformed inverted index state — MATCH queries still answered, but the
daemon’s mine-validation PRAGMA quick_check guard (correctly) failed every subsequent
mine with FTS5/SQLite quick_check failed: 1 issue(s). Fixed with the §4a rebuild
(quiesced, backed up): 385k docs rebuilt in ~2 min, quick_check ok, daemon resumed,
mines re-queued (mines are dedupe-idempotent, so the partial pre-failure writes were
harmless). Lesson: after any bulk sqlite-level upsert, run the §4a FTS rebuild before
resuming the daemon — treat it as part of the merge, not an optional verify.
Residuals: closets collection still 0 rows (regenerates via future daemon mines with
LLM env); repo-level mempalace.yaml/entities.json are intentionally absent — optional
mempalace init artifacts; the miner’s no-yaml fallback derives wing_canonical from the
canonical cwd (§5c posture) and entities live globally in
~/.mempalace/known_entities.json.
Two findings from one investigation: the MCP server stopped connecting, and the palace is
~30% raw machine dumps. Neither is corruption — §1’s triage passes clean throughout.
/mcp reports Failed to reconnect to plugin:mempalace:mempalace while the palace is
provably healthy (quick_check = ok, FTS MATCH answering, 0 drift segments, 4 lock files,
daemon state = running under launchd, hooks.daemon: true intact).
Cause.mempalace-mcp answers JSON-RPC initialize in 92.8s; Claude Code’s connect
budget is 30s. The blocker is a mandatory startup gate added upstream in #1818:
mcp_server.py:4806 → _refresh_sqlite_integrity_status() → repair.sqlite_integrity_errors()
→ PRAGMA quick_check over the entire chroma.sqlite3.
Startup stage (3.5.0, 3.2 GB palace)
Time
import mcp_server
0.5s
_refresh_sqlite_integrity_status()
80.2s
_refresh_vector_disabled_flag()
1.1s
total to initialize response
92.8s
ChromaDB is not implicated — PersistentClient + get_collection = 0.1s. Eager warmup is
opt-in and unset in the plugin config ({"command": "mempalace-mcp"}, no env), so it is
never the cause here. The gate cost is O(database size), so it crossed 30s as the palace grew;
in 3.5.0 there is no env knob to skip or budget it (the only vars are
MEMPALACE_MCP_ALLOW_PEER_WRITER, MEMPALACE_EAGER_WARMUP, MEMPALACE_MCP_IDLE_HOURS,
MEMPALACE_BACKEND*, MEMPALACE_PALACE_PATH, MEMPALACE_LOG_FILE, MEMPALACE_MCP_HTTP_TOKEN).
Reproduce (read-only, ~90s — pipe an initialize request and time the reply):
Terminal window
1
# Raw gate cost on its own:
2
time sqlite3 "file:$HOME/.mempalace/palace/chroma.sqlite3?mode=ro&immutable=1""PRAGMA quick_check;"
Why §1 misses it: every existing triage check is a data-health probe. This is a
latency failure — the palace is fine and the client still cannot connect. Measure startup
time, not just integrity.
Stopgap (applied 27/07):MCP_TIMEOUT: "180000" in ~/.claude/settings.json → env.
Claude Code reads the env block at launch, so it takes effect on the next session, not the
current one. MCP_TOOL_TIMEOUT is the separate per-tool-call budget; it is not involved here.
This is a stopgap that degrades — the gate re-runs per MCP server, per session, and grows with
the palace.
Durable: upgrade to 3.6.0 (uv tool upgrade mempalace). #1911/#1987 fix this directly —
stdio answers initialize immediately while preflight runs on a background thread, and the
startup SQLite probe is skipped entirely above MEMPALACE_STARTUP_INTEGRITY_MAX_MB
(default 512 MB). Our palace is 3.2 GB, so the probe would not run at all. Repair preflights
stay strict, so the safety property #1818 wanted is retained where it matters.
3.6.0 also carries several things this runbook has been working around:
#2010 — Claude Code tool-results/ sidecars excluded from conversation scans, “so raw
machine dumps cannot flood the embedding space”. Directly addresses §10.3.
#1957 — appended/rewritten transcripts are purged and re-filed by mtime, instead of later
turns being silently skipped. Reduces duplicate accumulation.
#1213/#1953 — exclude_patterns in mempalace.yaml for per-project mining exclusions.
#1890/#1891 — conversation drawers keep transcript time as authored_at; ties prefer the
more recently authored drawer; list_drawers gains since/before. Idempotent, dry-run-first
backfill script for existing palaces (no re-embedding).
#2016 — CLI search checks HNSW divergence before opening ChromaDB, routing a diverged
palace to the SQLite BM25 fallback.
#1945/#2015/#2017 — integrity checks wait up to 15s for transient writer contention rather
than calling a healthy palace corrupt; repair --mode from-sqlite now also rebuilds FTS5 and
vacuums, and requires a clean final quick_check. This subsumes §9’s manual lesson about
running the §4a FTS rebuild after a bulk upsert.
Treat the upgrade as a §4-class change: quiesce (launchctl bootout), back up, sandbox-off, and
re-record embedder identity afterwards if anything rebuilds (§7 trap). Re-verify per §6.
10.3 Palace-noise audit (the “search is noisy” complaint)
The duplicate-content head is agent-brief and dispatch-template boilerplate mined 156–468×
each, plus <local-command-caveat> blocks at 468×. ingest_mode is convos for 328,320
drawers. So the highest-signal surface (the diary) is outnumbered ~1000:1 by raw transcript,
and identical template text outranks it on lexical match.
# Redundant copies (rows beyond the first in each identical-content group):
3
sqlite3"$U""SELECT SUM(cnt), SUM(cnt-1) FROM (SELECT COUNT(*) cnt FROM embedding_metadata
4
WHERE key='chroma:document' GROUP BY string_value HAVING cnt>1);"
5
# Machine-dump drawers:
6
sqlite3"$U""SELECT COUNT(*) FROM embedding_metadata WHERE key='source_file'
7
AND string_value LIKE '%/tool-results/%';"
Cleanup levers. There is still no dedupe command — mempalace repair rebuilds indexes,
it does not merge or prune drawers. The available tools:
mempalace_delete_by_source (MCP tool, ships in 3.5.0 — available now). Exact-match,
dry-run by default, removes every drawer for one source_fileand its closet/AAAK index
entries. This is the lever for the 844 tool-results/ files. Also exposed as the
delete_by_source MCP tool; it counts as a mutation for peer-writer purposes, so run it
single-writer.
mempalace sync [dir] --wing W --apply (#1252) — prunes drawers whose source files are
gitignored, deleted, or moved. Dry-run is the default; --apply requires --wing or a project
root. Good for drawers orphaned by repo churn, but note §9: drawers whose sources were
intentionally retired are kept deliberately as historical memory (DR-010).
DO NOT RUN --apply on this palace — see §10.4.
mempalace compress --wing W --dry-run — AAAK-dialect compression, ~30× reduction. Shrinks
storage; does not remove duplicate drawers.
3.6.0 exclude_patterns — the durable prevention. Stops the machine dumps being mined again
rather than deleting them after the fact.
Deleting 30% of the palace is an owner decision, not a maintenance default — these drawers are
recoverable history, and DR-010 makes the palace a recall feed rather than a system-of-record.
Recommended order: upgrade to 3.6.0 first (which stops new sidecar ingest), then dry-run
delete_by_source over the sidecar files, then re-verify recall quality before pruning further.
Note on the §9 verification method: the FTS table is fts5(tokenize='trigram'), so any
phrase check here must use literal substrings — see §9’s methodology correction before drawing
conclusions from match rates.
10.3a Why bulk writes cannot get a clean window on a live machine (27/07)
Four consecutive attempts to bulk-delete the §10.3 sidecar drawers all deadlocked, even
after launchctl bootout of the daemon andpkill -f mempalace-mcp. The blocker is not the
daemon and not the MCP servers:
~/.claude/plugins/cache/mempalace/mempalace/3.6.0/hooks/mempal-stop-hook.sh → mempalace hook run spawns a palace writer at the END OF EVERY TURN, in EVERY live Claude session.
So the writer set regenerates continuously while any session is open — including the session
running the repair, whose own Stop hook fires between its turns. Symptom: the repair process
sits at 0% CPU with ~0.01s CPU time, blocked inside mine_palace_lock acquisition, rather
than erroring. MEMPALACE_MCP_ALLOW_PEER_WRITER=1 does not help — it bypasses the MCP
peer-writer guard only, not the per-palace mine_palace_lock flock.
This is the same mechanism behind §8’s failed 21/07 rebuild (“MineAlreadyRunning — a
SessionEnd hook writer grabbed the palace mid-rebuild”). Treat it as the general rule for any
bulk mutation, not just rebuilds.
Diagnosis (identifies the real holder, not just “something has it”):
Terminal window
1
psaux|grep-E"[m]empalace"# look for `mempalace hook run` + mempal-stop-hook.sh
2
ls-la~/.mempalace/locks/mine_palace_*.lock
3
# a repair stuck at 0% CPU is BLOCKED on the flock, not working — check before waiting longer
4
ps-opid,etime,time,%cpu-p<repair-pid>
Prerequisite for any bulk write window: close every other Claude Code session first, and run
the mutation from a plain terminal (not from inside a Claude session, whose own Stop hook
competes). Batch-hold the lock for the whole run — mempalace.palace.mine_palace_lock(palace)
is process-wide re-entrant, so acquiring it once around the loop stops per-item releases from
letting a hook-spawned writer steal the palace mid-batch. Per-item acquisition is what makes a
long batch unfinishable here.
Owner ruling (27/07/2026):.lavish/ and .user-scratch/ carry rich working context and
must be mined, despite being gitignored (canonical/.gitignore:142-143; both also exist in
knowledge-hub, procurement, ims, knowledge-hub-docs-site, gh-security,
re-brand-kick-off). Measured 27/07: 0 drawers from either — the miner’s gitignore guard has
been silently excluding them all along. Volume is small and dense (canonical: 133 files, 2.7 MB),
so this is a high-signal-per-drawer addition, the inverse of §10.3’s machine dumps.
Mine them with --include-ignored (present in 3.5.0 — no upgrade needed):
--include-ignored takes project-relative paths, repeatable or comma-separated. Do not
reach for --no-gitignore, which disables the guard wholesale and would pull in
node_modules/.next.
TRAP — mempalace sync --apply deletes exactly these drawers.sync.py:127 classifies any
drawer whose source is gitignored as gitignored, and line 276 prunes that bucket alongside
missing. sync has no include_ignored awareness — it cannot tell a deliberately
included path from an orphan. So once .lavish / .user-scratch are mined, any
sync --apply touching their wing silently destroys them. Treat sync --apply as
banned on this palace while this ruling stands; use delete_by_source (exact-match,
dry-run-first) for cleanup instead. sync --dry-run remains safe and is the way to preview.
10.5 Structural finding — the recall path is tuned in the hook, not in the tool
The noise complaint has a specific mechanism, visible by comparing the two recall paths:
Path
Diary ranking
Noise filters
Corpus
mempal-recall.sh (SessionStart digest)
room='diary' ranked first (line 109)
filters CHECKPOINT:%, %Base directory for this skill%, topic='checkpoint' (lines 105-108)
flat FTS, bounded to 6 rows
mempalace_search (MCP — what start-session §2a and recall-grounding actually mandate)
none
none
flat across all 445,848 drawers
The automatic digest is well-tuned; the tool the skills tell you to reach for is not. Those
hand-rolled filters in the hook are themselves evidence that noise has been a chronic, reactively
patched problem — and they exist in only one of the two paths.
Compounding it: wing_claude (the curated diary — the highest-signal surface, and the one the
hook deliberately ranks first) holds 304 drawers, 0.07% of the palace, against 308,089
sessions drawers. The signal is outnumbered ~1000:1, and identical boilerplate outranks it on
lexical match.
Implications worth acting on, in increasing order of effort:
Client-side filtering belongs in recall-grounding. That skill already documents a
client-side workaround for the #1665 wing-filter defect; it should carry the same
diary-first / drop-CHECKPOINT: discipline the hook encodes, so MCP recall matches the
digest’s precision. Currently that knowledge lives only in shell.
The diary is the thing that works — feed it. Cross-session recall quality tracks diary
volume far more than transcript volume. /handoff writing mempalace_diary_write at session
close is the highest-leverage habit here; 304 drawers is undernourished.
Prevention over pruning — 3.6.0 exclude_patterns (§10.2) stops the machine dumps being
re-mined, which matters more than the one-off deletion.
Unrelated staleness surfaced during this review (not MemPalace, flagged for
propagate-workflow-change): .dev-workflow/sdlc/.claude/agents/references/shared-discipline.md:131
and .dev-workflow/sdlc/.claude/skills/workflow-orchestration/SKILL.md:172 both instruct
bun scripts/ledger-cli.ts get task <id> status. That script no longer exists — the ledger is
ordna markdown files (tasks/id-N.md), per CLAUDE.md. Both cite it as the “cheap guard” for
verifying live task status, so the guard is currently un-runnable.
11. 28/07/2026 — sidecar bulk delete: the DELETE path is broken; rebuild-as-delete works
SUPERSEDED by §12 (31/07/2026).delete_by_source works again — the pathology below
was the poisoned WAL, eliminated by the 31/07 from-sqlite rebuild. Keep this section for
the failure anatomy; do NOT follow its rebuild-as-delete recipe.
The §10.3 sidecar cleanup (800 tool-results/ files, 92,252 drawers, ~19% of the
palace) executed S507 via the daemon mcp_tool route and surfaced a fourth
chromadb 1.5.9 pathology: delete_by_source fails deterministically with
Error in compaction: Failed to apply logs to the hnsw segment writer against the
from-sqlite-rebuilt vector segment. Anatomy, verified on a 10-file pilot + 1-file
canary before the full wave:
The metadata segment applies each delete (sqlite rows drop exactly); the
HNSW vector segment refuses the same log entries, which stay pending in
embeddings_queue. ADDs are unaffected (the same day’s mines were
vector-searchable); reads stay healthy throughout.
Because sqlite is ground truth and repair --mode from-sqlite rebuilds into a
fresh store, the rebuild IS the vector-side delete — and it eliminates the
poisoned WAL by construction. Working bulk-delete recipe on this chromadb:
full metadata-side wave via daemon mcp_tool jobs (each job reports
failed — expected; verify sqlite rows reach 0);
one repair --mode from-sqlite in a §10.3a solo window, then §6 verify.
Upstream-report candidate (d): delete-compaction vs rebuilt segment.
Regrowth guard: 3.6.0’s convo miner already hard-skips tool-results/
(CONVO_SKIP_DIRS, convo_miner.py:76) so per-turn transcript ingestion cannot
re-mine sidecars; the 844-file corpus was pre-3.6.0 legacy. The only regrowth
route is a deliberate mine --mode projects over ~/.claude/projects — carry an
explicit tool-results exclusion if one is ever run.
12. 31/07/2026 (S517) — the real cause of the recurring recall outage
backends/chroma.py sets the divergence tolerance to 2 x hnsw:sync_thresholdwhen the
collection stores that key explicitly; without it, tolerance is
max(2000, sqlite_count * 0.10). repair --mode from-sqlite writes sync_threshold: 2.
So a rebuilt palace tolerates 4 diverged drawers where an un-rebuilt one tolerates
39,765. Cross it and repair-status reports DIVERGED, MCP search sets
vector_disabled: true and silently degrades to bm25_only_via_sqlite. Normal mining
crosses 4 within hours. Every rebuild re-armed the trap it was run to clear.
Fix (ratified DR-110) — quiesced, ~1 minute, no rebuild:
launchctl bootout leaves a stale palace lock. It kills the daemon while it holds
~/.mempalace/locks/mine_palace_<hash>.lock; the file survives with a dead PID inside,
and the next writer blocks on it forever. Symptom is §10.3a’s: 0% CPU, state S, no
progress. Always verify the recorded PID is dead, then clear:
P=$(tr-d'\0'<"$L"|awk'{print $1}') # NUL-safe — see the defect note below
3
if [ -z"$P" ]; thenecho"UNPARSEABLE — inspect by hand, do not clear"
4
elifps-p"$P">/dev/null2>&1; thenecho"ALIVE — do not clear"
5
elserm-f"$L"; fi
Defect, found S518: the version of this guard published before 01/08/2026 never
fired. The lock file begins with a NUL byte, so awk '{print $1}' returned empty,
ps -p "" always failed, and the || rm -f branch always ran — the lock was cleared
unconditionally, live writer or not. The “verify-then-clear” safety property was never
real. The NUL is invisible in the doc and in cat; the only way this surfaced was
executing the recipe rather than reading it. Refusing on an empty parse is the fix.
The daemon holds that lock for its entire lifetime (not per-operation). Any second
writer process blocks. Sequence bulk work as: bootout -> clear stale lock -> run ->
bootstrap. Restarting the daemon before the work is the mistake.
auggie respawns its peer mempalace-mcp within a second of being killed
(auggie --mcp --mcp-auto-workspace). Killing the child is futile; quiescing it means
stopping auggie or removing mempalace from its MCP config.
The warm-vs-cold rule this section originally gave was WRONG and is withdrawn
(S518). It read: “Route bulk deletes through a warm MCP server, never a fresh
script.” S518 reproduced the same deadlock six ways — warm MCP search, warm MCP
delete_by_source dry-run, cold client with the daemon up, cold client with zero
palace holders, cold client with device=cpu, cold client with sync_threshold=2.
Process warmth is not the variable. Neither is the DR-110 pin: reverting it 1000 → 2
changed nothing.
A client calling into the chroma collection on a palace in this state hangs before
opening any palace file (flat RSS, 0% CPU, no .bin/sqlite fds). Stack:
An internal condvar that never signals. sqlite stays healthy throughout (quick_check ok,
FTS reads fine), so lock-free FTS is the only usable recall path while it lasts.
The actual variable is palace health, and the fix is a from-sqlite rebuild — proven
S519, see §13. Two operational consequences that do still hold:
--dry-run never calls get_collection() (convo_miner.py: collection = get_collection(...) if not dry_run else None), so it is immune to the deadlock. When a
subsystem hangs, find the code path that avoids the hanging resource and build against
that — the whole S518 mining redesign was validated this way on a deadlocked palace.
Route bulk writes through the daemon (mcp_tool jobs), not a second writer process —
for the single-writer reason (DR-009), not the warmth reason. See §13.
~/.mempalace/tools/mempalace-census.py — lock-free (mode=ro&immutable=1), never a peer
writer, safe with everything live, ~9s over 400k drawers. Census (wings, rooms, source
rollup, prune candidates, age histogram, redundancy) plus a --export markdown tree for
human review.
~/.mempalace/tools/prune-stage.py is superseded — do not run it. It opens its own
chroma client, which is a second writer beside the daemon (DR-009). The proven bulk-delete
route is the daemon mcp_tool job loop in §13.2.
_ingest_transcript mines path.parent — the whole ~/.claude/projects/<slug>/
dir, not the transcript (hooks_cli.py:879). Canonical: 2,547 files / 948 MB per fire.
This is deliberate: subagent transcripts live in a sibling subtree and are 66% of convo
drawers — and are where the highest-signal reasoning lives. Do not blanket-exclude them.
hooks.auto_save gates Stop and SessionEnd and PreCompact
(hooks_cli.py:1096/1277/1338). Flipping it false loses all transcript mining. Env
override MEMPALACE_HOOKS_AUTO_SAVE beats config.json.
No config separates the CHECKPOINT: diary write from convo mining — they are adjacent
in one function. exclude_patterns is projects-mode only (per-project
mempalace.yaml, path-matched); the convo path never reads it.
authored_at is written by one line (convo_miner.py:555), convo path only. Diary
(0/3,883) and project-mode (0/171,908) drawers never carry it, and the pre-3.6.0 convo
corpus can never gain it (mtime-skip + normalize_version=2). ~70% of the palace is
permanently undated — chronology must come from the register, ordna and git.
(Narrower than it reads: source_mtime is populated on 99% of projects-mode drawers
and content_date on 84%. authored_at specifically is the gap, not dating in general.)
13. 01/08/2026 (S519) — the rebuild clears the deadlock; the delete route that works
The S518 unresolved question is answered. ~/repair-palace-s518.sh ran 09:32–10:47 on
01/08 (log ~/repair-palace-s518.log) and both smoke tests passed — semantic search and a
delete_by_source dry-run, the two paths S518 could not get through at all.
The DR-110 re-pin executed and was verified in the same run
(mempalace_drawers|1000, mempalace_closets|1000). This is the step the older
repair-palace-s506.sh lacks — running that script as-is re-arms the outage.
Embedder identity re-recorded (minilm, dim 384) — the §7 trap.
Durability held. Re-verified 5 hours later from a fresh session: mempalace_status
answered, semantic search returned matched_via: drawer+closet with real cosine scores
(not the bm25_only_via_sqlite degradation), sqlite_integrity.ok. S517’s rebuild
re-broke within hours; this one did not.
One green repair-status is still not proof. What distinguishes this from S517 is the
re-check on a later, separate process plus a vector-path assertion (matched_via), not
the status line. Verify recall that way after any rebuild.
13.2 Bulk deletes: the daemon mcp_tool job loop (proven)
Supersedes §11’s rebuild-as-delete recipe and §12.3’s warm-MCP rule. Single-writer safe
by construction — the write happens inside the daemon process, so it never races the
daemon’s own mining (DR-009), and it needs no quiesce window.
mcp_tool accepts write-classified tools only (service.py:WRITE_TOOLS); read tools
are refused so verbatim palace content never lands in the queue DB.
Procedure, as executed on the S355 stage orphans (44 files / 3,472 drawers):
Build the manifest lock-free — select distinct string_value from embedding_metadata where key='source_file' and string_value like '<pattern>' against
file:…chroma.sqlite3?mode=ro&immutable=1.
Export the drawers with content to a JSONL first. That, not a 3 GB palace copy, is
what makes the delete reversible.
Dry-run every file; sum match_count and assert it equals the manifest total before
committing anything. Ours was 3,472 = 3,472, 0 failures.
Commit. Note delete_by_source returns no deleted_count — do not read a count back
from the result, verify externally.
Verify both sides. The S507 pathology was metadata applying while HNSW refused, and
it reported success: select count(*) … like '<pattern>' must reach 0, andmempalace repair-status must show divergence 0. Then run a real semantic search.
Result: sqlite 0 stage rows, drawers 394,967 = HNSW 394,967, divergence 0, search healthy.
The S507 delete-compaction pathology is gone — it was the poisoned WAL, as S517 read it.
Landed this session, after the repair proved the vector layer:
~/.mempalace/config.json → hooks.auto_save: false (backup .bak-s519). This kills
the plugin’s per-turn whole-slug mine and its CHECKPOINT: diary writes, which is
intended — one of those CHECKPOINT jobs is what deadlocked the daemon on 31/07.
Smoke-tested end to end with CANONICAL_MINE_DRY_RUN=1 before trusting it: OK slug=-Users-liamj-Documents-development-canonical, 69.3s, parents 206 / subagents 1,175,
correct per-layer split.
/handoff Step 2c is now the diary’s single point of failure. It writes via the daemon
queue, so a wedged daemon loses the entry silently — that is exactly how the S517 entry was
lost. Always verify the job reached succeeded; submission is not landing:
Ran end to end S519. Final state: 1,770 files / 55,466 drawers, one wing
(wing_canonical), nine rooms, zero general. Before: 1,996 files / 107,938 drawers,
100% general, four wings. Divergence 0 on both collections throughout.
Three defects in the design as written, all found by projecting the mine before running
it. None were visible by reading.
src/** + !src/content/docs/** re-includes nothing. gitignore never descends into
an excluded directory, so the negation is unreachable — the first projection scanned
zero docs files. The working form excludes a directory’s contents with dir/* and
re-includes the wanted child before excluding one level down. A bare *.md compounded it
by excluding markdown at every depth.
The archive room’s DR-106 keywords are unreachable.detect_room walks path parts
outermost-first and returns on the first part matching any room, so
intended-architecture and phase-0-investigation are claimed by initiatives, and
every _archive by its own family (specs, runbooks, reference,
continuation-prompts). Only archive and product-functionality resolve. Confirmed
post-mine: intended-architecture/01-vision.md and 03-tech-stack.md both land in
room=initiatives. T3 as a path-routed tier does not work; discriminating those
families is recall-path work (DR-112 Q6), keyed off source_file.
Corollary: DR-112 Q4 is a no-op.01-vision.md was never going to land in
archive — initiatives matches four levels earlier. It was not moved.
Corollary: Q2 must land at the repo root, not tasks/archive/. Anything under
tasks/ matches the tasks room at depth 1.
Survivors are skipped, and skipping keeps them wrong.file_already_mined(..., check_mtime=True) skips any unchanged file, so the 931 previously-mined survivors would
have kept room=general across four wings — the F5 incoherence the whole decision exists
to close. §8’s “those files are deleted or re-mined into wing_canonical … no separate
merge step needed” is wrong. Worse, make_drawer_id_from_chunk hashes the room into
the drawer id, so a re-mine at a new room adds drawers rather than replacing them.
The only self-consistent fix is delete-then-re-mine (owner-approved S519); patching
room metadata in place would leave ids that no longer match their recipe, and the next
edit to any such file would fork it into parallel drawers.
Step 9 of the design’s execution sequence — an owner-run repair --mode from-sqlite solo
window between delete and mine — was dropped. Its stated rationale was that the rebuild is
what makes vector-side deletion real (§11). §13.2 disproves that: deletes now apply
vector-side unaided. Nothing in the plan needed a terminal.
Gotcha — a warm MCP server goes stale across a big out-of-process mutation. After the
waves and mines (all run by the daemon), this session’s MCP server returned nearest-vector
hits with wing: unknown, room: unknown and empty text — vectors found, content
unresolvable — while repair-status read a clean divergence 0. mempalace_reconnect fixed
it instantly. Call it after any bulk mutation made by another process, and never read an
empty/degraded search as evidence about the palace.
Open, for the recall-path work (Q6):mempalace_search with room= returned 0
results for queries that match plainly when unfiltered — the #1665 filter defect the
recall-grounding skill documents for wing. It applies to room too, and now that rooms
are populated and meaningful, that defect is the thing standing between this mine and any
observable recall improvement. Filter client-side until it is fixed.
13.5 01/08/2026 (S520) — room routing: the priority pre-pass, and two §12 corrections
#1665 is not a filter defect.room=/wing= are genuine ChromaDB pre-filters —
build_where_filter feeds where straight into drawers_col.query (searcher.py:1214),
and total_before_filter is only the n_results * 3 over-fetch, already filtered. The
0-result shape is _query_drawers_with_filter_fallback (searcher.py:1100): chromadb
raises Error finding id on filtered queries when the HNSW index is inconsistent with
the metadata store, and the code retries unfiltered + post-filters in Python. The remedy
is repair + mempalace_reconnect, not a client-side workaround. S519 probed the filters
minutes after its own bulk mine — i.e. in exactly the stale state §13.4 documents. Never
judge a filter from a palace you have just mutated out-of-process.
Rooms may now be declared priority: true (DR-114).detect_room returns on the
shallowest matching path part, so a parent family always beats a deeper one — that is
why S519 found archive unreachable, and why reference as a decisions keyword swept
the whole north-star set into room=decisions (452 of 2,572 drawers were actually DRs).
A priority room is tested against every path part, in declaration order, before the
depth-ordered walk; exclude_paths (fnmatch, project-relative) carves out exceptions so a
directory marker is not all-or-nothing. Absent the flag, routing is byte-identical.
The patch lives in site-packages/mempalace/miner.py — uv tool upgrade re-arms the
defect. Same standing hazard as DR-110’s hnsw:sync_threshold re-pin. Backup:
miner.py.bak-s520. The daemon imports from site-packages, so it must be restarted
before a mine picks the patch up. Upstream issue owed against
github.com/MemPalace/mempalace.
$P$T/assert-room-routing-s520.py # PASS/FAIL per checkpoint, BEFORE anything mutates
4
$P$T/build-remine-manifest-s520.py manifest.tsv # lock-free diff: current room vs projected room
5
$P$T/export-affected-s520.py manifest.tsv out.jsonl # reversibility — content, not a 3 GB palace copy
6
$P$T/delete-affected-s520.py manifest.tsv # dry-run; asserts sum == manifest
7
$P$T/delete-affected-s520.py manifest.tsv --commit # then re-mine via a daemon `mine` job
match_count is drawers only — closets return closet_match_count separately, so a
manifest counted from embedding_metadata (which spans both collections) must assert
against the sum. On commitdelete_by_source returns no count at all, so an assertion
written against the dry-run shape reports a false failure on the irreversible pass.
Two §12.2 corrections, both from executing it rather than reading it:
“A stale lock file blocks the next writer” is FALSE.fcntl.flock is released by the
OS when the holder dies; the file is residue. Verified by acquiring the lock left behind
by S519’s dead SessionEnd hook (PID 50502). The NUL-safe parse is still right for
diagnostics, but the verify-then-clear ritual is not protecting anything.
The daemon does NOT hold the mine lock for its entire lifetime. It was running, idle
and unlocked. What blocks a writer is a live holder — usually a mine actually in
flight. Restarting the daemon before bulk work is therefore fine, and is required when
the work depends on a patched module.
14. 11/08/2026 (S552) — third recurrence; the wedge-spotting recipe; chunk-level deletes
Timeline: S504 (27/07) segfault outage → repair; S518 (31/07) collection-access
deadlock → S519 repair; 05–06/08 segfault crash-loop (launchd err.log, ~20 crash
reports) after which the restarted daemon deadlocked on its first job and sat wedged
for 5.6 days — zero jobs completed 05→11/08, 155 queued behind it including every
diary entry S535→S551. repair --mode from-sqlite (the patched s518 script) cleared it
again: rebuild 381,762 = 381,762 divergence 0, DR-110 re-pin verified on BOTH
collections, the formerly wedged job then succeeded on its final attempt, and the
queue drained — 18 diary_write jobs landed within minutes of the restart.
Under mining load this palace loses its reader/writer paths roughly weekly on
chromadb 1.5.9 (DR-098 accepted risk). Each repair is ~50 min plus a session’s
attention. That run-rate is evidence for re-opening the deferred pgvector migration
(id-299) — owner’s call, recorded here so the cost is visible.
14.2 How to SPOT the wedge (it does not look like an outage)
mempalace daemon status answers normally while wedged. The discriminators:
succeeded count static across checks while queued grows — the S552 wedge showed
the same succeeded total for a full day.
The active job’s started_at is days old (queue DB:
~/.mempalace/daemon/<id>/queue.sqlite3, open with mode=ro&immutable=1 — plain
mode=ro fails on the WAL).
sample <daemon-pid> 3 shows every sample in the same chromadb_rust_bindings frames
with _pthread_cond_wait hits (the §12.3 signature). A HEALTHY daemon mining burns
200%+ CPU; a wedged one sits near 0%.
14.3 Script + verification gotchas found by running it
Quiesce false-positive:pgrep -f mempalace matches any process whose command line
merely CONTAINS the plugin-cache path (…/plugins/cache/mempalace/…) — chrome-devtools-mcp
qualified, and the script failed its quiesce on a process holding no palace handle. Fixed
in ~/repair-palace-s518.sh: match "bin/mempalace".
The post-repair smoke test races the queue drain. The restarted daemon starts mining
the backlog immediately; a cold CLI search during that burst can fail with
backfill request to compactor … Error deserializing pickle file: EOF — the §6/S506
transient under live-mine contention, NOT a failed repair. Re-run once the burst
settles; S552’s search answered with real cosine scores minutes later.
immutable=1 reads can tear under heavy daemon writes (database disk image is malformed mid-query). Retry or wait for the drain; it is a snapshot artefact, not
palace corruption.
14.4 Chunk-level deletes are supported (id-411 {411.2})
mempalace_delete_drawer is in the daemon’s WRITE_TOOLS allowlist, so the §13.2
daemon-job loop works at drawer granularity — required for boilerplate classes that
live as chunks INSIDE transcript files (CHECKPOINT rows, <local-command-caveat> blocks,
skill-preamble echoes), where delete_by_source would destroy sibling argument chunks.
Same discipline: lock-free manifest → JSONL export with content → assert export ==
manifest → submit at LOW priority (behind mines) → verify residuals externally. Chunk
deletes need no closet cleanup — the file’s closet still summarises the remaining file.
14.5 Agreed trigger (owner, S552): the next recurrence reopens pgvector
If the segfault / collection-access-deadlock class recurs a fourth time, do not just
repair and move on — reopen the pgvector-backend conversation. Owner-agreed at S552
close. There is no id-299 task file; mint the task at that point, citing §14.1’s
economics (three full outages in six weeks, ~50 min repair plus a session’s attention
each, all on chromadb 1.5.9’s accepted risk DR-098).
