ID-65 {65.1} RESEARCH — Ledger-CLI v3 + ledger storage architecture

ID-65 {65.1} RESEARCH — Ledger-CLI v3 + ledger storage architecture

Status: Draft for ratification. Authored: S280 (29/05/2026), {65.1} Planner dispatch. Parent Task: ID-65 — Ledger-CLI v3 + ledger storage architecture — scoped writes, programmatic bulk-create, whole-file-re-escape elimination (priority high, spec_needed). Scope: S279/S280 token-efficiency friction cluster (#4, #5, #6) + the storage rethink Liam flagged. This RESEARCH determines the approach across all three before any PRODUCT/TECH spec.

Predecessor: ID-35 ledger-CLI (done, 44 subtasks). Spec chain at docs/specs/id-35-ledger-cli/ (incl. the nested ledger-cli-v2/ reorg). This builds on, does not replace, that surface.

---

1. Context — code-intelligence orientation

Per the “Always Do” section in .gitnexus/CLAUDE.md, orientation was run before authoring.

gitnexus_query({query: "ledger scoped serialise write record"}) — returned NO ledger / serialise execution flows. The top-ranked processes were all unrelated app-route flows (LibraryContent → UseUrlFilters priority 0.079; ContentOwnerManagement → CreateClient 0.076; NewItemPage → Cn 0.069). Verdict: greenfield surface for the execution-flow graph. This is expected and not a gap: scripts/ledger-cli.ts and lib/ledger/*.ts are a Bun CLI + vendored primitives, not part of the indexed Next.js app execution-flow corpus (GitNexus indexes 300 app processes; the ledger CLI is a standalone script).

gitnexus_context({name: "scopedSerialise"}) → Symbol 'scopedSerialise' not found. gitnexus_context({name: "escapeSerialise"}) → Symbol 'escapeSerialise' not found. Both confirm the ledger primitives are outside the GitNexus symbol index (45 952 symbols, none in lib/ledger/). ccc fallback was not needed — the domain is small, fully read directly (six lib/ledger/*.ts files + scripts/ledger-cli.ts, 3 573 lines), and the call-graph is linear (CLI dispatch → commitMutation → serialise/scopedSerialise → atomicWriteFile). Citing this directly is more accurate than a graph query the index does not cover.

The authoritative call-graph (read directly):

ledger-cli dispatch (scripts/ledger-cli.ts)
  ├─ field-edit cmds (flip-task, flip-subtask, update-task, update-subtask, update-roadmap,
  │    update-backlog, append-journal)
  │     → fieldPatchMutation (applyPatches, lib/ledger/patch-apply.ts)
  │     → commitMutation (scripts/ledger-cli.ts:~1640)
  │           ├─ scoped && scopedWrite → scopedSerialise(originalText, patch)   ← MINIMAL DIFF
  │           └─ else                  → serialise(detected) = escapeSerialise(detected.data)  ← WHOLE-FILE
  │     → atomicWriteFile (lib/ledger/atomic-write.ts)
  ├─ create cmds (add-subtask, open-task, create-theme, create-backlog)
  │     → insertRecord (lib/ledger/record-mutate.ts) / fieldPatchMutation
  │     → commitMutation WITHOUT scopedWrite → serialise(detected)             ← WHOLE-FILE ALWAYS
  └─ promote (cross-ledger)
        → parseJsonArg (positional only — NO readRecordInput)
        → insertRecord + removeRecord → stageAtomicWrite × 2-3 → commitStagedWrite  ← WHOLE-FILE

---

2. The headline problem (#5) — empirically re-characterised

The brief cites the S276 Curator finding (live in task-list.json Task 48’s {48.15} journal, line ~4993):

“bun scripts/ledger-cli.ts append-journal/flip-subtask --scoped re-serialises task-list.json with \uXXXX universal escapes across the whole file (~915-line spurious diff), defeating --scoped’s stated byte-preservation guarantee. Confirmed live this session — reverted CLI write + manual Edit applied.”

That finding is no longer reproducible. I re-verified empirically against the current on-disk ledgers (29/05/2026 — see §8 Verification):

Probe	Result
escapeSerialise(JSON.parse(text)) no-op round-trip vs on-disk, all 3 ledgers	BYTE-IDENTICAL (0 lines)
flip a single subtask/task field — scoped path	1 changed line
flip a single field — whole-file path	8 changed lines
add-subtask (insert a record) — whole-file path (the only path it has)	~5 910 changed lines of 6 393 total

So the OQ-LS-2 (S270) normalisation held: the on-disk ledgers are now byte-stable under the plain-parse escape-serialise the scoped path uses. The S276 915-line diff was a transient pre-normalisation artefact, or a --scoped-flag-omitted whole-file write. The byte-preservation primitive itself is sound today. This materially changes the problem statement.

The real, current defect is a COVERAGE GAP, not a broken primitive:

1. Record-creating commands have no scoped path at all. add-subtask / open-task / create-theme / create-backlog / promote all fall through to serialise(detected) = escapeSerialise(detected.data) (scripts/ledger-cli.ts:1074-1075). detected.data is the Zod-reparsed document; insertRecord (lib/ledger/record-mutate.ts:115-157) structuredClones and re-.parse()s the WHOLE ledger. Re-emitting the Zod-canonical doc both reorders every record’s keys into schema-declared order AND shifts every record after the insertion point. One add-subtask ≈ a whole-file diff. This is the catastrophic case Liam is reacting to.

2. The scoped path structurally cannot handle inserts. scopedSerialise(originalText, patch) (lib/ledger/scoped-serialise.ts:190-224) takes a SINGLE FieldPatch and mutates ONE leaf in the plain-parsed original. It has no record-insert/remove mode — it walkTaskLists to a {container, key} leaf and assigns patch.newValue. add-subtask already tries to ride the field-patch path with fieldPath: ['tasks', taskId, 'subtasks'], newValue: nextSubtasks (scripts/ledger-cli.ts:2397-2400) but then commitMutation is called WITHOUT a scopedWrite descriptor, so it whole-files anyway. Even if --scoped were threaded, re-emitting the entire subtasks[] array of a large Task (some have 40+ subtasks) is still a wide diff.

3. Why the divergence persists for the Zod path: escapeSerialise(detectSchema(text).data) already diverges from on-disk by 7 lines on a no-op round-trip (§8). The on-disk ledger carries 7 lines of key-order / default drift vs Zod-canonical. Any whole-file write bakes that in document-wide; the scoped path (plain-parse) never touches it.

The cmux-fleet race (S276 lost-journal risk) is the second-order driver. Four cmux terminals share docs/reference/task-list.json. A whole-file write is a large diff that collides on cherry-pick / git-apply across sibling worktrees; a single concurrent write can clobber another terminal’s just-appended journal block. Minimal-diff writes shrink the collision surface but do not eliminate the shared-mutable-file hazard — that is the real argument for the storage rethink.

---

3. Storage / diff approaches — options + trade-offs

The brief asks for the most logical approach, open to per-record files or an out-of-repo store — not just “add --scoped everywhere”. Four options evaluated.

Option A — Extend scoped-serialise to all create/flip commands (in-place)

Add a record-insert/remove mode to scopedSerialise (splice the new record’s JSON text into the plain-parsed-original at the collection’s end, preserving every untouched record’s bytes), and thread scoped: true + a scopedWrite descriptor through add-subtask / open-task / create-theme / create-backlog / promote. Make --scoped the default (with --whole-file as the escape hatch), since the on-disk ledgers are now byte-stable.

Option B — Per-record file storage (one file per Task) + build/assemble step

Split task-list.json into docs/reference/tasks/ID-N.json (one file per Task), with a build step assembling the canonical aggregate. A single-record edit touches exactly one small file.

Option C — Out-of-repo store (sibling dir) + change-only merge into repo

Keep the live ledger in a sibling/out-of-repo store; merge only changed records back into the committed repo copy on demand.

Option D — Status quo (whole-file everywhere)

Do nothing on storage; accept whole-file diffs on creates.

Trade-off matrix

Criterion	A: extend scoped	B: per-record files	C: out-of-repo store	D: status quo
Single-record edit diff	1 line (field) / ~record-sized (insert)	1 file, tiny	tiny (merge step)	~whole-file on insert
Eliminates cmux-fleet collision	Reduces, not eliminates (still one shared file)	Yes (different files = no git collision)	Yes (out of git path)	No
task-view editor compat (lib/ledger/ vendored detect-schema + scoped-serialise; task-view reads ONE JSON per ledger)	No change — same single-file schema	BREAKS — task-view expects one task-list.json; needs an assembled view + re-vendor	BREAKS unless the merged repo copy stays canonical (assemble-on-read)	No change
Mirror regen (scripts/regen-mirrors.sh, ledger-mirror-parity CI; mirrors docs/reference/{tasks,roadmap,backlog}/ID-N.md ALREADY exist)	No change	Build step must feed task-view’s mirror-generator.ts an assembled doc; double source-of-truth risk	Same as B for the committed copy	No change
CI guards (task-view-vendor-drift.yml watches lib/ledger/{atomic-write,detect-schema,patch-apply,record-mutate}.ts + lib/validation/*; doc-freshness.test.ts)	scoped-serialise.ts is KH-authored, NOT vendor-watched (lib/ledger/README.md:42-47) — free to extend	New assembler is KH-authored (no drift weight) but the per-file schema diverges from vendored single-file schema	Same	None
Implementation cost	Low-medium — one new mode + 5 call-site threads + tests	High — new storage layout, assembler, migration of live ledger, task-view re-vendor, CI rework	High — out-of-repo lifecycle, merge tooling, two-copy consistency	Zero
Reversibility	High (flag-gated)	Low (layout migration)	Medium	n/a
Atomicity / crash-safety	Preserved (atomicWriteFile / staged-write unchanged)	Per-file atomic; aggregate consistency is new surface	New surface	Preserved

Recommendation

Option A (extend scoped-serialise; make minimal-diff the default), with a deliberate decision to NOT pursue B/C now. Rationale, grounded in the code read:

1. The primitive already works — on-disk ledgers are byte-stable under the scoped path TODAY (§8). The defect is purely that creates/promote never call it. Closing the coverage gap is low-cost and directly kills the 5 910-line add-subtask diff.
2. B and C both BREAK task-view editor compatibility. task-view (task-view <ledger.json>, per CLAUDE.md) and the vendored detect-schema + mirror-generator assume ONE JSON file per ledger keyed by document_name. Per-record files require an assembled view + a task-view re-vendor and a second source-of-truth (per-file vs aggregate), exactly the drift the task-view-vendor-drift.yml guard exists to prevent. The cost is disproportionate to the token-efficiency goal.
3. The mirror system already gives 90% of B’s benefit. docs/reference/tasks/ID-N.md per-record mirrors already exist (54 files) and are CI-parity-gated. They provide per-record readability / diff-locality for humans WITHOUT splitting the canonical store. The canonical JSON stays single, minimal-diff-written.
4. The cmux-fleet collision is better addressed by minimal diffs + the existing record-set gate ({35.16}, rejects silent drop/dup on the bytes about to be written) than by a storage migration. A 1-line scoped insert almost never collides on cherry-pick; a 5 910-line whole-file write almost always does.

Residual risk Option A does not fully solve: two terminals editing the SAME record’s SAME field concurrently still race (last-writer-wins on a 1-line diff). If Liam judges that race material, the out-of-repo/per-record direction (B/C) is the only structural fix — surfaced as OQ-1.

Implementation shape for A (for the {65.3} TECH spec to detail, NOT to build here):

• New scopedSerialise insert/remove mode operating on plain-parsed-original (splice record text; preserve untouched bytes); Zod-validate the mutated doc before emitting (mirror the existing scopedSerialise validate-before-emit contract, lib/ledger/scoped-serialise.ts:215-221).
• Thread scoped default-true + scopedWrite through add-subtask / open-task / create-theme / create-backlog and the promote staged-write content derivation (scripts/ledger-cli.ts:3112-3115).
• Keep escapeSerialise(detected.data) as the --whole-file fallback + a one-shot re-normalise tool (precedent: scripts/ledger-normalise-oqls2.ts) to clear the 7-line residual drift.
• scoped-serialise.ts stays KH-authored / NOT vendor-watched — extending it carries no task-view-vendor-drift.yml weight.

---

4. Friction #4 — programmatic bulk-create-subtasks-from-PLAN + char budgets

Bulk-create today: absent. There is NO batch/bulk add path in scripts/ledger-cli.ts (grep for bulk-add / add-subtasks / --batch returns nothing). The {N.4} PLAN flow hand-crafts each add-subtask call; each is a separate whole-file write (so the diff cost compounds with #5).

Proposal: a add-subtasks <taskId> --file <plan-or-json> command that parses a PLAN.md’s TM-shape Subtask records (the {N.5+} blocks the planning-and-task-breakdown skill emits as JSON) and batch-inserts them in one write. Two viable input shapes (OQ-3):

• JSON array (cleanest): the Planner already returns Subtask records as TM-shape JSON; feed that array directly. No PLAN.md markdown parsing needed.
• PLAN.md markdown parse (richer but brittle): extract the SUBTASK RECORDS JSON block from a PLAN.md. Markdown parsing is fragile; prefer the JSON-array path and have the Planner/Orchestrator emit a sidecar .json.

Batch insert composes naturally with Option A: one scoped multi-record splice = one minimal write, one record-set-gate check (delta +N), one mirror regen. Reuse the existing readRecordInput --file/stdin resolver (scripts/ledger-cli.ts:394-441) and the {35.21} auto-id + {35.28}/{35.29} type-coercion logic per record.

Char budgets — review. Current budgets (lib/validation/ledger-budgets.ts, docs/reference/task-list-discipline.md §2):

Field	Budget	Class
task.description	1500	soft-warn + CLI hard-reject (unless --force)
task.status_note	300	soft-warn + reject
subtask.description	250	soft-warn + reject
subtask.testStrategy	300	soft-warn + reject
subtask.details	none	the append-only journal / dispatch-brief catch-all
theme.description / theme.notes	1500 / 300	soft-warn + reject
item.title / item.description	80 / 500	soft-warn + reject

Key facts grounding the budget question:

• Budgets are plain data, never Zod .max() (ledger-budgets.ts header) — raising a number is a one-line change with no schema/vendor-drift impact. The schema stays cap-free so the live ledger always parses.
• subtask.details is deliberately unbudgeted — it is THE home for load-bearing narrative (dispatch brief + <info added on …> journal). The discipline doc §2 is explicit: “When in doubt about which field carries narrative content, the answer is the Subtask details journal block.”
• The reported truncation pain is subtask.description (≤250) and task.description (≤1500) clipping content that is genuinely load-bearing. Per the discipline model, the CORRECT fix is usually “move it to details (unbudgeted) + a cross_doc_links pointer”, NOT raise the cap — raising caps re-opens the ID-34 drift the budgets were authored to close.

Recommendation: Do NOT reflexively raise budgets. Default to the discipline pattern (relocate to details + pointer). BUT surface specific raises as OQ-2 where the budgeted field is structurally the right home and the cap is merely conservative (candidate: subtask.description 250→400; item.title 80→120). Any raise is a one-liner in ledger-budgets.ts — cheap and reversible. Liam’s call.

---

5. Friction #6 — promote --file/stdin

Smallest item; confirmed. promote is the ONLY create-side command that is positional-JSON-only:

• Dispatch: case 'promote' reads const [backlogId, taskJson] = p and passes taskJson: string straight to promote(...) (scripts/ledger-cli.ts:2814-2833).
• promote() calls parseJsonArg('promote', taskJson) (scripts/ledger-cli.ts:2891) — NOT the readRecordInput resolver that gives add-subtask (:2312) and open-task (:2630) their --file / stdin / named-flag input.

Fix: route promote’s task body through readRecordInput (the {35.15} resolver, scripts/ledger-cli.ts:394) so it accepts positional JSON | --file <path> (- = stdin) | named flags, exactly like open-task. backlogId stays positional. Trivial; no primitive change.

---

6. CI / compatibility guard summary (must survive any change)

• task-view-vendor-drift.yml watches lib/ledger/{atomic-write,detect-schema,patch-apply,record-mutate}.ts
  • lib/validation/{task-list,roadmap,backlog}-schema.ts + work-status.ts (paths confirmed in the workflow, lines 39-48). scoped-serialise.ts and ledger-budgets.ts are KH-authored and explicitly NOT on this list (lib/ledger/README.md:42-47) → Option A + budget raises carry zero vendor-drift weight. Do NOT touch the four vendored primitives’ bodies.
• ledger-mirror-parity (ci.yml job ~920) regenerates docs/reference/{tasks,roadmap,backlog}/ID-N.md via task-view --check @ TASK_VIEW_TAG and gates on git diff --exit-code. Any write change must keep scripts/regen-mirrors.sh producing identical mirrors → another reason to keep the single-file canonical store (Option A) rather than split it (B/C).
• doc-freshness.test.ts runs on every test (a “guard test that breaks on structural changes” per CLAUDE.md Gotchas) — verify it doesn’t assert ledger path/shape if the storage layout ever changes.

---

7. Recommended scope for the ID-65 spec chain (for the Orchestrator)

Decompose ID-65 into three coherent slices, all building on the single-file canonical store:

1. Scoped-write coverage (#5) — extend scopedSerialise with record insert/remove; thread default---scoped through all create commands + promote’s staged write; --whole-file escape hatch + re-normalise tool. (Headline; highest token-efficiency payoff.)
2. Bulk-create + budget review (#4) — add-subtasks --file (JSON-array input preferred); budget-raise decisions per OQ-2. Composes with slice 1’s multi-record splice.
3. promote --file/stdin (#6) — route through readRecordInput. Small; can fold into slice 1’s PR or stand alone.

Estimated effort: medium (2-4h) for slice 1, low for slices 2-3. {65.4} PLAN warranted (chain-dependent: slice 2 depends on slice 1’s splice primitive; slice 3 independent).

---

8. Verification (pre-ratification empirical checks)

Date: 29/05/2026 (S280). Pinned: TypeScript via repo tsconfig; bun v1.3.4. All probes run against the live on-disk ledgers via throwaway scripts importing @/lib/ledger/* (removed after run).

Check	Symbol / path	Result
escapeSerialise(JSON.parse(text)) no-op round-trip, all 3 ledgers	lib/ledger/scoped-serialise.ts:75	BYTE-IDENTICAL (0 lines) — PRESENT, behaviour matches contract
escapeSerialise(detectSchema(text).data) no-op round-trip vs on-disk	scoped-serialise.ts:75 + detect-schema.ts:52	7 lines divergent — Zod-canonical drift confirmed
Single-field flip — scoped path	scopedSerialise scoped-serialise.ts:190	1 changed line
Single-field flip — whole-file path	serialise scripts/ledger-cli.ts:1074	8 changed lines
add-subtask insert — whole-file (only path)	insertRecord record-mutate.ts:115 + serialise	~5 910 / 6 393 lines — confirms the headline defect is the CREATE path
promote input mode	scripts/ledger-cli.ts:2891	positional-only (parseJsonArg) — --file ABSENT, confirms #6
Bulk-create command exists?	grep scripts/ledger-cli.ts	ABSENT — confirms #4 is greenfield
S276 ~915-line scoped diff reproducible?	live ledgers today	NOT reproducible — OQ-LS-2/S270 normalisation held

No external-library API citations requiring import-and-call verification (all symbols are internal KH lib/ledger/* / scripts/ledger-cli.ts; Zod is a pinned framework dep used via existing schemas).

---

9. OQs for Liam (ratification before {65.2} PRODUCT)

1. Storage direction. The byte-preservation primitive is sound today; the defect is a coverage gap (creates/promote never call scoped). I recommend Option A (extend scoped + make minimal-diff the default) and explicitly NOT splitting the store (B/C break task-view editor compat + the mirror parity model). The residual risk A leaves is two terminals editing the SAME record’s SAME field concurrently (last-writer-wins). Is that race material enough to justify the per-record-file / out-of-repo cost? Default: NO — ship A.

2. Char budgets. Default position: do NOT reflexively raise caps; relocate load-bearing content to the unbudgeted details + cross_doc_links pointer (the ID-34 discipline). Candidate raises if you disagree (one-line, zero vendor-drift): subtask.description 250→400, item.title 80→120. Raise any budgets, and to what? Default: NO raises; rely on details.

3. Bulk-create input shape. JSON array (Planner emits TM-shape Subtask JSON directly) vs PLAN.md markdown parse (extract the SUBTASK RECORDS block). I recommend JSON-array via add-subtasks --file (markdown parsing is brittle). Confirm JSON-array as canonical input?

4. Default flip. Should --scoped become the DEFAULT for all mutating commands (with --whole-file as the opt-out), given on-disk ledgers are now byte-stable? Default: YES — minimal-diff by default.

5. Re-normalise tool. A no-op escapeSerialise(detectSchema(...)) already diverges by 7 lines. Want a one-shot re-normalise (precedent: scripts/ledger-normalise-oqls2.ts) to clear that residual so even a forced whole-file write is minimal? Default: YES, low-cost.

6. promote --file scope. Fold #6 into the slice-1 PR, or ship as its own tiny Subtask? Default: fold into slice 1.