V1 ledger-mutation plan (S299 carry-across)

V1 ledger-mutation plan (S299 carry-across)

One-shot worklist for a dedicated ledger-cli session. S299 was research-only and wrote no ledgers. This doc carries the S299 investigation across so the work can be enacted without re-litigating it. Companion to the ordering board v1-completion-sequence.md.

Provenance: S299 — 6 parallel investigation agents + 1 bl-165 options agent + Liam feedback. Per-item evidence (file:line) in /tmp/claude/s299-research/cluster{1-6}-*.md + bl165-options.md (transient — the load-bearing facts are inlined below).

Executor: a general-purpose sub-agent driving bun scripts/ledger-cli.ts. Verify each --help before writing. Do not hand-edit the .md ledger mirrors — canonical is the JSON.

---

0. Process model (resolves Liam’s “update-then-promote vs promote-then-record” question)

• promote authors the new Task body at promotion time (promote <bl> --file <taskJson> / --title … / --capability-theme <id>). So fix stale content AT promotion — never pre-edit the backlog item then promote (double-handling). Same for add-subtask: author corrected content directly.
• “Close” = delete-backlog <id>. The backlog status enum has no terminal value (spec_needed|needs_research|parked|ready|blocked), so resolved/done/promoted items are removed (bl-185 precedent). promote removes the source item atomically; for add-subtask folds, follow with delete-backlog.
• Do NOT add implementation subtasks to a spec_needed Task (ID-57/58/59/60/61). Those tasks’ spec chains own subtask creation. Instead fold-in register: keep the backlog item, point its cross_doc_links at the target Task, and the planner consumes it at spec time. Add subtasks only to already-decomposed/in-flight Tasks (ID-64, ID-52).

CLI verbs used below: promote, add-subtask, add-subtasks --file, update-backlog, delete-backlog, update-task.

---

1. Master disposition table

Item	Verdict (S299)	Operation	Target
bl-158 entity_resolution zero	RESOLVED (wired stage_5.py:209→flow.py:2297, ID-53.14)	delete-backlog + record finding	—
bl-177 ContextKey clash	RESOLVED (ID-67 killed dual-path; repro 30 pass/0 clash)	delete-backlog + record finding	—
bl-216 deploy folder consolidation	DONE (implemented S299)	delete-backlog	—
bl-189 content_items col cleanup	promote+fold (pre-re-ingest)	add-subtasks ×2 → then delete-backlog	ID-64 {64.12}+{64.13}
bl-204 Path-C per-row failures	re-scope + merge TECH §10	add-subtask → then delete-backlog	ID-52 {52.22}
bl-27 opaque-Json RPC	keep scope; after-cutover	promote --file (new Task)	new Task (deps {64.8})
bl-135 scheduled jobs	dissolve (pre-pivot) — do NOT promote	delete-backlog; md-sync→ID-46 scope (retire-target), eval→ID-61 register	ID-46 + ID-61
bl-162 num_reprocesses	defer-v2	update-backlog notes + deferral-register row	keep backlog (v2)
bl-188 batch_reclassify	keep backlogged (Liam)	update-backlog notes (clarify)	keep backlog
bl-165 pydantic error-class	research complete → ready	update-backlog status ready + register	ID-61 fold-in
bl-166 PipelineRunStatus cast	do-now quick-fix	keep ready + register	ID-61 fold-in
bl-169 error_class runbook	rewrite Cloud Run→Coolify	update-backlog (fix text) + register	ID-61 fold-in
bl-176 operation-value case	tiny align (dormant)	keep + register	ID-61 fold-in
bl-186 mcp-tooling-audit spec	EXPAND → dedicated MCP Task (Liam reviewed 54 tools)	promote --file (new Task)	new MCP Task (lifts T13 MCP sub-part)
bl-70 bids.ts defer-rename	form_type-shape gated	register	ID-61 fold-in
bl-76 ts_rank measurement	rewrite (drop Cloud Run)	update-backlog (fix text) + register	ID-57 fold-in
bl-74 q_a_pair_history extend	keep	register	ID-58 + ID-60 fold-in
bl-136 Q&A copy-tracking UI	keep (needs effort est.)	register	ID-58 fold-in
bl-127 entity merge/split E2E	keep (dedup vs Stage-5)	register	ID-60 fold-in
bl-75 T6 WP3 test hardening	keep (3 nits)	register	ID-41/T6 fold-in

Items NOT in the S299 scope (bl-108/109/119 etc. from §5) keep their existing fold-targets — see canonical-pipeline-sequencing.md §5; not re-adjudicated here.

---

2. Authored content for promotes / add-subtasks

ID-64 ← bl-189 (two subtasks; pre-re-ingest schema cleanup)

add-subtasks 64 --file with this array (then delete-backlog 189):

• {64.12} — Drop 5 AI-telemetry token columns + rewrite item-provenance consumer. details: Drop content_items.classification_tokens_in, classification_tokens_out, classification_cache_creation_tokens, classification_cache_read_tokens, embedding_tokens (all at migration 20260416102457; 0 writers anywhere incl. cocoindex flow.py; sole reader = lib/provenance/item-provenance.ts:129-134). Rewrite that consumer: strip the 5 token cols + the estimateClassifyCost/estimateEmbedCost calls (:164-179) + the ./pricing import + the ItemProvenanceResponse token fields (:54-60); KEEP classification_model/embedding_model + their resolution (:147-161). Update __tests__/lib/provenance/item-provenance.test.ts + __tests__/migrations/classification-disputes-rls.test.ts:328-393. GitNexus impact FIRST. testStrategy: provenance API/UI tests green; no remaining refs to the 5 cols (ast-dataflow column-reads).
• {64.13} — Drop 4 orphan content_items columns + reader rewrites. details: Drop notes, parent_id, source_bid, source_document (migration 20260416102457:586-618). Pre-reqs: drop content_items_source_bid_fkey + the self-FK on parent_id (re-added 20260601180102:22-23) FIRST. source_document has ~8 display readers → rewrite to source_file ?? source_url (values already migrated by 20260428131822). notes/parent_id = 0 reader/0 writer. GitNexus impact FIRST. testStrategy: build + display-surface tests green; ast-dataflow confirms 0 reads of the 4 cols.
• NOT now: Part C content_items.summary_data stays ID-56-gated (NEW-OQ-B unruled — cocoindex does not write it yet). Record a one-line note on ID-56 to rule summary_data at its extraction-contract close.

ID-52 ← bl-204 (one subtask; re-scoped) — then delete-backlog 204

• {52.22} — Path-C catalogue idempotent re-run (merge of TECH §10 follow-up). details: Path C is an app-side service-role Supabase write (lib/catalogue/from-instance.ts:401-403), NOT a cocoindex-pipeline write — the cocoindex write-model does not govern it. {52.14} already shipped the per-row-resilient write + user-facing failure surface (CLI N written/declined/failed + non-zero exit; tested incl. 23505 dup __tests__/lib/catalogue/from-instance.test.ts:214). The real remaining gap = re-run safety: a plain .insert() re-raises 23505 on every prior row. Fix: switch to .upsert(..., { onConflict: 'template_name,template_version,section_ref,question_number' }) (the live UNIQUE form_template_requirements_unique_section). Fix the stale spec text claiming a “single-transaction wrapper” (TECH.md:789, PLAN.md:577) — the decision is per-row, de facto. testStrategy: re-cataloguing the same instance is a no-op (no 23505); upsert unit test. re-ingest relation: parallel-ok; live exercise after the Path-B form-smoke (Spine S1). After {52.22} lands, ID-52 can close (all other subtasks done).

New Task ← bl-27 (promote; after-cutover)

promote 27 --file <taskJson> — corrected body:

• title: “OQ-R9 opaque-Json RPC migration — Tier 1 + Tier 2 (RETURNS Json → RETURNS TABLE)”.
• description: Migrate 5 RPCs whose return type resolves to opaque Json, losing call-site type safety: Tier 1 get_user_tag_counts, get_workspace_counts, merge_entities; Tier 2 get_dashboard_attention_counts, get_filter_counts (per docs/specs/id-50-ast-dataflow-tool/investigations/R-WP12-opaque-json-rpcs.md:541-547). NONE are on the re-ingest read-path (no search_content_chunks/q_a_search/vector). It is a signature, not a data, change → schedule AFTER the {64.8} cutover gen types (one clean regen; doing it before forces a double regen). merge_entities also writes entity tables → safer on a stable post-cutover corpus. Re-derive line refs via bun scripts/audit-opaque-json-rpcs.ts at spec time (the cutover types-regen shifts them).
• dependencies: [64] (the {64.8} cutover). priority: medium. effort: ~7h. capability-theme: (ast-dataflow / pipeline — Liam’s call).
• After promote, this is a Lane-C (after-cutover) Task in the ordering board.

New Task ← bl-186 (promote; MCP tooling rationalisation) — Liam-driven

promote 186 --file <taskJson> — corrected/expanded body:

• title: “MCP tooling rationalisation — audit + consolidate + update + extend the 54-tool surface”.
• description: Audit the full MCP tool surface (54 tools — likely more than needed). Per Liam’s completed review, the work splits four ways: (a) update tools for terminology drift (bid→procurement etc.); (b) remove tools no longer required; (c) integrate/consolidate overlapping tools; (d) add new tooling where there are gaps. RESEARCH-first — {N.1} is seeded by Liam’s review notes (Liam to hand over). This absorbs the old bl-186 mcp-tooling-audit spec pair (this Task is that spec, expanded) and lifts the MCP-audit sub-part OUT of ID-61/T13 (→ also update-task 61 description to drop it). The MCP surface is the AI-consumption layer — parallel pre-launch, NOT a re-ingest gate. Spec chain {N.1 RESEARCH}→{N.2 PRODUCT}→{N.3 TECH}→{N.4 PLAN}.
• dependencies: []. priority: should. capability-theme: mcp-tooling (or Liam’s call). effort: TBD at RESEARCH.

---

3. Fold-in registers (keep in backlog; set cross_doc_links → target; fix stale text)

For each, update-backlog <id> cross_doc_links to point at the target Task, and a one-line note “S299: research carried across; consume at spec time.” Fix stale text where flagged.

• → ID-61/T13 (pre-launch ops + observability — the absorber): bl-165 (set status ready; option B+D recommended ~45-60min — wire the already-built classify_pydantic_error() extraction.py:474-483 + redact the leak, see §7), bl-166 (ready quick-fix; cast at record/route.ts:145), bl-169 (rewrite: Cloud Run→Coolify/{66.15}), bl-176 (align dormant _emit_upsert_log literal INSERT/UPDATE→upsert), bl-70, bl-135 eval-half (weekly-eval schedule — or a 1-line GH Actions schedule:). (The MCP-audit sub-part lifts OUT of ID-61/T13 into the new MCP Task — see §2; bl-186 promotes there, not here.)
• → ID-57/T10: bl-76 (fix text: drop “Cloud Run sidecar deploy”; dep = post-T7 corpus).
• → ID-58/T11: bl-74 (q_a_pair_history superseded_by+source_workspace_id), bl-136 (copy-tracking UI; add effort est.).
• → ID-60/T12: bl-74 (also), bl-127 (entity merge/split E2E — dedup against Stage-5/ID-53 scope first).
• → ID-41/T6: bl-75 (3 checker nits).

When ID-61 / ID-57 / ID-58 / ID-60 are specced, the planner pulls these registered items into the spec chain as {N.x} subtasks. The ordering board already lists the target IDs for parallel-work clarity.

---

4. Task status flips

• update-task 53 status done — all 16 ID-53 subtasks done + stage_5.py live. This unblocks ID-60’s Inv-19 gate (a). (Also fix id-53/PRODUCT.md Inv-19 stale “no T12 task exists” — T12 = ID-60.)
• ID-52 → done AFTER {52.22} (bl-204) lands — not yet (it hosts the new subtask).
• Leave ID-64 in_progress (hosts {64.7}/{64.8}/{64.12}/{64.13}).

---

5. Doc edits (non-ledger — can be done with the ledger session or separately)

• deferral-register.md §2 — add a V2 row for bl-162 (cocoindex num_reprocesses: unreachable from KH webhook emission without refactor; different axis from _FlowRetryCounter — revisit if engine API changes). Per Liam.
• 07-collapse-list.md — mark §5.1 (regenerateChunks/chunker) + §6 (analyse/route.ts) RETIRED-SINCE ({56.11}/{52.15}); fix the “Docling Cloud Run sidecar” ×3 framing (§3.2/§7/§11.3 — Cloud Run torn down, now IONOS/Coolify + trafilatura); note jsdom retained by ontology, pdfplumber retained by ID-52.
• PLAN.md (docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md) §4.12 + bl-74 notes — fix “ID-24” → “ID-74” (renumbered S61; current ID-24 is unrelated).
• canonical-pipeline-sequencing.md — add a one-line banner at the top: “Superseded as forward-map by v1-completion-sequence.md (S299); retained for history + architecture (§0-§6) + the write-model.” Fix §2.3 lines 367/372 (“Paths B and C UNBUILT” → built; Path-B not yet live-smoked).
• canonical-pipeline-sequencing.md §5 — strike the bl-185 row (promoted → ID-67 done); re-home the orphaned fold-targets (ID-49 done): bl-158→(closed), bl-162→(v2), bl-177→(closed).

---

6. Open Liam decisions before/within the ledger session

• bl-27 capability-theme for the new Task (ast-dataflow-tooling vs pipeline).
• bl-189 home: ID-64 subtasks (recommended — keeps schema-finalisation coherent) or a small standalone Task?
• bl-135 eval-schedule: do we even want a scheduled weekly MCP-eval pre-launch, and does the per-tenant×2-project fan-out collapse to single-host for single-tenant V1? (markdown-sync half is settled: dissolve into ID-46.)
• MCP Task (from bl-186): capability-theme; Liam to hand over his 54-tool review notes to seed {N.1} RESEARCH.
• ID-61/T13: spec it next? (It absorbs the most fold-ins — speccing it materialises bl-165/166/169/176/135-eval as subtasks; the MCP-audit sub-part is removed from its scope, now owned by the MCP Task.)

---

7. S299 findings to preserve (so they are not re-litigated)

• RESOLVED — close with these notes: bl-158 (entity_resolution count wired stage_5.py:209→flow.py:2297 via ID-53.14 — legitimately 0 only when nothing needed reconciliation); bl-177 (ID-67 canonicalised scripts.cocoindex_pipeline.* + added collection guard conftest.py:279; adverse-order repro = 30 pass / 0 ContextKey kh_pipeline_db clash).
• bl-176 ≠ bl-177 (Liam asked): different root cause/file/fix — a dormant-helper case-literal vs a pytest ContextKey-registry clash. Not duplicates.
• ⚠ SECURITY-ADJACENT (new S299 finding, relevant to ID-68 de-ID): the terminal failure webhook writes str(exc) un-redacted to pipeline_runs.error_message (flow.py:2255→:2335→record-run.ts:212), echoing LLM-extracted client content verbatim — unlike the redacted manifest path (flow.py:2024). Fix = the ~5-line redaction in bl-165 Option D, or track as a standalone hardening item. Surface to Liam alongside the ID-68 public-flip work.
• bl-135 answer (Liam’s question): the markdown_batch handler is “a thin wrapper around orchestrateMarkdownBatch({ phase: 'import' })” = OLD-pipeline markdown import, superseded by cocoindex source-binding. The kh-nightly-markdown-sync-* jobs were never built and are not required under the new pipeline → retire with the OLD pipeline (ID-46). Weekly-eval is independent (already in ci.yml).
• ID-52 / ID-53 are effectively complete (all subtasks done) but sit in_progress — the ID-53 flip is load-bearing for ID-60.