Canonical-pipeline → task-list migration (Linear-style umbrella/tag model) — PRODUCT

Canonical-pipeline → task-list migration (Linear-style umbrella/tag model) — PRODUCT

Spec slug: canonical-pipeline-task-list-migration Version: v1.0 (draft, S63 WP2) Status: DRAFT — authored by task-planner under write-product-spec per ID-31.2 dispatch. Awaits Checker promotion of Subtask 31.2 in_progress → done. Source briefs: docs/research/canonical-pipeline-task-list-migration-approach.md (895 L, S62 W2); 7 OQ ratifications in task-list.json ID-31.1 details journal block (S62 W3, 2026-05-21). Predecessor: RESEARCH (ID-31.1, done S62 W2). Successor: TECH (ID-31.3, fresh Planner) → PLAN (ID-31.4) → implementation Subtasks (ID-31.5+).

Summary

Migrate the canonical-pipeline implementation plan (docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md, T0-T14) into docs/reference/task-list.json as top-level Tasks while PLAN.md remains the rendered spec substrate. The migration adds (a) a retrospective umbrella Task T0 covering ~15 pre-phase doc landings (S180-S241); (b) six retrospective Tasks for T1-T6 already shipped, opened in done status with provenance journal blocks; (c) a new ledger docs/reference/umbrellas.json codifying multi-Task strategic umbrellas (Linear Initiative analogue) with curated task_ids[] + substrate_doc per umbrella. The migration honours seven Liam ratifications batched S62 W3 — most notably the T0-umbrella retro Task (OQ-1 ALT, cascading T1-T14 IDs +1), the T8=ID-28 anchor with cross-branch collision flag (OQ-2), just-in-time forward Task expansion (OQ-3), separate-document umbrellas schema (OQ-4), manual cross-branch ID discipline (OQ-5), shipping-cadence Subtask granularity (OQ-6), and manual cross-doc-links sync (OQ-7).

Problem

Knowledge Hub’s canonical-pipeline plan is the load-bearing architectural work surface for Phase 1: 14 enumerated implementation Tasks (T1-T14) across spec authoring, schema migrations, code refactors, RPC additions, RLS-pattern enforcement, retrieval substrate, cocoindex pipeline scaffolding, and pre-launch operational gates. T1-T6 have already shipped retrospectively across sessions S238-S251 without ever being represented in task-list.json — they preceded the task-list / Tn structure entirely. The remaining T7-T14 are tracked only inside the 700-line PLAN.md document; there is no Task-level traceability surface for status, dependencies, dispatch, or per-Subtask journal entries. Simultaneously, the project has multiple concurrent strategic umbrellas (canonical-pipeline, production-readiness, intelligence-workspaces, sales-proposals, docs-foundation) and the current flat task-list cannot express membership of a Task in one or more of these umbrellas. Without a migration scheme + umbrella model, the canonical-pipeline plan stays invisible to the Task ledger and multi-stream initiatives cannot be queried, navigated, or rendered in the future Astro docs site (ID-9).

Goals / Non-goals

Goals

• Open one Task per Tn in task-list.json covering T0-T14 (15 Tasks total per OQ-1 ALT).
• Populate retrospective Tasks T0-T6 in done status with provenance journal blocks citing commits and session references.
• Introduce a new ledger docs/reference/umbrellas.json (Option C from RESEARCH §7) with task_ids[] arrays + substrate_doc field per umbrella entry.
• Preserve docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md as the canonical spec substrate; Task descriptions are short and reference PLAN.md sections via cross_doc_links.
• Maintain sibling-only Subtask dependencies (existing schema invariant) for retrospective Tasks; cross-Task references live at the Task level only.
• Make cross-branch ID coordination explicit (manual MAX-ID check before assignment).

Non-goals

• No PLAN.md auto-sync (β / γ from RESEARCH §5.2) — manual cross-doc-links only (OQ-7).
• No CI guard for cross-branch ID collision detection — manual orchestrator discipline (OQ-5).
• No upfront opening of T7-T14 Tasks in pending status — JIT-only (OQ-3).
• No Astro docs renderer for umbrellas in this Task — that’s a downstream follow-up Task to be opened at PLAN dispatch (per OQ-4 implementation followup note).
• No edit of docs/reference/umbrellas.json runtime tooling (update-roadmap-backlog skill or successor) — implementation lands in ID-31.5+.
• No edit of PLAN.md to add backlinks — substrate stays read-only here (manual cross-doc-links flow per OQ-7 default).
• No re-decomposition of T1-T6 from scratch — retrospective Subtasks reflect actual shipping cadence (OQ-6 ratification).

Ratifications → invariant mapping

The seven OQ ratifications from S62 W3 map one-to-one onto Behavior invariants below. This is the load-bearing trace so a Checker can verify spec compliance per-ratification.

Ratification	Decision summary	Honoured by invariants
OQ-1 ALT	Open T0 umbrella retro Task; cascade T1-T14 IDs +1	1, 2, 3, 4
OQ-2	Keep T8 = ID-28 anchor; flag cross-branch collision (resolved S62 W4)	5
OQ-3	Just-in-time forward Task expansion (T7-T14 opened only when actioned)	6
OQ-4	Separate umbrellas.json document, Option C	7, 8, 9
OQ-5	Manual cross-branch ID discipline (orchestrator MAX-ID check pre-open)	10
OQ-6	Shipping-cadence Subtask granularity for retro Tasks	11, 12
OQ-7	Manual cross-doc-links sync (PLAN.md ↔ Task descriptions, Option α)	13, 14

Invariants 15-17 carry KH-wide hygiene bars (UK English, status-machine compliance, schema-roundtrip correctness) that apply automatically but warrant explicit numbering for Checker verification.

Behavior

Retrospective Task population (T0-T6)

1. A retrospective umbrella Task T0 opens in task-list.json covering pre-phase investigation work spanning roughly S180-S241. T0’s title is “T0 — Phase 0.9 investigation substrate (architecture sub-docs + drift audit + ratification arc)”. Its status at open is done. Its description is a single paragraph linking to docs/specs/id-31-0.9-canonical-pipeline/{PRODUCT,TECH}.md and the four architecture sub-docs (02-data-flow.md, 04-workspace-types.md, 05-qa-flow.md, 06-mcp-tooling.md, 08-new-features.md) plus the drift audit + implementation- readiness audit + STILL-OPEN consolidation. T0 has between 4 and 8 Subtasks per shipping-cadence granularity (invariant 11). Each Subtask is done with an <info added on ...> journal block citing the relevant session references (S180- S241 inclusive) without requiring per-commit SHAs (pre-Tn work predates the commit- citing convention).

2. Six retrospective Tasks open for T1 through T6, one per Tn. Each retrospective Task opens in done status with: (a) a title prefixed T{n} — <topic> matching the PLAN.md §2 + §4.n header text; (b) a slim description (≤200 words) that links to PLAN.md §4.n via cross_doc_links and gives a one-sentence summary of what shipped; (c) Subtasks populated per shipping cadence (invariant 11) with <info added on ...> journal blocks per invariant 12; (d) the commit_refs[] field populated with load-bearing commit SHAs from the journal block; (e) the session_refs[] field populated with the relevant S2NN session identifiers.

3. Final Task ID assignment for T0-T14 locks at the {31.4} PLAN dispatch, not at this PRODUCT spec. The proposed mapping per RESEARCH §3.2 (T1=ID-30, T2=ID-31, …, T6=ID-35, T8=ID-28 anchored, T7=ID-36, T9=ID-37, …, T14=ID-42) was authored before OQ-1 ALT was ratified. With T0 added, the cascade is T0 = next-available-ID, T1 shifts to T0+1, …, T6 shifts to T0+6, T8 stays anchored at ID-28, T7 + T9-T14 shift to T0+8 onwards. The exact mapping is finalised during {31.4} PLAN as a table that takes into account ID space already consumed by S62 W2-W4 (Tasks 30, 31, 32, 33 already opened — see task-list.json HEAD) and any further Tasks opened in the interim.

4. Cascade documentation invariant: PRODUCT.md does not bake in specific IDs for T0-T14. PRODUCT.md documents the shape (T0 umbrella + T1-T6 retro done + T7 plus T9-T14 forward pending JIT + T8 anchored at ID-28) and the cascade rule (T0 occupies the next-available-ID slot; T1-T6 plus T7 + T9-T14 cascade sequentially around the T8=ID-28 anchor). The TECH spec specifies the exact resolution algorithm; the PLAN dispatch executes it and writes the final table.

5. T8 = ID-28 anchor invariant: Task ID-28 represents T8 (Cocoindex flow scaffolding) per OQ-2 ratification. The cross-branch collision flagged at OQ-2 ratification (production-readiness ID-28 = “Orchestrator-of-orchestrators polish”) was resolved S62 W4 via P-R rename ID-28 → ID-33 (commit 9e498e2f per HEAD git log). Therefore at PRODUCT acceptance time, ID-28 is unambiguously reserved for T8 on all three branches (main, production-readiness, content-items- investigation) and the migration may rely on this without further coordination.

Forward Task population (T7 + T9-T14)

6. Forward Tasks open just-in-time, not upfront. T7, T9, T10, T11, T12, T13, and T14 are not opened by this Task. They open only when each becomes the next-up work item to dispatch. Each forward Task uses the full spec chain at open: {N.1 RESEARCH → N.2 PRODUCT → N.3 TECH → N.4 PLAN → N.5+ implementation}. The opening orchestrator records the cross-doc-link to PLAN.md §4.n in the Task description
   • cross_doc_links fields per invariants 13-14. Until they open, T7 and T9-T14 remain visible only in PLAN.md §4.7 + §4.9-§4.14 — not in task-list.json. T8 = ID-28 is the sole exception (already in-flight on content-items- investigation per OQ-2; not opened by this migration but explicitly referenced).

umbrellas.json ledger shape

7. A new ledger docs/reference/umbrellas.json exists with this top-level shape:

   {
     "document_name": "umbrellas",
     "document_purpose": "Cross-Task strategic umbrellas (Linear Initiative analogue) — curated task_ids[] arrays + substrate_doc per umbrella for navigation, querying, and rendering.",
     "last_updated": "YYYY-MM-DD",
     "related_documents": ["docs/reference/task-list.json", "docs/reference/product-roadmap.json", "docs/reference/product-backlog.json"],
     "umbrellas": [ { ... } ]
   }

   The top-level structure mirrors task-list.json / product-backlog.json / product-roadmap.json so the same Astro renderer family (ID-9 + ID-20) can consume it. Field order is fixed: document_name, document_purpose, last_updated, related_documents, umbrellas.

8. Each entry in umbrellas[] has these required fields:

   • id — kebab-case stable string identifier (e.g. canonical-pipeline, production-readiness). Used as the URL slug for the Astro Initiative page (ID-9 follow-up).
   • title — human-readable display name (Title Case, e.g. “Canonical Pipeline Implementation”).
   • substrate_doc — relative path from repo root to the canonical substrate document (e.g. docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md). This is the document the Astro renderer surfaces alongside the Task list on the Initiative page. Exactly one substrate_doc per umbrella; if multiple substrate docs apply (e.g. spec + plan + research), the substrate_doc is the primary entry-point and the others are linked from inside it.
   • task_ids — array of Task ID strings (e.g. ["28", "30", "31", "32"]). Order matters for rendering (insertion order = display order; orchestrator curates per umbrella convention). Each entry must reference a real Task in task-list.json; broken references fail the cross-doc validation (see invariant 9). A Task may appear in multiple umbrellas (many-to-many).
   • status — one of proposed, in_progress, done, archived. Maps to the aggregate phase of the umbrella, not derived automatically from member Tasks.
   • phase — short string identifier for the project phase the umbrella belongs to (e.g. Phase 1, Phase 2, Phase 3).

9. umbrellas.json must round-trip with task-list.json: every Task ID listed in any umbrellas[].task_ids[] array references a real Task entry in task-list.json. Conversely, every Task in task-list.json is expected to appear in at least one umbrella (the orchestrator is responsible for assigning on Task open; absence flags a missed assignment but is not a hard error). This cross-doc consistency is verified by a TECH-spec’d integration test (not a CI guard — manual run + advisory only per OQ-5 spirit).

Cross-branch ID coordination

10. The orchestrator assigns NEW Task IDs by querying all active branches. Before opening any NEW Task on any branch, the opening session runs git show origin/<branch>:docs/reference/task-list.json against each active long-lived branch (currently main, production-readiness, content-items- investigation) to compute MAX_ID_ACROSS_BRANCHES, then assigns the new Task ID as MAX_ID_ACROSS_BRANCHES + 1. The discipline is documented inside the workflow-orchestration skill body (not a separate CI guard per OQ-5). If a collision is later detected at merge time (e.g. due to a concurrent open that bypassed the discipline), the rename pattern is: the later-merged Task gets renumbered, with a commit message of the shape chore(s{NN}-w{N}): rename ID-{old} → ID-{new} (cross-branch collision resolve) and a single commit spanning the task-list.json rename + any inbound cross_doc_links pointing at the old ID. The S62 W4 resolution of ID-28 (9e498e2f) is the worked example.

Retrospective Subtask granularity

11. Retrospective Subtasks match the actual shipping cadence per OQ-6 ratification — neither finer (per-commit) nor coarser (one Subtask per Task). The granularity rule:

    • If a Task shipped in one session via one or two WP packages, it gets 1-3 Subtasks per WP package (with commit SHAs grouped in the Subtask title or details field).
    • If a Task shipped across multiple sessions, each session-WP combination is a Subtask (e.g. T5 = 7 Subtasks across S248 + S251 W1B Phases A-G).
    • If a Task has natural WP groupings inside one session (e.g. T6 = WP1 schema
      • WP2 RPCs + WP3 test + WP4 docs), each WP is a Subtask.
    • Subtask count per Task stays under the 25-soft-ceiling (per existing schema invariant); T2 at 12 Subtasks and T5 at 7-8 Subtasks are the upper end of observed retro Tasks.

12. Each retrospective Subtask carries an <info added on YYYY-MM-DDTHH:MM:SS.sssZ> journal block appended to its details field at retro-Task open time (timestamp = the retro open session, not the original work session). Block contents:

    • One sentence opening: “RETROSPECTIVE OPENING — Task opened in done status post-implementation. Original work happened S{NN} per docs/continuation-prompts/<file>.md.”
    • A Commits (S{NN}, <branch>): block listing the load-bearing commit SHAs (8-character prefix) with their original commit message line. The commit_refs[] field at the Task level mirrors these SHAs without the message text for queryability.
    • Migration file references where applicable (e.g. supabase/migrations/ <timestamp>_<name>.sql).
    • A PLAN.md §4.n acceptance criteria all met. close, citing the PLAN section whose acceptance criteria the retro Task delivered against.
    • Any follow-up flags or known-gaps explicitly noted (e.g. T1’s deferred cocoindex-ledger-api → v1.1; T6’s anon-EXECUTE leak fix landing S250 W1b).

PLAN.md ↔ Task description sync (manual cross-doc-links)

13. PLAN.md stays as the canonical substrate. This Task does not edit PLAN.md. Task descriptions for T0-T14 are short (≤200 words for retro Tasks per invariant 2; ≤300 words for forward Tasks at JIT-open per invariant 6) and reference PLAN.md §4.n via the cross_doc_links[] field. Each entry in cross_doc_links[] is an object: { "type": "spec_substrate", "path": "docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md", "section": "§4.n" }. The full canonical scope, acceptance criteria, and dependencies live in PLAN.md §4.n; the Task description is a summary + pointer.

14. PLAN.md backlinks to Tasks are added manually at Task-open time, not auto-generated. When a forward Task opens (JIT per invariant 6), the opening session optionally appends a See Task ID-NN parenthetical to the PLAN.md §4.n header text (e.g. ### §4.7 T7 — Phew Q&A first-ingest (see Task ID-36)). This is a hygiene discipline, not a hard requirement; absence does not break the umbrellas.json → task-list.json round-trip (invariant 9). For retrospective Tasks T0-T6, backlinks may be added in a single batch commit or skipped — the PLAN.md DONE-S2NN annotations and the Task’s cross_doc_ links[] provide the cross-reference either way.

KH-wide hygiene bars

15. UK English throughout umbrellas.json, retrospective journal blocks, and Task descriptions: organisation, colour, behaviour, DD/MM/YYYY date format. Display titles (e.g. “Canonical Pipeline Implementation”) use Title Case but UK spelling. ISO 8601 timestamps in journal blocks (YYYY-MM-DDTHH:MM:SS.sssZ) are language-neutral and unaffected.

16. Status-machine compliance: This Task’s lifecycle (and every retro / forward Task opened by this migration) honours the existing §6.3 state machine — Planner sets the initial pending state at Subtask creation; Executors flip pending → in_progress at start of work; Checkers promote to done; Orchestrators set deferred / cancelled. Retro Tasks deliberately skip the pending → in_progress → done walk because their work landed pre-task-list-adoption — they open directly in done per the user’s S250 ratification (cited in 31.1 RESEARCH §3.3). The retro-open is the lone legitimate exception to the standard state-machine walk.

17. Schema-roundtrip correctness: umbrellas.json parses against the TECH-spec’d Zod schema (specified in {31.3} TECH). Any orchestrator edit that breaks schema validation fails the test suite. Editing convention: orchestrator updates task-list.json and umbrellas.json in the same commit when opening, renaming, or retiring a Task that affects umbrella membership; the commit message names both files.

Open questions

Five new open questions surface from PRODUCT authoring. All have proposed defaults suitable for {31.3} TECH to assume unless Liam overrides. Numbered P-OQ-1..5 to distinguish from the seven RESEARCH OQs already ratified.

• P-OQ-1 — Initial umbrellas.json populated set. RESEARCH §7.3 listed five candidate umbrellas (canonical-pipeline, production-readiness, sales-proposals, intelligence-workspaces, docs-foundation). Does this Task author all five at first commit, or only canonical-pipeline (the load-bearing one) and let the others land JIT as their substrate docs ratify?

  • Default: canonical-pipeline only at first commit, plus stub entries (id + title + substrate_doc only, no task_ids[]) for the other four. This seeds the schema with multi-umbrella examples without over-committing on Task assignments for umbrellas whose substrate docs may still drift. Stubs get populated as the orchestrator does Task triage in subsequent sessions.
  • Alternative: Populate all five at first commit, including production-readiness (with the 15 existing P-R Tasks listed) and intelligence-workspaces (with T2 retrospectively cross-listed). Risk: requires deciding now whether ID-22 (etc.) belongs to canonical-pipeline OR production-readiness OR both.

• P-OQ-2 — Task-to-umbrella multi-membership policy. Invariant 8 allows a Task to appear in multiple umbrellas’ task_ids[] arrays. Is there a convention for minimum multi-membership (e.g. “every Task belongs to at least one umbrella”) or is it acceptable for a Task to have zero umbrella memberships?

  • Default: Zero memberships allowed but flagged as a soft warning. The cross-doc validation test (invariant 9) emits a warning per orphan Task but does not fail. Some Tasks (e.g. one-off ops tasks, cmux polish) genuinely don’t fit any umbrella and forcing an unassigned umbrella would be ledger noise.
  • Alternative: Hard-require ≥1 umbrella membership; add an unassigned umbrella as the catch-all. Pro: cleaner queryability. Con: many ops Tasks don’t belong to any strategic initiative.

• P-OQ-3 — Retro Task commit_refs[] shape vs journal SHA strings. Invariant 12 records SHAs in both commit_refs[] (at Task level) and the journal block (in Subtask details). Should these be a single source-of-truth (e.g. journal-only with commit_refs[] deprecated) or kept dual?

  • Default: Kept dual. commit_refs[] enables programmatic queryability (e.g. “show all Tasks touching commit X”); the journal block provides context (commit message + WP grouping). The dual store is duplicated but tiny; a TECH-spec’d helper can populate commit_refs[] from the journal block at retro-Task-open time so the orchestrator writes once.
  • Alternative: Journal-only. Cheaper to write but loses queryability until an Astro tooling pass parses journal blocks into a structured index.

• P-OQ-4 — T0 umbrella retro Task vs umbrellas.json canonical-pipeline entry: relationship? OQ-1 ALT opens a Task T0 representing pre-phase substrate. OQ-4 ratifies a umbrellas.json entry canonical-pipeline whose substrate_doc is PLAN.md. Are T0 (the Task) and canonical-pipeline (the umbrella) two distinct ledger items, or is T0 redundant given the umbrella?

  • Default: Two distinct items. T0 (Task) represents work done during pre-phase (the ~15 doc landings); canonical-pipeline (umbrella) represents the strategic initiative (Phase 1 of the project plan). T0’s Task ID appears in canonical-pipeline.task_ids[] alongside T1-T14’s IDs. The distinction is “umbrella = container; Task = unit of shipped work”.
  • Alternative: Collapse T0 into the umbrella’s substrate_doc body — encode pre-phase substrate as prose inside the umbrella entry, not as a Task. Pro: lighter ledger. Con: loses Subtask-level granularity for pre- phase work (the ~15 doc landings) which OQ-1 ALT specifically wanted captured at retro-Subtask resolution.

• P-OQ-5 — Retrospective Task commit batching at migration land. The migration opens 7 Tasks (T0 + T1-T6) in done status with full journal blocks. Is this one commit per Task or one commit covering all seven?

  • Default: One commit per Task. Each retro-Task open is its own commit with message chore(s{NN}-w{N}): open Task ID-NN T{n} retrospective (done) — <one-line summary>. Pro: per-Task auditability; small, reviewable commits; matches the orchestrator’s existing per-Task commit discipline (cf. ID-31.1 RESEARCH commit pattern).
  • Alternative: Single batched commit for all seven retro opens. Pro: fewer commits in the history. Con: harder to review per-Task and to revert individually.

---

End of PRODUCT.md. Ratification by Liam (or Checker promotion in absence of overrides) gates {31.3} TECH dispatch.