PLAN — ID-127: Platform ingestion pipeline standup (decomposition into implementation Subtasks)

PLAN — ID-127: Platform ingestion pipeline standup

{127.4} PLAN artefact. Decomposes the ratified {127.2} PRODUCT (15 Behavior invariants) + {127.3} TECH (per-BI proposed-change map + 5 resolved decisions) into ordered, verifiable implementation Subtasks, folding in all 9 {127.3} adversarial review notes (REVIEW-NOTES-127.3.md). The Subtask records (TM-shape JSON) the Orchestrator adds via bun scripts/ledger-cli.ts add-subtasks 127 are at the end of this doc.

• Task: ID-127 “Platform ingestion pipeline standup” (promoted from bl-301).
• Predecessors (read in full): {127.1} RESEARCH.md, {127.2} PRODUCT.md, {127.3} TECH.md (commit f9c027c5), REVIEW-NOTES-127.3.md (the 9 PASS_WITH_NOTES precision items). The PRODUCT “Corrected model” + the TECH Q-EX2 BEHAVIOUR_DRIFT resolution win over stale RESEARCH premises.
• Authored: 23/06/2026, agent (task-planner, fresh context per Q-PLANNER-2 — NOT the {127.2} or {127.3} author).
• Status: Draft. The Subtask JSON is returned to the Orchestrator for add-subtasks.
• Language: UK English throughout. Dates DD/MM/YYYY.

Scope guard (load-bearing — PRODUCT Non-goals): ID-127 delivers a *deployment + corpus

• topology + seed*, NOT new pipeline logic. The 6-stage flow, the worker HTTP server, the manifest resolver, the URL source, the /extract cleaner, and the in-app ingestion seams all exist in-repo and are UNCHANGED. Any pipeline-logic change surfaced during implementation is an ESCALATION to the Orchestrator (cf. OQ-127-MANIFEST-RENAME → bl-368), never authored as in-scope work.

---

Overview

Stand up the Platform-owned cocoindex Coolify deployment as a separate Coolify project (canonical-platform-pipeline) on the existing IONOS host 77.68.122.71, writing ONLY the Platform DB (zjqbrdctesqvouboziae). Deliverables, in dependency order: the seed chain (6 workspaces → feed_sources → feed_articles, gated on the asserted application_types), the synthetic corpus + mandatory root manifest, the deploy/coolify/docker-compose.platform.yaml compose (PullMD-free, ca-named, resource-limited), the parallel onprem-deploy.yml deploy job with the full guard composition, backup + monitor wiring, the net-new compose-parity guard + sentinel, the promote/topology runbook, and the operator standup runbook that ties live execution together (DSN/write-model verify, sync, isolation guards, __qa__ avoidance). One Python behaviour test (BI-7 fork routing + absent-manifest abort) and two TS seed-script behaviour tests are the only code-level tests; the rest is operational E2E + reviewer confirmation.

Code-intelligence orientation (cited verbatim per .gitnexus/CLAUDE.md; HEAD 23/06/2026)

Run before authoring the decomposition; cited so the Checker can verify the step ran.

• gitnexus_query({repo:'canonical', query:'cocoindex pipeline flow manifest feed ingestion'}) → returned app_main (scripts/cocoindex_pipeline/flow.py, L3330–3913) and _ingest_file_body (flow.py:1862–2049) as the load-bearing definitions, plus the fork-routing/qa-sidecar/form-write behaviour tests under scripts/tests/ (test_cocoindex_flow_fork_routing.py, test_qa_sidecar_roundtrip.py, test_cocoindex_form_write_real_body.py:_make_manifest). No processes returned (the entrypoint is runtime-invoked, not in-graph called). Confirms the worker is in-repo + complete — this is NOT a greenfield surface; the ccc fallback was unnecessary.
• gitnexus_context({repo:'canonical', name:'app_main', file_path:'scripts/cocoindex_pipeline/flow.py'}) → "incoming": {} (zero in-graph callers — the flow entrypoint cocoindex’s runtime invokes). Outgoing calls include load_workspace_manifest (workspace_resolver.py), FeedUrlSource.items (url_source.py), _run_stage_5_resolution (stage_5.py), _run_qa_dedup_proposer (qa_dedup_proposer.py), _emit_pipeline_run_webhook. Verdict: the Platform pipeline reuses this entrypoint UNCHANGED — additive deployment, LOW structural risk to the worker; no gitnexus_impact blast-radius applies (no symbol is modified). The design surface is config + corpus + topology + seed.
• Grep-verified line claims (ast-dataflow is TS-only; the pipeline/deploy substrate is Python + YAML + SQL — grep used per .ast-dataflow/CLAUDE.md):
  • flow.py:1649 _WORKSPACE_MANIFEST_FILENAME = ".kh-workspace-map.json"; flow.py:1781 — a file named .kh-workspace-map.json is SKIPPED during the walk (not ingested).
  • flow.py:3403–3427 — app_main loads the manifest UNCONDITIONALLY and raises manifest_missing/manifest_invalid (status failed, webhook emitted) if absent (NOTE 8 — the deployed content-only run is NOT manifest-free).
  • flow.py:2021–2037 — __qa__/ prefix without a qa_sidecar mapping emits qa_sidecar_route_missing + ingests as junk content (S297 BUG-B; NOTE 9).
  • lmdb-backup-cold.sh:57 — LMDB_HOST_DIR default literal embeds ${APP_UUID}_cocoindex-state (NOT derived from a renamed volume; LMDB_HOST_DIR IS an accepted override — NOTE 1).
  • onprem-deploy.yml:354–357 (prod guard ref==refs/heads/main && (push || inputs.deploy=='true'))
    • :459–462 (staging refs/heads/staging && …) — NOTE 3 guard-composition target.
  • docker-compose.staging.yaml:99–105 — Traefik router labels present (kh-cocoindex-staging-walk), but NO deploy:/cpus/mem_limit resource block exists (NOTE 5 — resource limits are net-new).
  • squash_baseline.sql:555–559 (feed_sources NOT-NULL workspace_id, name, url, source_type default) + :6184–6204 (feed_articles NOT-NULL workspace_id, feed_source_id, external_url, title; passed default false; extraction_method_check valid non-pullmd values = rss_content|fetch|jina_reader|firecrawl|summary_fallback|docling) — NOTE 6 + NOTE 7b.
  • adapters.py:181 reads PULLMD_SERVICE_URL (live adapter, dormant only because env unset — NOTE 7a, a one-line scope note, NOT a deploy-artefact failure).

---

Architecture decisions (PLAN-level, resolving the review notes)

1. ONE volume name, applied to BOTH the compose volume declaration AND the backup LMDB_HOST_DIR (NOTE 1 + NOTE 2 — coupling). Chosen name: ca-cocoindex-platform-state (honouring the TECH OQ-127-NAMING ca/canonical scheme, BI-5). Because the backup script’s default LMDB_HOST_DIR literal hardcodes _cocoindex-state (lmdb-backup-cold.sh:57), the Platform backup env-file MUST carry an explicit LMDB_HOST_DIR=/var/lib/docker/volumes/<platform-app-uuid>_ca-cocoindex-platform-state/_data/lmdb/mdb override (the TECH “auto-derivation” wording is FALSE — corrected here). The rsync sync dest path and the backup read path therefore both resolve to the SAME host volume <platform-app-uuid>_ca-cocoindex-platform-state. Sync and backup are coupled, not independent.

2. Deploy-job guard is a full 3-way composition (NOTE 3). The new Platform job guards if: github.event_name == 'workflow_dispatch' && github.event.inputs.deploy_platform == 'true', AND the existing prod (:354–357) + staging (:459–462) guards are EXTENDED with && github.event.inputs.deploy_platform != 'true'. Without the extension a Platform dispatch on main ALSO fires the client-prod deploy (both share the main checkout). A new deploy_platform boolean input is added to the workflow_dispatch block.

3. The compose-parity guard + the .never-auto-deploy-sentinel are NET-NEW (NOTE 4). bl-346 has ZERO repo refs — there is no existing deploy/compose-parity substrate (only check-token-parity.ts for CSS + schema-parity.yml for DB; neither watches deploy/coolify/**). The parity job is a brand-new warn-only (non-required) check, modelled on the token-parity posture. The .never-auto-deploy-sentinel is a Coolify setting (watch_paths), NOT a committed repo file — it is an OPERATOR standup step recorded with an explicit verification note (a Checker CANNOT confirm it from source).

4. HOST isolation is SOFT — concrete guards added (NOTE 5). A Coolify project is an organisational grouping, not a hard isolation primitive: the Platform rig shares one Docker daemon / Traefik / kernel / host-FS with the client PROD + STAGING stacks. The compose therefore adds (a) a deploy.resources.limits (CPU + memory) block + a mem_reservation on the cocoindex service, (b) a corpus/LMDB volume sized within a documented disk budget (operator-enforced quota note), and (c) DISTINCT Traefik router labels (ca-cocoindex-platform-walk, never colliding with kh-cocoindex-{prod,staging}-walk). The BI-9 intermittent shape + the separate Platform DB remain genuine mitigations; the project-isolation framing is explicitly the soft link these guards harden.

5. Seed sequencing is a strict chain with NOT-NULL completeness (NOTE 6). The squash baseline creates application_types with NO seeded rows — the 6 types exist ONLY if seed-platform-from-staging.ts has RUN against zjqbrdctesqvouboziae. Order: (assert application_types present) → seed 6 workspaces → seed ≥1 feed_sources → seed ≥1 feed_articles. NOT-NULL completeness per squash_baseline.sql: feed_sources needs name + url + workspace_id; feed_articles needs external_url + title + workspace_id + feed_source_id. The feed seed stamps a NON-pullmd extraction_method (fetch, or leaves it NULL — the column is nullable; NEVER a pullmd_* value, extraction_method_check, NOTE 7b).

6. Manifest ships as .kh-workspace-map.json (matching deployed code). The .ca-workspace-map.json rename is a CODE-CONSTANT change (flow.py:1649 + ~16 tests + client manifest) = a pipeline-logic change = OUT of scope (OQ-127-MANIFEST-RENAME → bl-368). The corpus manifest is mandatory at the corpus root (NOTE 8 absent-manifest abort); the synthetic corpus AVOIDS the __qa__/ reserved prefix OR carries an explicit qa_sidecar mapping (NOTE 9).

7. No cross-Task Subtask dependency surfaced — sibling-only constraint HONOURED. Every dependency is within ID-127 (the seed chain; compose→deploy-job; volume-name→backup; artefacts→runbook). ID-69 (content↔workspace M2M), ID-113 (tag-release), bl-346 (parity origin), bl-368 (manifest rename), ID-45 (client reingest) are REFERENCED, not depended-on. No Task-split/merge escalation required.

---

Dependency graph

S1 application_types ASSERT (verify-only, no author)
     │
     ▼
S2 6 workspaces seed ──────────────┐
     │                             │
     ▼                             ▼
S3 feed_sources + feed_articles    S4 corpus + root manifest (needs procurement workspace uuid from S2)
   seed (strict NOT-NULL chain)        (BI-3/7/8/9; __qa__ avoidance; forms-slice mapping)
                                       │
S5 docker-compose.platform.yaml ──────┤  (independent of seed; PullMD-free, ca-named,
   (BI-2/4/5; resource limits;          │   resource-limited, ONE volume name)
    distinct router; volume name) ──────┤
     │                                  │
     ▼                                  │
S6 onprem-deploy.yml platform job ──────┤  (needs S5 compose; full guard composition)
   (BI-2/13; deploy_platform input)      │
     │                                  │
     ▼                                  │
S7 backup env-file + LMDB_HOST_DIR ─────┤  (needs S5 volume name — COUPLED)
   + monitor wiring (BI-11)             │
                                        │
S8 compose-parity guard (NET-NEW)       │  (needs S5 path; warn-only; bl-346)
   (BI-15)                              │
                                        │
S9 promote + topology runbook ──────────┘  (needs S5/S6 naming + env settled)
   (BI-5/14)
     │
     ▼
S10 BI-7 fork-routing + absent-manifest behaviour test (Python; needs S4 manifest design)
     │
     ▼
S11 operator standup runbook + live E2E gate
    (BI-1/3/6/9/10/12; DSN/write-model verify; rsync sync; sentinel-set; __qa__ note;
     isolation-guard verification; webhook round-trip) — the live-execution wrapper, last.

Implementation phases + checkpoints

Phase A — DB seed foundation (S1→S2→S3)

Strict serial chain (FK ordering, NOTE 6). S2 and S4 may proceed in parallel once S2’s procurement workspace uuid is known, but S3 hard-depends on S2. Highest fail-fast value (the DSN/write-model verify in S11 gates first write, but the seed scripts surface schema drift early).

Checkpoint A: 6 workspaces rows + ≥1 feed_sources + ≥1 feed_articles (passed=true) exist in zjqbrdctesqvouboziae; both seed scripts idempotent + fail-loud on missing application_types; bun run test green for the two TS seed behaviour tests.

Phase B — corpus + deploy artefacts (S4, S5→S6, S7, S8) — parallelisable

S4 (corpus/manifest) is independent of the deploy YAML. S5 (compose) → S6 (deploy job) is serial. S7 (backup, COUPLED to S5’s volume name) and S8 (parity guard, needs S5’s path) both depend on S5. These four lanes (S4 · S5→S6 · S7 · S8) parallelise after S5 lands.

Checkpoint B: compose is PullMD-free (grep: zero pullmd/playwright/PULLMD_*), ca-named, resource-limited, distinct router; the deploy job’s full guard composition is in place (a Platform dispatch does NOT fire client-prod); backup env-file carries the explicit LMDB_HOST_DIR override on the SAME volume name; the parity guard surfaces a divergent compose on a main push.

Phase C — docs + tests + live standup (S9, S10, S11)

S9 (runbook) + S10 (behaviour test) parallelise. S11 (operator standup + live E2E) is LAST — it consumes every prior artefact.

Checkpoint C (Task close): a live Platform ingest lands content_items rows in zjqbrdctesqvouboziae and ZERO in any client DB; the webhook records a pipeline_runs row; the monitor reads idle correctly; the BI-7 absent-manifest abort is regression-guarded; the promote runbook is followable end-to-end; all 15 BIs + all 9 review notes discharged.

---

The 9 review-notes → coverage table

Each {127.3} review note is discharged by a concrete Subtask guard (NOT paraphrased away).

#	Review note	Discharged by	How
1	BACKUP: explicit LMDB_HOST_DIR override (script hardcodes _cocoindex-state; “auto-derivation” FALSE)	S7	Backup env-file carries explicit LMDB_HOST_DIR=/var/lib/docker/volumes/<platform-uuid>_ca-cocoindex-platform-state/_data/lmdb/mdb; TECH wording corrected in S7 details.
2	SYNC/volume coupling: ONE volume name in BOTH compose decl + backup LMDB_HOST_DIR	S5 + S7	S5 declares volume ca-cocoindex-platform-state; S7 backup override + S11 rsync dest both use the SAME <platform-uuid>_ca-cocoindex-platform-state host path.
3	BI-2 deploy-dispatch guard composition (new job guarded; prod/staging extended)	S6	New job if: workflow_dispatch && inputs.deploy_platform=='true'; prod/staging guards extended && inputs.deploy_platform != 'true'; new deploy_platform input.
4	BI-15 parity guard NET-NEW (bl-346 zero refs); sentinel is a Coolify SETTING	S8 + S11	S8 = brand-new warn-only compose-parity job. S11 records the .never-auto-deploy-sentinel watch_paths as an OPERATOR step with an explicit “Checker cannot confirm from source” verification note.
5	HOST isolation SOFT: add CPU/mem limits + disk quota + distinct router	S5	S5 compose adds deploy.resources.limits (CPU+mem) + mem_reservation, a documented disk budget, and a distinct ca-cocoindex-platform-walk router label.
6	SEED sequencing app_types→workspaces→feed_sources→feed_articles + NOT-NULL completeness	S1→S2→S3	S1 asserts application_types; S2 seeds workspaces; S3 seeds feed_sources (name+url+workspace_id) then feed_articles (external_url+title+workspace_id+feed_source_id), strict order.
7	BI-4 PullMD residue: (a) adapters.py live adapter scope note; (b) feed seed must NOT stamp pullmd_*	S3 + S5	S5 details carry the one-line adapters.py:181 dormant-adapter scope note (env unset → BI-4 met at the deploy artefact). S3 feed seed stamps fetch/NULL, NEVER pullmd_* (extraction_method_check).
8	BI-7 manifest-mandatory (resolved): ship root manifest; manifest gates feed slice too; add absent-abort guard	S4 + S10	S4 ships the mandatory root .kh-workspace-map.json (forms-only mapping). S10 behaviour test asserts content-only ingests with the manifest present AND asserts the absent-manifest manifest_missing abort.
9	BI-7 __qa__/ reserved-prefix hazard: avoid the prefix OR add a qa_sidecar mapping	S4 + S11	S4 corpus construction AVOIDS __qa__/ (or carries an explicit {path_prefix:'__qa__/', route:'qa_sidecar'} mapping). S11 operator runbook restates the hazard + the avoidance obligation.

---

Risks and mitigations

Risk	Impact	Mitigation
Corpus ships without root manifest → every run aborts manifest_missing (NOTE 8)	High	S4 ships the manifest as a non-optional corpus artefact; S10 regression-guards the absent-manifest abort.
Feed seed fails on NOT-NULL FK ordering (NOTE 6)	High	S3 runs the strict chain (application_types-assert → workspaces → feed_sources → feed_articles); fails loud if application_types absent.
Platform dispatch on main doubles deploy (fires client-prod) (NOTE 3)	High	S6 full guard composition: new job dispatch-only + prod/staging extended with && inputs.deploy_platform != 'true'.
Backup reads the wrong volume path (NOTE 1/2)	Med	S7 explicit LMDB_HOST_DIR override on the same ca-cocoindex-platform-state name; S5 declares that exact volume.
DSN guesswork / service-role JWT as PG password (ID-49.8 class)	Med	S11 BI-10 verify reads the aws-N prefix from the dashboard + asserts the DSN is a pooler postgres DSN.
Co-location contention beside client prod (NOTE 5)	Med	S5 resource limits + disk budget + distinct router; BI-9 intermittent shape + separate DB; backlog note to split hosts before client GA.
__qa__/ corpus file ingested as junk (NOTE 9)	Low	S4 avoids the prefix (or adds a qa_sidecar mapping); S11 runbook restates the hazard.
Supabase CLI / ledger-cli sandbox failure (“Operation not permitted”)	Low	Seed scripts run with the sandbox disabled + the Platform service-role key operator-gated (S11).

Open questions / escalations (returned to the Orchestrator)

• OQ-127-MANIFEST-RENAME (already escalated in {127.3}): the .kh-workspace-map.json → .ca-workspace-map.json rename is a pipeline-code change (code constant + ~16 tests), OUT of ID-127 scope → bl-368 (via the Curator). ID-127 ships .kh-workspace-map.json.
• No NEW open question. All 6 PRODUCT OQs are ratified (S406 re-ratified OQ-127-HOST); the TECH HOST/NAMING/SYNC/CADENCE resolutions stand; the sibling-only dependency constraint surfaced no Task-boundary conflict.

---

Subtask records (TM-shape JSON — for bun scripts/ledger-cli.ts add-subtasks 127 --file -)

The implementation Subtask records (S1–S11, id "1".."11") are returned to the Orchestrator in the dispatch reply (the planner-reporting {N.4} PLAN emit-template), not re-pasted here. Each carries id, title, description (≤250 chars), details (load-bearing/unbudgeted dispatch brief — review-note(s) discharged, file paths, gitnexus/grep citations, code-intel discipline), testStrategy (≤300 chars), and sibling-only dependencies. The budget gate (description ≤250, testStrategy ≤300) is HARD-enforced at add-subtasks time — the records below are authored within budget on the first pass.

Provenance pointers

• Predecessors: specs/id-127-platform-pipeline/{RESEARCH,PRODUCT,TECH}.md + REVIEW-NOTES-127.3.md (all read in full).
• Code-intel orientation: gitnexus_query/gitnexus_context on app_main (flow.py:3330–3913, zero incoming callers) cited verbatim above; grep-verified line claims for NOTE 1/3/5/6/7/8/9 cited inline.
• Promote substrate: specs/id-113-platform-release-versioning/TECH.md (PI-10) + onprem-deploy.yml:546–595 (tag-release, shipped).
• Manifest / M2M model: specs/id-69-client-corpus-onboarding/PRODUCT.md.

Changelog

Date	Session	Note
23/06/2026	task-planner (fresh ctx, Q-PLANNER-2 — not the {127.2}/{127.3} author), S406	{127.4} PLAN authored. 11 implementation Subtasks (S1–S11) across 3 phases (DB seed foundation · corpus+deploy artefacts · docs+tests+live standup) with sibling-only dependencies. All 9 {127.3} review notes discharged via the coverage table (each → a concrete Subtask guard, none paraphrased away). PLAN-level architecture decisions resolved the coupled review notes: ONE volume name ca-cocoindex-platform-state in both compose + backup LMDB_HOST_DIR override (NOTE 1+2); full 3-way deploy guard composition (NOTE 3); net-new warn-only compose-parity guard + sentinel-as-Coolify-setting (NOTE 4); soft-isolation hardening via resource limits + distinct router (NOTE 5); strict seed chain + NOT-NULL completeness (NOTE 6); non-pullmd extraction_method + adapters scope note (NOTE 7); mandatory root manifest + absent-abort guard (NOTE 8); __qa__/ avoidance (NOTE 9). Grep-verified every load-bearing line claim against HEAD. No NEW open question; OQ-127-MANIFEST-RENAME remains the only escalation (→ bl-368). Sibling-only constraint honoured (no cross-Task dep).