TECH — ID-127: Platform ingestion pipeline standup (Platform-owned cocoindex deploy + corpus + topology)

TECH — ID-127: Platform ingestion pipeline standup

{127.3} TECH artefact. This is the technical-plan phase: it translates the ratified {127.2} PRODUCT behaviour into an implementation plan grounded in the existing codebase. It maps Proposed changes one-to-one against PRODUCT’s BI-1..BI-15, resolves the two TECH-deferred decisions (HOST, NAMING) and the three TECH-mechanism decisions (CORPUS-SYNC, WORKSPACE-seed, CADENCE), and carries the Q-EX2 empirical re-verification results inline.

• Task: ID-127 “Platform ingestion pipeline standup” (promoted from bl-301).
• Predecessors (read in full): {127.1} RESEARCH.md (env/secrets/webhook matrices + current-state gap), {127.2} PRODUCT.md (15 ratified Behavior invariants + Ratified decisions S403 + Notes for TECH). The PRODUCT “Corrected model” wins over stale RESEARCH premises.
• Authoritative grounding: reference/platform-direction.md (workspace ≠ client/tenant; six baseline application_types; controlled-local-fs v1 store).
• Authored: 23/06/2026, agent (task-planner, fresh context per Q-PLANNER-2 — NOT the {127.2} PRODUCT author).
• Status: Draft, pending ratification of the two newly-resolved decisions (HOST, NAMING) and the one NEW open question surfaced (OQ-127-MANIFEST-RENAME, below).
• Language: UK English throughout. Dates DD/MM/YYYY.

Scope guard (PRODUCT Non-goals — load-bearing): ID-127 delivers a *deployment + corpus

• topology*, NOT new pipeline logic. The 6-stage flow, the worker HTTP server, the manifest resolver, the URL source, the /extract cleaner, and the in-app ingestion seams all exist in-repo and are unchanged. Any pipeline-logic change surfaced below is flagged as an ESCALATION, never authored as in-scope work (see OQ-127-MANIFEST-RENAME).

---

Context

What is being built

A third cocoindex Coolify deployment — Platform-owned, mirroring the two live client (Phew) apps in shape but writing ONLY the Platform DB (zjqbrdctesqvouboziae) — plus the synthetic corpus it ingests, the topology/naming rebuilt for extensibility, a parallel deploy job in onprem-deploy.yml, the 6-workspace + feed-row seed, backup + monitor wiring, and a documented promote runbook. The pipeline code is the artefact’s input, not its output.

Code-intelligence orientation (cited verbatim per .gitnexus/CLAUDE.md; HEAD 23/06/2026)

• gitnexus_context({name: "app_main", file_path: "scripts/cocoindex_pipeline/flow.py", repo: "canonical"}) → app_main (Function, L3330–3913). "incoming": {} — zero in-graph callers (it is the flow entrypoint cocoindex’s runtime invokes, not in-repo code). Outgoing calls include load_workspace_manifest (workspace_resolver.py), FeedUrlSource.items (url_source.py), _emit_pipeline_run_webhook, _run_stage_5_resolution, _run_qa_dedup_proposer. Verdict: the Platform pipeline reuses this entrypoint unchanged — additive deployment, LOW structural risk to the worker; the design surface is config + corpus + topology (matches PRODUCT’s orientation). No gitnexus_impact blast-radius warning applies — no symbol is modified.
• gitnexus orientation: existing symbols match — the cocoindex worker, manifest resolver, URL source, monitor, and webhook bridge are all already indexed and complete; this is NOT a greenfield surface. (The ccc fallback was unnecessary — the gitnexus query returned the full worker flow.)

Relevant code (file:line, all read-only for ID-127)

File	Lines	Role
deploy/coolify/docker-compose.production.yaml	38–305	Client prod compose (4 services: cocoindex + pullmd + playwright + trafilatura); name: kh-onprem; router kh-cocoindex-prod-walk; Host(kh-pipeline.aisolutionhub.co.uk); volumes cocoindex-state + pullmd-data-production. The structural template.
deploy/coolify/docker-compose.staging.yaml	55–310	Staging compose; the suffix-delta exemplar — name: kh-onprem-staging, container_name: kh-cocoindex-staging, router kh-cocoindex-staging-walk, Host(kh-pipeline-staging.aisolutionhub.co.uk), volume cocoindex-state-staging.
deploy/coolify/.env.staging.example	1–79	The ~18-var per-app encrypted env contract template.
.github/workflows/onprem-deploy.yml	112–113, 142–202, 204–312, 337–418 (prod deploy), 444–523 (staging deploy), 546–595 (tag-release)	The build (off-host buildpacks → ghcr.io sha-), the detect-changes full-compare gate, the per-branch PATCH-COCOINDEX_IMAGE_TAG+deploy jobs (hardcoded client uuids), the ID-113 semver re-tag job.
scripts/cocoindex_pipeline/flow.py	1649 (_WORKSPACE_MANIFEST_FILENAME), 1928–1990 (the fork), 3343–3427 (app_main manifest load), 3526–3530 (localfs.walk_dir), 3000–3018 (content-branch “workspace_ids NEVER written”), 3100–3106 / 3168 / 3238 (forms-branch workspace_id consumption)	The worker. The fork defaults route="content"; only route=="forms" consumes workspace_id.
scripts/cocoindex_pipeline/workspace_resolver.py	65 (RouteKind), 143–171 (WorkspaceManifest; empty mappings:[] “degenerate but legal”), 196–224 (load_workspace_manifest), 227–278 (resolve_route)	The manifest schema + resolver.
scripts/cocoindex_pipeline/url_source.py	64–69 (_PASSED_URLS_SQL), 213–226 (FeedUrlSource)	Reads feed_articles WHERE passed = true over the asyncpg pool.
scripts/cocoindex_pipeline/server.py	249 (/health), 265 (/stage), 419/439/451 (/walk → update_blocking(live=False)), /extract (calls extract.clean_html)	Worker HTTP surface (port 8080).
scripts/cocoindex_pipeline/extract.py	42 (import trafilatura), 72–84 (clean_html → trafilatura.extract), 129 (apply_quality_gate)	The cocoindex-based URL-cleaner — the PullMD replacement (ID-110/111/112).
deploy/onprem/backup/lmdb-backup-cold.sh	30–57	Env-parameterised cold LMDB→S3 backup: APP_UUID (req) derives the container + /var/lib/docker/volumes/${APP_UUID}_cocoindex-state/_data/lmdb/mdb; SECRETS_ENV_FILE carries BACKUP_S3_BUCKET + BACKUP_ENV_LABEL (default production).
deploy/onprem/monitor/datapath-watch.ts	64–90	The idle-aware stall-watch: COCOINDEX_SOURCE_PATH empty/unset → idle, no alert; INGESTION_EXPECTED default false; reads pipeline_runs (not /health).
app/api/internal/pipeline-runs/record/route.ts	21–30	The webhook bridge: verifyCronAuth (Bearer CRON_SECRET) → createServiceClient → recordPipelineRun; maxDuration=10. Already on canonical-platform.
lib/upload/folder-drop.ts (133+), lib/extraction/clean-via-worker.ts (63+)	—	The in-app ingest seams: POST ${COCOINDEX_WORKER_URL}/stage+/walk (Bearer CRON_SECRET) and /extract (Bearer EXTRACT_API_TOKEN).
supabase/migrations/20260617130000_squash_baseline.sql	702–716 (workspaces), 5368 (application_types), 6182–6207 (feed_articles), 6979 (pipeline_runs_status_check — now includes in_progress)	The schema the Platform DB carries (tracks main).
scripts/seed-platform-from-staging.ts	1–70	Already bootstraps the Platform DB canonical baseline (incl. application_types — in PAYLOAD_CONTRACT), explicitly DEFERRING reference_items “when the platform gains its own ingestion pipeline (ID-108 follow-up)” — that follow-up is ID-127.
scripts/seed-e2e-users.ts	88–90, 234–289	Asserts the pipeline service-account row a0000000-0000-4000-8000-000000000001 + auth.users/auth.identities shape — the BI-10 verify tool.
scripts/propagation/payload-contract.ts	77–121	PAYLOAD_CONTRACT — application_types present (stableKey ['key']). Confirms the 6 baseline types are seeded into Platform.

Live topology facts (PRODUCT + RESEARCH, read-only recon)

• Coolify host 77.68.122.71 = server knowledge-hub-phew, Ubuntu 24.04, 8 vCPU / ~16 GB, Traefik + letsencrypt. 1 project (knowledge-hub-phew, uuid wvmxb1yciztj4het6twi0v0g), 2 client apps — prod ybiczck7f7e1xbdev3bk89cr (branch release/v1.0.0), staging ljurtiab99kb6dst38k76419 (branch staging). 0 managed DBs (Supabase is external cloud).
• PullMD trio is GONE in scope for Platform (PRODUCT Corrected model #1): the Platform stack is cocoindex-only. With the trio gone, the Platform stack is ~one slim service + the cocoindex image (no ~3.7 GB playwright layer), materially lighter than a client stack.

---

Decision resolutions (the TECH-deferred + TECH-mechanism calls)

OQ-127-HOST — RESOLVED: separate Coolify project on the existing IONOS host (77.68.122.71)

SUPERSEDED (S408, ratified) — see AMENDMENT-staging-prod-two-server.md. The host-placement decision below (separate Coolify project on the shared client IONOS host, single prod-only Platform app, plus a new Platform GH environment) was ratified out in S408 on BOTH axes: (1) the Platform pipeline now runs on a dedicated Platform VPS (Server B), physically separate from the client box (two-server split) — NOT co-tenanted; (2) the Platform is staging + prod, not prod-only, each ca-suffix-isolated. One Coolify control-plane on Server B manages both servers; the existing Production/Staging Coolify creds are reused and there is no new GH environment (OQ-2 — this also supersedes the OQ-127-NAMING part 3 below that minted a Platform GH environment). The amendment is authoritative on topology; the body below is retained as the superseded rationale (its single- host dogfood / isolation analysis no longer holds post-split).

Decision: create a new Coolify project canonical-platform-pipeline on the existing IONOS host, holding the single Platform cocoindex app. Not a separate VPS; not a temporary Mac.

Liam’s three framing questions answered:

1. Would a real SMB host its pipeline on existing infrastructure rather than a dedicated VPS? Yes — and that is exactly the dogfood we want. A UK SMB on our v1 controlled- local-fs model runs ONE modest ingestion sidecar against ONE knowledge base; it would co-tenant it on whatever VPS it already pays for, not provision a dedicated box. Hosting the Platform rig the same way keeps the promote artefact (BI-14) faithful to the resource envelope a client actually deploys into — a dedicated-VPS Platform rig would make us blind to the single-host contention a real client hits.
2. With PullMD removed, is temporary Mac-hosting feasible? Feasible for the inner dev loop, rejected as the deliverable. Local-run (RESEARCH Option C) cannot exercise the Coolify deploy, Traefik routing, the PIPELINE_RUN_WEBHOOK_URL round-trip, or the COCOINDEX_WORKER_URL app→worker seams — i.e. most of the config that IS the promote artefact (BI-2/BI-14). A Mac is also not a faithful upstream (different arch, no Traefik, no Coolify env-injection). Keep Mac local-run as the fast inner loop only.
3. With PullMD removed, is co-locating Platform + client on the one IONOS VPC acceptable until launch, given the topology rebuild? Yes, until launch. The capacity objection (RESEARCH §5: “a 3rd full stack is tight”) is dissolved — the Platform stack is cocoindex-only (no ~3.7 GB playwright). The isolation objection (our dev rig beside a client prod) is mitigated by THREE structural guards: (a) a separate Coolify project (not just a third app in the client project) gives a clean resource/permissions boundary and is the topology-extensibility win BI-5 wants; (b) the Platform app writes a different DB (zjqbrdctesqvouboziae) and gets its own uuid-namespaced volume (<platform-uuid>_cocoindex-state); (c) the rig is intermittent (BI-9) — it is not a standing load beside the client. This is explicitly an “until launch” posture: a backlog note (see Follow-ups) tracks splitting Platform to its own host before client GA if contention or the isolation posture demands it.

Rationale (one line): cheapest, faithful to a real SMB’s single-host envelope, isolation adequate via separate-project + separate-DB + intermittent-shape, and the separate project IS the BI-5 topology-extensibility deliverable.

OQ-127-NAMING — RESOLVED: ca/canonical resource naming + reuse the unprefixed worker env contract, GitHub secrets per-environment

Decision (two parts):

1. Coolify/Traefik/volume resource names adopt the ca/canonical scheme (BI-5):
   • Coolify project: canonical-platform-pipeline.
   • Coolify app: ca-pipeline-platform (the kh-onprem-pipeline-* client apps stay as-is — ID-127 does not own the client rename; see Follow-ups).
   • Compose name: ca-pipeline-platform; container_name: ca-cocoindex-platform.
   • Traefik router/service ca-cocoindex-platform-walk; Host(ca-pipeline-platform.aisolutionhub.co.uk).
   • Volume ca-cocoindex-platform-state (the engine LMDB + corpus mount).
   • Image path is UNCHANGED: ghcr.io/${GHCR_OWNER:-ai-solution-hub}/kh-cocoindex-pipeline — the image is the shared promote artefact (BI-14); renaming it is out of scope and would fork the ID-113 substrate. The kh-cocoindex-pipeline image name is a deliberate exception to the ca rename.
2. Worker env contract: reuse the existing UNPREFIXED names (SUPABASE_*, COCOINDEX_*, EXTRACT_API_TOKEN, CRON_SECRET, PIPELINE_RUN_WEBHOOK_URL) scoped to the Platform DB — do NOT mint a KH_PLATFORM_* / CA_PLATFORM_* prefix. RESEARCH §6 + PRODUCT BI-5 starting point: the worker code reads the unprefixed names; a prefix buys nothing and forces a read-shim (a pipeline-code change = escalation). The kh→ca rename does NOT force the env rename because env keys are internal compose contract, never user-facing resource names. Confirmed: the only KH_PLATFORM_* vars in-repo (KH_PLATFORM_URL, KH_PLATFORM_SECRET_KEY) are read by seed-platform-from-staging.ts, never the pipeline.
3. GitHub Actions secret scoping: per-environment, not shared. Add a Platform GH environment holding COOLIFY_API_TOKEN + COOLIFY_BASE_URL (same shape as the existing Production / Staging environments the client deploy jobs use). Per-environment keeps the Platform deploy credential blast-radius isolated from the client deploy credentials and matches the established pattern (onprem-deploy.yml:343 environment: Production, :450 environment: Staging). A shared secret set would couple Platform and client deploy auth — rejected.

Rationale (one line): ca for human-facing resource names, unprefixed for the worker-read env (no shim, no pipeline-code change), per-env GH secrets for credential isolation matching the existing two-environment pattern.

OQ-127-SYNC — RESOLVED: a deploy-time rsync sync step from local-fs-platform/corpus → on-prem volume

Decision: v1 corpus-sync is a small operator-run rsync push from the local local-fs-platform/corpus to the Platform app’s on-prem corpus volume path (/cocoindex-state/corpus, via the host volume <platform-uuid>_ca-cocoindex-platform-state), documented as a runbook step and wrapped in a thin script (deploy/onprem/sync/corpus-sync-platform.sh, env-parameterised: PLATFORM_HOST, APP_UUID, LOCAL_CORPUS_DIR). Mechanism choice: rsync over scp/copy because re-syncs are incremental (only changed files transfer) and idempotent (BI-7 base-path stability is preserved — rsync to the same dest path never re-roots identity). NOT a deploy-job step in onprem-deploy.yml (the deploy job touches only COCOINDEX_IMAGE_TAG; corpus management stays operator-owned per the burn-safety posture). Pluggable external sources (SharePoint/Notion) are OUT (WS-6-gated). OKF is a forward note only (bl-367) — NOT designed for.

Rationale (one line): rsync is incremental + idempotent + base-path-stable, operator-run keeps it out of the burn-sensitive deploy path, and a thin env-parameterised script keeps it reusable for a future second environment.

OQ-127-CADENCE — RESOLVED: on-demand only for v1 (no scheduled walk)

Decision: the Platform rig has no scheduled /walk cadence in v1 — walks are purely operator-triggered (curl the bearer-gated /walk, or the in-app folder-drop seam). This honours BI-9 (dev-time/intermittent, not a standing schedule) most cleanly and avoids unattended Anthropic spend on a dev rig. (Boot is burn-safe regardless — lifespan-only, no boot walk — so leaving COCOINDEX_SOURCE_PATH set permanently is safe.) A light nightly can be added later via a Coolify scheduled task if E2E iteration wants it, but v1 ships on-demand.

---

Proposed changes — one-to-one against PRODUCT BI-1..BI-15

BI-1 → Platform deployment writes ONLY the Platform DB

Change: the Platform app’s COCOINDEX_DB_DSN, SUPABASE_URL, SUPABASE_*_KEY, NEXT_PUBLIC_SUPABASE_*, PIPELINE_RUN_WEBHOOK_URL all resolve to zjqbrdctesqvouboziae / https://kh.aisolutionhub.co.uk. COCOINDEX_DB_DSN = postgresql://postgres.zjqbrdctesqvouboziae:<POSTGRES_PASSWORD>@aws-<N>-eu-west-2.pooler.supabase.com:5432/postgres — read the aws-0/aws-1 prefix from the Supabase dashboard, never guess (RESEARCH §6; the load-bearing var). The client app uuids are NEVER referenced by any Platform path. The Platform app is in a separate Coolify project, so no client env is reachable.

• Maps to: BI-1 acceptance — ingest produces rows in zjqbrdctesqvouboziae and zero in any client DB; client uuids untouched.

BI-2 → Platform deploy mirrors the client shape; parallel deploy job, own trigger, never the client branch guard

Change A — compose (deploy/coolify/docker-compose.platform.yaml, NEW): a faithful clone of docker-compose.staging.yaml with (i) the ca-suffix renames (OQ-127-NAMING), (ii) the PullMD trio removed (BI-4 — no pullmd/playwright/trafilatura services, no depends_on: pullmd, no pullmd-data-* volume), and (iii) the corpus/LMDB volume renamed ca-cocoindex-platform-state. The cocoindex service keeps user: "0:0", the bash /dev/tcp healthcheck, COCOINDEX_DB: /cocoindex-state/lmdb, the ${COCOINDEX_IMAGE_TAG:?} hard-fail, and the Traefik label block scoped to /walk+/health+/extract.

Change B — deploy job (onprem-deploy.yml, NEW job deploy-cocoindex-platform): a parallel job modelled on deploy-cocoindex (prod), with:

• environment: Platform (the NEW GH environment, OQ-127-NAMING).

• env: COOLIFY_APP_UUID: <platform-app-uuid> (filled at app-create time).

• A distinct trigger guard — it must NOT reuse the client refs/heads/main guard, since Platform ALSO tracks main and that guard already fires the client-prod deploy. v1 guard: workflow_dispatch only (with a deploy_platform: true input), so a routine main push deploys the client prod app (unchanged) and the Platform deploy is an explicit operator dispatch. This is the cleanest “own trigger, never reuses the client branch guard” (BI-2 acceptance) and keeps Platform deploys deliberate (matching BI-9 intermittent shape). (Alternative considered + rejected: a paths:-narrowed main-push auto-deploy — rejected because two jobs firing on the same main push (client-prod + Platform) doubles the deploy surface and risks the client-prod deploy masking a Platform failure. The dispatch trigger is the conservative v1.)

• Same PATCH-COCOINDEX_IMAGE_TAG + GET-/deploy body as the client jobs (injection-safe env mapping; --fail-with-body).

• Maps to: BI-2 acceptance — compose-defined + onprem-deploy.yml-deployed; per-service/ env mapping to client counterparts; only deltas = removed PullMD trio + ca-suffixes; own trigger, never the client branch guard.

BI-3 → ingests the synthetic Platform corpus only

Change: the Platform COCOINDEX_SOURCE_PATH = /cocoindex-state/corpus, populated ONLY from local-fs-platform/corpus via OQ-127-SYNC. The corpus is assembled (operator task) to exercise ≥3 content shapes (Q&A, capability/methodology, sector intel, procurement) and ≥2 file formats (md + PDF/DOCX), plus the feed slice (BI-4). No file derives from local-fs-client/corpus (the ID-45 Phew set) or any sanitised client fixture.

• Maps to: BI-3 acceptance — COCOINDEX_SOURCE_PATH rooted at the synthetic set; ≥3 shapes / ≥2 formats; reviewer confirms no client-derived file.

BI-4 → no PullMD/Playwright; URL extraction via the cocoindex /extract cleaner; feed rows seeded

Change A — compose: zero pullmd/playwright/trafilatura services, zero pullmd-data volume, zero PULLMD_* env (Change A under BI-2 already removes them). Note: the cocoindex image itself bundles trafilatura in-process (extract.py imports trafilatura.extract) — the /extract route is served by the cocoindex worker, NOT a separate trafilatura service. So URL/feed extraction works with the trio removed.

Change B — feed-row seed (NEW seed script scripts/seed-platform-feed-articles.ts, or a documented SQL seed): insert ≥1 feed_articles row into the Platform DB with passed = true so FeedUrlSource._PASSED_URLS_SQL (SELECT … WHERE passed = true) enumerates it. Constraint (verified): feed_articles.workspace_id is NOT NULL with FK feed_articles_workspace_id_fkey → workspaces(id) ON DELETE CASCADE, and feed_source_id is also NOT NULL (FK). So the feed seed DEPENDS on (a) the 6 Platform workspaces existing (BI-8) and (b) a feed_sources row existing — seed those first. Use a real public URL so the /extract clean produces real content.

• Maps to: BI-4 acceptance — zero PullMD in compose/volumes/env; a URL ingest produces cleaned content via /extract; reviewer confirms no PullMD carried in.

BI-5 → topology rebuilt for extensibility; ca/canonical naming documented

Change: OQ-127-NAMING’s scheme, plus a documented topology-conventions section in the promote runbook (below) stating the project/server/source/destination roles and the ca-suffix-per-environment rule so a future second environment (or second client) slots in by suffix substitution, not bespoke rework. The separate Coolify project IS the extensibility unit.

• Maps to: BI-5 acceptance — documented conventions; Platform resources use ca/canonical; generalises to N environments. (Naming↔GH-secret interaction resolved: per-env GH environments.)

BI-6 → local-fs split cleanly separates the two configs

Change: documented in the corpus-sync runbook: local-fs-platform/corpus → Platform volume; local-fs-client/corpus → client volume; both reach their on-prem corpus via /cocoindex-state/corpus (per-app uuid-namespaced volume, so no cross-contamination). The Platform sync script (OQ-127-SYNC) hard-codes LOCAL_CORPUS_DIR default to the platform folder; no Platform config points at the client folder.

• Maps to: BI-6 acceptance — reviewer can state which local folder feeds which deployment and how each reaches its on-prem volume; no cross-pointing config.

BI-7 → stable base path is the canonical contract; manifest required ONLY for the forms slice — with a verified correction

Q-EX2 BEHAVIOUR finding (load-bearing — see Empirical verification below): the deployed app_main (flow.py:3403–3427) calls load_workspace_manifest(source_path / ".kh-workspace-map.json") unconditionally whenever COCOINDEX_SOURCE_PATH is set and exists, and ABORTS the flow (manifest_missing stage error, status="failed", raise) if the manifest file is absent. The PRODUCT BI-7 phrasing “the content slice ingests manifest-free” is true only of the per-item fork default (flow.py:1947 route="content" when manifest is None) — but app_main NEVER passes None; the manifest is None branch is reachable only by in-task unit-test callers. Therefore a deployed Platform content-only run still REQUIRES a manifest FILE at the corpus root.

Resolution (no pipeline-code change — within scope): the Platform corpus ships a single .kh-workspace-map.json at the corpus root carrying:

• For the content/q_a/feed slices: rely on the loader accepting a manifest whose mappings do not prefix those paths → each such file hits UnmappedPath → benign soft-warn → routes to content (workspace-agnostic, BI-7 intent preserved). A minimal {"schema_version": 1, "mappings": [<forms-only entries>]} satisfies the loader (empty mappings: [] is “degenerate but legal” per WorkspaceManifest, but we carry the forms entry anyway).
• For the forms slice (BI-8): one mapping {path_prefix: "forms/procurement/", workspace_id: "<procurement-workspace-uuid>", route: "forms"} routing the procurement bid-questions files to their Platform workspace.

So the operator obligation is: base path fixed (/cocoindex-state/corpus, identity-fork consequence documented) AND a manifest file present at the root (even for content-only runs). This is the precise correction the Checker must verify against — it differs from a naive reading of “manifest-free”.

• Maps to: BI-7 acceptance — base-path-stability obligation documented; the content/q_a/feed slices route to content (via UnmappedPath soft-warn, manifest present but non-matching); the forms slice carries a root manifest that loads without ManifestLoadError and routes to the intended workspace; re-running ingest re-stamps the same canonical identities (uuid5 of rel_path — deterministic).

ESCALATION (NOT in-scope) — OQ-127-MANIFEST-RENAME: PRODUCT BI-7 says the manifest is “to be renamed .ca-workspace-map.json”. But the filename is a code constant (flow.py:1649 _WORKSPACE_MANIFEST_FILENAME = ".kh-workspace-map.json"), exercised by ~16 tests. Renaming it is a pipeline-logic change — explicitly an ID-127 escalation, not in-scope work. Recommendation: ID-127 ships the corpus manifest as .kh-workspace-map.json (matching the deployed code); the .ca-workspace-map.json rename is a SEPARATE backlog item (a gitnexus_rename-discipline change to the constant + tests + any client manifest, never find-and-replace). Returned to the Orchestrator for routing to the Curator.

BI-8 → Platform workspaces follow the canonical entity model; 6 application_types, minimal forms slice

Change — 6-workspace seed (NEW seed script scripts/seed-platform-workspaces.ts, or a documented idempotent SQL seed): insert 6 workspaces rows into the Platform DB, one per baseline application_type (procurement, intelligence, sales_proposal, product_guide, competitor_research, training_onboarding). Each row sets name (NOT NULL, e.g. “Platform — Procurement”), application_type_id (NOT NULL FK — resolve by (SELECT id FROM application_types WHERE key = '<key>'), the stable cross-DB key), and created_by = a0000000-0000-4000-8000-000000000001 (the pipeline service account; nullable but stamped for provenance). Prerequisite (verified): the 6 application_types rows are already on the Platform DB via seed-platform-from-staging.ts (application_types is in PAYLOAD_CONTRACT) — the workspace seed asserts their presence and fails loud if absent (do NOT silently create application_types here; that is canonical-baseline territory).

Idempotency: upsert on a stable key (e.g. name) so re-seeding converges. The forms-slice manifest (BI-7) references the procurement workspace’s uuid (the ≥1 forms workspace).

• Maps to: BI-8 acceptance — every manifest workspace_id resolves to a Platform-DB workspaces row bound to an application_type; no config treats a workspace as a tenant; workspace count driven by the 6 application_types, not by client count.

BI-9 → dev-time / intermittent; boot never auto-walks

Change: no scheduled walk (OQ-127-CADENCE) — walks on explicit /walk trigger or the in-app folder-drop seam. COCOINDEX_SOURCE_PATH may stay set permanently (boot is lifespan-only, coco.start_blocking(), no walk — verified onprem-deploy.yml header + the server.py lifespan). The intermittent shape is documented in the runbook.

• Maps to: BI-9 acceptance — a Platform boot with corpus mounted does not auto-walk; walk only on explicit trigger; documented as intermittent/dev-time.

BI-10 → write-model prerequisites VERIFIED (not authored) before first ingest

Change — verify step (operator/CI, documented in the runbook):

1. Run scripts/seed-e2e-users.ts (or its verify-only mode) against zjqbrdctesqvouboziae to assert the pipeline service-account row a0000000-0000-4000-8000-000000000001 exists in auth.users with ≥1 auth.identities row (the S374 defect must be remediated by the squash baseline — seed-e2e-users.ts:255,289 are the exact assertions).
2. Confirm COCOINDEX_DB_DSN is a pooler DSN for the postgres role (RLS-exempt) with the correct aws-N prefix read from the dashboard — NOT the Supabase service-role JWT as the PG password (the removed ID-49.8 root-cause bug). The service-role JWT stays in SUPABASE_SERVICE_ROLE_KEY (used by the monitor + app, not the worker write path).
3. A first ingest stamps created_by = a0000000-0000-4000-8000-000000000001.

This Task authors NO migration and NO RLS — the prerequisites are verified, per Non-goals.

• Maps to: BI-10 acceptance — verify step confirms svc-acct + auth triggers; DSN is a pooler postgres DSN; reviewer confirms no service-role JWT as PG password; first ingest stamps the svc-acct created_by.

BI-11 → backup + monitor are standard, non-optional

Change A — backup: reuse deploy/onprem/backup/lmdb-backup-cold.sh for the Platform app with APP_UUID=<platform-app-uuid> (derives the container + the <platform-uuid>_ca-cocoindex-platform-state/_data/lmdb/mdb host dir automatically — the volume rename flows through the APP_UUID-derived path) and a Platform secrets env-file with BACKUP_ENV_LABEL=platform (so backups land at s3://…/${BACKUP_S3_PREFIX}/platform/lmdb-<ts>.tar.zst, never colliding with the client production label). Documented as a non-optional standup step (operator runs it / wires a Coolify scheduled task at standup, not “later”).

Change B — monitor: reuse deploy/onprem/monitor/datapath-watch.ts against the Platform DB (SUPABASE_URL/SUPABASE_SERVICE_ROLE_KEY → Platform). Set INGESTION_EXPECTED=false (default) and leave COCOINDEX_SOURCE_PATH readable so the monitor’s idle-boot detection treats Platform’s intermittent idleness as normal (no alert) — the BI-11 idle-aware requirement is satisfied by the monitor’s existing COCOINDEX_SOURCE_PATH empty/unset → idle

• INGESTION_EXPECTED gate logic (datapath-watch.ts:77–82). The Vercel cron app/api/cron/datapath-watch already covers the Platform app natively once it ingests.

• Maps to: BI-11 acceptance — both backup + monitor present in delivered scope; monitor does not raise on expected Platform idleness.

BI-12 → pipeline-run webhook records Platform runs against the Platform app

Change: Platform worker PIPELINE_RUN_WEBHOOK_URL = https://kh.aisolutionhub.co.uk/api/internal/pipeline-runs/record + a shared Platform CRON_SECRET (matching the value on the Platform Vercel app’s env). The endpoint already exists on canonical-platform; _emit_pipeline_run_webhook is best-effort (skips if env unset, never raises). Verified bonus: the squash baseline’s pipeline_runs_status_check constraint (L6979) NOW includes 'in_progress' (and 'cancelled') — so the flow-start in_progress row lands on the Platform DB (the datapath-watch GOTCHA-A is resolved on main, which Platform tracks).

• Maps to: BI-12 acceptance — a pipeline_runs row recorded via the Platform endpoint; webhook points at the Platform app + shares CRON_SECRET; best-effort (unreachable/unset does not fail the run).

BI-13 → deliberate image-tag pin (ID-113 semver), never :latest/rolling

Change: Platform COCOINDEX_IMAGE_TAG pins a concrete vX.Y.Z (ID-113 tag-release aliases a known-good sha- image to the semver tag without rebuild). The compose’s ${COCOINDEX_IMAGE_TAG:?} hard-fails if unset. Rollback = repoint to a prior vX.Y.Z (PATCH + redeploy, no rebuild). Where no vX.Y.Z release yet exists at standup, pin a deliberate sha- tag and migrate to vX.Y.Z at the next release cut.

• Maps to: BI-13 acceptance — COCOINDEX_IMAGE_TAG resolves to vX.Y.Z (or pinned sha-), never :latest; compose hard-fails on unset; rollback by repoint works.

BI-14 → documented promote mechanism (manual runbook) pushes a proven Platform config to a client config

Change — promote runbook (NEW, lands in the docs-site alongside the ID-113 per-client-release-pins crib; see “The promote runbook” below). v1 = manual. The promote artefact is the kh-cocoindex-pipeline GHCR image (build-once); promote = pin the proven vX.Y.Z (ID-113 substrate) + apply a config-delta template (Platform-suffix→client-suffix, Platform-DB→client-DB re-key). References ID-113 PI-10 + the cut runbook.

• Maps to: BI-14 acceptance — a documented runbook from “Platform proven at main” to “client pinned to the release”, covering compose delta + env re-key + image pin; references ID-113; states manual.

BI-15 → Platform config guarded against silent drift from main

Change A — sentinel: the Platform Coolify app sets watch_paths = deploy/coolify/.never-auto-deploy-sentinel (the same never-auto-deploy guard the client staging app uses — a path that never changes, so Coolify never auto-deploys; the GH Action / operator dispatch is the SOLE deploy trigger). Note: this is a Coolify setting, not a committed repo file (verified — no such file exists in deploy/coolify/).

Change B — parity guard (bl-346): the Platform compose is cut off main and lives in deploy/coolify/docker-compose.platform.yaml (a main-tracked path, unlike the client composes that live on the staging/release/v1.0.0 deploy branches). The drift risk is LOWER for Platform (its compose is on main), but to satisfy BI-15 wire a compose-parity check: extend the existing parity-guard pattern (cf. schema-parity.yml, token-parity.yml) with a deploy/coolify/** parity job that flags when the Platform compose diverges from the shared structural template (the cocoindex service block, env keyset) — surfacing drift on a main push rather than letting it ship silently. v1 may be a warn-only check (non-required), matching the token-parity posture.

• Maps to: BI-15 acceptance — Platform app carries the .never-auto-deploy-sentinel watch-path guard; a divergent Platform-compose change is surfaced by the parity/drift guard, not silently shipped.

---

The promote runbook (BI-14 deliverable — lands in the docs-site)

Lands at runbooks/platform-to-client-promote.md (or folded into runbooks/per-client-release-pins.md — one ops surface, per ID-113 PI-7). Steps:

1. Topology conventions (BI-5). State the project/server/source/destination roles and the ca-suffix-per-environment rule. A new environment = a new Coolify project + ca-<env>-suffixed app/router/volume + a docker-compose.<env>.yaml cut from the template + a per-env GH environment holding COOLIFY_*.
2. Prove on Platform. Develop/test the whole config (compose + env keyset + corpus) on the Platform rig at main; confirm a real ingest lands Platform content_items rows, the webhook records a pipeline_runs row, and the monitor reads idle correctly.
3. Cut the release (ID-113). At known-good main: git tag vX.Y.Z, fast-forward release/vX.Y.Z, and re-tag the sha-<sha> cocoindex image as vX.Y.Z via the tag-release workflow_dispatch (source_sha + release_version).
4. Config-delta template (Platform → client). Apply, per target client:
   • Compose: ca-pipeline-platform → kh-onprem-pipeline-<client>; router/volume suffixes platform → <client>; Host(ca-pipeline-platform.…) → the client FQDN. (The client stack re-adds the PullMD trio IF that client ingests URLs via the legacy path — Platform’s trio-free compose is the cocoindex-only baseline; the client delta is additive.)
   • Env re-key: COCOINDEX_DB_DSN / SUPABASE_* / PIPELINE_RUN_WEBHOOK_URL / NEXT_PUBLIC_APP_URL Platform values → client values (client DB ref + FQDN). CRON_SECRET / EXTRACT_API_TOKEN minted per client.
   • Image: pin the client’s COCOINDEX_IMAGE_TAG to the proven vX.Y.Z (same immutable image; build-once / config-per-environment).
5. Pin + record. Set the client Coolify COCOINDEX_IMAGE_TAG to vX.Y.Z (PATCH + deploy); append the upgrade row to the ID-113 append-only crib (from→to version, date, migration delta).

---

Testing and validation

Behaviour-first per reference/test-philosophy.md. Each BI maps to a concrete check. The verification is mostly operational (a deployment + corpus + topology, not new code), so “tests” are largely live E2E assertions + reviewer confirmations, plus the two seed scripts which DO get behaviour tests.

BI	Verification
BI-1	Live E2E: an ingest run produces content_items/source_documents rows in zjqbrdctesqvouboziae and zero in rovrymhhffssilaftdwd/turayklvaunphgbgscat; reviewer confirms no client uuid in the Platform deploy path.
BI-2	Reviewer maps each Platform compose service+env key to a client counterpart; the only deltas are the removed PullMD trio + ca-suffixes; the deploy job uses environment: Platform + a non-main-branch-guard trigger.
BI-3	Reviewer confirms COCOINDEX_SOURCE_PATH rooted at the synthetic set, ≥3 shapes / ≥2 formats, no client-derived file.
BI-4	grep sweep: zero pullmd/playwright/PULLMD_* in the Platform compose+env; a live URL ingest lands cleaned content via /extract (real public URL).
BI-5	Reviewer confirms documented topology conventions + ca resource names + per-env GH environment.
BI-6	Reviewer states which local-fs folder feeds which deployment from the sync runbook; no cross-pointing config.
BI-7	Behaviour test (Python, scripts/tests/): assert a content-only corpus with the root .kh-workspace-map.json present ingests (no manifest_missing); assert a content file under a non-forms prefix routes to content (soft-warn UnmappedPath, row lands workspace-agnostic); assert the forms file under forms/procurement/ routes to forms and writes form_templates.workspace_id = the procurement workspace. Re-run = same uuid5 identities (no duplication). Also assert the absent-manifest abort (manifest_missing) so the operator obligation is regression-guarded.
BI-8	Behaviour test on seed-platform-workspaces.ts: 6 workspaces rows, one per application_type, each FK-valid; re-seed idempotent; fails loud if an application_type row is missing. Reviewer confirms no tenant-as-workspace treatment.
BI-9	Live: a Platform boot with corpus mounted lands zero pipeline_runs rows (no auto-walk); a /walk trigger then lands rows. Reviewer confirms documented intermittent shape.
BI-10	seed-e2e-users.ts verify-mode against Platform passes (svc-acct row + identities); reviewer confirms DSN is a pooler postgres DSN (not the service-role JWT); first ingest stamps created_by = a0000000-…001.
BI-11	Backup: a Platform LMDB cold-backup produces an object at the platform S3 label. Monitor: with INGESTION_EXPECTED=false and idle source, the monitor reports idle (no alert).
BI-12	Live: after a Platform ingest, a pipeline_runs row (incl. the in_progress flow-start row) lands in the Platform DB via the Platform endpoint; unreachable-endpoint test confirms the run still completes.
BI-13	Reviewer confirms COCOINDEX_IMAGE_TAG = a concrete vX.Y.Z/sha-, never :latest; compose hard-fails on unset; rollback-by-repoint verified.
BI-14	Reviewer follows the promote runbook end-to-end on paper from “Platform proven at main” to “client pinned vX.Y.Z”.
BI-15	Reviewer confirms the Platform app’s .never-auto-deploy-sentinel watch-path; a divergent Platform-compose push is surfaced by the parity check.

Quality bars inherited (PRODUCT KH quality bars): the two new TS seed scripts use sb() / tryQuery() from @/lib/supabase/safe for any Supabase-client read/write (the worker write path is asyncpg, exempt — BI-10); no barrel re-exports (direct file imports); bun run test (never bun test) for TS, python3 -m pytest scripts/tests/ for the BI-7 Python behaviour test; UK English; no proxy.ts publicRoutes edit (the ingest/webhook routes are /api/*, bypass the login redirect, and enforce bearer auth internally — verified RESEARCH §7.1).

---

Risks and mitigations

• BI-7 manifest-mandatory drift (HIGHEST). If the corpus ships without a root .kh-workspace-map.json, the deployed flow aborts every run with manifest_missing — a content-only run is NOT manifest-free at the deployed entrypoint (Q-EX2 finding). Mitigation: ship the root manifest as a non-optional corpus artefact; the BI-7 behaviour test regression- guards the absent-manifest abort.
• Feed-seed FK ordering. feed_articles.workspace_id (NOT NULL FK) + feed_source_id (NOT NULL FK) mean the feed seed fails unless the 6 workspaces AND a feed_sources row exist first. Mitigation: the seed scripts run in order (application_types-assert → workspaces → feed_sources → feed_articles); document the dependency.
• Two deploy jobs on one main push. A naive main-push Platform auto-deploy would fire alongside the client-prod deploy. Mitigation: the Platform deploy job uses a workflow_dispatch-only trigger (OQ-127-NAMING / BI-2), never the client main branch guard.
• DSN guesswork (ID-49.8 class). Guessing the aws-N pooler prefix or using the service-role JWT as the PG password silently fails ingest. Mitigation: BI-10 verify step reads the prefix from the dashboard and asserts the DSN role.
• Co-location isolation posture. Platform dev rig beside client prod on one host. Mitigation: separate Coolify project + separate DB + intermittent shape; backlog note to split hosts before client GA (Follow-ups).
• Supabase CLI sandbox. Any supabase CLI / bun scripts/ledger-cli.ts call fails in-sandbox (“Operation not permitted”) — run with the sandbox disabled. The seed scripts that hit the Platform DB need the Platform service-role key in env (operator-gated).

---

Empirical verification (Q-EX2 — re-verified at implementation depth, 23/06/2026)

External-library symbols + the load-bearing manifest/forms-route claim, re-verified against the pinned versions (requirements.txt) and HEAD code:

Symbol / claim	Pinned version	Result
cocoindex (module)	cocoindex[postgres]==1.0.7	PRESENT — installed cocoindex 1.0.7 matches pin.
cocoindex.connectors.localfs.walk_dir	1.0.7	PRESENT — hasattr(localfs, 'walk_dir') == True (re-confirmed; flow.py:3526 calls localfs.walk_dir(source_path, live=True, recursive=True)).
cocoindex.connectors.localfs.declare_file	1.0.7	PRESENT — hasattr(localfs, 'declare_file') == True.
cocoindex.start_blocking / stop_blocking	1.0.7	PRESENT — top-level attrs confirmed. Boot is lifespan-only (start_blocking, no walk); the on-demand /walk calls KH_PIPELINE_APP.update_blocking(live=False, full_reprocess=…) (server.py:439).
trafilatura.extract (the /extract cleaner path)	trafilatura==2.0.0	PRESENT — trafilatura 2.0.0, hasattr(trafilatura, 'extract') == True; extract.py:84 trafilatura.extract(html, url=url, **TRAFILATURA_CONFIG). In-process in the cocoindex image — no separate trafilatura service needed (BI-4).
BI-7 manifest/forms-route claim	HEAD	BEHAVIOUR_DRIFT (recorded + resolved inline). Verified at flow.py:1947 (fork default route="content" ONLY when manifest is None), flow.py:3403–3427 (app_main loads the manifest UNCONDITIONALLY and ABORTS manifest_missing if absent — manifest is None is unreachable from the deployed entrypoint), flow.py:3000–3018 (content branch: “workspace_ids NEVER written (BI-7)”), flow.py:3100–3106 (forms branch: “Only this Path-B form-write consumes a workspace_id”). Conclusion: content records ARE workspace-agnostic (no workspace_id; content_item_workspaces M2M unpopulated by ingest, ID-69 v1.1) and only the forms route consumes workspace_id — PRODUCT BI-7’s substance holds — BUT a deployed content-only run REQUIRES a root manifest FILE (not manifest-free). Resolved by shipping the root manifest (BI-7 Proposed change). No ABSENT/SIGNATURE_DRIFT.
pipeline_runs_status_check includes in_progress	HEAD (squash baseline)	PRESENT (migrations/20260617130000_squash_baseline.sql:6979) — the datapath-watch GOTCHA-A defect is resolved on main; the BI-12 in_progress flow-start row lands on the Platform DB.
feed_articles shape (passed, workspace_id NOT NULL FK)	HEAD (squash baseline)	PRESENT (:6182–6207, :9659) — feed seed must satisfy the workspace_id/feed_source_id NOT-NULL FKs (BI-4/BI-8 ordering).
application_types in propagation PAYLOAD_CONTRACT	HEAD	PRESENT (payload-contract.ts:114, stableKey ['key']) — the 6 baseline types are seeded into Platform by seed-platform-from-staging.ts; the workspace seed asserts, not authors, them.

No ABSENT / SIGNATURE_DRIFT findings. The one BEHAVIOUR_DRIFT (BI-7 manifest-mandatory) is recorded inline and surfaced to the Orchestrator; it is resolved by a corpus artefact (the root manifest), not a code change.

---

Follow-ups (deferred, cross-referenced)

• OQ-127-MANIFEST-RENAME (ESCALATION → backlog): rename _WORKSPACE_MANIFEST_FILENAME .kh-workspace-map.json → .ca-workspace-map.json (code constant + ~16 tests + client manifest) via gitnexus_rename discipline. A pipeline-logic change, out of ID-127 scope.
• Platform host split before client GA (backlog note): if co-location contention or the dev-rig-beside-client-prod isolation posture demands it, split the Platform pipeline to its own Coolify server (RESEARCH §5 Option B). Tracked as an “until launch” posture.
• Client kh→ca resource rename (backlog): ID-127 renames only the Platform resources; the kh-onprem-pipeline-* client apps + composes are left as-is (BI-5 says “where the rename is safe to land in the same pass” — the client rename touches live client deploys and is deliberately deferred).
• OKF-shaped corpus layout trial (bl-367, forward-looking): the BI-7 stable corpus layout is the cheapest place to trial an OKF-shaped layout — NOT an ID-127 dependency.
• Light nightly walk cadence (optional): if E2E iteration wants it, add a Coolify scheduled task POSTing /walk (OQ-127-CADENCE shipped on-demand-only).

---

Provenance pointers

• Predecessors: specs/id-127-platform-pipeline/{RESEARCH,PRODUCT}.md (read in full).
• Promote substrate: specs/id-113-platform-release-versioning/TECH.md (PI-10) + .github/workflows/onprem-deploy.yml:546–595 (tag-release, shipped).
• Manifest / M2M model: specs/id-69-client-corpus-onboarding/PRODUCT.md (content-agnostic; M2M unpopulated by ingest; v1 operator-side, v1.1 ingest-side).
• Entity glossary: reference/platform-direction.md (workspace ≠ client/tenant).
• Code-intelligence orientation: gitnexus_context({name:"app_main", repo:"canonical"}) (zero incoming callers) cited verbatim in Context; Q-EX2 re-verify cited in Empirical verification.

Changelog

Date	Session	Note
23/06/2026	task-planner (fresh ctx, Q-PLANNER-2 — not the {127.2} author)	{127.3} TECH authored. Per-BI proposed-change map (BI-1..BI-15). Resolved OQ-127-HOST (separate Coolify project on the existing IONOS host — answered Liam’s 3 framing questions), OQ-127-NAMING (ca/canonical resource names + reuse unprefixed worker env + per-env GH Platform environment), OQ-127-SYNC (operator rsync push, idempotent/base-path-stable), OQ-127-CADENCE (on-demand only, v1). Q-EX2 re-verify: cocoindex 1.0.7 localfs + start_blocking + trafilatura.extract all PRESENT; one BEHAVIOUR_DRIFT recorded — the deployed app_main REQUIRES a root manifest FILE (aborts manifest_missing if absent), so a content-only run is NOT manifest-free — resolved by shipping the root manifest (corpus artefact, no code change). Surfaced NEW escalation OQ-127-MANIFEST-RENAME (the .ca-workspace-map.json rename is a pipeline-code change, out of scope → backlog).