RESEARCH — bl-301: Platform ingestion pipeline standup (own cocoindex deploy targeting the Platform DB)

RESEARCH — bl-301: Platform ingestion pipeline standup

Backlog item: bl-301 (CLI id 301 — bun scripts/ledger-cli.ts show backlog 301; the bl- prefix is a label, NOT a CLI-resolvable id).
Status: DECISION-SUPPORT RESEARCH (scoping/investigation only — NOT a PRODUCT/TECH spec, builds nothing).
Origin: ID-108 / S348 (the platform-tail decision that deliberately deferred ingestion).
Author date: 21/06/2026 (S384, sub-orchestrator subo-bl301).
Method: 6-lane multi-modal sweep — runbooks, pipeline-reference/write-model, historical ledger+specs+archive, recent mempalace memory, current repo code/infra, and live Coolify + Supabase MCP inventory (read-only, no mutations).

Purpose (ratified framing — Liam, S384)

The Platform pipeline IS the actual cocoindex data-ingestion setup, Platform-owned and wired to the Platform DB — giving us our own full Dev/test end-to-end pipeline configuration. The value is a controlled, Platform-owned E2E environment that is the UPSTREAM of client deployments: we develop + test the entire pipeline E2E against the Platform DB, then promote that proven configuration out to client configs (IONOS/Coolify). This is the dogfood / E2E rationale. It settles two things that earlier looked open:

Source corpus = our own dev/test content (not a client corpus; not “nothing-yet”).
Scope = the whole pipeline configuration, deploy shape included — because the artefact being developed/tested is precisely what gets pushed downstream to clients. A faithful upstream must mirror the client Coolify deployment, not just fill the DB. This flips the target recommendation toward a Platform-owned Coolify app (§5).

This also makes the Platform-tracks-main / clients-pin-vX.Y.Z model (ID-113) coherent end to end: dev/test on Platform main → cut a release → pin clients downstream.

TL;DR

The gap is real and narrow. The Platform DB (zjqbrdctesqvouboziae, Supabase project canonical-platform, ACTIVE_HEALTHY, eu-west-2) has no ingestion compute pointed at it. Live Coolify (host 77.68.122.71) runs exactly two cocoindex apps, both CLIENT (Phew) — kh-onprem-pipeline-production (uuid ybiczck7f7e1xbdev3bk89cr, branch release/v1.0.0) and kh-onprem-pipeline-staging (uuid ljurtiab99kb6dst38k76419, branch staging). Neither writes the Platform DB.
There is nothing to invent — only to clone + repoint. The client stack IS the build template: a 4-service Coolify docker-compose app (cocoindex + pullmd + playwright + trafilatura) defined in deploy/coolify/docker-compose.production.yaml, deployed by .github/workflows/onprem-deploy.yml, with a ~20-var per-app encrypted env contract templated in deploy/coolify/.env.staging.example. A Platform instance is a third clone whose env targets zjqbrdctesqvouboziae.
The “service-account seed” is two things, both mostly already done. Pipeline DB writes go over a direct asyncpg pooler DSN (COCOINDEX_DB_DSN, the RLS-exempt postgres DB-owner role) — NOT the Supabase service-role REST API. The created_by identity is the seeded row a0000000-0000-4000-8000-000000000001, provisioned by the squash-baseline migration (S374 found Platform briefly missing the auth.users triggers + user_profiles mirror; remediated via the baseline — re-verify before first write).
The purpose answers the two questions that earlier looked open. Corpus = our own dev/test content; scope = the whole pipeline config (deploy shape included) because the Platform deployment is the promote artefact for clients. So the in-app ingestion seams (folder-drop / URL import) and the Coolify deploy/webhook wiring are in scope by definition — a local-run-only path would not exercise the config that gets pushed downstream. The residual decisions are scoping, not blocking (host placement, stack breadth, the promote mechanism — §10).
Recommendation (§5): stand up a Platform-owned Coolify app that mirrors the client stack (so the config is promotable), tracking main, with its env targeting the Platform DB. Local-run is useful only as a fast inner dev loop, not the deliverable. bl-301 remains post-v1 / behind ID-45 (the live client-reingest gate) and should adopt ID-113’s semver image-tag pin from day one (§9); sequence accordingly.

1. The question and the load-bearing distinction

Question: what is required to set up the Platform’s own cocoindex ingestion pipeline?

The distinction (confirmed verbatim across all 6 lanes — this is what a prior nearby research doc reportedly conflated):

	PLATFORM	CLIENT (Phew)
Web	`kh.aisolutionhub.co.uk`	`kh.phew.org.uk`
Vercel project	`canonical-platform`	`canonical-phew`
Tracks branch	`main`	`release/v1.0.0`
Supabase	`zjqbrdctesqvouboziae` (acts as both prod & staging)	own prod (`rovrymhhffssilaftdwd`) + staging (`turayklvaunphgbgscat`) pair
Nature	our own dev / CI / showcase instance — NOT a client	a real client deployment
Cocoindex pipeline	NONE — that gap IS bl-301	the on-prem Coolify stack on `77.68.122.71` (under the aisolutionhub domain until handover-flip)

bl-301 = give the Platform DB a cocoindex ingestion path of its own. The verbatim record (ledger-cli show backlog 301): “Platform DB (zjqbrdctesqvouboziae) is a full dev instance per ID-108, but the on-prem cocoindex pipeline (IONOS/Coolify) is the CLIENT pipeline writing to the client prod/staging pair — platform has no ingestion path. Stand up a platform-owned cocoindex deployment: target decision (separate Coolify app vs local-run), env/secrets wiring (KH_PLATFORM_ in .env.local), webhook endpoints, pipeline service-account seed. Origin: ID-108, S348.”* — type=feature, status=parked, priority=should, track=unsorted, dependencies=[], effort_estimate=null, notes=null.

2. Current state — what exists, what’s missing

2.1 Live infra inventory (verified via Coolify + Supabase MCP, read-only)

Coolify host 77.68.122.71 = server knowledge-hub-phew, Ubuntu 24.04, 8 vCPU / ~16 GB, Traefik proxy + letsencrypt. 1 Coolify project (knowledge-hub-phew, uuid wvmxb1yciztj4het6twi0v0g), 2 applications, 0 managed databases, 0 services.
Both apps are build_pack=dockercompose, git ai-solution-hub/canonical, base_directory /deploy/coolify, both currently pin COCOINDEX_IMAGE_TAG=sha-b92aee5e3cc1405d4abcce61a7581472faccb136:
- prod kh-onprem-pipeline-production — uuid ybiczck7f7e1xbdev3bk89cr, branch release/v1.0.0, Host kh-pipeline.aisolutionhub.co.uk.
- staging kh-onprem-pipeline-staging — uuid ljurtiab99kb6dst38k76419, branch staging, Host kh-pipeline-staging.aisolutionhub.co.uk, watch_paths = deploy/coolify/.never-auto-deploy-sentinel (deliberate never-auto-deploy guard).
Supabase: canonical-platform ref zjqbrdctesqvouboziae, eu-west-2, ACTIVE_HEALTHY, org ptacqvoidhqznzdnzpbg, pg 17.6.1.127 — confirmed live. The sibling canonical-phew ref rovrymhhffssilaftdwd (same org) is the DB the on-prem stack writes today. Coolify reports 0 managed databases → the Supabase DBs are external cloud Supabase, not Coolify-hosted Postgres.

→ There is zero platform-targeted ingestion compute. A Platform instance is net-new.

2.2 The reference architecture (what we’d clone)

The 4-service compose (deploy/coolify/docker-compose.production.yaml, name kh-onprem):

cocoindex — the engine sidecar, image ghcr.io/${GHCR_OWNER:-ai-solution-hub}/kh-cocoindex-pipeline:${COCOINDEX_IMAGE_TAG}, aiohttp wrapper (scripts/cocoindex_pipeline/server.py) on port 8080. Runs user: 0:0 (root — fresh named-volume mount is root-owned; uid-1000 buildpack user would hit EACCES on LMDB init). Healthcheck via bash /dev/tcp/127.0.0.1/8080 (slim image has no curl). depends_on: pullmd.
pullmd (aeternalabshq/pullmd:2.0.0) + playwright + trafilatura — HTML→Markdown render/extract helpers; needed only if Platform ingests URLs.
Named volumes: cocoindex-state (LMDB at /cocoindex-state/lmdb + corpus at /cocoindex-state/corpus) and pullmd-data-production (SQLite share-link store).
Per-env deltas (prod vs staging) are ONLY: Traefik Host(...) rule, service/container/volume -staging suffixes, COOLIFY_BRANCH, and the encrypted env values. Compose structure + env-var key set are identical → a Platform clone needs a new FQDN, its own volume namespace, and Platform-valued env — nothing structurally new.

2.3 Deploy mechanism (`onprem-deploy.yml`) — and the conflation trap

.github/workflows/onprem-deploy.yml: push on branches [main, staging] → detect-changes → build-and-push image off-host (Cloud Native Buildpacks → ghcr.io, content-addressable sha-<commit> tag, never :latest, pull-verified) → a deploy job that PATCHes COCOINDEX_IMAGE_TAG on the Coolify app’s /api/v1/applications/<uuid>/envs then GETs /api/v1/deploy?uuid=<uuid>. The host never builds; it pulls the pinned tag. Rollback = re-PATCH a prior sha- tag + redeploy (no rebuild). Needs GH env secrets COOLIFY_BASE_URL (=http://77.68.122.71:8000) + COOLIFY_API_TOKEN.

Trap: the main-guarded deploy job hardcodes the CLIENT prod app uuid ybiczck7f7e1xbdev3bk89cr. So a main push already redeploys the client pipeline. Because Platform also tracks main, a Platform deploy cannot reuse the branch guard — it needs a parallel job (or a parameterised target / workflow_dispatch), not a new branch.

2.4 What’s missing today (the bl-301 delta)

No Platform Coolify app, no deploy/coolify/docker-compose.platform.yaml, no Platform FQDN/cert, no Platform deploy job.
.env.local has CRON_SECRET, SUPABASE_*, POSTGRES_PASSWORD, ANTHROPIC_API_KEY, OPENAI_API_KEY, PLATFORM_PROJECT_REF/PROD_PROJECT_REF/STAGING_PROJECT_REF. It is missing COCOINDEX_DB_DSN, EXTRACT_API_TOKEN, PIPELINE_RUN_WEBHOOK_URL, COCOINDEX_WORKER_URL, COCOINDEX_SOURCE_PATH, COCOINDEX_IMAGE_TAG, PULLMD_*, and there is no KH_PLATFORM_* prefix anywhere (see §6 naming note).
No defined Platform corpus; no Platform /walk schedule.

3. The build template — client on-prem stack as the analog

The client cocoindex pipeline (id-66 B1 → deploy/coolify/* → onprem-deploy.yml) is the literal recipe. Standing up Platform = the same compose + GH-action with env repointed at the Platform DB. Reusable, surface-agnostic assets already in-repo:

deploy/coolify/docker-compose.{production,staging}.yaml + .env.staging.example (the env contract).
deploy/onprem/{backup,monitor,verify}/ — lmdb-backup-cold.sh, datapath-watch.ts, live-verify.sh, verify_driver.py. All env-parameterised → reusable for Platform. (Correction to brief: deploy/onprem/ DOES exist; what’s absent is a pipeline compose there — the composes live under deploy/coolify/. See Appendix B.)
The monitor app/api/cron/datapath-watch is a Vercel cron native to the Platform app — it would cover Platform automatically once Platform ingests.

Not a live alternative: the Cloud Run path (cloud-run-phase-1.md, cocoindex-deploy.md) is RETIRED (S298) — manifests deleted. On-prem/Coolify is the sole deployed pipeline path; local-run is the only non-Coolify option (§5).

4. Requirements checklist

Grouped by the bl-301 axes + what the sources surface.

4.1 Target (ratified: Platform-owned Coolify app — see Purpose + §5)

Deploy target = a Platform-owned Coolify app mirroring the client stack (so the config is a promotable E2E upstream). Local-run is the inner dev loop only, not the deliverable. (Was the core open axis; resolved by the ratified dogfood/E2E-upstream purpose.)
Coolify-app build: author deploy/coolify/docker-compose.platform.yaml (platform-suffixed names/volumes, new Host(kh-pipeline-platform.aisolutionhub.co.uk) Traefik router); create a 3rd Coolify app (kh-onprem-pipeline-platform, base_dir /deploy/coolify, tracks main); add DNS A-record + letsencrypt cert; replicate the .never-auto-deploy-sentinel guard; add a Platform deploy job to onprem-deploy.yml (parallel to / parameterised from the main push — cannot reuse the client guard).
Decide host placement — reuse 77.68.122.71 (cheapest, but co-locates OUR dev/CI/showcase with a CLIENT prod; 8 vCPU/16 GB already runs 2 stacks → a 3rd ~3.7 GB-playwright stack is tight) vs a new Coolify server vs local-run.
Decide stack breadth — full 4-service (URL ingest) vs cocoindex-only (file/ folder-drop corpus only, drop pullmd/playwright/trafilatura).

4.2 Env / secrets wiring

Mint COCOINDEX_DB_DSN for the Platform pooler (§6, §7).
Populate the ~20-var Coolify encrypted env (or .env.local for local-run), Platform-valued (§6 matrix).
Resolve the KH_PLATFORM_* naming question (§6 note) before wiring.

4.3 Webhook endpoints

Point PIPELINE_RUN_WEBHOOK_URL → Platform app /api/internal/pipeline-runs/record (the endpoint already exists on canonical-platform); share a Platform CRON_SECRET.
Set COCOINDEX_WORKER_URL on the Platform Vercel app to the Platform worker ingress
- EXTRACT_API_TOKEN (§7) — the in-app ingestion seams are in scope (full E2E config).

4.4 Pipeline service-account seed

Verify the Platform DB has row a0000000-0000-4000-8000-000000000001 (created_by identity) + the 3 auth.users triggers + user_profiles mirror (S374 defect — should be remediated by the squash baseline; re-verify). Verifiable via scripts/seed-e2e-users.ts.
Confirm write-model schema prerequisites hold (5 cross-target FKs absent; ID-64 content_items/source_documents NOT-NULL+rename reconciliations; role-based USING(true) RLS) — applied via the squash baseline + ID-64; should already hold since Platform tracks main.

4.5 Surfaced by the sources (not in the original axes)

Image: pin COCOINDEX_IMAGE_TAG to a sha- tag (compose hard-fails on unset; refuses :latest). Same shared ghcr.io/.../kh-cocoindex-pipeline image — no new build needed. Per ID-113 PI-10, pin a deliberate vX.Y.Z semver, not a rolling main sha (§9).
Source path / corpus: define COCOINDEX_SOURCE_PATH + corpus — undefined anywhere (OQ, §8). URL path additionally needs passed=true rows in feed_articles.
Walk cadence: stand up a Coolify scheduled task POSTing bearer-gated /walk (ratified 0 2 * * * nightly; optional hourly fallback). Boot never walks (ID-83/bl-221 burn gate) → SOURCE_PATH may stay set permanently, burn-safe.
Backup/monitor (optional, low priority): Platform LMDB cold-backup cron (reuse deploy/onprem/backup/lmdb-backup-cold.sh); LMDB loss = memo rebuild only.
CI parity guard (bl-346): composes live on deploy branches that never auto-sync from main; a Platform app cut off main could diverge silently — wire the parity guard / sentinel.

5. Target-decision options + recommendation

Option	Cost	Isolation	Ops burden	In-app ingest UI works?	Scheduled cadence?
A. 3rd Coolify app on `77.68.122.71`	low (shared host)	weak — OUR showcase co-located with CLIENT prod	medium (new compose+FQDN+cert+env+deploy job)	✅	✅
B. 3rd Coolify app on a new server	medium (extra VPS)	strong	high	✅	✅
C. Local-run / on-demand CLI (`scripts/ingest.py` or `python3 -m scripts.cocoindex_pipeline` against the Platform DSN)	minimal	strong (no shared infra)	minimal	❌ (no reachable worker → folder-drop 502, URL-clean 503)	❌ (manual / local cron)
D. Cocoindex-only slim Coolify app (drop pullmd trio; file corpus only)	low	per-host	medium	folder-drop ✅, URL ✗	✅

Recommendation: a Platform-owned Coolify app that mirrors the client stack (Option A or D).

The ratified purpose — a promotable, faithful E2E upstream of client deployments — decides the target. The Platform pipeline must be the same shape the client runs (a Coolify docker-compose app deployed by onprem-deploy.yml, with the webhook + in-app ingestion seams wired) because the configuration we develop/test on Platform is exactly what gets pushed downstream to clients. Local-run (Option C) cannot be that upstream: it never exercises the Coolify deploy, Traefik routing, the PIPELINE_RUN_WEBHOOK_URL round-trip, or the COCOINDEX_WORKER_URL app→worker seams — most of the config that is the actual promote artefact. Keep local-run only as a fast inner dev loop.

Concretely:

Author deploy/coolify/docker-compose.platform.yaml (platform-suffixed names/volumes, a new Host(kh-pipeline-platform.aisolutionhub.co.uk) Traefik router) and create a 3rd Coolify app kh-onprem-pipeline-platform tracking main (Platform is the newest-config head; clients pin vX.Y.Z downstream per ID-113).
Option A vs D (OQ-301-5, stack breadth): full 4-service (Option A — if Platform E2E must cover URL ingestion via pullmd) vs cocoindex-only (Option D — if the dev/test corpus is file/folder-drop only). Since the goal is full E2E coverage of the client config, lean Option A unless URL ingestion is deliberately out of the dev/test matrix.
Host placement (OQ-301-4): isolation matters more for a standing dev/test rig than for a throwaway showcase. Weigh a separate small Coolify server vs the shared client host 77.68.122.71 (8 vCPU/16 GB, already 2 stacks; a 3rd full stack is tight and co-locates our rig with a client’s prod). Replicate the .never-auto-deploy-sentinel guard and wire the bl-346 parity guard so the Platform config does not silently drift from main.
Define the promote path (OQ-301-7): how a proven Platform config (compose + env keyset + pinned image tag) is pushed to a client config. This is the point of bl-301 and ties directly to ID-113’s release-cut / client-upgrade runbook — formalise it in the PRODUCT/TECH chain.

6. Env / secrets matrix

Worker-side (Coolify per-app encrypted env, or .env.local for local-run). “Exists today” = present for Platform specifically.

Var	Lives in	Exists for Platform?	Notes
`COCOINDEX_DB_DSN`	Coolify env / `.env.local`	NO	`postgresql://postgres.zjqbrdctesqvouboziae:<pw>@aws-N-eu-west-2.pooler.supabase.com:5432/postgres`. The load-bearing var. Read `aws-0`/`aws-1` prefix from the dashboard — never guess. NOT the LMDB path; NOT the service-role JWT.
`COCOINDEX_DB`	compose literal	n/a	`/cocoindex-state/lmdb` (LMDB engine path) — baked in compose, boot-required.
`COCOINDEX_IMAGE_TAG`	Coolify env	NO	pinned `sha-`/`vX.Y.Z`; shared image, compose hard-fails if unset.
`GHCR_OWNER`	Coolify env	client only	defaults `ai-solution-hub`.
`COCOINDEX_SOURCE_PATH`	Coolify env	NO	corpus dir — depends on the undecided corpus (§8).
`COCOINDEX_LMDB_MAP_SIZE`	Coolify env	client only	default `4294967296` (4 GiB).
`PIPELINE_RUN_WEBHOOK_URL`	Coolify env	NO	→ `https://kh.aisolutionhub.co.uk/api/internal/pipeline-runs/record`.
`CRON_SECRET`	Coolify env + Platform Vercel	partial	exists in `.env.local`; gates `/walk` IN and the webhook OUT — value must match worker↔Platform app.
`EXTRACT_API_TOKEN`	Coolify env + Platform Vercel	NO	dedicated `/extract` bearer (distinct from `CRON_SECRET`); fails closed 401 if unset. (Absent from `.env.staging.example`; added for client only at S382 — do not assume it exists.)
`SUPABASE_URL` / `SUPABASE_SERVICE_ROLE_KEY` / `SUPABASE_PUBLISHABLE_KEY` / `NEXT_PUBLIC_SUPABASE_*`	Coolify env + `.env.local`	partial	Platform values exist in `.env.local` but are NOT in any Coolify app; must target `zjqbrdctesqvouboziae`.
`NEXT_PUBLIC_APP_URL`	Coolify env	NO	`https://kh.aisolutionhub.co.uk`.
`ANTHROPIC_API_KEY` / `OPENAI_API_KEY`	Coolify env + `.env.local`	yes (reuse; consider distinct keys for cost isolation)
`POSTGRES_PASSWORD`	input to `COCOINDEX_DB_DSN`	yes (`.env.local`)	the pooler password component.
`SENTRY_AUTH_TOKEN`	Coolify env	client only	shared across envs.
`PULLMD_API_TOKEN` / `PULLMD_ADMIN_EMAIL` / `PULLMD_ADMIN_PASSWORD`	Coolify env	NO	pullmd single-admin bootstrap; only if URL ingest.
`PULLMD_SERVICE_URL`	compose literal	n/a	`http://pullmd:3000`.
`PIPELINE_CLIENT_ORG`	Coolify env (optional)	NO	`holder_rule.py` first-party fail-closed; leave empty for Platform.
`COCOINDEX_WORKER_URL`	Platform Vercel app env	NO	app→worker base URL; gates folder-drop + URL-clean (§7).
`COOLIFY_API_TOKEN` / `COOLIFY_BASE_URL`	GH environment	client only	a Platform deploy job needs these in a Platform GH env (the `COOLIFY_API_TOKEN` was the single blocker that gated the staging arm).

Naming note — `KH_PLATFORM_*` is aspirational, not real

The bl-301 description says “env/secrets wiring (KH_PLATFORM_* in .env.local)”, but the worker code reads UNPREFIXED names (SUPABASE_* / COCOINDEX_* / EXTRACT_API_TOKEN). The only KH_PLATFORM_* vars in the repo (KH_PLATFORM_URL, KH_PLATFORM_SECRET_KEY) are consumed by scripts/seed-platform-from-staging.ts — not the pipeline. So bl-301 must decide: adopt a new KH_PLATFORM_* prefix (and add a read-shim) vs reuse the existing unprefixed contract scoped to the Platform DB. Recommend reuse — the worker already reads those names; a prefix buys nothing and adds a shim. (OQ-301-3.)

7. Webhook + service-account wiring

7.1 Webhooks (two directions)

OUT (worker → app): flow.py:_emit_pipeline_run_webhook POSTs a per-run rollup to PIPELINE_RUN_WEBHOOK_URL with Authorization: Bearer <CRON_SECRET>; best-effort (skips if env unset, never raises). Lands at app/api/internal/pipeline-runs/record/route.ts → verifyCronAuth (lib/cron-auth.ts:16-25, compares to Bearer ${CRON_SECRET}) → createServiceClient + recordPipelineRun. Status-only, maxDuration=10 — a heavy promote/embed batch cannot run in this webhook. The endpoint already exists on canonical-platform; only the worker’s PIPELINE_RUN_WEBHOOK_URL + a shared Platform CRON_SECRET are new.
IN (app → worker), only if in-app UI must work: via COCOINDEX_WORKER_URL — lib/upload/folder-drop.ts POSTs {WORKER}/stage then {WORKER}/walk (Bearer CRON_SECRET); lib/extraction/clean-via-worker.ts POSTs {WORKER}/extract (Bearer EXTRACT_API_TOKEN). Without a reachable worker, /api/ingest/folder-drop 502s and /api/ingest/url HTML cleaning 503s.
proxy.ts: /api/* routes bypass the login redirect (isApiRoute); auth is enforced inside each route (bearer for the webhook; getAuthorisedClient(['admin', 'editor']) for ingest/extract). No publicRoutes edit is required for bl-301.

7.2 Worker HTTP surface (port 8080)

server.py: GET /health (public), POST /stage (multipart byte-drop, compose-internal only — NOT Traefik-routed, Inv-13), POST /walk (one-shot update_blocking(live=False), Bearer CRON_SECRET, single-flight → 2nd concurrent = 409), POST /extract (HTML→text, Bearer EXTRACT_API_TOKEN, 20 MB cap). Traefik routes exactly /walk + /health + /extract.

7.3 Service-account / write identity (two layers)

Write path = the direct asyncpg COCOINDEX_DB_DSN (RLS-exempt postgres DB-owner role over the Supabase pooler). This is what “service-account seed” means for writes — provision a Platform DB password + build the correct aws-N pooler DSN. There is no app-user row to seed for the write path itself. (The service-role JWT must NOT be used as the PG password — that was the removed root-cause bug ID-49.8.)
created_by identity = seeded row a0000000-0000-4000-8000-000000000001 (flow.py:1395, label “Pipeline (system)”), provisioned by the squash-baseline migration along with the auth.users triggers + user_profiles mirror. S374 found Platform briefly missing these → re-verify on the Platform DB before first write (scripts/seed-e2e-users.ts asserts the row).

8. Source corpus (ratified)

Corpus = our own dev/test content (Liam, S384). The Platform pipeline is a dogfood / E2E rig, so it ingests content we control for developing and testing the pipeline end to end — not a client corpus (ID-45’s Phew set) and not “nothing-yet”. A sanitised copy of client fixtures is explicitly out (re-introduces the ID-68 IP-separation concern).

The remaining work here is curation, not a decision: assemble a representative dev/test set (file corpus under COCOINDEX_SOURCE_PATH, and — if URL ingestion is in the E2E matrix — seed feed_articles rows for the URL path) that exercises the content shapes the pipeline must handle (Q&A, capability/methodology, sector intel, procurement; md/PDF/DOCX/URL). Size it for fast E2E iteration, not coverage of a real client’s volume.

9. Sequencing & dependencies

bl-301 is the UPSTREAM of client deployments (promote model). The Platform pipeline is where the full pipeline config is developed/tested; a proven config is then promoted to client IONOS/Coolify deployments. Practically: Platform tracks main (newest config), and the promote artefact is the compose + env keyset + a pinned image tag → a client’s pinned vX.Y.Z. This is exactly ID-113’s “Platform tracks main, clients pin vX.Y.Z” model; bl-301’s PRODUCT/TECH should formalise the promote/cut step (see OQ-301-7).
bl-301 is post-v1 / behind ID-45 in priority (ID-45 full client re-ingest is the live v1 gate; v1-completion-sequence.md calls bl-301 “parallel and post-client-launch, not a Spine gate”) — but note bl-301’s purpose makes it the dev/test rig that would ideally exist before large client-config changes. Sequence is Liam’s call (OQ-301-6); the dependency is priority, not a hard technical gate.
ID-113 (release versioning) PI-10 tie-in: the same semver that tags the Vercel build re-tags the cocoindex image vX.Y.Z. Recommendation in id-113 (and here): ship ID-113 first so the Platform Coolify COCOINDEX_IMAGE_TAG pins a deliberate vX.Y.Z from day one rather than a rolling main sha that must later be re-pinned. “No hard ordering, but noted in the crib.”
bl-346 (deploy-branch parity guard): composes live on deploy branches (staging, release/v1.0.0) that never auto-sync from main (S382). A Platform app cut off main could diverge silently → wire the parity guard / sentinel.
Verify on Platform DB before first write: service-account row + auth.users triggers
- user_profiles mirror (S374), write-model schema prerequisites (ID-64, squash baseline). Should already hold (Platform tracks main) — explicit check is cheap.
cocoindex version pin: requirements.txt pins cocoindex[postgres]==1.0.7, but _coco_api.py/url_source.py docstrings reference 1.0.3 behaviour. Verify the import façade resolves under 1.0.7 if a fresh Platform image is built.

10. Open questions for Liam

Resolved by the S384 clarification (no longer open): corpus = our own dev/test content (§8); target = a Platform-owned Coolify app mirroring the client stack, in-app ingestion seams in scope (§5). Remaining questions are scoping, none blocking:

OQ-301-4 (host): Put the Platform rig on the client host 77.68.122.71 (cheapest, but co-locates our dev/test rig with a client’s prod; capacity-tight at a 3rd full stack) vs a separate small Coolify server (better isolation for a standing rig)? §5.
OQ-301-5 (stack breadth): Include URL ingestion in the Platform E2E matrix (full pullmd 4-service stack, Option A) or file/folder-drop only (cocoindex-only, Option D)? Lean A for full client-config coverage. §5.
OQ-301-3 (naming): Adopt a new KH_PLATFORM_* env prefix (per the bl-301 wording) or reuse the existing unprefixed SUPABASE_*/COCOINDEX_*/EXTRACT_API_TOKEN contract scoped to the Platform DB? (Recommend reuse — the worker already reads those names.) §6.
OQ-301-7 (promote mechanism — the bl-301 payoff): How does a proven Platform config get pushed to a client config? (compose copy + env re-key + image-tag pin, manual vs scripted; its relationship to ID-113’s release-cut/upgrade runbook.) This is the point of the rig — formalise it in PRODUCT/TECH. §9.
OQ-301-6 (priority/sequencing): bl-301 is currently post-v1 / behind ID-45 — but its purpose is to be the dev/test rig upstream of client-config work. Keep it post-v1 or pull it forward? §9.

11. Recommended next step + effort estimate

Recommended next step: promote bl-301 to a Task with the {N.1} RESEARCH (this doc, port/adopt) → {N.2} PRODUCT → {N.3} TECH → {N.4} PLAN chain. The ratified dogfood/E2E-upstream purpose makes this the Coolify-app route, and the promote-to-client mechanism (OQ-301-7) is a genuine design surface that warrants the spec chain rather than a one-shot change.

Scope of the build (TECH/PLAN):

deploy/coolify/docker-compose.platform.yaml (platform-suffixed names/volumes, new Traefik Host), a 3rd Coolify app kh-onprem-pipeline-platform tracking main, FQDN + DNS + letsencrypt, the ~20-var Platform env (DSN minted for zjqbrdctesqvouboziae), a Platform deploy job in onprem-deploy.yml (parallel to the main push — cannot reuse the client guard), a scheduled /walk, COCOINDEX_WORKER_URL + PIPELINE_RUN_WEBHOOK_URL wiring, the .never-auto-deploy-sentinel + bl-346 parity guard, and the curated dev/test corpus (§8).
Verify-first (cheap): confirm the Platform DB has the pipeline service-account row + auth.users triggers + user_profiles mirror + write-model schema (§7.3, §4.4).
Define the promote path (OQ-301-7) as a first-class PRODUCT/TECH concern.

Effort: M (~1–2 days) for the standing Coolify app + wiring, plus corpus curation; sequence after ID-113’s semver image-tag convention (so Platform pins vX.Y.Z day one) and mindful of ID-45 priority (OQ-301-6). A local-run inner loop is ~0.5 day and can be set up immediately as a stopgap to start exercising flow/extractor code against the Platform DSN while the Coolify app is built — but it is not the deliverable.

Ledger intents (for the parent/Liam to apply on MAIN — this worker did NOT mutate the ledger):

Update bl-301 (backlog id 301) notes → link this RESEARCH (specs/bl-301-platform-pipeline/RESEARCH.md) + one-line conclusion: “Scoped S384: gap confirmed (no platform ingestion compute; only client cocoindex apps exist). Purpose ratified (Liam): Platform-owned E2E dogfood rig, upstream of client deploys (dev/test → promote config to client IONOS/Coolify); corpus = our own dev/test content. Recommend a Platform-owned Coolify app mirroring the client stack; promote bl-301 to a Task with the {N.1→N.4} spec chain. Effort M (~1–2d) + corpus curation; sequence after ID-113, mind ID-45 priority. Open scoping OQs: host (301-4), stack breadth (301-5), env naming (301-3), promote mechanism (301-7), priority (301-6).”
Recommend (do NOT auto-execute): promote bl-301 to a Task (Coolify route is ratified by purpose); set status off parked when the parent opens it. Liam/parent decides priority vs ID-45.

Appendix A — Citation index (file:line / uuid / ref)

bl-301 record — product-backlog.json id 301; bun scripts/ledger-cli.ts show backlog 301.
Deploy workflow — .github/workflows/onprem-deploy.yml:73,112-113,337-403 (build+deploy, main→client-prod uuid ybiczck7f7e1xbdev3bk89cr).
Compose — deploy/coolify/docker-compose.production.yaml:38-57,82-89,91-169,300-305; docker-compose.staging.yaml (-staging deltas); .env.staging.example:1-79.
Worker — scripts/cocoindex_pipeline/server.py:210-215,452-511,555-600,697-700; flow.py:807-856 (webhook OUT), 1395 (svc-acct UUID), 1520-1556,3806 (_build_dsn/ asyncpg pool), adapters.py:175-218 (pullmd), holder_rule.py:88 (PIPELINE_CLIENT_ORG).
App seams — app/api/internal/pipeline-runs/record/route.ts (+ lib/cron-auth.ts:16-25); lib/upload/folder-drop.ts:133-140; lib/extraction/clean-via-worker.ts:63-72; proxy.ts:105-117; app/api/cron/datapath-watch.
Live infra — Coolify server knowledge-hub-phew 77.68.122.71; apps ybiczck7f7e1xbdev3bk89cr (prod) / ljurtiab99kb6dst38k76419 (staging); project wvmxb1yciztj4het6twi0v0g; pinned sha-b92aee5e3cc1405d4abcce61a7581472faccb136. Supabase canonical-platform zjqbrdctesqvouboziae / canonical-phew rovrymhhffssilaftdwd, org ptacqvoidhqznzdnzpbg, eu-west-2.
Reference/write-model — themes/canonical-pipeline/reference/cocoindex-write-model.md (R1–R5, boot-never-walks), v1-completion-sequence.md:66-82,449-457 (topology + bl-301 lane), operational-pre-decisions-register.md (walk cadence 0 2 * * *), pipeline-writepath-schema-gap-s295.md (ID-64 reconciliations).
Runbooks — runbooks/onprem-b1-deploy.md (§B2 cadence, §{66.11-66.15}), _archive/staging-coolify-cocoindex.md (2nd-instance template; archived S491), onprem-secrets-migration.md.
Historical — id-66 (B1 standup recipe), id-95 (per-client topology), id-108 (platform tail — deferred ingestion, ledger-cli get task 108), id-113 PI-10 (id-113-platform-release-versioning/{RESEARCH,TECH}.md — semver/image-tag tie-in).
Memory — drawer adf91cac13e6558b1 “Platform DB + Cocoindex Pipeline (ID-45 Context)” (2026-06-16); claude diary S374/S375 (svc-acct/auth defects), S382 (deploy-branch drift, bl-346).

Appendix B — Corrections to the brief’s premises

deploy/onprem/ DOES exist (backup/, monitor/, verify/) — env-parameterised and reusable. What is genuinely absent is a pipeline compose there (composes live under deploy/coolify/) and a Platform-targeted compose/app/deploy-job.
Cloud Run is RETIRED (S298) — not a live alternative deploy target; treat cloud-run-phase-1.md/cocoindex-deploy.md as historical only (the two-DB boot model they describe still applies).
KH_PLATFORM_* does not exist for the pipeline — the worker reads unprefixed names (§6 naming note).
EXTRACT_API_TOKEN is absent from .env.staging.example — added for the client only at S382; do not assume a Platform value exists.