Staging Coolify cocoindex stand-up runbook (ID-66.21)

RETIRED (S491): The staging stand-up this runbook drove has been executed — live staging-stack operations are now covered by runbooks/onprem-b1-deploy.md (incl. the idle-mode boot-failure triage carried over from here). Project refs (turayklvaunphgbgscat/rovrymhhffssilaftdwd) and the pullmd content are stale-era. Retained for historical context only.

Stand up a staging mirror of the on-prem cocoindex ingestion stack (cocoindex + pullmd

Playwright + Trafilatura) on Coolify, targeting the staging Supabase branch (turayklvaunphgbgscat). The staging stack lets the operator validate ingest, the {66.16} E2E smoke, and the ID-62 reachability hand-off without touching the live on-prem prod corpus or prod Supabase.

This runbook is operator-only for the Coolify-UI steps — the agent that authored the templates holds no Coolify or host access. It mirrors the house style of onprem-b1-deploy.md, cocoindex-deploy.md, and onprem-secrets-migration.md.

Status legend: ✅ done · 🔶 in progress / blocked on operator · ⬜ not started.

Artefacts

Artefact	Purpose
`deploy/coolify/docker-compose.staging.yaml`	Staging compose template (mirror of `deploy/coolify/docker-compose.production.yaml`).
`deploy/coolify/.env.staging.example`	Env-var template — names + safe placeholders + per-var notes.
`deploy/coolify/docker-compose.production.yaml`	The prod source this staging compose mirrors.
`docs/runbooks/onprem-secrets-migration.md`	GCP Secret Manager → Coolify env mapping (names + fetch commands).
`docs/runbooks/cocoindex-deploy.md`	The two boot-required DBs + DSN minting (§0.1).

How staging mirrors prod — and where it deliberately differs

The staging compose is byte-for-byte the prod on-prem compose with three deliberate divergences:

Supabase target — staging points at the turayklvaunphgbgscat branch (NOT prod rovrymhhffssilaftdwd). Wired purely via Coolify env (SUPABASE_* / COCOINDEX_DB_DSN); no ref is hardcoded in the compose. The operator pastes staging-branch values.
Separate corpus + engine volume — cocoindex-state-staging (distinct from prod’s cocoindex-state), so staging ingest never reads or mutates the live prod corpus, and a staging teardown never destroys prod state.
Distinct names + co-residence — staging service/container names (*-staging) and a separate compose project name (kh-onprem-staging), so the staging stack can co-reside with prod on the same Coolify server.

Everything else (image-tag discipline, the root-user LMDB fix, the /dev/tcp healthcheck, the pullmd→Playwright/Trafilatura wiring, the secrets-via-Coolify-env discipline) is identical to prod. Re-pin the Playwright/Trafilatura :2 digests in lockstep with the prod compose when upstream bumps the tag.

Prerequisites

Coolify dashboard reachable (per onprem-b1-deploy.md {66.6} — http://77.68.122.71:8000, or the {66.13} FQDN once DNS/SSL lands).
The cocoindex image is built + pushed to ghcr.io by {66.7} and you have a pinned sha- tag to set as COCOINDEX_IMAGE_TAG (never :latest).
A staging COCOINDEX_DB_DSN for the turayklvaunphgbgscat branch. Build it from the staging POSTGRES_PASSWORD and the region-qualified pooler host (aws-<n>-eu-west-2.pooler.supabase.com — read the aws-0/aws-1 prefix from the Supabase dashboard, do not guess). See cocoindex-deploy.md §0.1 for the mint/verify commands. Treat confirming the staging DSN as a stand-up prerequisite.
The staging Supabase keys (SUPABASE_*, NEXT_PUBLIC_SUPABASE_*) and the shared PULLMD_API_TOKEN / PULLMD_ADMIN_* credentials to hand.

Operator stand-up steps (Coolify UI)

Create the staging application in Coolify using the Docker Compose build pack. Source it from deploy/coolify/docker-compose.staging.yaml (point Coolify at this repo + path, or paste the compose). The project name is kh-onprem-staging.
Set environment variables on the staging application. Use deploy/coolify/.env.staging.example as the contract — for every row, set the key from the example and paste the real staging value:
- COCOINDEX_IMAGE_TAG — the pinned sha- tag from {66.7}.
- COCOINDEX_DB_DSN — the staging-branch pooler DSN (prerequisite above).
- SUPABASE_* / NEXT_PUBLIC_SUPABASE_* / NEXT_PUBLIC_APP_URL — staging-branch values.
- ANTHROPIC_API_KEY, OPENAI_API_KEY, CRON_SECRET, SENTRY_AUTH_TOKEN — staging values.
- PULLMD_ADMIN_EMAIL, PULLMD_ADMIN_PASSWORD, PULLMD_API_TOKEN — staging values (the API token is shared between the cocoindex and pullmd apps).
- Leave COCOINDEX_SOURCE_PATH empty for first boot (idle-mode smoke below).
- PIPELINE_RUN_WEBHOOK_URL — point at the staging app host’s app/api/internal/pipeline-runs/record, reconciled with the {66.13} cutover plan (NOT the prod app URL). Leave empty until {66.13} confirms the staging repoint.
Mark every secret row as secret/encrypted. Paste values directly into the UI; never write them to a file or commit them (same discipline as onprem-secrets-migration.md).
Confirm the staging corpus volume. The compose declares the named volume cocoindex-state-staging (distinct from prod’s cocoindex-state). Coolify creates it on first deploy; it is disk-backed under /var/lib/docker/volumes and persists across redeploys. No prod volume is touched. When you later activate ingest, stage the corpus in a subdir under this volume (e.g. /cocoindex-state/corpus) and set COCOINDEX_SOURCE_PATH.
Deploy. Coolify pulls the pinned image, starts the four staging services, and gates on the cocoindex /dev/tcp healthcheck.

Idle-mode smoke check (first boot, no corpus)

With COCOINDEX_SOURCE_PATH empty, the worker must boot into idle mode and stay running — this is the O-Q8 idle contract (see scripts/tests/test_cocoindex_flow_idle_mode.py).

Verify after deploy:

Container stays running — the cocoindex-staging service is up and the healthcheck is passing (Coolify shows it healthy). No crash-loop / restart churn.
Idle log message — the cocoindex logs show the idle-mode message and no worker crash: no cocoindex background thread crashed, no asyncpg gaierror / TargetServerAttributeNotMatched (which would mean the staging DSN host is wrong — re-check the region-qualified pooler host), and no ValueError: Environment settings must provide Settings.db_path (which would mean COCOINDEX_DB / the volume mount is missing).
pullmd-staging healthy — pullmd booted (single-admin creds present) and the Playwright/Trafilatura sidecars are up.

If all three hold, the staging stack is correctly idle and ready for an operator to stage a corpus when the {66.16} smoke / ID-62 hand-off needs it.

Teardown / rollback

Roll back an image — re-set COCOINDEX_IMAGE_TAG to the prior pinned sha- tag and redeploy. Image tags are pinned, so rollback is deterministic.
Tear down the staging stack — stop/delete the kh-onprem-staging application in Coolify. Because staging uses the separate cocoindex-state-staging volume and *-staging service names, this leaves the prod stack and its cocoindex-state volume untouched.
Discard staging state — delete the cocoindex-state-staging volume in Coolify. LMDB loss costs only a memo-rebuild on the next ingest; Supabase is canonical, so no KH data is lost. Never delete the prod cocoindex-state volume from here.
Revert ingest target — the staging stack is env-only; nothing in code points at it, so removing the app fully reverts the change.

Status

{66.21} ⬜→🔶 — templates + this runbook are authored (the CODE/DOC deliverable). The actual Coolify-UI stand-up is operator-only and is tracked/escalated separately (OQ).