cocoindex sidecar Cloud Run deploy — Liam-side gcloud operations

Archived S504 (ruling R3): RETIRED since S298. Current deploy path: runbooks/onprem-b1-deploy.md; current topology: reference/deployment-architecture.md.

cocoindex sidecar Cloud Run deploy — Liam-side gcloud operations

RETIRED (S298): Cloud Run is fully decommissioned. The ingestion pipeline now runs on-prem (two-server IONOS/Coolify split — see runbooks/onprem-b1-deploy.md for the client-box execution record and reference/deployment-architecture.md for the current topology). This runbook and its referenced cloudrun/ manifests are removed from the active deploy path; retained for historical context only — its sibling Cloud-Run-era runbooks (cloud-run-phase-1.md, pullmd-deploy.md) already live in runbooks/_archive/. Monitoring has moved too: the gcloud logging / cloud_run_revision commands in §3 below are historical — live pipeline-health monitoring is now the datapath-watch Vercel cron (app/api/cron/datapath-watch, */15, detectStalls predicate), alert sink MONITOR_ALERT_WEBHOOK_URL (deliberately unset → Sentry fallback, S311). (Retirement banner re-verified 22/07/2026 — S491 W4 tier-2 sweep; no content below this line is current.)

Spec / context: docs/audits/cocoindex-state-db-connection-crash-2026-05-26.md (the two-DB crash analysis); Task ID-49 (canonical-pipeline follow-on). Manifests: cloudrun/services/{staging,prod}-{phew,kpf}-cocoindex.yaml. Owner: Liam (gcloud-authenticated operator).

This runbook documents the operational steps the worktree subagent cannot execute — there are no gcloud credentials in the sandbox, and the Executor that authored the manifests holds no GCP or Secret Manager access. It mirrors the house style of docs/runbooks/pullmd-deploy.md.

The cocoindex sidecar is a single Cloud Run Service per (env, tenant) running a long-lived Rust + LMDB engine (kh-cocoindex-pipeline-{phew,kpf} on kh-staging-494815 and kh-prod-494815), ingress: internal, minScale=maxScale=1, execution-environment: gen2.

---

§0. The two boot-required databases (read this first)

The cocoindex sidecar depends on two distinct databases, and both must be present in the applied revision or the worker thread crashes at boot while /health stays green on its own thread (the failure mode in the 2026-05-26 audit). They are easy to conflate — they are not the same thing.

Env var	What it is	How it is wired	Why boot-required
COCOINDEX_DB_DSN	The KH-owned asyncpg pool — a Postgres connection string for the app’s own writes (flow.py:_build_dsn() / @coco.lifespan).	secretKeyRef in the manifest (name: COCOINDEX_DB_DSN, key: latest). Value lives in Secret Manager.	The @coco.lifespan pool is created before /health can serve; a post-replace --set-secrets races the health-gated deploy (ID-49.8).
COCOINDEX_DB	The cocoindex ENGINE’s internal LMDB state store — a filesystem path, NOT a Postgres DSN. cocoindex’s internal store is LMDB-only; there is no Postgres backend for it (audit §2/§3).	Plain value: env (/cocoindex-state/lmdb), backed by an in-memory emptyDir volume mounted at /cocoindex-state. No secret.	environment.start_sync() raises ValueError("Environment settings must provide Settings.db_path (or set COCOINDEX_DB environment variable)") when unset — the next boot crash after the DSN pool (ID-49.9).

COCOINDEX_LMDB_MAP_SIZE (plain env, 268435456 = 256 MiB) bounds the LMDB map below the volume sizeLimit (the engine default is 4 GiB, which over-reserves the tmpfs). Idle-mode footprint is tiny; revisit when the pipeline is activated (ID-42 T7).

§0.1 Secret prerequisites

• Prod COCOINDEX_DB_DSN — value minted out-of-band; CONFIRMED correct by Liam. The DSN is a region-qualified Supabase pooler string (postgres.<project-ref> user + aws-<n>-eu-west-2.pooler.supabase.com host — NOT <project-ref>.pooler.supabase.com, which never resolves; audit §3 #1).
• Staging COCOINDEX_DB_DSN — may need minting / refresh before a staging deploy. Build it from .env.local’s POSTGRES_PASSWORD and the region-qualified pooler host (aws-<n>-eu-west-2.pooler.supabase.com; read the aws-0 vs aws-1 prefix from the Supabase dashboard / supabase CLI pooler-url — do not guess). Treat confirming/minting the staging secret as a staging-deploy prerequisite.
• COCOINDEX_DB — no secret. It is a plain path env baked into the manifest.

Mint / refresh a DSN secret (per project):

PROJECT=kh-staging-494815   # or kh-prod-494815
DSN='postgresql://postgres.<ref>:<POSTGRES_PASSWORD>@aws-<n>-eu-west-2.pooler.supabase.com:5432/postgres'

# First mint:
printf '%s' "$DSN" | gcloud secrets create COCOINDEX_DB_DSN \
  --project="$PROJECT" --replication-policy=automatic --data-file=-

# If it already exists, add a new version instead:
printf '%s' "$DSN" | gcloud secrets versions add COCOINDEX_DB_DSN \
  --project="$PROJECT" --data-file=-

Verify resolution + connection BEFORE redeploy (from a context with the real creds):

python3 -c "import asyncio,asyncpg; asyncio.run(asyncpg.connect('$DSN'))"
# or, minimally, that the host resolves:
nslookup aws-<n>-eu-west-2.pooler.supabase.com

The runtime SA ({phew,kpf}-pipeline-sa@<project>.iam.gserviceaccount.com) already holds roles/secretmanager.secretAccessor at project scope (docs/runbooks/cloud-run-phase-1-handover.md §3) — no new grant is needed.

---

§1. Volume choice + persistence open question

The LMDB store currently lives on an in-memory emptyDir (medium: Memory, sizeLimit: 512Mi) mounted at /cocoindex-state. This is the correct boot-time fix and loses nothing in idle mode (no memo state exists to lose yet), but it is ephemeral — the store is empty on every cold start.

• Why emptyDir, not a Cloud Storage volume: tmpfs is a real local FS, so LMDB’s mmap + file locking work. GCS FUSE does NOT support mmap/locking, so a Cloud Storage volume is LMDB-incompatible.
• Why RAM-backed is safe: the volume counts against the 4Gi container memory; 512Mi is safe headroom for idle mode and bounds COCOINDEX_LMDB_MAP_SIZE (256 MiB) above it.
• True cross-cold-start persistence is deferred. The only Cloud-Run-native option that gives LMDB a real persistent FS is Filestore NFS (~$200+/mo, 1 TiB minimum, plus LMDB-over-NFS locking caveats). That cost/architecture decision is deferred to the parent / Liam and should be revisited at ID-42 T7 (first-ingest pipeline activation), when there is actual memo state worth persisting across restarts.

---

§2. Deploy trigger

Two paths. The WIF-CI path is the norm; workflow_dispatch / manual replace is the recovery fallback.

§2.1 WIF-CI path (the norm)

.github/workflows/cloud-run-deploy.yml path-triggers on cloudrun/** pushes:

• Push to production-readiness → deploys to staging (kh-staging-494815).
• Push to main → deploys to prod (kh-prod-494815).

It also accepts workflow_dispatch (the environment input is the lowercase production / staging choice — the capital-first Production / Staging are the GitHub Environment names the workflow maps to internally, not the dispatch value):

gh workflow run cloud-run-deploy.yml --ref main -f environment=production
# or, for staging:
gh workflow run cloud-run-deploy.yml --ref production-readiness -f environment=staging

§2.2 Manual fallback — gcloud run services replace

# Staging
gcloud run services replace cloudrun/services/staging-phew-cocoindex.yaml \
  --project=kh-staging-494815 --region=europe-west2

# Production
gcloud run services replace cloudrun/services/prod-phew-cocoindex.yaml \
  --project=kh-prod-494815 --region=europe-west2

replace is health-gated — the new revision must pass its startup probe before traffic shifts. That is precisely why both boot fixes must be present in the applied manifest: a replace revision missing COCOINDEX_DB_DSN crashes on the asyncpg pool, and one missing COCOINDEX_DB crashes on the LMDB ValueError — either way the startup probe times out and the deploy fails before any follow-up --set-secrets/--update-env-vars step could repair it. Both are now baked into the manifest, so the replace revision boots clean.

The remaining 14 deploy-time secrets (ANTHROPIC_API_KEY … PULLMD_SERVICE_URL) are still applied by the workflow’s --set-secrets layer and are not boot-gated — do not move them into the manifest. (Note: a bare gcloud run services replace strips deploy-time-only secrets; for a PULLMD_SERVICE_URL-style single-secret refresh use --update-secrets, per docs/runbooks/pullmd-deploy.md §3.)

---

§3. Verify GREEN after deploy

PROJECT=kh-staging-494815   # or kh-prod-494815
TENANT=phew                 # or kpf

# 1. Latest ready revision == latest created revision (no failed roll-forward).
LATEST_READY=$(gcloud run services describe kh-cocoindex-pipeline-$TENANT \
  --project="$PROJECT" --region=europe-west2 \
  --format='value(status.latestReadyRevisionName)')
LATEST_CREATED=$(gcloud run services describe kh-cocoindex-pipeline-$TENANT \
  --project="$PROJECT" --region=europe-west2 \
  --format='value(status.latestCreatedRevisionName)')
echo "ready=$LATEST_READY created=$LATEST_CREATED"   # must match

# 2. No worker crash in the logs (the audit's smoking gun).
gcloud logging read 'resource.type="cloud_run_revision"
  resource.labels.service_name="kh-cocoindex-pipeline-'"$TENANT"'"' \
  --project="$PROJECT" --region=europe-west2 --limit=50 --freshness=15m \
  --order=desc --format='value(textPayload, jsonPayload.message)'

Expected:

• latestReadyRevisionName matches latestCreatedRevisionName (the new revision went Ready, not the prior one holding traffic after a failed deploy).
• The logs show no cocoindex background thread crashed, no asyncpg gaierror / TargetServerAttributeNotMatched, and no ValueError: Environment settings must provide Settings.db_path — the two boot crashes this manifest fixes.

Failure modes:

• cocoindex background thread crashed + asyncpg gaierror -2 → COCOINDEX_DB_DSN is absent or the host does not resolve. Confirm the secret exists and the DSN uses the region-qualified pooler host (§0.1).
• ValueError … Settings.db_path (or set COCOINDEX_DB …) → COCOINDEX_DB is unset or the volume is not mounted. Confirm the manifest carries the COCOINDEX_DB env + the cocoindex-state volumeMount + the in-memory emptyDir volume (guarded by scripts/tests/test_cocoindex_service_manifests.py).
• latestReadyRevisionName lags latestCreatedRevisionName → the new revision failed its startup probe; read the revision logs above before retrying.

---

§4. Rollback

gcloud run services replace retains the prior revision. To roll back:

gcloud run revisions list --service=kh-cocoindex-pipeline-$TENANT \
  --project="$PROJECT" --region=europe-west2 \
  --format='table(metadata.name,status.conditions[0].status,metadata.creationTimestamp)'

gcloud run services update-traffic kh-cocoindex-pipeline-$TENANT \
  --project="$PROJECT" --region=europe-west2 \
  --to-revisions=<last-good-revision>=100