TECH — On-prem deployment pivot: B1 co-locate cocoindex + pullmd on one host (ID-66.3)

TECH — On-prem deployment pivot: B1 co-locate cocoindex + pullmd on one host (ID-66.3)

Task: ID-66 — On-prem deployment pivot — B1 co-locate cocoindex + pullmd on one host. Subtask: {66.3} TECH. Author date: 29/05/2026. Predecessors (ratified): RESEARCH.md ({66.1}), PRODUCT.md ({66.2}), OQ-CLOSURE.md (OQ-66-1/2/3/4/6 closed). This TECH consumes all three as given and does not re-litigate them. Authored by a fresh Planner instance per Q-PLANNER-2 — PRODUCT.md was read in full as input, not assumed.

Filename convention (ID-48.4): canonical {N.3} artefact is TECH.md. The companion {66.4} PLAN.md (decomposition) is a separate fresh-Planner instance.

Where validation lives. Per the write-tech-spec structure, PRODUCT.md has no Validation section by design — each numbered Proposed change below carries its own Verify line that maps to the PRODUCT invariant(s) it satisfies. The coverage map at the end of the Proposed changes section confirms all 25 invariants are claimed by one or more Proposed changes.

---

Context

B1 re-platforms the ingestion pipeline only — the cocoindex sidecar Service plus the pullmd extractor Service (and pullmd’s Playwright/Trafilatura sidecars) — off Google Cloud Run onto one IONOS VPS (8 vCPU / 16 GB / 480 GB NVMe, ~£21/mo, per OQ-66-1) managed by Coolify as the PaaS layer. The product app (Vercel + managed Supabase) is untouched; the only app↔pipeline seam is the single inbound pipeline-runs/record webhook. B1 dissolves the {62.1} reachability defect by co-location and eliminates the Cloud Run cost floor by using local disk. See PRODUCT.md for the 25 numbered behaviour invariants this plan implements; this section grounds the current-state artefacts the pivot replaces/repoints.

Code-intelligence orientation (cited verbatim per the planner-block)

This is an infrastructure/ops artefact. The orientation tools were run against the KH corpus per .gitnexus/CLAUDE.md; the gitnexus index is STALE (last indexed b8d9f89), so the line numbers below are treated as approximate and the file paths are authoritative.

• gitnexus_query({query: "cocoindex pipeline pullmd adapter", repo: "knowledge-hub"}) → "processes": [] (no execution flows indexed for this surface — the pipeline has never run end-to-end, consistent with PRODUCT §Problem). It returned the pipeline definitions as definitions[]: _pullmd_to_markdown (scripts/cocoindex_pipeline/adapters.py:115-188, module Cocoindex_pipeline), extract_source_provenance (scripts/cocoindex_pipeline/adapters.py:233-278), ingest_file (scripts/cocoindex_pipeline/flow.py:1039-1489), _build_db_ctx (scripts/cocoindex_pipeline/flow.py:651-673), and the cocoindex server-thread symbols _health_handler (scripts/cocoindex_pipeline/server.py:131-144), start_cocoindex_thread (server.py:168-225), _target (server.py:203-217). This matches the {66.2} PRODUCT orientation exactly — the cocoindex→pullmd adapter is a self-contained Python leaf and the /health handler runs on its own thread.
• The webhook emitter _emit_pipeline_run_webhook (scripts/cocoindex_pipeline/flow.py:552-645) reads both PIPELINE_RUN_WEBHOOK_URL and CRON_SECRET from os.environ and is best-effort (missing env or HTTP error logs but does not raise — flow.py:576-577, 588-598). The inbound route app/api/internal/pipeline-runs/record/route.ts:96-175 gates on verifyCronAuth (lib/cron-auth.ts), parses with a Zod BodySchema, and calls recordPipelineRun (lib/pipeline/record-run.ts) → pipeline_runs. The PRODUCT orientation already recorded these as LOW blast radius (_emit_pipeline_run_webhook: 1 direct caller app_main; _pullmd_to_markdown: 2 direct callers, both in adapters.py).

External-library grounding (pre-ratification empirical check — OQ-3 / Q-EX2). This artefact asserts no importable third-party library symbol shape as load-bearing, so no import-and-call check is required:

• The cocoindex→pullmd repoint is env-only (PULLMD_SERVICE_URL), empirically confirmed at scripts/cocoindex_pipeline/adapters.py:125 (os.environ.get("PULLMD_SERVICE_URL"), fail-fast, no hardcoded host, no default) by OQ-66-2 this-session grep. No cocoindex symbol shape is asserted; cocoindex[postgres]==1.0.3 is already empirically pinned (requirements.txt; RESEARCH §6).
• mdb_copy is a CLI tool shipped with LMDB, not an importable Python/TS symbol; its MVCC hot-copy safety is grounded in OQ-CLOSURE OQ-66-4’s verified web research (lmdb.tech / mdb_copy(1) man page).
• Coolify is a hosted-platform capability set, not an importable API; its Compose / per-app-env / S3-DB-backup / deploy-webhook capabilities are grounded in RESEARCH §2 + OQ-CLOSURE OQ-66-3/4/6.

Current-state artefacts this pivot replaces / repoints (files of record)

Artefact	Current state (file-grounded)	B1 disposition
cloudrun/services/{staging,prod}-phew-cocoindex.yaml	run.googleapis.com/ingress: internal (l.23); minScale=maxScale=1 (l.28-29); image …/kh-cocoindex-pipeline:latest; COCOINDEX_SOURCE_PATH="" (l.45-46, idle); COCOINDEX_DB=/cocoindex-state/lmdb (l.70-71); COCOINDEX_LMDB_MAP_SIZE=268435456 (256 MiB, l.75-76); LMDB on emptyDir{medium: Memory, sizeLimit: 512Mi} (l.99-103); COCOINDEX_DB_DSN via secretKeyRef (boot-required, l.55-59); /health startup+liveness probe on :8080 (l.80-92)	Replaced by the cocoindex service in the co-location compose; tmpfs → persistent volume; /health no longer trusted as datapath signal. Manifests remain in git (reversibility).
cloudrun/services/{staging,prod}-pullmd.yaml	pullmd ingress container :3000 (aeternalabshq/pullmd:2.0.0, pinned not :latest, l.81); Playwright sidecar (aeternalabshq/pullmd-playwright:2, l.161-166) over PLAYWRIGHT_URL=http://localhost:8002/render (l.105-106); Trafilatura sidecar (:2, l.173-178) over TRAFILATURA_URL=http://localhost:8001/extract (l.107-108); PULLMD_AUTH_MODE=single-admin (l.92-93, Bearer-gated); CACHE_DB=/tmp/pullmd-cache.db (l.118-119); no /health — probes GET / on :3000 (l.140-154); admin creds + PULLMD_API_TOKEN via secretKeyRef (boot-required, l.125-139); ingress: internal (l.59-60)	Carried over verbatim into the compose service set (already a localhost-coupled compose unit); the cocoindex service joins the same compose network.
.github/workflows/cloud-run-deploy.yml	WIF auth (l.140-153, vars.GCP_WIF_PROVIDER / vars.GCP_DEPLOY_SA_EMAIL); two-image Cloud Build (l.171-211); for TENANT in phew kpf loops for job deploy (l.225), cocoindex-service deploy (l.264), IMAGE_SHA inject (l.310), secret-mount (l.362, l.389), smoke (l.414); NEXT_PUBLIC_CLIENT_ID bake-assert guard (l.222-247); --set-secrets declarative strings (base set l.347-358; COCOINDEX_SECRETS l.387; pullmd l.409); no --vpc-egress/--network flag anywhere (root of {62.1})	Retired by the cutover. Replaced by a thin GH Action (build+push to ghcr.io → Coolify deploy). WIF + deploy-SA retired, not migrated. File remains in git (reversibility).
scripts/cocoindex_pipeline/adapters.py:125	pullmd_url = os.environ.get("PULLMD_SERVICE_URL") — fail-fast (l.126-130), request built f"{pullmd_url}/api" (l.143); PULLMD_API_TOKEN env-only Bearer (l.131-138)	No source change. Repoint is the env var only (OQ-66-2).
scripts/cocoindex_pipeline/flow.py:552-645	_emit_pipeline_run_webhook reads PIPELINE_RUN_WEBHOOK_URL + CRON_SECRET from env (l.588-589); best-effort POST via aiohttp (l.624-645)	No source change. Webhook repoint is the env var only.
app/api/internal/pipeline-runs/record/route.ts + lib/pipeline/record-run.ts + lib/cron-auth.ts	Inbound webhook: verifyCronAuth (route l.97), Zod BodySchema parse, recordPipelineRun → pipeline_runs (route l.143-157)	Zero change. Host-agnostic by construction; lands the same row from any host holding CRON_SECRET.
cloudrun/jobs/{prod,staging}-kpf.yaml + kpf refs	kpf decommissioned but live in 6 files + the for TENANT in phew kpf loop (RESEARCH §4-iii)	kpf dead-manifest cleanup — labelled side-task, see Proposed change 9.
docs/runbooks/cocoindex-deploy.md §0/§1	The two-DB distinction: COCOINDEX_DB = cocoindex engine LMDB path vs COCOINDEX_DB_DSN = KH’s asyncpg Supabase pool; Filestore ~$200/mo + GCS-FUSE/LMDB incompatibility rationale	Superseded for the deploy mechanics by a new B1 runbook (Proposed change 1/8); the engine-LMDB-vs-DSN distinction carries forward unchanged.

---

Proposed changes

Each change is mapped to the PRODUCT invariant number(s) it implements and carries a Verify line (the acceptance step the Checker / operator uses). The implementation is infra plumbing — Compose YAML, a thin GH Action, a backup cron, env values, and a monitor — not KH application code (cocoindex adapters.py / flow.py and the Vercel route are unchanged; OQ-66-2).

1. Co-location docker-compose.yml (Coolify “Docker Compose” build pack)

Author a single docker-compose.yml (new file; proposed location deploy/onprem/docker-compose.yml) declaring four services on one Coolify-managed compose network:

• cocoindex — image ghcr.io/<org>/kh-cocoindex-pipeline:<pinned-tag> (ghcr.io, see change 4; replaces the Cloud Run …/kh-cocoindex-pipeline:latest). Env sets PULLMD_SERVICE_URL=http://pullmd:3000 (compose service alias; http://localhost:3000 is the equivalent if the two share a network namespace) — repoint is env-only, no source change (adapters.py:125, OQ-66-2). Retains PULLMD_API_TOKEN Bearer auth as defence-in-depth (kept even on host-local network). COCOINDEX_SOURCE_PATH and COCOINDEX_DB / COCOINDEX_DB_DSN carry over from the current manifest (boot-required discipline preserved — both must be present before /health serves, per staging-phew-cocoindex.yaml:51-71).
• pullmd — aeternalabshq/pullmd:2.0.0 (pinned, carried verbatim from staging-pullmd.yaml:81); ports 3000; PULLMD_AUTH_MODE=single-admin; PLAYWRIGHT_URL=http://playwright:8002/render, TRAFILATURA_URL=http://trafilatura:8001/extract (compose-alias form of the existing in-pod localhost wiring — invariant 4); CACHE_DB=/data/pullmd-cache.db on a named volume (pullmd-data-staging:/data) — repointed from the original /tmp form by {75.18} (S320): the SQLite store holds share-link ids whose persistence reference_items.pullmd_share_id depends on across container recreates.
• playwright — aeternalabshq/pullmd-playwright:2 (resolve :2 → digest before first deploy per pullmd-deploy.md §4).
• trafilatura — aeternalabshq/pullmd-trafilatura:2.

The shape is additive and reversible: pullmd’s half is already a localhost-coupled compose unit (RESEARCH §1-b); B1 adds the cocoindex service into the same network and repoints one env var. Coolify ingests the Compose build pack unchanged (RESEARCH §2.3 names this as the direct fit; the images are the same ones Cloud Run ran). No Cloud Run probe semantics carry over — the compose cocoindex service may keep a /health healthcheck for container-restart purposes, but it is explicitly not the datapath signal (change 6).

Verify (invariants 1, 2, 3, 4, 11): an operator inspecting the host sees all four containers running on one compose network (inv 1); issuing GET {PULLMD_SERVICE_URL}/api?url=… -H "Authorization: Bearer <PULLMD_API_TOKEN>" from inside the cocoindex container returns a pullmd response over the alias, never a *.run.app URL (inv 2); there is no ingress: internal classification and no VPC/NAT/egress dependency anywhere in the cocoindex→pullmd path — the cross-service hop that could not route on Cloud Run does not exist because the two services share a host-local network, so a test that previously failed for want of a route now succeeds (inv 3 — the {62.1} dissolution, end-to-end-confirmed at migration step 9); pullmd→Playwright/Trafilatura wiring is preserved (inv 4); PULLMD_SERVICE_URL is a non-secret host-local value (inv 11).

2. LMDB persistent volume (replaces the tmpfs)

Replace the Cloud Run emptyDir{medium: Memory, sizeLimit: 512Mi} (the in-RAM LMDB) with a Docker named volume / Compose volumes: mount on the host’s local NVMe disk, mounted at the cocoindex COCOINDEX_DB parent path (/cocoindex-state, so the engine store lands at /cocoindex-state/lmdb). The volume is disk-backed, not tmpfs. Local NVMe mmaps and locks natively (LMDB’s hard requirement; GCS-FUSE never could — RESEARCH §1-c). The same volume can host the corpus directory (COCOINDEX_SOURCE_PATH, currently "") when the pipeline is activated.

Sizing guidance: the Cloud Run LMDB map size was bounded at 256 MiB (COCOINDEX_LMDB_MAP_SIZE=268435456, staging-phew-cocoindex.yaml:75-76) and the tmpfs at 512 MiB; on disk, free space is cheap (480 GB NVMe), so set COCOINDEX_LMDB_MAP_SIZE generously (e.g. a few GiB — LMDB map size is a sparse virtual reservation, not a disk pre-allocation) to avoid a MDB_MAP_FULL re-tune once the pipeline runs at volume. The engine-LMDB-vs-asyncpg-DSN distinction (cocoindex-deploy.md §0) is preserved: this volume backs COCOINDEX_DB (engine store), not COCOINDEX_DB_DSN (the Supabase pool).

Verify (invariants 5, 6, 7): an operator inspecting the mount sees a disk-backed volume, not tmpfs (inv 5); writing memo state, restarting the cocoindex container without destroying the volume, and observing the state intact confirms persistence (inv 6); the spec text documents that LMDB loss forces a memo-rebuild only, never KH data loss — Supabase is canonical (inv 7).

3. Secrets migration to Coolify-native per-app encrypted env

Migrate the ~8–10 runtime secret values out of GCP Secret Manager into Coolify’s per-app encrypted env (hidden-after-save; OQ-66-3 — Infisical deferred to B2). The set, derived from cloud-run-deploy.yml:347-409:

• Base set the cocoindex service needs: ANTHROPIC_API_KEY, OPENAI_API_KEY, SUPABASE_URL, SUPABASE_PUBLISHABLE_KEY, SUPABASE_SERVICE_ROLE_KEY, NEXT_PUBLIC_SUPABASE_URL, NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY, NEXT_PUBLIC_APP_URL, CRON_SECRET, SENTRY_AUTH_TOKEN.
• cocoindex-specific: COCOINDEX_DB_DSN (Supabase pooler string, boot-required), PIPELINE_RUN_WEBHOOK_URL (change 7), PULLMD_API_TOKEN. PULLMD_SERVICE_URL largely disappears as a secret — it becomes the non-sensitive http://pullmd:3000 (inv 11).
• pullmd service: PULLMD_ADMIN_EMAIL, PULLMD_ADMIN_PASSWORD, PULLMD_API_TOKEN.

Mechanics: one-time manual copy — for each value run gcloud secrets versions access latest --secret=<name> to read it, then paste into Coolify’s per-app env UI. Dropped, not migrated: the GCP deploy-SA / WIF credentials — B1 eliminates gcloud run deploys entirely (change 4), so no deploy-SA JSON ever lands on the host.

Verify (invariants 10, 11, 12): auditing Coolify env shows the runtime secret set stored encrypted/hidden, not read from GCP Secret Manager at runtime (inv 10); PULLMD_SERVICE_URL is a plain non-secret value (inv 11); a host secrets audit finds no GCP deploy-SA JSON — only runtime secrets (inv 12).

4. Thin GitHub Action — build off-host → push pinned images to ghcr.io → Coolify pull

Author a new thin workflow (proposed .github/workflows/onprem-deploy.yml) that replaces the build/deploy chain of cloud-run-deploy.yml:

1. On push to the deploy branch (paths scripts/**/*.py, requirements.txt, deploy/onprem/**), build the cocoindex image on GitHub-hosted runners (the heavy ~9 GB two-image build — cocoindex ~5.3 GB + Playwright ~3.7 GB — never runs on the 16 GB host; OQ-66-6: Coolify warns resource-intensive on-host builds crash the server).
2. Push the image to ghcr.io under a pinned, content-addressable tag (e.g. :sha-<gitsha> or :<semver>, never :latest — mirrors the existing pullmd no-:latest discipline asserted at cloud-run-deploy.yml:458).
3. Call Coolify’s deploy webhook/API to pull and deploy the pinned image. The host pulls finished images; it does not build.

Rollback = re-point image tag, no rebuild: to roll back, set the deployed image tag to a prior pinned tag via Coolify’s API/UI and redeploy (near-instant pull; OQ-66-6 confirms Coolify supports tag-edit rollback). Rolling forward is the same op with a newer tag.

Retirement: cloud-run-deploy.yml + its WIF auth are no longer the active deploy path once B1 is live; the file and the Cloud Run manifests remain in git (supporting invariant 24). NEXT_PUBLIC_CLIENT_ID is NOT a B1-host concern (OQ-66-6): the Next.js app builds on Vercel (no Dockerfile in repo), and neither cocoindex nor pullmd is the app — the B1 host build does not touch the bake. The cloud-run-deploy.yml:222-247 slug-pattern assert retires with the workflow; if multi-tenant pipeline deploys ever resume under B2, the equivalent build-arg + slug-pattern check is re-implemented in this thin Action (RESEARCH §1-d preservation requirement — flagged for B2, not B1).

Verify (invariants 13, 14, 15, 16): a build runs on GH runners with no host CPU/RAM contention (inv 13); a push pushes a pinned (non-:latest) image to ghcr.io then triggers a Coolify pull (inv 14); rollback re-points a tag with no host rebuild (inv 15); cloud-run-deploy.yml/WIF are not the active path post-cutover but remain in git (inv 16).

5. LMDB mdb_copy → S3 backup job

Author a scheduled backup job (a Coolify “scheduled task” / host cron, or a tiny sidecar container sharing the LMDB volume; proposed deploy/onprem/backup/lmdb-backup.sh) because Coolify’s native S3 backups are database-only and do not cover a raw named volume (OQ-66-4). The job:

1. Runs mdb_copy -c /cocoindex-state/lmdb /snapshot (the -c flag compacts and drops free pages). mdb_copy copies through an LMDB read transaction — LMDB’s MVCC read-snapshot is immutable for the txn lifetime regardless of concurrent writes — so it produces a consistent hot snapshot with no quiesce/stop (OQ-66-4). A naive cp/tar of the live data.mdb is never used (it can capture a torn page set).
2. tar (or restic) the snapshot and ship it to the same S3-compatible bucket Coolify already targets for its DB backups.

Schedule: B1’s write volume is low and the LMDB is small (≤256–512 MiB historically), so the job is cheap — a daily (or twice-daily) cron is ample; retention is the operator’s choice within the 100 GB IONOS backup allowance (OQ-66-1).

Restore procedure (documented in the new runbook): fetch the most recent S3 snapshot, untar onto the persistent volume at /cocoindex-state/lmdb, and resume the cocoindex container. The restored store is a valid cocoindex memo state; if the snapshot is stale or unavailable, the fallback is a memo-state rebuild (invariant 7), not data loss.

Verify (invariants 17, 18, 19, 20): a scheduled job lands an LMDB snapshot in the S3 bucket, separate from Coolify’s DB backup (inv 17); a snapshot taken during active writes restores to a consistent, openable LMDB (inv 18 — the load-bearing mdb_copy safety claim); the documented restore path is exercisable and resumes the engine (inv 19); the backup path touches Supabase not at all (inv 20).

6. Datapath monitoring — pipeline_runs row-arrival watch (not /health)

Design a datapath monitor (proposed deploy/onprem/monitor/datapath-watch.ts or a small scheduled query) that watches pipeline_runs row arrival in Supabase, because the cocoindex /health probe is a liveness no-op on a separate thread (green ≠ ingesting; RESEARCH §4-i, server.py:131-144). The monitor:

• Treats “container up + /health green” as insufficient evidence the datapath works.
• Queries pipeline_runs for the most recent row (filter to active-ingestion windows) and compares the newest created_at/run timestamp against a configurable stall threshold (proposed default: no terminal-status row within the expected window of a triggered ingestion → flag a datapath stall). A successful ingestion is confirmed by row arrival, not by the probe.
• Routes a stall alert via Coolify’s notification channels (Discord/Telegram/email; RESEARCH §1-f) or Sentry.

The threshold/window values are operator-tunable and recorded in the runbook; this design does not trust container liveness as the health signal.

Verify (invariants 21, 22): with containers up and /health green but no pipeline_runs rows in the expected window, the monitor flags a datapath stall, not “healthy” (inv 21, 22); a real ingestion is confirmed by the arriving row (inv 22).

7. Webhook repoint — host env PIPELINE_RUN_WEBHOOK_URL (zero app change)

Set the cocoindex container’s host env PIPELINE_RUN_WEBHOOK_URL to the same Vercel app URL the Cloud Run deploy mounted (cloud-run-deploy.yml:387), with the same CRON_SECRET (change 3). _emit_pipeline_run_webhook (flow.py:588-589) reads both from os.environ — no source change. There are zero changes to app code, proxy.ts, the route handler (route.ts:96-175), or Vercel config. The endpoint, its auth (verifyCronAuth), and its row-writing (recordPipelineRun → pipeline_runs) are byte-for-byte as today.

Verify (invariants 8, 9): the only change is two host env vars; the app/route/proxy.ts are untouched (inv 8); a completed/failed run on the new host lands a pipeline_runs row of the same shape it would have had from Cloud Run (inv 9).

8. Cost-floor removal + reversibility (revert path)

Deleted (cost floor gone, invariant 23): no Filestore ($200/mo — local disk mmaps the LMDB natively); one host instead of two warm minScale=maxScale=1 Services; no per-deploy two-image Cloud Build (builds on GH runners, change 4). Standing cost = the single IONOS VPS line (£21/mo all-in; OQ-66-1) in place of the warm-×2 + Filestore + NAT/Artifact-Registry floor.

Kept (reversibility, invariants 24, 25): the Cloud Run manifests (cloudrun/services/**, cloudrun/jobs/**) and cloud-run-deploy.yml + WIF remain in git history. Revert path: cocoindex→pullmd URL reverts with one env var (PULLMD_SERVICE_URL, OQ-66-2); webhook target reverts with one env var (PIPELINE_RUN_WEBHOOK_URL, change 7); secrets move back to Secret Manager; deploy path reverts to the WIF workflow. No B1 step requires an irreversible code change. The LMDB memo state is the single genuinely stateful artefact (invariant 25) — its loss costs a rebuild, not KH data loss; everything else is re-pointable config; the KB is never at risk (Supabase canonical + independently backed up).

Verify (invariants 23, 24, 25): standing cost shows one VPS line, no Filestore/warm-×2 (inv 23); each B1 item has a one-env-var or config revert documented and the Cloud Run artefacts are present in git (inv 24); the runbook names the LMDB as the one stateful artefact (inv 25).

9. kpf dead-manifest cleanup (clearly-labelled B1-cutover SIDE-TASK, not a core step)

This is not a core B1 migration step. It is recorded so {66.4} PLAN can scope it as a side-task. It rides the B1 cutover because retiring cloud-run-deploy.yml (change 4 / invariant 16) is the natural moment to drop the retired-but-still-deployed kpf tenant (RESEARCH §4-iii). It must not block or bloat the core migration.

Exact deletion / edit set (file-grounded this session):

1. Delete cloudrun/jobs/prod-kpf.yaml (dedicated kpf Job manifest).
2. Delete cloudrun/jobs/staging-kpf.yaml (dedicated kpf Job manifest).
3. Edit cloudrun/jobs/staging-phew.yaml:44 — kpf reference in a comment ([[ "$NEXT_PUBLIC_CLIENT_ID" =~ ^(phew|kpf)$ ]]); drop kpf from the slug pattern.
4. Edit cloudrun/services/prod-pullmd.yaml:3 — “kpf decommissioned” comment (cosmetic; update or drop).
5. Edit cloudrun/services/staging-pullmd.yaml:3 — same “kpf decommissioned” comment.
6. Edit cloudrun/services/README.md — kpf SA-mapping line (l.47) + the two kh-cocoindex-pipeline-kpf smoke-verify command blocks (l.65, l.76) that target a Service with no manifest (cocoindex/pullmd are phew-only).
7. Remove the for TENANT in phew kpf loop from cloud-run-deploy.yml (l.225, l.264, l.310, l.362, l.389, l.414) — mooted if change 4 retires the whole workflow, so the simplest realisation is: the workflow retirement subsumes item 7; items 1–6 are the residual cleanup of files the retirement does not delete.

Verify: grep -rl kpf cloudrun/ returns zero files after the cleanup (down from the 6 of RESEARCH §4-iii); no kpf tenant is referenced by any active deploy path.

Invariant coverage map

Every PRODUCT invariant 1–25 is claimed by one or more Proposed changes above (inv 11 is legitimately claimed twice — Change 1 for the compose-wiring topology fact, Change 3 for the secret-disappearance fact). Change 9 (kpf cleanup) is in-scope-adjacent and maps to no core invariant by design.

PRODUCT invariant	Proposed change
1 — Single-host co-location	1
2 — cocoindex reaches pullmd host-locally	1
3 — Reachability defect dissolved, not patched	1
4 — pullmd sidecar wiring preserved	1
5 — LMDB on persistent local disk	2
6 — LMDB survives container restart	2
7 — LMDB is engine store, not KH datastore	2
8 — Webhook repoint is one host env var; app unchanged	7
9 — A run on the new host lands a pipeline_runs row	7
10 — Secrets in Coolify-native encrypted env	3
11 — PULLMD_SERVICE_URL largely disappears as a secret	1, 3
12 — GCP deploy identity retired, not migrated	3
13 — Build happens off-host on GitHub runners	4
14 — Push-to-deploy pushes pinned images, then Coolify pulls	4
15 — Rollback is re-pointing an image tag, no rebuild	4
16 — cloud-run-deploy.yml + WIF retired by the cutover	4
17 — LMDB volume backed up on a schedule to S3	5
18 — Snapshot taken safely while the store is live (mdb_copy)	5
19 — A restore path is defined and exercisable	5
20 — Supabase canonical, backed up independently	5
21 — Monitoring watches pipeline_runs row arrival, not /health	6
22 — A datapath stall is observably distinct from liveness	6
23 — The Cloud Run cost floor is gone	8
24 — Every B1 item re-pointable back to Cloud Run	8
25 — LMDB memo state is the single genuinely stateful artefact	8

---

Migration plan / ordering

The ~1.5–2.5-day step sequence (the spine is OQ-CLOSURE OQ-66-1’s effort table; sequencing is chosen so each step’s output is verifiable before the next depends on it):

1. Stand up the IONOS VPS + base hardening (~1–2 h). Provision the 8 vCPU / 16 GB / 480 GB box; SSH/firewall hardening.
2. Install Coolify (one-script install) (~0.5–1 h). Verify the dashboard + compose build pack are available.
3. Author + deploy the co-location compose (change 1) (~2–4 h). cocoindex + pullmd + Playwright + Trafilatura over the compose network; set PULLMD_SERVICE_URL=http://pullmd:3000. Depends on the ghcr.io cocoindex image existing → can be unblocked by a first manual GH Action run (step 6) or a manual push.
4. Create + mount the LMDB persistent volume (change 2) + the corpus volume (~0.5–1 h). Replaces the tmpfs; must precede the first real ingest so memo state persists.
5. Migrate ~8–10 secret values into Coolify env (change 3) (~1–2 h). Gates the cocoindex boot (COCOINDEX_DB_DSN) and the webhook (CRON_SECRET).
6. Thin GH Action → ghcr.io → Coolify deploy webhook (change 4) + begin retiring cloud-run-deploy.yml/WIF (~3–5 h). The heaviest step; produces the pinned image steps 3 consume.
7. Repoint the inbound webhook env (change 7) + DNS/SSL via Coolify auto-Let’s-Encrypt for any host-facing endpoint (~1–2 h). No app change.
8. LMDB mdb_copy → S3 backup job (change 5) (~1–2 h).
9. First end-to-end ingest smoke + datapath monitor (change 6) (~2–3 h). Watch pipeline_runs row arrival, not /health, as the success signal — this is the step that proves the {62.1} defect is dissolved (invariant 3).
10. kpf dead-manifest cleanup (change 9) — folded into the cloud-run-deploy.yml retirement of step 6; the residual file edits (items 1–6) land alongside.

The cutover order keeps B1 reversible at every step: nothing irreversible happens until the old Cloud Run path is torn down, and even that is git-recoverable.

---

Risks and mitigations

• 16 GB RAM is the one dimension neither VPS exceeds (OQ-66-1). cocoindex (~cpu2/mem4Gi)
  • Playwright (~3.7 GB) + Trafilatura + LMDB + monitor share 16 GB. Mitigation: builds run off-host (change 4) so the host RAM is never contended by a build; a RAM bump is the likely first scale step if corpus + concurrent ingest grow. Watch host memory in Coolify’s monitoring.
• Long-running mdb_copy under heavy writes transiently grows the on-disk file (the read txn pins old pages from reuse; OQ-66-4). Mitigation: negligible at B1’s low write volume; the snapshot is small and fast; -c compaction keeps the copy lean. Re-evaluate if write volume rises.
• Off-host build depends on GitHub Actions minutes (change 4). Mitigation: KH is a private repo with 2,000 free minutes/mo and GH cut runner prices ~40% on 01/01/2026 (OQ-66-6); a periodic two-image build fits comfortably. GH Actions cache + ghcr.io layer cache make rebuilds durable.
• Coolify raw-volume backup gap (OQ-66-4). Coolify will not back up the LMDB named volume natively. Mitigation: the dedicated mdb_copy→S3 job (change 5) — not a blocker, a small bespoke job.
• /health-green false assurance (RESEARCH §4-i). Trusting container liveness would re-introduce the Cloud Run blind spot. Mitigation: the datapath monitor (change 6) treats /health as insufficient and watches pipeline_runs row arrival.
• Image-tag drift on sidecars. Playwright/Trafilatura are pinned to the floating :2 major tag (staging-pullmd.yaml:162,174). Mitigation: resolve :2 → a digest before first deploy per pullmd-deploy.md §4; pin the digest in the compose file.

---

Notes for the {66.4} PLAN planner

• Heaviest decomposition area: change 4 (thin GH Action + ghcr.io + Coolify deploy webhook + cloud-run-deploy.yml/WIF retirement) — multiple discrete sub-slices (build-and-push job; Coolify webhook wiring; rollback-by-tag mechanics; workflow retirement). Likely 3–4 Subtasks on its own.
• Chain dependencies (sibling-only): the compose deploy (change 1) depends on the ghcr.io image existing (change 4’s build-and-push slice). The first end-to-end smoke (change 6) depends on the volume (change 2), secrets (change 3), and webhook (change 7). These are all within Task 66 → sibling-only deps hold; no cross-Task dep is needed.
• kpf cleanup (change 9) is a clean standalone side-task Subtask — keep it separate from the core migration Subtasks; it should depend on the cloud-run-deploy.yml retirement slice of change 4.
• Estimated effort > 2h overall, multi-migration, chain-dependent → {66.4} PLAN is WARRANTED (not skippable). Roughly maps to a 9–10 step migration (the ordering above) plus the kpf side-task — expect ~10–14 Subtasks, comfortably under the 25 soft ceiling.
• No KH application code changes in any Subtask: adapters.py / flow.py / the Vercel route are unchanged (OQ-66-2). Subtask details should make this explicit so an Executor does not “fix” the env-only repoint in source.