On-prem B1 deploy runbook (ID-66)

On-prem B1 deploy runbook (ID-66)

⚠️ SUPERSEDED (S436, 2026-07-02): current topology is the two-server split — Server A client 88.208.255.178, Server B platform 87.106.72.59 (ONE Coolify, on B); the 77.68.122.71 build IP throughout this record is the original single box, and Coolify now runs on Server B. Pipeline ingress is HTTPS on all four apps with path scope /walk|/health|/extract (only /stage unrouted) — but it is no longer public on all four: the two PLATFORM apps on Server B went private (S558, 2026-08-12) behind a Cloudflare Tunnel + Access service-token policy, with Server B’s inbound 443+80 closed at the VPS firewall. Every “reachable over public HTTPS” statement below therefore still holds for the two CLIENT apps on Server A and is superseded for the platform pair; probes against a platform FQDN need CF-Access-Client-Id/CF-Access-Client-Secret alongside the bearer. See runbooks/private-ingress-cutover.md and reference/deployment-architecture.md §2/§4/§9.

⚠️ SECRET SPLIT ({127.18}; dual-accept retired S457 — noted 22/07/2026): every CRON_SECRET bearer on /walk in this record is HISTORICAL. The live /walk (and /producer-run, and the pipeline-runs webhook route) bearer is the dedicated PIPELINE_TRIGGER_SECRET — CRON_SECRET no longer authenticates there (it survives only on the app’s Vercel /api/cron/* routes). Any command below that reads $CRON_SECRET / os.environ["CRON_SECRET"] against /walk must use PIPELINE_TRIGGER_SECRET today — the copy-paste forms in §B2 are corrected inline. Also historical: PullMD is retired — no compose file carries pullmd services any more (the {75.18} section below is an execution record of a removed stack).

SUPERSEDED TOPOLOGY — read this first (S408 / ID-127.15). This runbook is the historical execution record of the ID-66 Cloud-Run→on-prem migration: the first client (Phew) pipeline stood up as prod + staging co-tenanted on ONE IONOS VPS (77.68.122.71), under the kh-onprem-pipeline-* names and the TEMPORARY aisolutionhub.co.uk domain, before the platform pipeline existed. The ratified target topology is now the two-server split (S413/S414): Server A = the client pipeline (kh-onprem-pipeline-*, the existing stack this runbook describes); Server B = a dedicated platform pipeline VPS (ca-pipeline-platform-*), with platform staging + prod isolated by ca-suffix. One Coolify control-plane (on Server B) manages both servers; client vs platform is distinguished by server + app UUID + deploy input, NOT by separate credentials, and there is no new GH environment (OQ-2). Naming migrates kh→ca / knowledge-hub→canonical; the client domain flips off aisolutionhub.co.uk to kh.phew.org.uk at the ID-45 handover. Read this runbook for HOW the client box was built and is operated; read specs/id-127-platform-pipeline/AMENDMENT-staging-prod-two-server.md (ratified)

• specs/id-127-platform-pipeline/GREENFIELD-standup-runbook.md for the two-server / ca-naming target. Where the two conflict, the amendment wins.

B1 re-platforms the ingestion pipeline only (cocoindex + pullmd + Playwright + Trafilatura) off Google Cloud Run onto one IONOS VPS managed by Coolify. The product app (Vercel + managed Supabase) is untouched. See docs/specs/ID-66-onprem-pivot/ (RESEARCH / PRODUCT / TECH / PLAN) for the full design and the 25 PRODUCT invariants.

Scope note (post-split). “one IONOS VPS” below describes the client box (Server A) as built. It is no longer the whole picture: the platform pipeline lives on a separate VPS (Server B) per the two-server split — see the superseded-topology banner above. This runbook is not the place to stand up the platform pipeline; use the ID-127 amendment + GREENFIELD standup runbook.

Status legend: ✅ done · 🔶 in progress / blocked on operator · ⬜ not started.

Cloud Run retirement (S298)

Cloud Run is fully removed — not reversible. This supersedes the spec’s one-env-var revert framing: there is no Cloud Run deploy path to fall back to. The cloudrun/ manifest directory and .github/workflows/cloud-run-deploy.yml have been deleted from the repository (full teardown). On-prem / Coolify is now the SOLE pipeline deploy path.

What this removes:

• Cost floor. No Filestore instance (~$200/mo). No warm ×2 Cloud Run Services holding minScale = maxScale = 1. No per-deploy two-image Cloud Build.
• The dedicated Cloud Run operational runbooks (cloud-run-phase-1.md, cocoindex-deploy.md, pullmd-deploy.md) are retained for historical context only and carry RETIRED banners — their referenced cloudrun/ manifests no longer exist.

What replaces it:

• Standing cost = a single IONOS VPS (~£21/mo). No warm-instance floor.
• LMDB is the primary genuinely stateful artefact. Its loss means a memoisation rebuild (recompute, slower next run) — never KH data loss. All canonical KH data lives in managed Supabase, which is untouched by this teardown. (Since S320 the pullmd share-link store is a second, lower-stakes stateful artefact — see the {75.18} section below.)

Host

Field	Value
Provider	IONOS (IONOS Images, Ubuntu 24.04.4 LTS)
Public IPv4	77.68.122.71
Spec	8 vCPU · ~16 GB RAM · 480 GB NVMe (/dev/vda1 → /, 462 G free)
Kernel	6.8.0-124-generic

Access (✅)

SSH is key-only (password auth disabled). Dedicated key generated for this host:

ssh -i ~/.ssh/kh_ionos_ed25519 root@77.68.122.71

• Private key: ~/.ssh/kh_ionos_ed25519 (ed25519, no passphrase — usable by CI later). Back this key up outside the laptop; losing it locks out the host (password auth is off).
• The original IONOS root password was used once to install the key and is stored in .env.local as IONOS (gitignored). It is now unusable for SSH (password auth disabled) but kept for the IONOS Cloud Panel login.

{66.5} Base hardening (✅)

Applied 30/05/2026:

• SSH key-only — /etc/ssh/sshd_config.d/00-kh-hardening.conf (sorts before 50-cloud-init.conf; sshd uses the first-obtained value). Sets PasswordAuthentication no, KbdInteractiveAuthentication no, ChallengeResponseAuthentication no, PubkeyAuthentication yes, PermitRootLogin prohibit-password. Verified: key login works, password auth refused.
• Firewall (ufw, active) — inbound allow 22, 80, 443, 8000, 6001, 6002 (tcp, v4+v6); default deny incoming. These are the SSH + Coolify ports.
• OS updates — apt upgrade (image already current at provision time); unattended-upgrades installed + enabled (/etc/apt/apt.conf.d/20auto-upgrades).

sshd drop-in precedence gotcha: the cloud image ships 50-cloud-init.conf (PasswordAuthentication yes) and 60-cloudimg-settings.conf (no). sshd uses the first match across sshd_config.d/*.conf in lexical order, so the yes won until the 00- override was added. Do not delete 00-kh-hardening.conf.

{66.6} Coolify install (✅ done — port 8000 open, admin created, dashboard validated S285)

• Installed via curl -fsSL https://cdn.coollabs.io/coolify/install.sh | sudo bash. Coolify 4.1.1. Containers healthy: coolify, coolify-db, coolify-realtime, coolify-redis, coolify-sentinel. Docker installed by the script.
• Dashboard responds host-locally (curl localhost:8000 → HTTP 302).

✅ External dashboard access — RESOLVED (S285; runbook corrected S287)

Option A was taken in S285 (this section previously read “blocked” and was never updated — corrected S287): TCP 8000 was opened in the IONOS Cloud Panel cloud firewall (host ufw already allowed it), the Coolify admin account was created at http://77.68.122.71:8000, and the dashboard was validated. Re-confirmed reachable S287: http://77.68.122.71:8000 → HTTP 302 (was timeout); 80 → 404 (Coolify proxy answering). The {66.6} acceptance items (Compose build pack, per-app encrypted env, S3 backup, deploy-webhook, auto-Let’s-Encrypt) are operator-verifiable in the UI now, and Coolify config for {66.10} secrets / {66.11} deploy-webhook is no longer firewall-blocked.

Option B (still the preferred long-term hardening, ties into {66.13}) — dashboard behind a domain on 443: point a DNS A record (e.g. coolify.kh.phew.org.uk) → 77.68.122.71, set it as Coolify’s instance FQDN so the dashboard serves over https://… on 443 and 8000 can be re-closed at the provider (auto-Let’s-Encrypt issues the cert).

⚠️ Back up the Coolify env

install.sh warns: back up /data/coolify/source/.env (Coolify app key + DB creds) to a password manager — required to restore the Coolify control plane.

{66.11} Burn-safe deploy + activation (🔶 wired in code; live activation = operator)

The .github/workflows/onprem-deploy.yml deploy-cocoindex job wires the deploy. This section is the load-bearing operating procedure: the on-prem pipeline burned Anthropic tokens twice across S294-S297, so “no push path can restart the app on a stale/broken image or trigger an Anthropic crash-loop” carries equal weight to wiring the deploy itself.

The automated flow

Every push to main or staging starts the workflow, as does a manual workflow_dispatch (the workflow-level paths: filter is retired — see the change-detection note below):

1. detect-changes (ID-86 / bl-236) diffs the full before..sha compare and emits relevant=true only when the push touched scripts/**/*.py, requirements.txt, or deploy/coolify/** (a workflow_dispatch is always relevant). Irrelevant pushes stop here — the heavy jobs below skip, visibly, in the run log.
2. build-and-push-cocoindex (gated on relevant == 'true') builds the image off-host and pushes ghcr.io/ai-solution-hub/kh-cocoindex-pipeline:sha-<commit>, then pull-verifies it.
3. deploy-cocoindex (needs: the detection + build jobs) PATCHes COCOINDEX_IMAGE_TAG to that exact sha-<commit> (update-in-place, never a duplicate key) …
4. … then triggers a Coolify deploy. Coolify re-clones the git compose and pulls the freshly-built pinned tag (the host never builds).

Change detection replaces the paths: filter (ID-86 / bl-236). GitHub’s push-event paths: filter inspects only the first 300 changed files of a compare, so align-class pushes (S319: 1,522 files, first scripts/ hit at position 1,467) silently skipped the workflow on both main and staging — no error, no run. That silent-skip class is fixed by the in-job detect-changes gate, which diffs the full compare with no file-count ceiling. The manual workflow_dispatch fallback remains available as belt-and-braces (it always evaluates relevant=true), but is no longer the only recourse after a large push.

Because the deploy job needs: the build job, it can never deploy a tag that was not just built, pushed and pull-verified — the “never a stale image” ordering guard.

B1 — race reconciliation: the Action must be the SOLE deploy trigger

There are two mechanisms that can trigger a Coolify deploy:

• Coolify git-auto-deploy — fires immediately on a git push to main. The build takes ~15 min, so an auto-deploy would redeploy on the STALE current COCOINDEX_IMAGE_TAG long before the new image exists. This is the stale-image footgun.
• The Action’s deploy-cocoindex job — fires only after the build + tag bump.

To guarantee “never a stale image”, Coolify git-auto-deploy MUST be disabled so the Action is the sole trigger. Two disable mechanisms:

• (a) RECOMMENDED — disable auto-deploy in the Coolify UI for kh-onprem-pipeline-production (the per-app auto-deploy toggle). This toggle is not exposed by the Coolify public API / MCP, so it is a manual UI action — flagged for the operator.
• (b) API-achievable alternative — watch_paths sentinel. Set the app’s watch_paths to a path that can never match a real change (e.g. .coolify-never-auto-deploy) so a git push never satisfies the auto-deploy filter, while the Action’s API deploy still works.

Coordination hazard: apply the disable at the same time the workflow change lands on main and the GH secrets (COOLIFY_BASE_URL) are created. If you disable auto-deploy before the Action can deploy, deploy-on-push silently stops working in the gap; if you land the Action before disabling auto-deploy, a push double-fires (stale + fresh).

B2 — Boot-never-walks burn gate + the /walk trigger (ID-83 / bl-221)

The manual SOURCE_PATH-blanking valve is RETIRED (ID-83 / bl-221). The burn gate is no longer “keep COCOINDEX_SOURCE_PATH empty” — it is architectural: the cocoindex worker boots lifespan-only (coco.start_blocking() enters the @coco.lifespan — asyncpg pool + LMDB engine — and returns) and runs ZERO corpus walk at boot, regardless of whether COCOINDEX_SOURCE_PATH is set. A deploy / auto-restart can therefore never auto-walk the corpus or burn Anthropic tokens. COCOINDEX_SOURCE_PATH MAY now stay set permanently to /cocoindex-state/corpus on both envs — no operator must remember to blank it.

The corpus walk fires only on an explicit signal: a bearer-gated POST /walk on the co-resident aiohttp server (scripts/cocoindex_pipeline/server.py). The handler runs one non-live update pass (KH_PIPELINE_APP.update_blocking(live=False)), reusing the cached lifespan env entered at boot, then returns the worker to idle.

• Auth: Authorization: Bearer $PIPELINE_TRIGGER_SECRET ({127.18} — originally CRON_SECRET; the dual-accept window was retired S457 and CRON_SECRET no longer authenticates here). Missing/wrong bearer → 401; PIPELINE_TRIGGER_SECRET unset → 503 (fail closed).
• Single-flight: a second /walk while one is in flight → 409 (never two concurrent walks burning Anthropic). A failed walk releases the lock (does not wedge it).
• Idle source: if COCOINDEX_SOURCE_PATH is unset/missing → named 400 (“nothing to walk”), not a silent no-op.
• Optional body: {"full_reprocess": true} → a full, cache-invalidating re-walk (default is incremental).
• Async: returns 202 Accepted + requestId immediately; the walk runs on a worker thread and its completion is observed via the existing pipeline_runs webhook + the {66.15} datapath monitor — NOT inline in the /walk response.

Per-env Coolify scheduled-task stand-up (operator step)

✅ CREATED — disabled (S312); command form FIXED + verified (ID-62.12, 06/06/2026). Both /walk scheduled tasks exist but enabled: false (mechanism in place; no walk fires, no Anthropic burn until cutover): prod task uuid gqi55umn3m447gguf2vbp5iq (container cocoindex), staging z1197cpafcghwlfsxa22xzlq (container cocoindex-staging — corrected from cocoindex in {62.12}); command = the absolute-interpreter-path python3 + urllib loopback POST /walk form below (the S312 bare-python3 form exits 127 in the buildpack container — see the caution at the end of this section); cadence 0 2 * * * (placeholder — Liam picks the final cadence, or switches to purely event-driven, when enabling at cutover).

The walk schedule is a per-app Coolify scheduled task — per-app config, not git-tracked (the same operability class as “git-auto-deploy disabled”). The mechanism (/walk route + boot decouple) is fully git-tracked + tested; only the cron cadence lives in Coolify. Stand up one scheduled task per env:

Env	Coolify app uuid	Loopback host
prod (kh-onprem)	ybiczck7f7e1xbdev3bk89cr	http://127.0.0.1:8080/walk
staging (kh-onprem-staging)	ljurtiab99kb6dst38k76419	http://127.0.0.1:8080/walk

No curl in the image. The slim buildpack cocoindex image ships no curl (its healthcheck uses bash /dev/tcp for exactly this reason). The scheduled-task command MUST therefore be dependency-free. Use one of:

• Absolute-path python3 + urllib (RECOMMENDED — the verified-202 form from {62.12}, deployed on both tasks 06/06/2026). Invoking the interpreter by its buildpack layer path needs no PATH injection (the exec PATH omits the python layers — the bare-python3 trap below), and no PYTHONUSERBASE (the poke is stdlib-only: os + urllib). Failure signalling is correct: urlopen raises on any non-2xx, so the task execution records success + message 202 on a good walk and a failed execution otherwise:

  /layers/google.python.runtime/python/bin/python3 -c 'import os,urllib.request as u;print(u.urlopen(u.Request("http://127.0.0.1:8080/walk",method="POST",headers={"Authorization":"Bearer "+os.environ["PIPELINE_TRIGGER_SECRET"]}),timeout=30).status)'

  ({127.18} correction: the form was verified live in {62.12} with CRON_SECRET; the env var read MUST now be PIPELINE_TRIGGER_SECRET — confirm the stored Coolify task commands were re-pointed too before enabling at cutover.) Coolify command-length limit (~255 chars, {62.12}). The Coolify API rejects (HTTP 500) scheduled-task command strings longer than ~255 characters. The otherwise-equivalent env PYTHONUSERBASE=… PATH=…-prefixed form (§B5 env block) is ~358 chars and cannot be stored — the absolute-interpreter-path form above is the fix. The env-prefix itself was proven to execute fine in-container (python3 --version → Python 3.13.13); it is only the stored-command length that rules it out.

• bash /dev/tcp (no Python interpreter spin-up; PATH-immune fallback): beware its exit code does NOT reflect the HTTP status (a 401/500 response still records success) — prefer the python form for the standing tasks:

  bash -c 'exec 3<>/dev/tcp/127.0.0.1/8080; printf "POST /walk HTTP/1.0\r\nHost: 127.0.0.1\r\nAuthorization: Bearer %s\r\nContent-Length: 0\r\n\r\n" "$PIPELINE_TRIGGER_SECRET" >&3; cat <&3'

• the committed helper python3 -m scripts.cocoindex_pipeline.trigger_walk — only stand this up if the inline forms above prove awkward in the Coolify task editor (the helper is NOT committed by default; the two inline forms cover the need).

Create it via the Coolify MCP scheduled_tasks(action='create', …) or the UI: container: the cocoindex service container; command: one of the above; frequency: a cron expr (cadence is an operator/product decision — OQ-1, e.g. nightly 0 2 * * *); enabled: true. For a supervised manual walk, run the same loopback POST by hand (Coolify “Run now” on the task, or docker exec <cocoindex-container> <command>).

docker exec + python3 = exit 127 without the buildpack env (S316) — RESOLVED for the scheduled tasks (ID-62.12, 06/06/2026). The buildpack image’s exec PATH does not include the python runtime layer. Any manual docker exec … python3 … form MUST inject -e PYTHONUSERBASE=/layers/google.python.pip/pip -e PATH=/layers/google.python.pip/pip/bin:/layers/google.python.runtime/python/bin:/usr/bin:/bin (the bash /dev/tcp form is unaffected). Coolify scheduled-task commands also execute inside the container, so the S312 bare-python3 task command would have exited 127 at OQ-1 enable. {62.12} re-pointed both task commands at the absolute interpreter path (/layers/google.python.runtime/python/bin/python3 — no PATH or PYTHONUSERBASE needed for the stdlib-only poke) and verified the form live: a temporary every-minute staging task executed it with result success / message 202 (bearer-authed walk accepted) before being deleted. Manual docker exec invocations of anything beyond the stdlib poke still need the full env block above.

Standing cadence — BI-19 / ID-75 D-3 (staging live 06/06/2026)

The documented end-to-end cadence for URL-sourced content (BI-19 acceptance requires a DOCUMENTED cadence; mechanism ratified by Liam S318, OQ-T2):

1. Poller discovery ≤ 15 minutes — the Vercel cron poller runs every 15 minutes (vercel.json, bearer-gated by CRON_SECRET), so a newly published article is discovered within 15 minutes of appearing in a feed.
2. Nudge-triggered walk ≈ minutes — when a poller run passes ≥ 1 article, it fires a fire-and-forget POST {COCOINDEX_WORKER_URL}/walk with Authorization: Bearer {PIPELINE_TRIGGER_SECRET} ({127.18} — the nudge in lib/intelligence/pipeline.ts now sends PIPELINE_TRIGGER_SECRET as the SOLE bearer and skips the nudge loudly if it is unset; the original CRON_SECRET form is retired). The walk lands the new URLs within minutes. The nudge CODE shipped with {75.13}; a failed nudge is catch-and-log — a delay, never a loss (snapshot enumeration loses nothing).
3. Worst case (nudge lost) ≤ 1 hour — an HOURLY Coolify scheduled task on the staging worker app is the belt-and-braces fallback: task uuid ixptsepifshcwtr22zf6zrzo (cocoindex-walk-hourly-fallback, app ljurtiab99kb6dst38k76419, container cocoindex-staging), frequency 0 * * * *, enabled, command = the verified absolute-interpreter-path form above. Staging only — the prod hourly fallback lands with {75.15} after Liam’s prod ratification.

Env (staging only, set 06/06/2026): COCOINDEX_WORKER_URL exists on Vercel as a Preview-environment variable scoped to the staging git branch (vercel env ls preview staging), value http://77.68.122.71:8080 (the IONOS host + the worker’s canonical port). Production is NOT set — that lands with {75.15}.

Reachability — RESOLVED by the ID-88.1 Traefik ingress (compose labels landed; live cert issuance + COCOINDEX_WORKER_URL repoint = operator). The {75.13}-era caveat (worker publishes no host port; host ufw does not allow inbound 8080, so every nudge catch-and-logged) is superseded: both compose files now carry path-scoped Traefik router labels on the cocoindex service — see §“Pipeline HTTPS ingress — Traefik path-scoped FQDN (ID-88.1)” below. Once the cert issues, repoint COCOINDEX_WORKER_URL to https://kh-pipeline-staging.aisolutionhub.co.uk (staging; prod mirrors with https://kh-pipeline.aisolutionhub.co.uk at {75.15}). The compose still publishes no host port and ufw still blocks raw 8080 — that stays deliberate: opening raw 8080 would transit the bearer in cleartext. The hourly fallback bounds nudge latency to 1 hour until the repoint is done.

Pipeline HTTPS ingress — Traefik path-scoped FQDN (ID-88.1)

⚠️ SUPERSEDED (S436, 2026-07-02): refreshed against the live compose files — path scope is /walk|/health|/extract (not just /walk+/health) across FOUR apps on two servers (client 88.208.255.178 + platform 87.106.72.59, ONE Coolify on B), all hostnames ca-*, Let’s Encrypt certs. Authoritative per-app matrix: reference/deployment-architecture.md §4/§9.

Further superseded for the PLATFORM pair (S558, 2026-08-12): the two Server B hostnames are no longer publicly reachable — a proxied Cloudflare Tunnel CNAME fronts them, a Cloudflare Access service-token policy gates them, TLS terminates at the Cloudflare edge (not Let’s Encrypt), and Server B’s inbound 443+80 are closed. The Traefik path scope below is unchanged and still load-bearing — the tunnel routes through coolify-proxy precisely to preserve it. The two Server A client hostnames remain public HTTPS with LE certs exactly as described. See runbooks/private-ingress-cutover.md.

The cocoindex worker is reachable over HTTPS via the coolify-proxy (Traefik), routed by compose labels on the cocoindex service in the four compose files (deploy/coolify/docker-compose.{staging,production,platform,platform-staging}.yaml).

App (server)	Hostname (all LE-certed)	Router name (verbatim in compose)
client staging (A)	ca-client-pipeline-staging.aisolutionhub.co.uk	kh-cocoindex-staging-walk (legacy name, current host)
client prod (A)	ca-client-pipeline.aisolutionhub.co.uk	kh-cocoindex-prod-walk (legacy name, current host)
platform staging (B)	ca-platform-pipeline-staging.aisolutionhub.co.uk	ca-cocoindex-platform-staging-walk
platform prod (B)	ca-platform-pipeline.aisolutionhub.co.uk	ca-cocoindex-platform-walk

Path scope (Inv-13). The router rule covers exactly PathPrefix(/walk) + PathPrefix(/health) + PathPrefix(/extract) — nothing else. /stage is deliberately NOT routed: it stays compose-internal only, and this path scope is the public-exposure guard now that an FQDN exists. /walk is bearer-gated by the app itself (Authorization: Bearer ${PIPELINE_TRIGGER_SECRET} — {127.18}; was CRON_SECRET when this section was written); /extract is bearer-gated by its own Bearer ${EXTRACT_API_TOKEN} (rate-limited, 20 MB cap); /health is intentionally public-readable (liveness only, no data) — on the two client hosts only since the platform pair went behind Access (S558, 2026-08-12). Any unrouted path 404s at the Traefik edge.

Cert resolver. tls.certresolver=letsencrypt — the Coolify v4 default ACME resolver name in the coolify-proxy Traefik config (instance: Coolify 4.1.1, Traefik 3.6.17). The coolify-proxy is already attached to both app networks (ljurtiab99kb6dst38k76419 staging, ybiczck7f7e1xbdev3bk89cr prod), so the labels alone route — no compose networks: change was needed. Service port: container 8080 (the worker’s $PORT default in server.py).

Operator note — labels-only routing. The Coolify app FQDN (“Domains”) field stays UNSET for both pipeline apps — this is the implemented approach. Coolify MERGES its own generated Traefik labels when an app FQDN is set; those generated routers are path-UNSCOPED (they would route the entire container root, defeating the Inv-13 guard). If the FQDN field is ever set regardless, it MUST match the hostnames above exactly, and the distinct router names + explicit router→service bindings in the compose labels keep the path-scoped routers unambiguous alongside the merged Coolify labels.

Naming (S321, TEMPORARY → resolved by the two-server split, S408). aisolutionhub.co.uk is the platform-owner domain — a TEMPORARY home pending the commercial-brand decision. The deployment model is per-client: this B1 stack is the Phew deployment’s pipeline (Server A in the two-server split), and the hostnames will be revisited per client once the brand decision lands. (The earlier caveat’s cocoindex-staging.kh.phew.org.uk suggestion is superseded by this S321 decision.)

Reconciliation (ID-127.15). The two ingress hosts above (kh-pipeline.aisolutionhub.co.uk / kh-pipeline-staging.aisolutionhub.co.uk) and the kh-cocoindex-*-walk router names are the client box’s kh-prefixed, pre-split, temporary-domain artefacts. Under the ratified two-server / ca-naming model: (a) the client pipeline keeps the kh-onprem-pipeline-* app names for now (ID-127 does not own the client rename), but its public domain flips off aisolutionhub.co.uk to kh.phew.org.uk at the ID-45 re-ingest / handover; (b) the platform pipeline on Server B uses the ca-scheme (ca-pipeline-platform, ca-cocoindex-platform-walk, Host(ca-pipeline-platform.aisolutionhub.co.uk)) — see specs/id-127-platform-pipeline/AMENDMENT-staging-prod-two-server.md §2

• TECH OQ-127-NAMING. The kh-cocoindex-pipeline GHCR image name is a deliberate exception that stays unchanged (shared promote artefact, ID-113).

B2a — Workspace manifest: route tag (ID-80 forms/content fork)

Spec: 80.2-forms-content-separation.md §B.1–B.4 + §B.8 — OQ-80.2-A/B/C all RATIFIED (Liam, S314, 05/06/2026). Shipped {80.6}–{80.10} (S314).

What a walk WRITES is now forked per manifest prefix. Each entry in the workspace manifest (<COCOINDEX_SOURCE_PATH>/.kh-workspace-map.json — on B1 that is /cocoindex-state/corpus/.kh-workspace-map.json) carries an optional route: "forms" | "content" tag:

• Default "content" — existing manifests parse unchanged; every untagged prefix stays on the content path (Path-A). Zero behaviour change until an operator opts a prefix into "forms".
• route: "forms" prefixes take the forms branch (Path-B): the walk writes form_templates + form_template_fields rows ONLY. A form lands ZERO content rows — no content_items / source_documents / content_chunks / q_a_extractions / entity_mentions [RATIFIED OQ-80.2-A].

Folder contract (operator-owned). The route tag trusts folder placement — it never sniffs file content:

• BLANK form instruments (unanswered templates — ID-52 Mode-3) live under forms-mapped prefixes.
• ANSWERED/completed forms are knowledge containers — they stay with the client corpus under route: "content" prefixes and land content rows as today [RATIFIED OQ-80.2-A caveat, Liam, 05/06/2026].

Mis-wire behaviour (all loud, never silent):

• An invalid route value (typo / non-enum) → ManifestLoadError at the manifest-load gate: the flow aborts loudly at start, before any file is processed (spec §B.8).
• A .md / .txt / .html file under a forms prefix → loud cocoindex.stage_error (extraction_validation_failed) and ZERO rows for that file; the rest of the walk continues (spec §B.3 suffix guard).
• Ambiguous manifest — operator-visible behaviour change (S314): pre-fork, an ambiguous manifest resolution still landed content rows BEFORE the error surfaced; post-fork an AmbiguousResolution at the fork lands ZERO rows on all targets for the affected file (spec §B.1/§B.8). When diagnosing an ambiguous-manifest walk, do NOT expect partial rows — their absence is correct behaviour, not data loss.

Re-tagging caveat. Flipping an existing prefix content ↔ forms orphans the rows the prior branch wrote (they are neither migrated nor deleted). This is an operator action with operator-owned consequences; a detection sweep (branch-flip by storage_path) is a recorded follow-up, out of {80.2}’s automated scope (spec §B.8).

Terminal webhook semantics [RATIFIED OQ-80.2-C]. Per-item faults never flip the walk’s terminal status. The flow-end webhook reports flow_status: "completed" plus an itemFailures tally ({forms: n, content: m}) when only per-item faults occurred — including a 100%-failure walk, where itemFailures equal to items processed is a loud signal, not a silent green. "failed" is reserved for walk-wide faults only (manifest load, Stage-5, mount errors).

Staging re-smoke brief (PARENT-owned — Orchestrator executes at the Spine-S1 smoke)

Operator block — do NOT run from an Executor worktree. This is a supervised staging burn, executed by the Orchestrator/operator via the {62.9} live-verify trigger (see the “ID-62 {62.9} live-verify trigger” section below): stage the two files into the staging corpus, then run deploy/onprem/verify/live-verify.sh (stage → walk → assert).

One staging walk containing two staged files:

1. the charnwood blank form (ITT Services.docx from docs/testing/test-data/templates/itt-services-charnwood/) under a forms-mapped prefix (route: "forms" in the staging manifest), and
2. one ordinary content doc under a content-mapped (or untagged) prefix.

Assert all four:

#	Assertion
1	charnwood lands form_templates = 1 plus form_template_fields rows
2	the content doc lands content rows (content_items + chunks/extractions)
3	the flow-end webhook reports flow_status = "completed"
4	charnwood writes NO content_items row

B3 — crash-loop backstop + kill-switch

• The load-bearing burn gate is now architectural (ID-83 / bl-221): the worker boots lifespan-only and never walks the corpus at boot, so any auto-restart is harmless by construction regardless of COCOINDEX_SOURCE_PATH’s value. The corpus walk runs only on an explicit POST /walk signal (B2). The old empty-SOURCE_PATH idle-boot gate is retired.
• restart: unless-stopped (compose default for the cocoindex service) is kept for availability — the app auto-recovers after a host reboot, and recovering into a non-walking lifespan-only boot is safe.
• During a supervised walk (a manual POST /walk), watch docker logs. The operator kill-switch is, in order of convenience: Coolify control stop (MCP) / the UI Stop button / SSH docker stop <container>. Stopping the container aborts an in-flight walk; the next /walk starts a fresh pass (the single-flight lock dies with the process).
• Boot-failure triage: asyncpg gaierror / TargetServerAttributeNotMatched in the boot logs ⇒ wrong region-qualified pooler host in COCOINDEX_DB_DSN; ValueError: Environment settings must provide Settings.db_path ⇒ COCOINDEX_DB / the state-volume mount is missing. An empty COCOINDEX_SOURCE_PATH is NOT a failure — the worker boots idle, no crash-loop (idle contract: scripts/tests/test_cocoindex_flow_idle_mode.py).
• Optional hardening (NOT applied): swap restart: unless-stopped → restart: on-failure:N on the cocoindex service to cap crash-restarts at N. Trade-off: it loses idle auto-restart after a clean stop or Docker-daemon restart, so the app would not come back idle after a host reboot. Present as an option for a future smoke-hardening pass, not the default.

Operator activation checklist

These steps cannot be done from the agent terminal (they are live burn-safety + repo-admin actions):

• (a) Create GH secret COOLIFY_BASE_URL=http://77.68.122.71:8000 in the Production environment (case-sensitive). COOLIFY_API_TOKEN already exists there (added S294).
• (b) Disable Coolify git-auto-deploy for kh-onprem-pipeline-production — UI toggle (B1.a, recommended) or set the watch_paths sentinel via API (B1.b). Coordinate timing with the workflow landing on main (see the B1 coordination hazard).
• (c) Confirm the ghcr package ghcr.io/ai-solution-hub/kh-cocoindex-pipeline is public (made public S288 → Coolify pulls anonymously, no registry creds). If it is ever made private, add ghcr registry credentials in Coolify.
• (d) ✅ DONE S312 — Set COCOINDEX_SOURCE_PATH=/cocoindex-state/corpus on both Coolify apps (it may now stay set permanently — boot never walks, so this is burn-safe; see B2). The manual blanking step is RETIRED — no need to confirm it empty before a deploy.
• (e) Run a deploy dry-run via workflow_dispatch (deploy enabled) and confirm the tag-bump + deploy produce a lifespan-only boot — /health serves 200, no corpus walk, no ingest, no burn — even with COCOINDEX_SOURCE_PATH set.
• (f) Enable the per-env Coolify scheduled task that POSTs to /walk (B2 — pick the cadence, OQ-1; both tasks already exist disabled with the verified command form), then trigger a supervised manual /walk for the first end-to-end live-ingest verification. This is FOCUS/operator work. Command-form verification step (ID-62.12): before flipping enabled: true, confirm each task’s command is the absolute-interpreter-path form from §B2 (/layers/google.python.runtime/python/bin/python3 -c … — NOT bare python3, which exits 127 in the buildpack container) and that the staging task’s container is cocoindex-staging (prod: cocoindex). After the first scheduled run, check the task execution log shows success + message 202.

{66.12} Rollback by tag (✅ documented S308)

Rollback and roll-forward are the same operation with a different COCOINDEX_IMAGE_TAG value — re-point the image tag, no host rebuild. The only stateful artefact (the LMDB memo store) is untouched by an image swap. The mechanism is the deploy-cocoindex job’s PATCH+deploy, already proven by every onprem-deploy.yml run (each deploy is a tag pull, never a host build).

Two equivalent paths:

1. Coolify UI (operator): app kh-onprem-pipeline-production (uuid ybiczck7f7e1xbdev3bk89cr) → Environment Variables → edit COCOINDEX_IMAGE_TAG (env uuid l92p743mjory7qaopxqjx58e) to the prior pinned sha-<commit> → Save → Deploy. Coolify re-clones the git compose and pulls the prior tag (near-instant; the host never builds).
2. API (scriptable — mirrors the Action):

   # PATCH = update-in-place. POST would create a DUPLICATE key (the §6 footgun).
   curl --fail-with-body -sS -X PATCH \
     "${COOLIFY_BASE_URL}/api/v1/applications/ybiczck7f7e1xbdev3bk89cr/envs" \
     -H "Authorization: Bearer ${COOLIFY_API_TOKEN}" \
     -H "Content-Type: application/json" \
     -d '{"key":"COCOINDEX_IMAGE_TAG","value":"sha-<PRIOR_COMMIT>","is_buildtime":true,"is_runtime":true}'
   curl --fail-with-body -sS -X GET \
     "${COOLIFY_BASE_URL}/api/v1/deploy?uuid=ybiczck7f7e1xbdev3bk89cr&force=false" \
     -H "Authorization: Bearer ${COOLIFY_API_TOKEN}"

Burn-safe by construction: a rollback PATCHes only COCOINDEX_IMAGE_TAG — never COCOINDEX_SOURCE_PATH (env uuid rqjlptckuuqvt4q8nz0wd98t). The rolled-back image boots lifespan-only and never walks the corpus at boot (ID-83 / bl-221 — the architectural burn gate, see {66.11} B2), so a rollback can never re-arm the burn footgun regardless of whether COCOINDEX_SOURCE_PATH is set. The corpus walk fires only on an explicit POST /walk — a rollback does not issue one.

Find rollback targets: gh api /orgs/ai-solution-hub/packages/container/kh-cocoindex-pipeline/versions (or the ghcr UI / onprem-deploy.yml run history — each run pins TAG=sha-<github.sha>). The image is public — Coolify pulls anonymously, no registry creds needed.

Operator verification (one-time): PATCH COCOINDEX_IMAGE_TAG from the current tag to the prior pinned tag and redeploy; confirm the Coolify deploy log shows a pull, not a build, the container reports the prior image within seconds, and the LMDB volume is untouched. (Satisfies inv 15. The mechanism is already exercised by every production deploy — this confirms the rollback direction specifically.)

{66.14} LMDB → S3 backup (✅ ARMED + verified S312 — COLD snapshot to Supabase Storage)

The cocoindex engine keeps its memo state in an LMDB store on the cocoindex-state named volume (mounted at /cocoindex-state, COCOINDEX_DB=/cocoindex-state/lmdb). Coolify’s native S3 backups are database-only (scoped to a database_uuid) and do not cover a raw named volume (OQ-66-4), so the LMDB store needs its own backup job.

Pin-bump LMDB key migration — id-128 S507

The cocoindex pin moved 1.0.7 → 1.0.18 (id-128 S507; see /reference/cocoindex-pipeline/). 1.0.9 moved the UserState LMDB db key (upstream #2102). CI’s LMDB is ephemeral per run, but the staging/prod VPS /cocoindex-state/lmdb volume is not — on first deploy at this pin, verify the engine-managed migration completes cleanly, or reset the LMDB volume (deliberately drop the cocoindex-state named volume and accept a memo rebuild on the next /walk). Coordinate with the id-127 deploy topology before rolling to Coolify. The COLD snapshot above is a non-critical memo cache (inv 20/25), so a reset is safe — loss costs a memo rebuild, never KH data loss.

DR-146 landing step — REQUIRED engine-store wipe (id-448, owner ruling D1)

When rolling the DR-146 producer change (git_sync becomes the sole bundle writer; write_bundle stops declaring bundle paths as cocoindex target states) onto any host whose LMDB store predates it, wipe the engine store BEFORE the first producer run: stop the pipeline app, remove /cocoindex-state/lmdb/mdb (or drop the cocoindex-state volume’s lmdb subtree), and restart. Reason, measured S553: the store holds declarations for every bundle path from pre-DR-146 runs (24 paths on platform-staging). Once the producer stops declaring them, they drop out of the declared keyset and the engine’s orphan-delete is free to delete the entire bundle from the working tree on the next flow update. The wipe costs a memo rebuild only (same safety argument as the pin-bump caution above — Supabase remains canonical, bundles are synthetic pre-launch). Verify the first post-wipe producer run deletes no bundle file (git status in the bundle checkout must show no unexplained deletions; the run’s log.md block must show no unexpected Removed entries). This step must be replayed on every host with a pre-DR-146 store — platform-production and any client lane provisioned from a snapshot.

✅ ARMED + verified (S312) — COLD snapshot (the mdb_copy design further below is SUPERSEDED)

The hot mdb_copy snapshot (lmdb-backup.sh, detailed in the subsections below) cannot be used: cocoindex’s Rust engine bundles an LMDB whose env the distro mdb_copy refuses to open — MDB_VERSION_MISMATCH (-30794), reproduced with lmdb-utils 0.9.24 (Debian bookworm) and 0.9.31 (Debian trixie) — even though the data file is unmistakably standard LMDB (meta magic 0xBEEFC0DE, data version 1). Rather than pin + build cocoindex’s exact bundled LMDB, the backup takes a consistent COLD snapshot, justified because the LMDB is a non-critical memo cache (inv 20/25 — loss costs a memo rebuild, never KH data loss), is ~256 KiB, and the worker boots lifespan-only (ID-83 — no walk at boot) so the stop/start window is a couple of seconds at a quiet hour.

What runs (all committed + verified live S312):

• deploy/onprem/backup/lmdb-backup-cold.sh (host): resolves the running cocoindex container + on-host volume from APP_UUID, then docker stop → cp data.mdb → docker start (immediate), then tar+ships via the sidecar. The container is always restarted (EXIT trap), even on copy/upload failure.
• deploy/onprem/backup/Dockerfile.tools → kh-lmdb-backup-tools:latest (host-built): the upload sidecar carrying awscli + zstd (the host has neither — Ubuntu 24.04 dropped the awscli apt package). The cold script runs it with --env-file so the host shell never sees the S3 secret values.
• deploy/onprem/backup/restore-verify.sh: pulls the newest object + validates the restored data.mdb via its meta-magic (mdb_stat can’t open it either — same version skew).

Live armed state (prod app ybiczck7f7e1xbdev3bk89cr):

Item	Value
Store path (on-host)	/var/lib/docker/volumes/ybiczck7f7e1xbdev3bk89cr_cocoindex-state/_data/lmdb/mdb/ — note the /mdb subdir: cocoindex 1.x nests the env one level under COCOINDEX_DB=/cocoindex-state/lmdb. (Corrects the earlier “expect data.mdb directly under /cocoindex-state/lmdb” assumption.)
Bucket	onprem-backups (Supabase Storage S3) · key lmdb/cocoindex-state/production/lmdb-<UTC-ISO8601>.tar.zst
Secrets	/root/kh-secrets/lmdb-backup.env (chmod 600 — BACKUP_S3_BUCKET + 4×AWS_*); Liam-placed, never in the repo
Schedule	host crontab 30 3 * * * (after the /walk 02:00 window) → >> /var/log/lmdb-backup.log
Verified S312	one backup run (object landed, 3.9 KiB compressed) + restore-verify (restored data.mdb = 262144 B, magic dec0efbe) + container returned healthy.

Restore: stop the cocoindex container; run restore-verify.sh to fetch + validate, or aws s3 cp the newest object via the sidecar; untar data.mdb into the on-host …/lmdb/mdb/ dir; start the container. Retention: objects are ~4 KiB each (negligible); add a Supabase Storage lifecycle rule or a --prune >90d flag only if it ever matters.

Artefact: deploy/onprem/backup/lmdb-backup.sh — a self-contained, surface-agnostic hot-snapshot → S3 job. It is committed unarmed: no cron, no Coolify schedule, no bucket names, no credentials. Activation is operator work (checklist below).

Target: Supabase Storage (S3-compatible) — chosen S311 (premise corrected)

Premise correction (S311). Earlier drafts said “reuse the S3 bucket Coolify already targets for its DB backups.” That bucket does not exist: a live check (coolify list_databases) shows Coolify manages zero databases — the canonical Postgres is Supabase (managed separately from Coolify), so Coolify has no DB-backup S3 target to reuse. The backup target is therefore chosen fresh.

Chosen target: Supabase Storage, which exposes an S3-compatible endpoint the backup script already supports via AWS_ENDPOINT_URL. Rationale: lowest-friction (no new vendor; the Supabase project + credentials already exist), and the LMDB store is a non-critical memo cache (loss = a memo rebuild on next ingest, never KH data loss — inv 20/25), so co-locating its snapshot with Supabase is acceptable. Follow-up (tracked): migrate the target to IONOS object storage to remove the Supabase co-location — see the backlog item “lmdb-backup S3 target → IONOS object storage” (raised S311). The script needs no change for that move — only the AWS_ENDPOINT_URL / bucket / creds env values differ.

Provision steps (Supabase Storage S3):

1. In the Supabase Dashboard → Project Settings → Storage, create a private bucket (e.g. onprem-backups) and generate an S3 access key (access key id + secret).
2. Note the S3 endpoint + region shown there — AWS_ENDPOINT_URL is https://<project-ref>.storage.supabase.co/storage/v1/s3; AWS_REGION is the project region (e.g. eu-west-2).
3. Set the script env: BACKUP_S3_BUCKET=onprem-backups, AWS_ENDPOINT_URL=<above>, AWS_REGION=<above>, AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY=<the S3 key>. Leave BACKUP_S3_PREFIX / BACKUP_ENV_LABEL at defaults (lmdb/cocoindex-state / production).

Where these values are stored (the repo goes PUBLIC — never commit them). The backup runs as a host cron + tooling container on the IONOS VPS (surface (b) below), so the operational home for the creds is a root-owned env file on the host, not the repo and not Coolify:

• Operational copy (what the cron reads): a chmod 600, root-only file on the IONOS host, e.g. /root/kh-secrets/lmdb-backup.env, holding the five *=value lines above. The host cron passes it to the tooling container with docker run --env-file /root/kh-secrets/lmdb-backup.env … (replaces the inline -e KEY=… flags in the surface-(b) recipe). The host shell cannot read a Coolify app’s injected env, so a host file is the correct home for a host-level cron.
• Secret-of-record (so they are never lost): your password manager — the same place the runbook already says to keep the IONOS SSH key and the Coolify /data/coolify/source/.env.
• Repo (public-safe): carries only deploy/onprem/backup/lmdb-backup.sh (reads from env — no secrets) and .env.*.example placeholders. ID-68 going public exposes none of these. The on-prem secrets convention this follows is onprem-secrets-migration.md ({66.10} — archived).

Do not put these in .env.local, any committed .env*, or a Coolify app env intended for the slim cocoindex image — none of those is read by a host cron, and the first two risk the public flip.

What the script does

1. Consistency (inv 18): snapshots with mdb_copy -c "$LMDB_SRC" "$SNAPSHOT_DIR". mdb_copy copies through an LMDB read transaction — LMDB’s MVCC read-snapshot is immutable for the txn lifetime regardless of concurrent writes, so it yields a consistent hot snapshot with no container stop / no quiesce. The -c flag compacts and drops free pages. A naive cp/tar of the live data.mdb is never used (torn-page risk). mdb_copy is a CLI tool from liblmdb (Debian/Ubuntu package lmdb-utils), not an importable symbol.
   • Source-path gotcha: cocoindex opens COCOINDEX_DB=/cocoindex-state/lmdb as a directory env, so the mdb_copy source is that directory (holding data.mdb/lock.mdb), not the data.mdb file. On first run confirm the layout: ls /cocoindex-state/lmdb → expect data.mdb lock.mdb. (If the env were ever MDB_NOSUBDIR, the source would be the data.mdb file with mdb_copy -n.)
2. Package + ship: tars the snapshot (zstd if available, else gzip) and uploads to the chosen S3-compatible bucket (Supabase Storage — see “Target” above; with AWS_ENDPOINT_URL set). Object key: s3://${BACKUP_S3_BUCKET}/${BACKUP_S3_PREFIX}/<env>/lmdb-<UTC-ISO8601>.tar.<zst|gz> (default prefix lmdb/cocoindex-state, default <env> segment production). Uses aws s3 cp and honours --endpoint-url "$AWS_ENDPOINT_URL" for non-AWS S3.
3. Fail loudly: set -euo pipefail, a trap cleaning the temp snapshot + tarball on EXIT, and preconditions that exit non-zero with a package-naming error if mdb_copy or aws is missing, if BACKUP_S3_BUCKET is unset, or if LMDB_SRC has no data.mdb.
4. Supabase: touched not at all (inv 20). The LMDB store is a memo cache; Supabase is canonical.

The script is surface-agnostic — all paths and S3 config come from env (LMDB_SRC, SNAPSHOT_DIR, BACKUP_S3_BUCKET, BACKUP_S3_PREFIX, BACKUP_ENV_LABEL, AWS_ENDPOINT_URL, AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY).

Execution surfaces (pick one at schedule time)

• (a) Coolify scheduled task inside the cocoindex container (scheduled_tasks: resource=application, uuid=ybiczck7f7e1xbdev3bk89cr, container=cocoindex). The container already mounts the volume at /cocoindex-state, but the slim buildpack image ships no mdb_copy, no zstd, no aws — so this surface needs those tools baked into the image (rejected: bloats the image / breaks the Inv-9 boundary). Not viable as-is.
• (b) Host cron + a throwaway tooling container (RECOMMENDED) — a tiny image carrying lmdb-utils + awscli, bind-mounting the on-host volume read-only:

  docker run --rm \
    -v /var/lib/docker/volumes/ybiczck7f7e1xbdev3bk89cr_cocoindex-state/_data:/cocoindex-state:ro \
    -e BACKUP_S3_BUCKET=... -e AWS_ACCESS_KEY_ID=... -e AWS_SECRET_ACCESS_KEY=... \
    -e AWS_ENDPOINT_URL=... -e AWS_REGION=... \
    -v /path/to/deploy/onprem/backup/lmdb-backup.sh:/lmdb-backup.sh:ro \
    <tooling-img> /lmdb-backup.sh

  Keeps the cocoindex image slim and the backup tooling independent.
• (c) Host cron directly on the IONOS host if lmdb-utils + awscli are installed there. Simplest; point LMDB_SRC at the on-host _data/lmdb path directly.

Recommended surface: (b). Schedule daily (or twice-daily) — B1 write volume is low and the LMDB is small (≤256–512 MiB historically). Retention is the operator’s choice within the 100 GB IONOS backup allowance (OQ-66-1); record the chosen cron + retention here once armed.

Restore procedure

1. Stop the cocoindex container — Coolify Stop, or docker stop <cocoindex>. (LMDB must not be open while you replace its files.)
2. Fetch the latest snapshot tar from S3:

   aws s3 cp "s3://${BACKUP_S3_BUCKET}/lmdb/cocoindex-state/production/<lmdb-...tar.zst>" /tmp/restore.tar.zst

   (List with aws s3 ls s3://${BACKUP_S3_BUCKET}/lmdb/cocoindex-state/production/ and take the newest by ISO-8601 timestamp in the key.)
3. Untar onto the persistent volume so the env dir lands at /cocoindex-state/lmdb. The tar holds the env files at top level, so untar into the lmdb dir on the on-host volume path:

   VOL=/var/lib/docker/volumes/ybiczck7f7e1xbdev3bk89cr_cocoindex-state/_data/lmdb
   rm -rf "$VOL" && mkdir -p "$VOL"
   tar -xf /tmp/restore.tar.zst -C "$VOL"   # restores data.mdb (+ lock.mdb if present)

4. Restart the container (Coolify Start / docker start). The restored store is a valid cocoindex memo state; the engine resumes with memo hits.

Fallback (inv 7 / inv 20): if the snapshot is stale or unavailable, the cost is only a memo-state rebuild on the next ingest — slower, but never KH data loss, because Supabase is canonical (inv 20). LMDB is a cache, not the system of record.

Operator activation checklist (UNARMED until done)

The script is committed but does nothing until an operator provisions the following. These cannot be done from an agent terminal (live S3 credentials + host scheduling):

• (a) Provision the Supabase Storage bucket + S3 access key (see “Target” above — Coolify has no DB-backup bucket to reuse); decide the prefix (default lmdb/cocoindex-state) and <env> segment (default production).
• (b) Provision S3 credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, and AWS_ENDPOINT_URL if the bucket is non-AWS S3) scoped to write the chosen prefix.
• (c) Choose the execution surface — (b) host cron + tooling container is recommended; install lmdb-utils + awscli into that image (or the host for surface (c)).
• (d) First-run check: ls /cocoindex-state/lmdb (or the on-host _data/lmdb) → confirm data.mdb lock.mdb (directory env, not MDB_NOSUBDIR).
• (e) Add the cron entry / Coolify schedule (daily or twice-daily) and record the chosen cron expression + retention policy in this section.
• (f) Verify one run lands a lmdb-<UTC-ISO8601>.tar.<zst|gz> object in the bucket, then exercise the restore procedure once into a scratch path and confirm the restored env opens (mdb_stat <restored-dir>).

{75.18} pullmd share-link store persistence (✅ compose wired S320; live recreate-verify = operator)

⚠️ SUPERSEDED — PullMD RETIRED (verified 22/07/2026). No compose file under deploy/coolify/ carries pullmd / playwright / pullmd-data-* services, volumes, or PULLMD_* env any more — URL extraction is served in-process by the cocoindex worker’s /extract (trafilatura) path in every environment. This section is retained as the execution record of the removed stack; do not stand any of it up.

Premise — pullmd’s SQLite store must survive container recreates. pullmd keeps everything in one SQLite DB at the CACHE_DB path: the conversions table holds the 8-hex share ids that GET /s/:id serves (KH persists them as reference_items.pullmd_share_id), and the auth layer (users, sessions, minted pmd_ tokens) reuses the same db handle (upstream server.js: createCache(process.env.CACHE_DB || './data/cache.db') then createAuth({ db: cache.db, … })). Until S320 the compose pointed CACHE_DB at /tmp/pullmd-cache.db — container-ephemeral — so every Coolify recreate/redeploy invalidated ALL persisted pullmd_share_id values (S319 live-observed: a pre-recreate /s/:id returned 404 post-recreate, failing the ID-75 share-link AC).

The fix (staging compose, deploy/coolify/docker-compose.staging.yaml):

• CACHE_DB: "/data/pullmd-cache.db" — repointed from /tmp into the upstream-blessed /data store dir (the pullmd Dockerfile runs mkdir -p /data && chown -R app:app /app /data before USER app; upstream’s own compose mounts a volume at /data).
• Named volume pullmd-data-staging:/data on the pullmd-staging service + a top-level pullmd-data-staging local-driver volume. Named volumes survive Coolify recreates/redeploys (same mechanism as cocoindex-state-staging); only an explicit volume teardown destroys the store. A fresh named volume inherits the image’s app:app mountpoint ownership (Docker copies image content + ownership into an empty named volume), so no user: override is needed.

Recreate-survival semantics: after this change a redeploy/recreate keeps every minted share id resolvable — GET /s/:id 200 for ids created before the recreate, and persisted reference_items.pullmd_share_id values stay live across the container lifecycle. The volume also makes the PULLMD_AUTH_TOKEN static-bearer bootstrap (S319, 686ee5dd) less load-bearing: minted pmd_ tokens now survive too, though the static bearer stays as the documented auth path. Loss budget: losing this volume costs share-link 404s only — the extracted markdown is canonical in Supabase (source_documents), never in pullmd. This makes the pullmd store the second stateful artefact on B1 alongside LMDB (lower stakes: permalink invalidation vs memo-rebuild; neither is KH data loss).

Operator steps (live verify — ACs 2/3 of {75.18}, operator-gated):

• (a) Redeploy the staging stack so the new compose (volume + CACHE_DB repoint) takes effect; confirm docker volume ls shows the pullmd-data-staging volume and docker exec kh-pullmd-staging ls /data shows pullmd-cache.db after first use.
• (b) Mint a share id (any pipeline HTML fetch, or GET /api?url=… with the Bearer token — the X-Share-Id response header), confirm GET /s/<id> → 200.
• (c) Recreate the container (Coolify redeploy/restart), then re-issue GET /s/<id> for the pre-recreate id → expect 200 (was 404 pre-fix).
• (d) Spot-check one persisted source_documents.pullmd_share_id value resolves (column lives on source_documents, not reference_items — S321 correction).
• (e) Note the first recreate after this change still invalidates ids minted into the old /tmp store — they were never persisted anywhere recoverable. Re-ingest (or accept 404) for share ids minted before S320.

Prod mirror: ✅ landed S321 (7dd4b153, ahead of the {75.15} cutover) — the prod compose now carries both the PULLMD_AUTH_TOKEN static bearer and the persistent store (volume name pullmd-data-production, mounted at /data). Inert until a manual prod deploy materialises it; the {75.15} execution checklist owns that leg.

{66.15} Datapath monitor — pipeline_runs stall-watch (✅ re-homed to a Vercel cron, S311; alert sink = operator)

Premise — row arrival, not /health. “container up + /health green” is insufficient evidence the datapath works. /health (server.py:_health_handler) reflects only the cocoindex worker thread’s liveness — it returns 503 only if that thread crashed. It says nothing about whether ingestion is producing rows. Datapath health is confirmed by pipeline_runs row arrival in Supabase, never by the probe. (This is not CocoInsight — Gotcha-B — which is a dev-time LMDB inspector, not a prod stall-watch.)

Artefact: deploy/onprem/monitor/datapath-watch.ts — a standalone, read-only stall-watch (it only SELECTs from pipeline_runs; no writes, no DDL, no KH pipeline/app code change). The core is a pure predicate (detectStalls(rows, config, now)) unit-tested in __tests__/deploy/onprem/monitor/datapath-watch.test.ts. The standalone artefact carries no schedule of its own; its predicate is consumed by the Vercel-cron home below.

Home (S311): a Vercel cron — not host cron. Because the monitor is a read-only Supabase poll with zero on-prem dependency (its premise is pipeline_runs row arrival, never /health / container state), it is re-homed to a Vercel cron route — app/api/cron/datapath-watch/route.ts, scheduled every 15 min in vercel.json, bearer-gated by the shared CRON_SECRET (the same pattern as the other eight app crons). The route reuses the tested detectStalls predicate + loadConfig + types verbatim from the standalone artefact and does app-native I/O (createServiceClient) + alerting (MONITOR_ALERT_WEBHOOK_URL, else the app logger.error → Sentry bridge). This keeps a Node runtime off the slim cocoindex image (Inv-9) and co-locates the monitor with the other app crons + their alerting. The host-cron surfaces further below are retained only as alternatives (e.g. if you later want host/container-state checks too), not the chosen home.

The three stall conditions + idle-mode guard

• (A) Stuck-in-flight: an op_id with an in_progress row but no terminal-status row (completed / completed_with_errors / failed) for the same op_id after STALL_THRESHOLD_MINUTES. Keyed off op_id + status (is a terminal row present?), not off completed_at — because recordPipelineRun stamps completed_at = now() on every insert incl. the flow-start row (Gotcha-C), so completed_at is not a reliable in-flight signal. ⚠ Gated on the Gotcha-A constraint fix (below) — until it lands, no in_progress row ever exists, so (A) is a dead signal by DB state.
• (B) Terminal failure: a recent status='failed' row in the window. recordPipelineRun already Sentry-alerts on failed / completed_with_errors; this monitor adds a host-side cross-check / digest so a missed Sentry alert is still caught.
• (C) Silence during active ingestion: when ingestion is known active (INGESTION_EXPECTED=true) and the deploy is not idle, no new pipeline_runs row within EXPECTED_RUN_WINDOW_MINUTES → datapath stall (“rows should be arriving but aren’t”).
• Idle-mode guard (load-bearing): when COCOINDEX_SOURCE_PATH is empty (idle boot) or INGESTION_EXPECTED is false, no rows is the CORRECT state — condition (C) is suppressed, so the monitor does not page on every burn-safe idle deploy.

Config (env — all operator-set at activation)

Env	Purpose	Default
SUPABASE_URL	Supabase project URL (REQUIRED)	—
SUPABASE_SERVICE_ROLE_KEY	Service-role reader (bypasses RLS structurally, like the cron handlers — not the app’s request-scoped getAuthorisedClient) (REQUIRED)	—
STALL_THRESHOLD_MINUTES	Condition (A) threshold	30
EXPECTED_RUN_WINDOW_MINUTES	Condition (C) tolerated silence	60
INGESTION_EXPECTED	Condition (C) gate — "true" when a corpus is staged/active	false
COCOINDEX_SOURCE_PATH	Idle-boot detector — empty/unset ⇒ idle, (C) suppressed	—
MONITOR_ALERT_WEBHOOK_URL	Alert sink — Discord/Telegram/email webhook (POST)	—
SENTRY_DSN	Fallback sink — Sentry.captureMessage (operator wires init at activation)	—

Execution surface — alternatives to the Vercel cron (NOT the chosen home)

The chosen home is the Vercel cron above. These host-side surfaces are retained only as fallbacks (e.g. if you later want the monitor to also check host/container state, which the Vercel cron deliberately does not). As with {66.14}, the slim buildpack cocoindex image ships no Node/tsx runtime, so the monitor must not run inside the cocoindex container. If you ever move it host-side, the standalone runOnce() entry point reads SUPABASE_URL + SUPABASE_SERVICE_ROLE_KEY directly (host env names), via:

• (a) Host cron + a tiny tooling container — a small Node/bun/tsx image that runs datapath-watch.ts (runOnce()) on a schedule. Keeps the cocoindex image slim.
• (b) Host cron directly on the IONOS host if a Node/bun/tsx runtime is installed there.
• (c) Scheduled Supabase SQL alternative — an equivalent pg_cron/external-cron psql query returning the same boolean stall flags (TECH.md §6).

Alert sink options

A single env-configurable sink — the operator picks the channel by env, no code change:

• MONITOR_ALERT_WEBHOOK_URL set → alerts POST as JSON to that webhook (Discord/Telegram/email).
• else SENTRY_DSN set → route via Sentry.captureMessage (wire the @sentry/node init at activation; the committed artefact leaves this unarmed — no DSN baked in, no import).
• neither → the artefact writes a stderr note so a misconfigured deploy is visible.

Gotcha-A — RESOLVED (S309, migration #106). This block previously flagged that the live pipeline_runs_status_check constraint omitted 'in_progress', so flow-start inserts were rejected and condition (A) was a dead signal by DB state. Migration #106 (20260604113356, applied + pg_get_constraintdef-verified on BOTH staging turayklvaunphgbgscat and prod rovrymhhffssilaftdwd, parity held at 106) widened the constraint to {running,in_progress,completed,completed_with_errors,failed,cancelled} — it now admits 'in_progress' (added alongside ID-76’s 'cancelled'; the two landed convergently). So in_progress rows now land and condition (A) is live. No further migration is required. Conditions (B) and (C) were always live.

Operator activation checklist (Vercel-cron home — mostly wired, S311)

The route + schedule + test are committed; the cron goes live on the next prod Vercel deploy (read-only + idle-guarded, so shipping it active is safe). What’s already wired vs. what the operator still does:

Already wired (S311):

• ✅ Execution surface — Vercel cron route app/api/cron/datapath-watch, no on-prem runtime.
• ✅ Supabase read — the route uses the app’s validated env via createServiceClient (NEXT_PUBLIC_SUPABASE_URL + SUPABASE_SERVICE_ROLE_KEY); no separate SUPABASE_URL to set.
• ✅ Schedule — */15 * * * * in vercel.json (git-tracked; change it there, not in Coolify).
• ✅ Condition (A) prerequisite — the Gotcha-A in_progress constraint widening landed (migration #106, S309); condition (A) is live.

Operator status (Vercel project env):

• ✅ Alert sink — DECIDED (S311): Sentry fallback only. MONITOR_ALERT_WEBHOOK_URL is not set by deliberate choice — alerts route through the app logger.error → Sentry bridge (already wired), so the monitor is never silent. A dedicated webhook can still be added later if a separate channel is wanted; not required.
• ✅ COCOINDEX_SOURCE_PATH — set in Vercel (S311). Harmless on its own: it only flips the condition-(C) idle detector to “not idle”. Condition (C) also requires INGESTION_EXPECTED=true, so with COCOINDEX_SOURCE_PATH set but INGESTION_EXPECTED unset/false, (C) stays quiet — no false pages. Conditions (A)/(B) ignore both.
• Condition (C) toggle (per active re-ingest): set INGESTION_EXPECTED=true while a corpus is actively ingesting; unset / false when idle. Conditions (A)/(B) are live as soon as the cron deploys and need neither toggle.
• Thresholds (optional): override STALL_THRESHOLD_MINUTES / EXPECTED_RUN_WINDOW_MINUTES on the Vercel env if the 30 / 60 defaults need tuning.

ID-62 {62.9} live-verify trigger (operator/host hook — on-demand, Inv-28)

Status: built, operator-gated. The trigger artefact is the committed host script deploy/onprem/verify/live-verify.sh. The live proof (a supervised on-host run) is FOCUS/operator work — the same supervised-burn discipline as the S297 smokes. The on-prem pipeline writes to live Supabase (prod-wired today) — fixtures are test-prefixed and cleaned by the test layer’s dropFixture.

The trigger replaces the OLD cocoindex-live-verify.yml WIF + gcloud run jobs execute mechanism: NO gcloud, NO WIF, NO Cloud Run Job anywhere in the trigger path, and it is NOT inlined into the PR-blocking ci.yml integration job (that job is unchanged) [ID-62 Inv-28]. It is on-demand today; the launch flip to a scheduled cadence is a one-line uncomment — see the # FLIP ON AT LAUNCH (ID-62 Inv-29) block at the bottom of the script [ID-62 Inv-29].

The sequence (ID-83 / bl-221 aware — stage → walk → assert)

Staging bytes no longer ingests on its own (boot is lifespan-only; ingestion fires only on a bearer-gated POST /walk, see §B2). The script therefore threads the walk through the trigger path:

1. Preflight — in-container /health probe (loopback, python3 + urllib).
2. Fixture-corpus seed (step 0b) — the buildpack image packages only scripts/ + requirements.txt; the committed docs/testing/test-data/** corpus is not in the image (verified live on B1, S316). The script docker cps $KH_REPO_DIR/docs/testing into the container at <app-dir>/docs/testing (default /workspace — override via COCOINDEX_APP_DIR) so verify_driver’s repo-relative fixture reads resolve. Idempotent: the /. source-suffix form overwrites in place on re-run ({62.11}, ratified option a — no image change, no compose change).
3. Driver stage — docker exec <cocoindex> python3 -m scripts.cocoindex_pipeline.verify_driver --fixtures templates ({62.7}, stage-only). A non-zero driver exit fails before the Vitest step.
4. POST /walk — bearer = the container’s own PIPELINE_TRIGGER_SECRET ({127.18} — was CRON_SECRET at authoring; the secret never touches the host; the POST runs via docker exec). 409 = walk already in flight.
5. Wait for completion — /walk is async (202 + requestId); the script watches docker logs for the worker’s "/walk completed (requestId=…)" terminal line (timeout WALK_TIMEOUT_SECS, default 900).
6. Vitest go-live tier + walk pump — the cocoindex-scoped Vitest invocation runs on the host with a background walk pump (POST /walk every WALK_PUMP_INTERVAL_SECS, default 30; 409s tolerated). The pump is load-bearing: Tier-1/Tier-2 files stage their own fixtures inside beforeAll during the Vitest run, so the single pre-Vitest walk cannot ingest them — without the pump every Tier-1 poll times out against nothing. Each pump pass is incremental (unchanged files memo-hit and burn nothing).

Buildpack docker exec env (S316 — exit-127 trap). The cocoindex image is a Google buildpack build: python3 lives at /layers/google.python.runtime/python/bin/python3 and site-packages resolve via the pip user-site (PYTHONUSERBASE=/layers/google.python.pip/pip). A bare docker exec … python3 exits 127 because the exec PATH omits the buildpack layers. Every in-container python3 step above therefore runs with -e PYTHONUSERBASE=/layers/google.python.pip/pip -e PATH=/layers/google.python.pip/pip/bin:/layers/google.python.runtime/python/bin:/usr/bin:/bin injected — the script does this for you; any MANUAL docker exec … python3 needs the same env block.

Two verify drivers — pick by proof type (ID-62 quality-review note, S321). There are TWO distinct verify_driver.py modules with different concerns: scripts/cocoindex_pipeline/verify_driver.py is the file-fixture driver (stage-only, loopback /stage, no DB access — invoked in step 3 above and by live-verify.sh); deploy/onprem/verify/verify_driver.py is the URL-mode driver (the {62.10} O4 landing-set proof: service-role Supabase seed → walk → pullmd share-id round-trip → idempotency walk). Running the wrong one gives a confusing partial proof — file-fixture proofs never touch Supabase; URL-mode proofs never read docs/testing fixtures.

Host prerequisites (operator)

• A knowledge-hub repo checkout on the host + bun install (KH_REPO_DIR).
• A host secrets env file (default /root/.kh-live-verify.env, never committed) with NEXT_PUBLIC_SUPABASE_URL + SUPABASE_SERVICE_ROLE_KEY (live, non-placeholder — hasRealLiveDbCredentials() rejects placeholders).
• The compose stack publishes no host port for the sidecar, so the spec’s http://localhost:<port> env block is realised host-side as the container’s bridge IP (resolved via docker inspect; still host-local). To use the literal localhost form instead, add ports: ["127.0.0.1:8080:8080"] to the cocoindex compose service and export COCOINDEX_STAGING_URL before invoking. The ID-88.1 Traefik HTTPS ingress (§“Pipeline HTTPS ingress — Traefik path-scoped FQDN (ID-88.1)”) does NOT change this: it routes only /walk + /health, and the live-verify harness drives /stage, which is deliberately unrouted (Inv-13) — so the harness stays host-local by design.
• The image does not carry the committed fixture corpus (it packages only scripts/
  • requirements.txt) — the script seeds it itself: step 0b docker cps $KH_REPO_DIR/docs/testing into the container at <COCOINDEX_APP_DIR>/docs/testing (default /workspace). No operator pre-seed needed; just make sure KH_REPO_DIR is a full checkout. To spot-check post-seed: docker exec <cocoindex> ls /workspace/docs/testing/test-data/templates.

Env block the script exports for the Vitest step: COCOINDEX_STAGING_URL + COCOINDEX_FIXTURE_STAGING_URL (host-local sidecar URL), COCOINDEX_SOURCE_PATH (/cocoindex-state/corpus — the real B1 value, not the spec’s /corpus literal), PULLMD_SERVICE_URL (pullmd sibling-container URL — gates Tier-4), plus the live-Supabase creds from the secrets file.

Per-file go-live ledger (TECH §Testing tier map / OQ-62-TECH-12 — none silently green)

The Vitest invocation’s --exclude list IS the defer ledger. Every excluded file is named with its reason — the deferred files are not claimed verified.

Tier	Files	Decision
Tier 1 (20 stage+poll files)	admin-merge-coexistence … file-change-detection (all call stageFixture in beforeAll)	LIVE — via the walk pump
Tier 1b	extract-memoisation	DEFER — staging half deferred at authoring (“FUTURE … 28.18”) and never landed; seededContentIds stays empty so the file fails deterministically when enabled. Needs its own stage + a second walk pass (re-ingest), not a Tier-1 sibling’s stage.
Tier 1b	memo-hit-pipeline-run	DEFER — never stages its own prefix; assumes the retired continuous poll-cycle model (pre-ID-83). Needs its own stage + two explicit walks.
Tier 1b	stage-5-failure-non-destructive	DEFER — the ?failStage5=<mode> destPath directive its injectStage5Failure helper sends IS now implemented by the live /stage route (id-414 AC-5/AC-6, S535): server.py allowlists it, strips it off the path, and arms a one-shot walk-scoped credential override that _run_walk consumes and restores in a finally. It stays excluded on a NARROWER, still-honest ground: nothing has yet verified end-to-end against a live sidecar that the injected credential yields the documented AuthenticationError on the MOCK tier (a mock provider need not 401 on a bogus key), nor that flow.py then resolves status='failed' with error_class='entity_resolution_failed'. Lift the exclude once that live-sidecar verification lands.
Tier 2	form-extraction, form-extraction-rls (integration root)	LIVE — self-stage; need the seeded /cocoindex-state/corpus/.kh-workspace-map.json ({62.6})
Tier 3	health-probe	LIVE — pure /health fetch, fully self-sufficient
Tier 3	transient-retry	LIVE — observational over recent pipeline_runs (tolerates an empty DB)
Tier 3	sidecar-cold-start	DEFER — needs an operator-driven cold-start cycle (COCOINDEX_COLD_START=true opt-in gate + deliberate restart); the Cloud-Run scale-to-zero framing needs a B1 re-anchor
Tier 3	sidecar-mime-coverage	DEFER — beforeAll staging is a 28.18-era stub; polls prefixes nothing stages (deterministic fail). HTML branch additionally ID-75-gated (PullMD cannot read staged local files)
Tier 3	sidecar-version-metadata	DEFER — beforeAll empty; polls its own random prefix nothing stages (28.18 stub)
Tier 3	stage-topology	DEFER — beforeAll staging stubbed (28.18 stub)
Tier 3	latency-budget	DEFER — beforeAll staging stubbed; the 30 s per-file budget also needs an ID-83 re-anchor (walk-pump cadence now dominates measured latency)
Tier 3	audit-log-shipping	DEFER — polls its own random prefix nothing stages (28.18 stub); the v1.1 audit_log half self-gates on table existence
Tier 4	agpl-boundary	LIVE — when PULLMD_SERVICE_URL resolves (pullmd sibling-container IP)
Tier 5 (5 no-gate files)	extractor-version-cross-ref, faiss-pin, no-partial-row-writes, non-pipeline-write, persistent-failure-dlq	LIVE — already run everywhere; unaffected

Tracked note (Curator routing): the six deferred Tier-3 files + the extract-memoisation / memo-hit-pipeline-run Tier-1b files are 28.18/53.14-era substrates whose staging halves were deferred at authoring and never completed. They need test-layer completion work (own stageFixture calls; a second-walk mechanism for the re-ingest pair) before they can join the live tier — that is code work outside {62.9}’s artefact set (YAML/runbook/doc/env only). The stage-5-failure-non-destructive file’s server-side blocker (?failStage5= directive support) landed at id-414 AC-5/AC-6 (S535); the remaining blocker there is live-sidecar verification of the MOCK-tier 401 path.

Operator activation checklist ({62.9})

• Repo checkout + bun install on the host; secrets env file in place.
• Fixture corpus: nothing to pre-seed — the script’s step 0b docker cps it from KH_REPO_DIR (spot-check ls above if in doubt).
• Run KH_REPO_DIR=… deploy/onprem/verify/live-verify.sh under supervision (watch docker logs -f; kill-switch = docker stop <container>).
• Confirm dropFixture cleanup left no test-prefixed rows behind (write-model R1 — no ON DELETE CASCADE; children are deleted explicitly).
• At launch: flip the schedule (one-line uncomment at the bottom of the script — # FLIP ON AT LAUNCH (ID-62 Inv-29)), or stand up the equivalent Coolify scheduled task.

Remaining subtasks (⬜)

{66.7} GH Action build→ghcr.io (off-host, no host dep — dispatchable now) · {66.8} co-location compose · {66.9} LMDB + corpus volumes · {66.10} secrets → Coolify env — see onprem-secrets-migration.md (archived; needs operator GCP Secret Manager access) · {66.11} deploy webhook (🔶 wired in code — see the {66.11} section above; live activation is operator) · {66.12} rollback-by-tag (✅ documented above) · {66.13} webhook repoint + DNS/SSL · {66.14} LMDB→S3 backup · {66.15} datapath monitor (✅ re-homed to a Vercel cron, S311 — see the {66.15} section above; operator sets the alert sink) · {66.16} E2E smoke + ID-62 reachability hand-off · {66.17} retire cloud-run-deploy.yml · {66.18} kpf dead-manifest cleanup.

ID-62 gate: ID-62’s Task-level dependency on ID-66 is satisfied only by {66.16} (datapath proven + HTTPS ingress reachable). ID-62’s {62.5} /stage route + {62.8} multipart are host-agnostic code and may be built ahead of the host.

Revert path (inv 24)

cocoindex→pullmd reverts with one env var (PULLMD_SERVICE_URL); inbound webhook reverts with one env var (PIPELINE_RUN_WEBHOOK_URL). The Cloud Run leg of inv-24 is void post-S298: there is no .github/workflows/cloud-run-deploy.yml to revert to (deleted in the full teardown — see the Cloud Run retirement section above) and no GCP Secret Manager; secrets are Coolify env vars and on-prem / Coolify is the sole deploy path. LMDB is the primary genuinely stateful artefact — its loss costs a memo-rebuild, never KH data loss (inv 25). The pullmd share-link store ({75.18} section above) is the only other stateful artefact — its loss costs share-link 404s, likewise never KH data loss.