Skip to content

S410 greenfield VPS standup runbook (two new IONOS boxes)

S410 — Greenfield VPS standup runbook (two new IONOS boxes)

Section titled “S410 — Greenfield VPS standup runbook (two new IONOS boxes)”

Date: 2026-06-24 (session S410) Audience: Liam (non-technical operator) + Claude (agent). PLANNING DOCUMENT — nothing in here has been executed yet.

Every step is tagged [OPERATOR] (Liam does it, by hand / in a web UI / by pasting one command) or [AGENT] (Claude does it, once Liam has handed over what the step needs).


0. Plain-language situation — what is actually down

Section titled “0. Plain-language situation — what is actually down”

Nothing that a user can see is down. Here is the picture in plain terms:

ThingWhat it isStatus right now
Phew product website (https://kh.phew.org.uk)The actual app a client would use. Runs on Vercel + Supabase — completely separate machines from the cancelled box.UP — responds HTTP 307 (a normal redirect, the site is alive). It was never on the cancelled box.
Phew databases (Supabase rovry… prod, turay… staging)Where all the real knowledge lives. Managed by Supabase.UP — untouched. The cancelled box never held any canonical data.
Old IONOS box (77.68.122.71)Only ran the ingestion pipeline (the background job that reads documents into the database) + the Coolify control panel.DOWN / cancelled — SSH times out, confirmed gone.
The ingestion pipeline itselfThe background “walk” that pulls new content in.Was already switched OFF (the nightly /walk tasks were disabled; Phew is pre-launch, ID-45 re-ingest still pending). So nothing was actively running on it when the box went away.

Bottom line for Liam: the client’s site and data are fine and were never on the cancelled box. What we lost is the content-ingestion machine, which was idle anyway. We are rebuilding it fresh on the two new boxes. There is no outage to firefight; this is a clean rebuild on our own schedule.

Reachability of the two new boxes (checked, no login attempted):

BoxNameIPSSH port 22
Server Bcanonical-platform-vps87.106.72.59reachable
Server Acanonical-client-vps-phew88.208.255.178reachable

Both boxes are alive and listening. We have not logged in (no credentials handed over yet). The existing SSH key ~/.ssh/kh_ionos_ed25519 (which we’ll reuse) exists on the laptop and survived the old box — confirmed.


1. The bootstrap mechanism decision (how we get from password → key-only SSH)

Section titled “1. The bootstrap mechanism decision (how we get from password → key-only SSH)”

The two new boxes ship with an initial root password (IONOS default). The target state (matching the old box’s hardening, runbook {66.5}) is key-only SSH — no password login at all, only the ~/.ssh/kh_ionos_ed25519 key.

sshpass is NOT installed on the laptop (checked: which sshpass → not found). That tool would have let the agent script the one-time password login non-interactively. Without it, the cleanest path that keeps the agent from ever seeing or typing the password interactively is:

Chosen mechanism: Liam pastes the 2 passwords into .env.local temporarily, agent reads them from there

Section titled “Chosen mechanism: Liam pastes the 2 passwords into .env.local temporarily, agent reads them from there”

This is exactly what Liam offered, and it’s the cleanest option. The agent reads the password from an environment variable (never interactively, never echoed), uses it once to push the SSH key, immediately hardens the box to key-only, and then Liam deletes the passwords.

  • .env.local is gitignored (it already holds the old IONOS Cloud-Panel password the same way) — it never gets committed.
  • The passwords are single-use: the moment the key is installed and password-login is switched off, they are dead for SSH (they remain only the IONOS Cloud-Panel login, like before).

Alternative if Liam prefers not to put passwords in a file at all: Liam can paste each box’s public key directly in the IONOS Cloud Panel (most IONOS images have an “SSH keys” field per server), OR run one copy-paste command himself per box (the ! <command> prompt form). Both are spelled out in Step 2 as the fallback. The default below is the .env.local route because Liam offered it and it lets the agent do the work.

Security posture (flag for Liam): the passwords go into a temporary, gitignored file, are used once, and Liam removes them immediately after Step 2c reports success on both boxes. The agent will never print them. After hardening, password SSH is off, so even if a password leaked it could not be used to log in.


2. VPS bootstrap — both boxes (password → key-only SSH + base hardening)

Section titled “2. VPS bootstrap — both boxes (password → key-only SSH + base hardening)”

Target per box (identical to old box {66.5}): SSH key-only, ufw firewall, automatic security updates. Server B additionally needs the Coolify ports (8000/6001/6002); Server A does not (only Server B runs the control panel).

Step 2a — [OPERATOR] Put the two initial passwords into .env.local (temporary)

Section titled “Step 2a — [OPERATOR] Put the two initial passwords into .env.local (temporary)”

Liam opens /Users/liamj/Documents/development/canonical/.env.local and adds these two lines at the bottom (paste the real initial passwords IONOS gave for each box):

IONOS_SERVER_B_INITIAL_PW=<the initial root password for 87.106.72.59 (canonical-platform-vps)>
IONOS_SERVER_A_INITIAL_PW=<the initial root password for 88.208.255.178 (canonical-client-vps-phew)>

Save the file. Exact variable names matter — the agent looks for precisely IONOS_SERVER_B_INITIAL_PW and IONOS_SERVER_A_INITIAL_PW.

Tell the agent “passwords are in .env.local” when done.

If Liam prefers the no-password-in-file route instead: skip 2a/2b and use the fallback at the end of Step 2 (Cloud-Panel key paste, or a single self-run command), then resume at Step 2d.

Step 2b — [AGENT] Install the SSH key on both boxes (one-time, using the temp password)

Section titled “Step 2b — [AGENT] Install the SSH key on both boxes (one-time, using the temp password)”

For each box, the agent does the equivalent of ssh-copy-id non-interactively (reading the password from the env var, never echoing it). Because sshpass isn’t present, the agent will first install it locally in a throwaway way, OR drive the copy via a tiny expect-free mechanism — concretely:

  • Read IONOS_SERVER_B_INITIAL_PW / IONOS_SERVER_A_INITIAL_PW from .env.local.
  • Append ~/.ssh/kh_ionos_ed25519.pub to /root/.ssh/authorized_keys on each box (creating ~/.ssh with 700 and authorized_keys with 600).
  • Verify key login works: ssh -i ~/.ssh/kh_ionos_ed25519 -o IdentitiesOnly=yes root@<ip> 'echo key-ok' returns key-ok on both boxes.

The agent reports “key login confirmed on both boxes” before touching anything else.

Agent note: if sshpass cannot be installed in the sandbox, the agent will tell Liam and fall back to the Cloud-Panel paste (below) — it will NOT ask Liam to type the password into the agent terminal.

Step 2c — [AGENT] Base-harden both boxes (key-only SSH, firewall, auto-updates)

Section titled “Step 2c — [AGENT] Base-harden both boxes (key-only SSH, firewall, auto-updates)”

Once key login works, on each box the agent (over SSH with the key) reproduces the old box’s hardening verbatim:

  1. SSH key-only — write /etc/ssh/sshd_config.d/00-kh-hardening.conf (sorts before 50-cloud-init.conf, so it wins) with exactly:
    PasswordAuthentication no
    KbdInteractiveAuthentication no
    ChallengeResponseAuthentication no
    PubkeyAuthentication yes
    PermitRootLogin prohibit-password
    then systemctl reload ssh (or sshd). Confirm a password login is now refused.
  2. Firewall (ufw) — default deny incoming, then allow:
    • Server B (87.106.72.59): 22, 80, 443, 8000, 6001, 6002 tcp (the SSH + Coolify ports — Coolify control panel lives here).
    • Server A (88.208.255.178): 22, 80, 443 tcp only (no Coolify ports — it’s just a managed pipeline box). ufw enable.
  3. Automatic security updates — install + enable unattended-upgrades (/etc/apt/apt.conf.d/20auto-upgrades = Update-Package-Lists “1” + Unattended-Upgrade “1”).
  4. apt update && apt upgrade -y once.

Agent reports the ufw status numbered + the refused-password check for both boxes.

[OPERATOR] note — IONOS Cloud-Panel firewall is separate from ufw. ufw is the firewall inside the box. IONOS also has a provider-level firewall in the Cloud Panel in front of it. Liam said the Cloud-Panel firewall is already configured — good. The one thing to double-check in the Cloud Panel: Server B must allow inbound TCP 8000 (the Coolify dashboard) and 80/443; Server A must allow 80/443 (and 22 on both for SSH). If those provider-firewall ports aren’t open, the agent’s ufw rules won’t be reachable from outside. This is the only Cloud-Panel item; everything else is the agent’s.

Step 2d — [OPERATOR] Remove the temporary passwords

Section titled “Step 2d — [OPERATOR] Remove the temporary passwords”

As soon as the agent reports “both boxes hardened, key-only confirmed”, Liam deletes the two IONOS_SERVER_*_INITIAL_PW lines from .env.local and saves. (They’re now useless for SSH anyway — password login is off.)

Fallback bootstrap (only if NOT using .env.local)

Section titled “Fallback bootstrap (only if NOT using .env.local)”
  • [OPERATOR] Cloud-Panel key paste: in the IONOS Cloud Panel, for each server, paste the contents of ~/.ssh/kh_ionos_ed25519.pub into the server’s SSH-keys field and reboot / re-image if prompted. Then tell the agent “key is on both boxes” and resume at Step 2c. (Agent will get ~/.ssh/kh_ionos_ed25519.pub content for Liam to copy.)
  • [OPERATOR] one self-run command per box: at the agent’s prompt, Liam types ! ssh-copy-id -i ~/.ssh/kh_ionos_ed25519.pub root@87.106.72.59 (then the same for 88.208.255.178), entering the initial password when prompted. The ! prefix runs it in the session so the agent sees the result. Then resume at Step 2c.

3. Coolify install on Server B (the relocated control plane) — {66.6}

Section titled “3. Coolify install on Server B (the relocated control plane) — {66.6}”

Only Server B gets Coolify. It is the single control plane that will manage both boxes. (Greenfield install — there is no old Coolify DB to restore; that decision is already made.)

Step 3a — [AGENT] Run the Coolify installer on Server B

Section titled “Step 3a — [AGENT] Run the Coolify installer on Server B”

Over SSH to 87.106.72.59, the agent runs the official installer (same as the old box):

curl -fsSL https://cdn.coollabs.io/coolify/install.sh | sudo bash

This installs Docker + Coolify (control-plane DB, Redis, Traefik proxy, etc.). The agent confirms the dashboard answers host-locally: curl -sS -o /dev/null -w '%{http_code}' http://localhost:8000 → expect 302. Agent reports “Coolify installed, localhost:8000 → 302”.

Step 3b — [OPERATOR] Create the Coolify admin account in the browser

Section titled “Step 3b — [OPERATOR] Create the Coolify admin account in the browser”

Coolify’s first-run admin account can only be created in the web UI (no API for it).

  1. Liam opens http://87.106.72.59:8000 in a browser. (Port 8000 is already open per Liam’s firewall config; if the page times out, re-check the Cloud-Panel firewall allows 8000 — see Step 2c note.)
  2. Coolify shows a “Register” / “Create admin” screen on first visit. Liam sets the admin email + a strong password (store both in the password manager).
  3. That’s it for the UI. Tell the agent “Coolify admin created” + hand over an API token (next step).

Step 3c — [OPERATOR] Create a Coolify API token and hand it to the agent

Section titled “Step 3c — [OPERATOR] Create a Coolify API token and hand it to the agent”

So the agent can drive Coolify (create apps, set env, deploy):

  1. In Coolify → Keys & Tokens / API tokens (top-right profile menu → “API tokens”), create a new token with read/write scope. Copy it.
  2. Hand it to the agent the same temporary way as the passwords — paste into .env.local as:
    COOLIFY_API_TOKEN_NEW=<the new Coolify API token>
    COOLIFY_BASE_URL_NEW=http://87.106.72.59:8000
    Tell the agent “Coolify token is in .env.local”. (The agent uses these to talk to the new control plane via the Coolify MCP / API. Liam can remove COOLIFY_API_TOKEN_NEW after standup, OR keep it — it’s the live control-plane token going forward; see Step 5 where the same token value also goes into GitHub.)

Step 3d — [OPERATOR] Back up the Coolify env file (safety net)

Section titled “Step 3d — [OPERATOR] Back up the Coolify env file (safety net)”

The Coolify installer writes /data/coolify/source/.env on Server B (the control-plane’s own app-key + DB creds). Coolify’s installer explicitly says to back this up.

  • [AGENT] can read it and present the key names (not values) + write the file to the password manager’s import format, OR
  • [OPERATOR] simplest: once standup is done, Liam (with the agent’s help) copies /data/coolify/source/.env from Server B into the password manager as “Coolify control-plane env (Server B)”. This is a restore-safety backup, not something we copy from the old box (the old one is gone; this is the new install’s own file).

4. Add Server A as a managed server in Coolify (so B can deploy to A)

Section titled “4. Add Server A as a managed server in Coolify (so B can deploy to A)”

Coolify on Server B needs to be able to SSH into Server A to deploy the client pipeline there. Coolify generates its own SSH key for this; we authorise it on Server A.

Step 4a — [AGENT] Create/locate Coolify’s deploy SSH key and add Server A as a server

Section titled “Step 4a — [AGENT] Create/locate Coolify’s deploy SSH key and add Server A as a server”

Using the Coolify API/MCP (with COOLIFY_API_TOKEN_NEW):

  1. Coolify has a private key for connecting to remote servers (it generates one on install). The agent retrieves its public half (via the Coolify private-keys API, or by generating a dedicated new keypair in Coolify for Server A).
  2. The agent then authorises that public key on Server A — append it to /root/.ssh/authorized_keys on 88.208.255.178 (the agent still has key access to Server A from Step 2).
  3. The agent adds Server A as a new server in Coolify (name: canonical-client-vps-phew, IP 88.208.255.178, user root, the key from step 1) and runs Coolify’s “Validate server” so Coolify installs its helper + confirms it can reach Docker on Server A.

Agent reports “Server A validated in Coolify (reachable, Docker present)”.

Step 4b — [OPERATOR] Eyeball the two servers in the Coolify UI (verification gate)

Section titled “Step 4b — [OPERATOR] Eyeball the two servers in the Coolify UI (verification gate)”

Liam opens Coolify → Servers and confirms he sees two servers, both green/healthy: the localhost one (Server B itself, where Coolify runs) and canonical-client-vps-phew (Server A). This is the “Liam’s eyes on it” gate before we build apps. If Server A shows red/unreachable, stop and tell the agent.


5. Client (Phew) apps on Server A — canonical-* prod + staging cocoindex apps

Section titled “5. Client (Phew) apps on Server A — canonical-* prod + staging cocoindex apps”

⚠ THIS STEP WAITS FOR TASK ID-129 (PullMD decommission) TO MERGE TO main. The compose files in git (deploy/coolify/docker-compose.production.yaml + docker-compose.staging.yaml) still contain pullmd/playwright/trafilatura today (ID-129 subtasks 1/2/4 are in_progress, subtask 3 pending). If we build the apps now, they’d deploy the AGPL pullmd stack we’re trying to remove. Once ID-129 is merged to main (and, for the client apps, the release/v1.0.0 branch the prod app tracks carries the change — see the branch note below), the apps deploy from the already-pullmd-free compose and Step 5 proceeds. Do not start Step 5 until ID-129 is merged.

Branch note: the prod client app tracks git branch release/v1.0.0, the staging app tracks staging (carry-across inventory §4). ID-129 landing on main is necessary but the prod app sees it only once it’s on release/v1.0.0 (next release cut) and staging sees it once on staging. At standup we can point the apps at the pullmd-free refs; coordinate the exact ref with the ID-45 release cut. Flag to confirm with Liam at the time: which ref each client app should track so it gets the pullmd-free compose.

This step is mostly [AGENT] (Coolify MCP + the carry-across inventory), with two [OPERATOR] secret hand-offs.

Step 5a — [AGENT] Create the Coolify project + the two apps (pointed at Server A)

Section titled “Step 5a — [AGENT] Create the Coolify project + the two apps (pointed at Server A)”

Using the Coolify API/MCP against Server B’s control plane, targeting Server A:

  1. Create a Coolify project for the client pipeline (e.g. canonical-client-phew).

  2. Create the prod app and the staging app as Docker-Compose (git-sourced) apps, each with the same shape as the old box (carry-across inventory §4):

    FieldProd appStaging app
    Git repoai-solution-hub/canonical (GitHub App source)same
    Branchrelease/v1.0.0 (confirm ref carries ID-129 — see note)staging
    Build packDocker Composesame
    Base directory/deploy/coolifysame
    Compose file/docker-compose.production.yaml/docker-compose.staging.yaml
    ServerServer A (canonical-client-vps-phew)Server A
    FQDN / Domains fieldleave UNSET (labels-only routing, inventory §4)leave UNSET

    Record the new app UUIDs the agent gets back — they’re needed for Step 6 (GitHub).

Step 5b — [AGENT + OPERATOR] Set the app environment variables (with the 9 drifts fixed)

Section titled “Step 5b — [AGENT + OPERATOR] Set the app environment variables (with the 9 drifts fixed)”

The full key list + values/dispositions are in CARRY-ACROSS-inventory.md §1a (prod) / §1b (staging). The agent sets all the non-secret keys verbatim from the inventory. The secret keys need their live values — handed over by Liam (from the password manager / the old Coolify env / the GH environments). For each secret key the agent will list exactly which it needs; Liam pastes them into .env.local (temporary) as e.g. PHEW_PROD_ANTHROPIC_API_KEY=… and removes them after, OR pastes directly into the Coolify app env UI himself.

The 9 drifts to FIX during this step (do NOT reproduce the old box’s mistakes):

  1. Duplicate EXTRACT_API_TOKEN — the old box had two identical EXTRACT_API_TOKEN env rows on each app (a POST-not-PATCH footgun). Create ONE row only, each app.
  2. /extract routing — the old Traefik router covered /walk ‖ /health ‖ /extract. Decide with Liam whether /extract should be publicly routed; the runbook’s invariant says only /walk + /health. Default: scope the new router to /walk + /health unless Liam wants /extract exposed.
  3. Resource limits — the old box had zero CPU/memory limits. Set real limits on the right-sized box (the final tier comes from ID-127.14’s on-box metrics; until then set sane caps, e.g. a couple of GB memory / a CPU or two per app, so a runaway can’t take the host down).
  4. prod auto-deploy sentinel — staging had a watch_paths sentinel (deploy/coolify/.never-auto-deploy-sentinel) to stop git-auto-deploy; prod had none (relied on a UI toggle). Set the watch_paths sentinel on BOTH apps so git-auto-deploy never fires (the GitHub Action is the sole deploy trigger). This is the API-visible, reproducible mechanism.
  5. LMDB backup tools image — on the old box the backup image got pruned by Coolify’s nightly force_docker_cleanup, so backups silently failed for weeks. On Server A: after apps exist, build kh-lmdb-backup-tools:latest from deploy/onprem/backup/Dockerfile.tools and protect it from cleanup (or pin it) — see Step 7c.
  6. Backup script location — scripts live in /root/kh-backup/ (only the secrets .env lives in /root/kh-secrets/). Place them there on Server A.
  7. Coolify docker auto-cleanup — the old box ran force_docker_cleanup=true nightly at 80% threshold (that’s what pruned the backup image). On Server B’s Coolify, either turn that off for the backup image or ensure the backup tools image is excluded/pinned.
  8. SENTRY_AUTH_TOKEN shared — prod + staging used the same value. Note for rotation hygiene; carry as-is is acceptable but flag it.
  9. pullmd removal — because we’re waiting for ID-129, the new compose has no pullmd. Confirm none of the pullmd env keys (PULLMD_*, CACHE_DB) are set on the new apps — they’re gone with the decommission.

Step 5c — [AGENT] Recreate the 3 scheduled /walk tasks (carry-across inventory §2)

Section titled “Step 5c — [AGENT] Recreate the 3 scheduled /walk tasks (carry-across inventory §2)”

After the apps exist, the agent creates the scheduled tasks via the Coolify MCP (scheduled_tasks(action='create')), matching the inventory exactly:

AppNameContainerFrequencyEnabled
prodcocoindex-walkcocoindex0 2 * * *false (off until cutover)
stagingcocoindex-walkcocoindex-staging0 2 * * *false
stagingcocoindex-walk-hourly-fallbackcocoindex-staging0 * * * *true

Command (all three, verified absolute-interpreter-path form — bare python3 exits 127 in the buildpack image):

/layers/google.python.runtime/python/bin/python3 -c 'import os,urllib.request as u;print(u.urlopen(u.Request("http://127.0.0.1:8080/walk",method="POST",headers={"Authorization":"Bearer "+os.environ["CRON_SECRET"]}),timeout=30).status)'

Keep prod + staging-nightly disabled so no walk fires (no Anthropic token burn) until Liam deliberately enables at cutover. The staging hourly fallback was enabled on the old box; match that only if Liam wants staging actively walking — otherwise leave all three off until ID-45 cutover.

Step 5d — [OPERATOR] Eyeball the apps in Coolify (verification gate)

Section titled “Step 5d — [OPERATOR] Eyeball the apps in Coolify (verification gate)”

Liam opens Coolify → the client project and confirms two apps exist on Server A, both configured (env present, no pullmd, scheduled tasks listed). Don’t deploy yet — DNS + GitHub rewiring (Step 6) should land together with the first deploy to avoid the stale-image race.


6. DNS + GitHub rewiring (point everything at the new boxes)

Section titled “6. DNS + GitHub rewiring (point everything at the new boxes)”

Two halves: (a) DNS so the pipeline hostnames resolve to Server A; (b) GitHub so the deploy Action talks to Server B’s Coolify and the new app UUIDs.

Step 6a — [OPERATOR] Repoint the pipeline DNS A-records to Server A

Section titled “Step 6a — [OPERATOR] Repoint the pipeline DNS A-records to Server A”

The two pipeline hostnames currently point at the dead box (77.68.122.71). In the DNS provider for aisolutionhub.co.uk (where these records live), change both A records to Server A’s IP 88.208.255.178:

  • kh-pipeline.aisolutionhub.co.uk88.208.255.178
  • kh-pipeline-staging.aisolutionhub.co.uk88.208.255.178

(Let’s Encrypt will re-issue the HTTPS certs automatically once DNS points to Server A and the Traefik labels apply — nothing to copy from the old box.) Tell the agent when DNS is changed; the agent can verify with a DNS lookup.

Step 6b — [OPERATOR] Update the GitHub Production + Staging environment secrets

Section titled “Step 6b — [OPERATOR] Update the GitHub Production + Staging environment secrets”

The deploy Action (.github/workflows/onprem-deploy.yml) reads COOLIFY_BASE_URL + COOLIFY_API_TOKEN from the GitHub Production and Staging environments. Both must now point at Server B’s control plane. In GitHub → ai-solution-hub/canonical → Settings → Environments:

  • In Production and in Staging, set/replace:
    • COOLIFY_BASE_URL = http://87.106.72.59:8000 (Server B)
    • COOLIFY_API_TOKEN = the new token from Step 3c
  • (Reuse the same Coolify creds for both environments — the AMENDMENT says distinguish prod/staging/client by server + app UUID + deploy input, not by separate credentials. No new GitHub environment is created.)

This is the cleanest as an [OPERATOR] step (GitHub secret editing). The agent can prepare the exact values for Liam to paste, but cannot write GitHub environment secrets itself.

Step 6c — [AGENT] Update the app UUIDs in the deploy workflow + open a PR

Section titled “Step 6c — [AGENT] Update the app UUIDs in the deploy workflow + open a PR”

The deploy workflow hardcodes the old client app UUIDs:

  • prod deploy-cocoindex job → COOLIFY_APP_UUID: ybiczck7f7e1xbdev3bk89cr (onprem-deploy.yml:339)
  • staging deploy-cocoindex-staging job → COOLIFY_APP_UUID: ljurtiab99kb6dst38k76419 (onprem-deploy.yml:444)

The agent replaces these two with the new app UUIDs from Step 5a, on a branch, and opens a PR for Liam to merge. (These are non-secret config — safe in the workflow file.) After merge, a push / workflow_dispatch deploys to the new boxes.

Timing (the stale-image race — runbook B1): land the GitHub secret change (6b), the UUID change (6c), and the first deploy together, and keep Coolify git-auto-deploy disabled (the watch_paths sentinel from Step 5b). If auto-deploy were on, a push would redeploy on a stale tag before the build finished. With the sentinel set and the Action as the sole trigger, this is safe.


7. Platform apps on Server B + final verification

Section titled “7. Platform apps on Server B + final verification”

Step 7a — [LATER / id-127 build wave] Platform pipeline apps on Server B

Section titled “Step 7a — [LATER / id-127 build wave] Platform pipeline apps on Server B”

The Platform pipeline (ca-pipeline-platform-*, prod + staging, wired to Platform Supabase zjqbr… prod / rbwqew… staging) is built on Server B as the id-127 build-wave phase (subtasks 127.1–127.11 + 127.13) — a separate, already-specced piece of work. Not expanded here. It follows the same app-creation shape as Step 5 but with the ca- naming, the platform DBs, the docker-compose.platform*.yaml composes, and the two platform deploy jobs. Sequence it after Server B’s Coolify is up (Step 3) and the client apps are settled.

Step 7b — [AGENT, supervised] First health + walk verification (client, Server A)

Section titled “Step 7b — [AGENT, supervised] First health + walk verification (client, Server A)”

Once Step 5/6 are done and ID-129 is merged:

  1. [AGENT/OPERATOR] Deploy the staging client app (Coolify deploy, or push to staging). Confirm the Coolify deploy log shows a pull, not a host build, and the container boots lifespan-only (no walk, no token burn).
  2. [AGENT] /healthhttps://kh-pipeline-staging.aisolutionhub.co.uk/health → 200 (once DNS + cert are live).
  3. [OPERATOR-supervised] one manual /walk — the agent triggers a single supervised walk (Coolify “Run now” on the staging task, or the loopback POST) while watching docker logs; kill-switch = Coolify Stop / docker stop. Confirm a pipeline_runs row lands in Supabase. This is FOCUS/operator work (it spends Anthropic tokens) — do it deliberately, not as a background step.

Step 7c — [AGENT] Arm the LMDB backup on Server A (fixing the broken-backup drift)

Section titled “Step 7c — [AGENT] Arm the LMDB backup on Server A (fixing the broken-backup drift)”
  1. [AGENT] copy deploy/onprem/backup/{lmdb-backup-cold.sh,Dockerfile.tools,restore-verify.sh} to /root/kh-backup/ on Server A (drift §6 — correct location).
  2. [OPERATOR] provide the S3 backup creds (Supabase Storage onprem-backups bucket key + AWS_*), placed in /root/kh-secrets/lmdb-backup.env (chmod 600). Regenerate these (old ones are gone). The agent can place the file once Liam hands the values over the temp way.
  3. [AGENT] build the tools image: docker build -t kh-lmdb-backup-tools:latest -f /root/kh-backup/Dockerfile.tools /root/kh-backup — and protect it from Coolify’s nightly cleanup (drift §5/§7) so backups don’t silently break again.
  4. [AGENT] add the host crontab line with the new prod app UUID: 30 3 * * * APP_UUID=<new-prod-uuid> /root/kh-backup/lmdb-backup-cold.sh >> /var/log/lmdb-backup.log 2>&1.
  5. [AGENT, supervised] run one backup + one restore-verify.sh and confirm an object lands in the bucket. (Drift §2 — this is the test the old box silently failed.)

Immediate [OPERATOR] next steps (the short list Liam acts on now)

Section titled “Immediate [OPERATOR] next steps (the short list Liam acts on now)”

These are the only things blocking the agent from starting. In order:

  1. Decide the bootstrap route. Default = put the two initial root passwords into .env.local as IONOS_SERVER_B_INITIAL_PW and IONOS_SERVER_A_INITIAL_PW, then tell the agent. (Or pick the Cloud-Panel key-paste fallback.)
  2. Confirm the Cloud-Panel firewall allows: Server B → 22/80/443/8000; Server A → 22/80/443. (Liam said it’s configured — just confirm 8000 on Server B.)
  3. After the agent hardens both boxes: create the Coolify admin account at http://87.106.72.59:8000, then create a Coolify API token and drop it into .env.local as COOLIFY_API_TOKEN_NEW (+ COOLIFY_BASE_URL_NEW=http://87.106.72.59:8000).
  4. Remove the temporary passwords from .env.local once the agent says both boxes are key-only.
  5. Hold Step 5 (client apps) until Task ID-129 is merged to main — it’s in_progress now. The agent will flag when it’s clear.

(GitHub secret edits, DNS repoint, and the per-app secret hand-offs come later, in Steps 5/6 — Liam doesn’t need to do them up front.)