S410 greenfield VPS standup runbook (two new IONOS boxes)
S410 — Greenfield VPS standup runbook (two new IONOS boxes)
Section titled “S410 — Greenfield VPS standup runbook (two new IONOS boxes)”Date: 2026-06-24 (session S410) Audience: Liam (non-technical operator) + Claude (agent). PLANNING DOCUMENT — nothing in here has been executed yet.
Every step is tagged [OPERATOR] (Liam does it, by hand / in a web UI / by pasting one command) or [AGENT] (Claude does it, once Liam has handed over what the step needs).
0. Plain-language situation — what is actually down
Section titled “0. Plain-language situation — what is actually down”Nothing that a user can see is down. Here is the picture in plain terms:
| Thing | What it is | Status right now |
|---|---|---|
Phew product website (https://kh.phew.org.uk) | The actual app a client would use. Runs on Vercel + Supabase — completely separate machines from the cancelled box. | UP — responds HTTP 307 (a normal redirect, the site is alive). It was never on the cancelled box. |
Phew databases (Supabase rovry… prod, turay… staging) | Where all the real knowledge lives. Managed by Supabase. | UP — untouched. The cancelled box never held any canonical data. |
Old IONOS box (77.68.122.71) | Only ran the ingestion pipeline (the background job that reads documents into the database) + the Coolify control panel. | DOWN / cancelled — SSH times out, confirmed gone. |
| The ingestion pipeline itself | The background “walk” that pulls new content in. | Was already switched OFF (the nightly /walk tasks were disabled; Phew is pre-launch, ID-45 re-ingest still pending). So nothing was actively running on it when the box went away. |
Bottom line for Liam: the client’s site and data are fine and were never on the cancelled box. What we lost is the content-ingestion machine, which was idle anyway. We are rebuilding it fresh on the two new boxes. There is no outage to firefight; this is a clean rebuild on our own schedule.
Reachability of the two new boxes (checked, no login attempted):
| Box | Name | IP | SSH port 22 |
|---|---|---|---|
| Server B | canonical-platform-vps | 87.106.72.59 | reachable ✅ |
| Server A | canonical-client-vps-phew | 88.208.255.178 | reachable ✅ |
Both boxes are alive and listening. We have not logged in (no credentials handed over
yet). The existing SSH key ~/.ssh/kh_ionos_ed25519 (which we’ll reuse) exists on the
laptop and survived the old box — confirmed.
1. The bootstrap mechanism decision (how we get from password → key-only SSH)
Section titled “1. The bootstrap mechanism decision (how we get from password → key-only SSH)”The two new boxes ship with an initial root password (IONOS default). The target state
(matching the old box’s hardening, runbook {66.5}) is key-only SSH — no password
login at all, only the ~/.ssh/kh_ionos_ed25519 key.
sshpass is NOT installed on the laptop (checked: which sshpass → not found). That
tool would have let the agent script the one-time password login non-interactively. Without
it, the cleanest path that keeps the agent from ever seeing or typing the password
interactively is:
Chosen mechanism: Liam pastes the 2 passwords into .env.local temporarily, agent reads them from there
Section titled “Chosen mechanism: Liam pastes the 2 passwords into .env.local temporarily, agent reads them from there”This is exactly what Liam offered, and it’s the cleanest option. The agent reads the password from an environment variable (never interactively, never echoed), uses it once to push the SSH key, immediately hardens the box to key-only, and then Liam deletes the passwords.
.env.localis gitignored (it already holds the oldIONOSCloud-Panel password the same way) — it never gets committed.- The passwords are single-use: the moment the key is installed and password-login is switched off, they are dead for SSH (they remain only the IONOS Cloud-Panel login, like before).
Alternative if Liam prefers not to put passwords in a file at all: Liam can paste each
box’s public key directly in the IONOS Cloud Panel (most IONOS images have an
“SSH keys” field per server), OR run one copy-paste command himself per box (the
! <command> prompt form). Both are spelled out in Step 2 as the fallback. The default
below is the .env.local route because Liam offered it and it lets the agent do the work.
Security posture (flag for Liam): the passwords go into a temporary, gitignored file, are used once, and Liam removes them immediately after Step 2c reports success on both boxes. The agent will never print them. After hardening, password SSH is off, so even if a password leaked it could not be used to log in.
2. VPS bootstrap — both boxes (password → key-only SSH + base hardening)
Section titled “2. VPS bootstrap — both boxes (password → key-only SSH + base hardening)”Target per box (identical to old box
{66.5}): SSH key-only,ufwfirewall, automatic security updates. Server B additionally needs the Coolify ports (8000/6001/6002); Server A does not (only Server B runs the control panel).
Step 2a — [OPERATOR] Put the two initial passwords into .env.local (temporary)
Section titled “Step 2a — [OPERATOR] Put the two initial passwords into .env.local (temporary)”Liam opens /Users/liamj/Documents/development/canonical/.env.local and adds these two
lines at the bottom (paste the real initial passwords IONOS gave for each box):
IONOS_SERVER_B_INITIAL_PW=<the initial root password for 87.106.72.59 (canonical-platform-vps)>IONOS_SERVER_A_INITIAL_PW=<the initial root password for 88.208.255.178 (canonical-client-vps-phew)>Save the file. Exact variable names matter — the agent looks for precisely
IONOS_SERVER_B_INITIAL_PW and IONOS_SERVER_A_INITIAL_PW.
Tell the agent “passwords are in .env.local” when done.
If Liam prefers the no-password-in-file route instead: skip 2a/2b and use the fallback at the end of Step 2 (Cloud-Panel key paste, or a single self-run command), then resume at Step 2d.
Step 2b — [AGENT] Install the SSH key on both boxes (one-time, using the temp password)
Section titled “Step 2b — [AGENT] Install the SSH key on both boxes (one-time, using the temp password)”For each box, the agent does the equivalent of ssh-copy-id non-interactively (reading the
password from the env var, never echoing it). Because sshpass isn’t present, the agent
will first install it locally in a throwaway way, OR drive the copy via a tiny expect-free
mechanism — concretely:
- Read
IONOS_SERVER_B_INITIAL_PW/IONOS_SERVER_A_INITIAL_PWfrom.env.local. - Append
~/.ssh/kh_ionos_ed25519.pubto/root/.ssh/authorized_keyson each box (creating~/.sshwith700andauthorized_keyswith600). - Verify key login works:
ssh -i ~/.ssh/kh_ionos_ed25519 -o IdentitiesOnly=yes root@<ip> 'echo key-ok'returnskey-okon both boxes.
The agent reports “key login confirmed on both boxes” before touching anything else.
Agent note: if
sshpasscannot be installed in the sandbox, the agent will tell Liam and fall back to the Cloud-Panel paste (below) — it will NOT ask Liam to type the password into the agent terminal.
Step 2c — [AGENT] Base-harden both boxes (key-only SSH, firewall, auto-updates)
Section titled “Step 2c — [AGENT] Base-harden both boxes (key-only SSH, firewall, auto-updates)”Once key login works, on each box the agent (over SSH with the key) reproduces the old box’s hardening verbatim:
- SSH key-only — write
/etc/ssh/sshd_config.d/00-kh-hardening.conf(sorts before50-cloud-init.conf, so it wins) with exactly:thenPasswordAuthentication noKbdInteractiveAuthentication noChallengeResponseAuthentication noPubkeyAuthentication yesPermitRootLogin prohibit-passwordsystemctl reload ssh(orsshd). Confirm a password login is now refused. - Firewall (
ufw) — default deny incoming, then allow:- Server B (
87.106.72.59):22, 80, 443, 8000, 6001, 6002tcp (the SSH + Coolify ports — Coolify control panel lives here). - Server A (
88.208.255.178):22, 80, 443tcp only (no Coolify ports — it’s just a managed pipeline box).ufw enable.
- Server B (
- Automatic security updates — install + enable
unattended-upgrades(/etc/apt/apt.conf.d/20auto-upgrades= Update-Package-Lists “1” + Unattended-Upgrade “1”). apt update && apt upgrade -yonce.
Agent reports the ufw status numbered + the refused-password check for both boxes.
[OPERATOR] note — IONOS Cloud-Panel firewall is separate from
ufw.ufwis the firewall inside the box. IONOS also has a provider-level firewall in the Cloud Panel in front of it. Liam said the Cloud-Panel firewall is already configured — good. The one thing to double-check in the Cloud Panel: Server B must allow inbound TCP 8000 (the Coolify dashboard) and 80/443; Server A must allow 80/443 (and 22 on both for SSH). If those provider-firewall ports aren’t open, the agent’sufwrules won’t be reachable from outside. This is the only Cloud-Panel item; everything else is the agent’s.
Step 2d — [OPERATOR] Remove the temporary passwords
Section titled “Step 2d — [OPERATOR] Remove the temporary passwords”As soon as the agent reports “both boxes hardened, key-only confirmed”, Liam deletes the two
IONOS_SERVER_*_INITIAL_PW lines from .env.local and saves. (They’re now useless for SSH
anyway — password login is off.)
Fallback bootstrap (only if NOT using .env.local)
Section titled “Fallback bootstrap (only if NOT using .env.local)”- [OPERATOR] Cloud-Panel key paste: in the IONOS Cloud Panel, for each server, paste the
contents of
~/.ssh/kh_ionos_ed25519.pubinto the server’s SSH-keys field and reboot / re-image if prompted. Then tell the agent “key is on both boxes” and resume at Step 2c. (Agent will get~/.ssh/kh_ionos_ed25519.pubcontent for Liam to copy.) - [OPERATOR] one self-run command per box: at the agent’s prompt, Liam types
! ssh-copy-id -i ~/.ssh/kh_ionos_ed25519.pub root@87.106.72.59(then the same for88.208.255.178), entering the initial password when prompted. The!prefix runs it in the session so the agent sees the result. Then resume at Step 2c.
3. Coolify install on Server B (the relocated control plane) — {66.6}
Section titled “3. Coolify install on Server B (the relocated control plane) — {66.6}”Only Server B gets Coolify. It is the single control plane that will manage both boxes. (Greenfield install — there is no old Coolify DB to restore; that decision is already made.)
Step 3a — [AGENT] Run the Coolify installer on Server B
Section titled “Step 3a — [AGENT] Run the Coolify installer on Server B”Over SSH to 87.106.72.59, the agent runs the official installer (same as the old box):
curl -fsSL https://cdn.coollabs.io/coolify/install.sh | sudo bashThis installs Docker + Coolify (control-plane DB, Redis, Traefik proxy, etc.). The agent
confirms the dashboard answers host-locally: curl -sS -o /dev/null -w '%{http_code}' http://localhost:8000 → expect 302. Agent reports “Coolify installed, localhost:8000 →
302”.
Step 3b — [OPERATOR] Create the Coolify admin account in the browser
Section titled “Step 3b — [OPERATOR] Create the Coolify admin account in the browser”Coolify’s first-run admin account can only be created in the web UI (no API for it).
- Liam opens
http://87.106.72.59:8000in a browser. (Port 8000 is already open per Liam’s firewall config; if the page times out, re-check the Cloud-Panel firewall allows 8000 — see Step 2c note.) - Coolify shows a “Register” / “Create admin” screen on first visit. Liam sets the admin email + a strong password (store both in the password manager).
- That’s it for the UI. Tell the agent “Coolify admin created” + hand over an API token (next step).
Step 3c — [OPERATOR] Create a Coolify API token and hand it to the agent
Section titled “Step 3c — [OPERATOR] Create a Coolify API token and hand it to the agent”So the agent can drive Coolify (create apps, set env, deploy):
- In Coolify → Keys & Tokens / API tokens (top-right profile menu → “API tokens”), create a new token with read/write scope. Copy it.
- Hand it to the agent the same temporary way as the passwords — paste into
.env.localas:Tell the agent “Coolify token is inCOOLIFY_API_TOKEN_NEW=<the new Coolify API token>COOLIFY_BASE_URL_NEW=http://87.106.72.59:8000.env.local”. (The agent uses these to talk to the new control plane via the Coolify MCP / API. Liam can removeCOOLIFY_API_TOKEN_NEWafter standup, OR keep it — it’s the live control-plane token going forward; see Step 5 where the same token value also goes into GitHub.)
Step 3d — [OPERATOR] Back up the Coolify env file (safety net)
Section titled “Step 3d — [OPERATOR] Back up the Coolify env file (safety net)”The Coolify installer writes /data/coolify/source/.env on Server B (the control-plane’s
own app-key + DB creds). Coolify’s installer explicitly says to back this up.
- [AGENT] can read it and present the key names (not values) + write the file to the password manager’s import format, OR
- [OPERATOR] simplest: once standup is done, Liam (with the agent’s help) copies
/data/coolify/source/.envfrom Server B into the password manager as “Coolify control-plane env (Server B)”. This is a restore-safety backup, not something we copy from the old box (the old one is gone; this is the new install’s own file).
4. Add Server A as a managed server in Coolify (so B can deploy to A)
Section titled “4. Add Server A as a managed server in Coolify (so B can deploy to A)”Coolify on Server B needs to be able to SSH into Server A to deploy the client pipeline there. Coolify generates its own SSH key for this; we authorise it on Server A.
Step 4a — [AGENT] Create/locate Coolify’s deploy SSH key and add Server A as a server
Section titled “Step 4a — [AGENT] Create/locate Coolify’s deploy SSH key and add Server A as a server”Using the Coolify API/MCP (with COOLIFY_API_TOKEN_NEW):
- Coolify has a private key for connecting to remote servers (it generates one on install). The agent retrieves its public half (via the Coolify private-keys API, or by generating a dedicated new keypair in Coolify for Server A).
- The agent then authorises that public key on Server A — append it to
/root/.ssh/authorized_keyson88.208.255.178(the agent still has key access to Server A from Step 2). - The agent adds Server A as a new server in Coolify (name:
canonical-client-vps-phew, IP88.208.255.178, userroot, the key from step 1) and runs Coolify’s “Validate server” so Coolify installs its helper + confirms it can reach Docker on Server A.
Agent reports “Server A validated in Coolify (reachable, Docker present)”.
Step 4b — [OPERATOR] Eyeball the two servers in the Coolify UI (verification gate)
Section titled “Step 4b — [OPERATOR] Eyeball the two servers in the Coolify UI (verification gate)”Liam opens Coolify → Servers and confirms he sees two servers, both green/healthy:
the localhost one (Server B itself, where Coolify runs) and canonical-client-vps-phew
(Server A). This is the “Liam’s eyes on it” gate before we build apps. If Server A shows
red/unreachable, stop and tell the agent.
5. Client (Phew) apps on Server A — canonical-* prod + staging cocoindex apps
Section titled “5. Client (Phew) apps on Server A — canonical-* prod + staging cocoindex apps”⚠ THIS STEP WAITS FOR TASK ID-129 (PullMD decommission) TO MERGE TO
main. The compose files in git (deploy/coolify/docker-compose.production.yaml+docker-compose.staging.yaml) still contain pullmd/playwright/trafilatura today (ID-129 subtasks 1/2/4 arein_progress, subtask 3pending). If we build the apps now, they’d deploy the AGPL pullmd stack we’re trying to remove. Once ID-129 is merged tomain(and, for the client apps, therelease/v1.0.0branch the prod app tracks carries the change — see the branch note below), the apps deploy from the already-pullmd-free compose and Step 5 proceeds. Do not start Step 5 until ID-129 is merged.Branch note: the prod client app tracks git branch
release/v1.0.0, the staging app tracksstaging(carry-across inventory §4). ID-129 landing onmainis necessary but the prod app sees it only once it’s onrelease/v1.0.0(next release cut) and staging sees it once onstaging. At standup we can point the apps at the pullmd-free refs; coordinate the exact ref with the ID-45 release cut. Flag to confirm with Liam at the time: which ref each client app should track so it gets the pullmd-free compose.
This step is mostly [AGENT] (Coolify MCP + the carry-across inventory), with two [OPERATOR] secret hand-offs.
Step 5a — [AGENT] Create the Coolify project + the two apps (pointed at Server A)
Section titled “Step 5a — [AGENT] Create the Coolify project + the two apps (pointed at Server A)”Using the Coolify API/MCP against Server B’s control plane, targeting Server A:
-
Create a Coolify project for the client pipeline (e.g.
canonical-client-phew). -
Create the prod app and the staging app as Docker-Compose (git-sourced) apps, each with the same shape as the old box (carry-across inventory §4):
Field Prod app Staging app Git repo ai-solution-hub/canonical(GitHub App source)same Branch release/v1.0.0(confirm ref carries ID-129 — see note)stagingBuild pack Docker Compose same Base directory /deploy/coolifysame Compose file /docker-compose.production.yaml/docker-compose.staging.yamlServer Server A ( canonical-client-vps-phew)Server A FQDN / Domains field leave UNSET (labels-only routing, inventory §4) leave UNSET Record the new app UUIDs the agent gets back — they’re needed for Step 6 (GitHub).
Step 5b — [AGENT + OPERATOR] Set the app environment variables (with the 9 drifts fixed)
Section titled “Step 5b — [AGENT + OPERATOR] Set the app environment variables (with the 9 drifts fixed)”The full key list + values/dispositions are in CARRY-ACROSS-inventory.md §1a (prod) /
§1b (staging). The agent sets all the non-secret keys verbatim from the inventory. The
secret keys need their live values — handed over by Liam (from the password manager /
the old Coolify env / the GH environments). For each secret key the agent will list exactly
which it needs; Liam pastes them into .env.local (temporary) as e.g.
PHEW_PROD_ANTHROPIC_API_KEY=… and removes them after, OR pastes directly into the Coolify
app env UI himself.
The 9 drifts to FIX during this step (do NOT reproduce the old box’s mistakes):
- Duplicate
EXTRACT_API_TOKEN— the old box had two identicalEXTRACT_API_TOKENenv rows on each app (a POST-not-PATCH footgun). Create ONE row only, each app. /extractrouting — the old Traefik router covered/walk ‖ /health ‖ /extract. Decide with Liam whether/extractshould be publicly routed; the runbook’s invariant says only/walk+/health. Default: scope the new router to/walk+/healthunless Liam wants/extractexposed.- Resource limits — the old box had zero CPU/memory limits. Set real limits on the right-sized box (the final tier comes from ID-127.14’s on-box metrics; until then set sane caps, e.g. a couple of GB memory / a CPU or two per app, so a runaway can’t take the host down).
- prod auto-deploy sentinel — staging had a
watch_pathssentinel (deploy/coolify/.never-auto-deploy-sentinel) to stop git-auto-deploy; prod had none (relied on a UI toggle). Set thewatch_pathssentinel on BOTH apps so git-auto-deploy never fires (the GitHub Action is the sole deploy trigger). This is the API-visible, reproducible mechanism. - LMDB backup tools image — on the old box the backup image got pruned by Coolify’s
nightly
force_docker_cleanup, so backups silently failed for weeks. On Server A: after apps exist, buildkh-lmdb-backup-tools:latestfromdeploy/onprem/backup/Dockerfile.toolsand protect it from cleanup (or pin it) — see Step 7c. - Backup script location — scripts live in
/root/kh-backup/(only the secrets.envlives in/root/kh-secrets/). Place them there on Server A. - Coolify docker auto-cleanup — the old box ran
force_docker_cleanup=truenightly at 80% threshold (that’s what pruned the backup image). On Server B’s Coolify, either turn that off for the backup image or ensure the backup tools image is excluded/pinned. SENTRY_AUTH_TOKENshared — prod + staging used the same value. Note for rotation hygiene; carry as-is is acceptable but flag it.pullmdremoval — because we’re waiting for ID-129, the new compose has no pullmd. Confirm none of the pullmd env keys (PULLMD_*,CACHE_DB) are set on the new apps — they’re gone with the decommission.
Step 5c — [AGENT] Recreate the 3 scheduled /walk tasks (carry-across inventory §2)
Section titled “Step 5c — [AGENT] Recreate the 3 scheduled /walk tasks (carry-across inventory §2)”After the apps exist, the agent creates the scheduled tasks via the Coolify MCP
(scheduled_tasks(action='create')), matching the inventory exactly:
| App | Name | Container | Frequency | Enabled |
|---|---|---|---|---|
| prod | cocoindex-walk | cocoindex | 0 2 * * * | false (off until cutover) |
| staging | cocoindex-walk | cocoindex-staging | 0 2 * * * | false |
| staging | cocoindex-walk-hourly-fallback | cocoindex-staging | 0 * * * * | true |
Command (all three, verified absolute-interpreter-path form — bare python3 exits 127 in
the buildpack image):
/layers/google.python.runtime/python/bin/python3 -c 'import os,urllib.request as u;print(u.urlopen(u.Request("http://127.0.0.1:8080/walk",method="POST",headers={"Authorization":"Bearer "+os.environ["CRON_SECRET"]}),timeout=30).status)'Keep prod + staging-nightly disabled so no walk fires (no Anthropic token burn) until Liam deliberately enables at cutover. The staging hourly fallback was enabled on the old box; match that only if Liam wants staging actively walking — otherwise leave all three off until ID-45 cutover.
Step 5d — [OPERATOR] Eyeball the apps in Coolify (verification gate)
Section titled “Step 5d — [OPERATOR] Eyeball the apps in Coolify (verification gate)”Liam opens Coolify → the client project and confirms two apps exist on Server A, both configured (env present, no pullmd, scheduled tasks listed). Don’t deploy yet — DNS + GitHub rewiring (Step 6) should land together with the first deploy to avoid the stale-image race.
6. DNS + GitHub rewiring (point everything at the new boxes)
Section titled “6. DNS + GitHub rewiring (point everything at the new boxes)”Two halves: (a) DNS so the pipeline hostnames resolve to Server A; (b) GitHub so the deploy Action talks to Server B’s Coolify and the new app UUIDs.
Step 6a — [OPERATOR] Repoint the pipeline DNS A-records to Server A
Section titled “Step 6a — [OPERATOR] Repoint the pipeline DNS A-records to Server A”The two pipeline hostnames currently point at the dead box (77.68.122.71). In the DNS
provider for aisolutionhub.co.uk (where these records live), change both A records to
Server A’s IP 88.208.255.178:
kh-pipeline.aisolutionhub.co.uk→88.208.255.178kh-pipeline-staging.aisolutionhub.co.uk→88.208.255.178
(Let’s Encrypt will re-issue the HTTPS certs automatically once DNS points to Server A and the Traefik labels apply — nothing to copy from the old box.) Tell the agent when DNS is changed; the agent can verify with a DNS lookup.
Step 6b — [OPERATOR] Update the GitHub Production + Staging environment secrets
Section titled “Step 6b — [OPERATOR] Update the GitHub Production + Staging environment secrets”The deploy Action (.github/workflows/onprem-deploy.yml) reads COOLIFY_BASE_URL +
COOLIFY_API_TOKEN from the GitHub Production and Staging environments. Both
must now point at Server B’s control plane. In GitHub →
ai-solution-hub/canonical → Settings → Environments:
- In
Productionand inStaging, set/replace:COOLIFY_BASE_URL=http://87.106.72.59:8000(Server B)COOLIFY_API_TOKEN= the new token from Step 3c
- (Reuse the same Coolify creds for both environments — the AMENDMENT says distinguish prod/staging/client by server + app UUID + deploy input, not by separate credentials. No new GitHub environment is created.)
This is the cleanest as an [OPERATOR] step (GitHub secret editing). The agent can prepare the exact values for Liam to paste, but cannot write GitHub environment secrets itself.
Step 6c — [AGENT] Update the app UUIDs in the deploy workflow + open a PR
Section titled “Step 6c — [AGENT] Update the app UUIDs in the deploy workflow + open a PR”The deploy workflow hardcodes the old client app UUIDs:
- prod
deploy-cocoindexjob →COOLIFY_APP_UUID: ybiczck7f7e1xbdev3bk89cr(onprem-deploy.yml:339) - staging
deploy-cocoindex-stagingjob →COOLIFY_APP_UUID: ljurtiab99kb6dst38k76419(onprem-deploy.yml:444)
The agent replaces these two with the new app UUIDs from Step 5a, on a branch, and opens
a PR for Liam to merge. (These are non-secret config — safe in the workflow file.) After
merge, a push / workflow_dispatch deploys to the new boxes.
Timing (the stale-image race — runbook B1): land the GitHub secret change (6b), the UUID change (6c), and the first deploy together, and keep Coolify git-auto-deploy disabled (the
watch_pathssentinel from Step 5b). If auto-deploy were on, a push would redeploy on a stale tag before the build finished. With the sentinel set and the Action as the sole trigger, this is safe.
7. Platform apps on Server B + final verification
Section titled “7. Platform apps on Server B + final verification”Step 7a — [LATER / id-127 build wave] Platform pipeline apps on Server B
Section titled “Step 7a — [LATER / id-127 build wave] Platform pipeline apps on Server B”The Platform pipeline (ca-pipeline-platform-*, prod + staging, wired to Platform
Supabase zjqbr… prod / rbwqew… staging) is built on Server B as the
id-127 build-wave phase (subtasks 127.1–127.11 + 127.13) — a separate, already-specced
piece of work. Not expanded here. It follows the same app-creation shape as Step 5 but
with the ca- naming, the platform DBs, the docker-compose.platform*.yaml composes, and
the two platform deploy jobs. Sequence it after Server B’s Coolify is up (Step 3) and the
client apps are settled.
Step 7b — [AGENT, supervised] First health + walk verification (client, Server A)
Section titled “Step 7b — [AGENT, supervised] First health + walk verification (client, Server A)”Once Step 5/6 are done and ID-129 is merged:
- [AGENT/OPERATOR] Deploy the staging client app (Coolify deploy, or push to
staging). Confirm the Coolify deploy log shows a pull, not a host build, and the container boots lifespan-only (no walk, no token burn). - [AGENT]
/health—https://kh-pipeline-staging.aisolutionhub.co.uk/health→ 200 (once DNS + cert are live). - [OPERATOR-supervised] one manual
/walk— the agent triggers a single supervised walk (Coolify “Run now” on the staging task, or the loopback POST) while watchingdocker logs; kill-switch = Coolify Stop /docker stop. Confirm apipeline_runsrow lands in Supabase. This is FOCUS/operator work (it spends Anthropic tokens) — do it deliberately, not as a background step.
Step 7c — [AGENT] Arm the LMDB backup on Server A (fixing the broken-backup drift)
Section titled “Step 7c — [AGENT] Arm the LMDB backup on Server A (fixing the broken-backup drift)”- [AGENT] copy
deploy/onprem/backup/{lmdb-backup-cold.sh,Dockerfile.tools,restore-verify.sh}to/root/kh-backup/on Server A (drift §6 — correct location). - [OPERATOR] provide the S3 backup creds (Supabase Storage
onprem-backupsbucket key +AWS_*), placed in/root/kh-secrets/lmdb-backup.env(chmod 600). Regenerate these (old ones are gone). The agent can place the file once Liam hands the values over the temp way. - [AGENT] build the tools image:
docker build -t kh-lmdb-backup-tools:latest -f /root/kh-backup/Dockerfile.tools /root/kh-backup— and protect it from Coolify’s nightly cleanup (drift §5/§7) so backups don’t silently break again. - [AGENT] add the host crontab line with the new prod app UUID:
30 3 * * * APP_UUID=<new-prod-uuid> /root/kh-backup/lmdb-backup-cold.sh >> /var/log/lmdb-backup.log 2>&1. - [AGENT, supervised] run one backup + one
restore-verify.shand confirm an object lands in the bucket. (Drift §2 — this is the test the old box silently failed.)
Immediate [OPERATOR] next steps (the short list Liam acts on now)
Section titled “Immediate [OPERATOR] next steps (the short list Liam acts on now)”These are the only things blocking the agent from starting. In order:
- Decide the bootstrap route. Default = put the two initial root passwords into
.env.localasIONOS_SERVER_B_INITIAL_PWandIONOS_SERVER_A_INITIAL_PW, then tell the agent. (Or pick the Cloud-Panel key-paste fallback.) - Confirm the Cloud-Panel firewall allows: Server B → 22/80/443/8000; Server A → 22/80/443. (Liam said it’s configured — just confirm 8000 on Server B.)
- After the agent hardens both boxes: create the Coolify admin account at
http://87.106.72.59:8000, then create a Coolify API token and drop it into.env.localasCOOLIFY_API_TOKEN_NEW(+COOLIFY_BASE_URL_NEW=http://87.106.72.59:8000). - Remove the temporary passwords from
.env.localonce the agent says both boxes are key-only. - Hold Step 5 (client apps) until Task ID-129 is merged to
main— it’sin_progressnow. The agent will flag when it’s clear.
(GitHub secret edits, DNS repoint, and the per-app secret hand-offs come later, in Steps 5/6 — Liam doesn’t need to do them up front.)