Private ingress cutover — Cloudflare Tunnel + Access (DR-046)
Private ingress cutover — Cloudflare Tunnel + Access (DR-046)
Section titled “Private ingress cutover — Cloudflare Tunnel + Access (DR-046)”What this runbook is. The execution sequence that retires the “pipeline origin is
directly public” property for one pipeline FQDN at a time. The design is already
ratified as DR-046 — Cloudflare Tunnel + Access
service token, routed through the existing Traefik path-scope, bearer retained as a second
factor, external /health monitor retired. This runbook implements DR-046; it does not
re-open it. Option A vs mTLS is settled: mTLS is a last-resort per-client escape hatch
(§7), never the platform default.
Provenance. Mechanics, option comparison and the ordered migration derive from the
{127.20} TECH-slice assessment (specs/id-127-platform-pipeline/notes/id-127.20-private-ingress-assessment.md
authored S454) — cited here as settled evidence, not as an open
question. Current public surface facts:
reference/deployment-architecture.md §4.
Status: executed for both PLATFORM hosts — staging S555, prod S558 (2026-08-12); Server B firewall closed, §5 verified. The two CLIENT hosts (Server A) remain on the public path, deliberately deferred to the id-45 domain flip per the §1.1 client-host caveat — their cutover is owned by the dedicated client-cutover task.
Read the ordering rule first. Every step is reversible until step §3.6 (firewall close). Do the parallel-run (§3.5) before you cut DNS, and cut DNS before you close the firewall — never the other way round. Rollback (§6) unwinds in the reverse order.
1. Preconditions
Section titled “1. Preconditions”All four must hold before the owner window opens. None are checked automatically.
| # | Precondition | How to confirm | Blocker if unmet |
|---|---|---|---|
| 1 | {127.38} has landed — the qa_sidecar app-side write half is retired (route MATERIALISE leg + lib/q-a-pairs/sidecar-path.ts), so nothing mints __qa__/ paths | tasks/id-127.md shows {127.38} checked; no __qa__ path minting in app/api/q-a-pairs/[id]/route.ts | HARD GATE — DR-086a. {127.38} must land before the ingress flip makes /walk reachable on the new path. Do not start §3 until it has. |
| 2 | A Cloudflare account exists and the pipeline zone is (or can be moved) onto Cloudflare nameservers | Cloudflare dashboard → the zone is listed and Active | Tunnel + Access service tokens need the zone on Cloudflare NS. Zero Trust free tier covers both. |
| 3 | The Cloudflare DPA is accepted (§2.2) | Cloudflare dashboard, account scope | Cloudflare terminates TLS at its edge and therefore sees ingress cleartext — the DPA is the contractual cover for that (DR-043’s “terms-protected transit egress” carve-out). |
| 4 | The target FQDN’s app is healthy and walking clean today | a green /walk on the current public path; a fresh pipeline_runs row | Never cut over a host that is already broken — you will not be able to tell tunnel faults from pre-existing ones. |
1.1 Which FQDNs move, and in what order
Section titled “1.1 Which FQDNs move, and in what order”Four pipeline hostnames exist today, all on the platform-owned (temporary)
aisolutionhub.co.uk zone, each Traefik-routed to exactly /walk|/health|/extract:
| Host | Coolify app | Server | Cutover order |
|---|---|---|---|
ca-platform-pipeline-staging.aisolutionhub.co.uk | ca-pipeline-platform-staging | B (87.106.72.59) | 1st — DONE (S555, 2026-08-12) |
ca-platform-pipeline.aisolutionhub.co.uk | ca-pipeline-platform-production | B | 2nd — DONE (S558, 2026-08-12) |
ca-client-pipeline-staging.aisolutionhub.co.uk | ca-pipeline-client-staging | A (88.208.255.178) | 3rd |
ca-client-pipeline.aisolutionhub.co.uk | ca-pipeline-client-production | A | 4th |
(App names corrected S555 to the live Coolify names — the original table carried
the pre-rename ca-cocoindex-*/kh-cocoindex-* names.)
Run the sequence once per host. Steps §2–§6 are per-FQDN; the account-level owner steps (§2.1–§2.2) are one-time.
Client-host caveat. The two client hosts flip off aisolutionhub.co.uk onto a
client-owned domain at the id-45 handover
(runbooks/per-client-release-pins.md). If the handover is
near, cut those hosts over after the domain flip and mint their Tunnel + Access
application on the client’s own zone — a tunnel minted on the platform zone would have to
be re-minted anyway.
2. Owner steps — [OWNER — Liam]
Section titled “2. Owner steps — [OWNER — Liam]”Every step in this section needs a human with account credentials. Nothing here is agent-executable. Each step names the surface it happens on.
2.1 Nameserver move for the pipeline zone — [OWNER — Liam] · registrar + Cloudflare dashboard · one-time
Section titled “2.1 Nameserver move for the pipeline zone — [OWNER — Liam] · registrar + Cloudflare dashboard · one-time”- Cloudflare dashboard → Add a site →
aisolutionhub.co.uk→ Free plan. Cloudflare scans and imports the existing DNS records — review the imported set before proceeding and confirm the four pipeline A-records plus every other record on the zone (app, Coolify dashboard, mail) came across intact. A missed MX/TXT record is the classic way this step causes an unrelated outage. - At the domain’s registrar (wherever
aisolutionhub.co.uk’s NS records are set): replace the current nameservers with the two Cloudflare nameservers the dashboard shows. - Cloudflare dashboard → wait for the zone to report Active (minutes to ~24h).
- Set every pipeline A-record to DNS-only (grey cloud) for now. Nothing changes behaviour yet — this step only moves authority, and leaving records unproxied keeps the pre-cutover posture byte-identical.
This is the one hard prerequisite of DR-046. If the zone cannot move to Cloudflare, stop and read §7 — do not improvise an alternative here.
2.2 Accept the Cloudflare DPA — [OWNER — Liam] · Cloudflare dashboard, account scope · one-time
Section titled “2.2 Accept the Cloudflare DPA — [OWNER — Liam] · Cloudflare dashboard, account scope · one-time”In the Cloudflare dashboard at account scope (Manage Account → the account-level configuration/privacy surface — Cloudflare relabels this periodically; it is account-level, not per-zone), accept the Customer Data Processing Addendum. Record the acceptance date in the private env-topology table alongside the other processor records.
Cloudflare terminates TLS at its edge, so it is a data processor for ingress cleartext. Precondition 3 exists because DR-043 admits terms-protected transit egress as a bounded exception — the DPA is what makes it terms-protected. A client that cannot accept a third party in that position takes the §7 escape hatch instead.
2.3 Create the Tunnel — [OWNER — Liam] · Cloudflare dashboard → Zero Trust · per FQDN
Section titled “2.3 Create the Tunnel — [OWNER — Liam] · Cloudflare dashboard → Zero Trust · per FQDN”- Zero Trust → Networks → Tunnels → Create a tunnel → connector type Cloudflared.
- Name it after the host, dropping the domain — see the naming convention in §2.6.
- Cloudflare shows a connector install command containing a tunnel token. Copy the token only (the long opaque string). It is a credential — treat it like any other secret: it goes to Coolify in §2.5 and nowhere else, and it never appears in a transcript, a commit, or a chat message.
- Do not add a public hostname in the dashboard yet. The hostname route is what publishes the Tunnel CNAME into DNS, and that is §3.5’s parallel-run step — adding it here cuts traffic over before the connector is verified.
2.4 Create the Access application + mint the service token — [OWNER — Liam] · Cloudflare dashboard → Zero Trust · per FQDN
Section titled “2.4 Create the Access application + mint the service token — [OWNER — Liam] · Cloudflare dashboard → Zero Trust · per FQDN”- Zero Trust → Access → Service Auth → Service Tokens → Create Service Token. Name it per §2.6. Pick the shortest duration you are willing to rotate on (Cloudflare defaults to 1 year).
- Cloudflare shows the Client ID and Client Secret. The secret is shown once. Put it straight into the surfaces in §2.5 — do not park it in a note, a scratch file, or a chat window.
- Zero Trust → Access → Applications → Add an application → Self-hosted. Application domain = the pipeline FQDN, path left empty (the whole host).
- Add one policy: action Service Auth, include rule Service Token → the token from step 1. No other include rules — no email, no IdP, no country.
- Do NOT add a
/healthBypass policy. DR-046 retires the external/healthmonitor outright (§4); a bypass policy would re-create an unauthenticated public path for no consumer.
2.5 Where each secret goes — [OWNER — Liam] · per FQDN
Section titled “2.5 Where each secret goes — [OWNER — Liam] · per FQDN”Three values come out of §2.3–§2.4. Each has exactly one home.
| Value | Goes to | Surface | Consumed by |
|---|---|---|---|
| Tunnel token | Coolify (Server B for platform hosts, Server A for client hosts) — env var on the cloudflared service added in §3.1 | Coolify dashboard → the app → Environment Variables | the cloudflared connector |
CF_ACCESS_CLIENT_ID | Vercel project env — Production scope for a prod FQDN, Preview scope for a staging FQDN; and the matching GitHub environment (Production / Staging) if CI smoke calls that host | Vercel project settings → Environment Variables; GitHub → Settings → Environments → Secrets | the three app→pipeline fetches (§3.3) |
CF_ACCESS_CLIENT_SECRET | same as above | same as above | same as above |
GitHub environment names are case-sensitive (Production / Staging) — see
runbooks/github-environments.md §2. The CF_ACCESS_* pair is
Vercel-and-CI only: the pipeline container never sends them, because it is the origin.
Not a new surface: PIPELINE_TRIGGER_SECRET and EXTRACT_API_TOKEN are unchanged and
stay exactly where they are. Access is an additional factor in front of the bearer, not a
replacement for it.
Follow-up (not part of this cutover): once a real token exists, add a row for the
Access service token to runbooks/secret-rotation.md §1 with its
expiry-driven cadence, and the tunnel token alongside it.
2.6 Naming convention — [OWNER — Liam]
Section titled “2.6 Naming convention — [OWNER — Liam]”Derived from the existing ca- scheme so the tunnel, the Access app and the FQDN are
greppable as one unit. <scope> is platform or the client slug; <env> is prod or
staging.
| Object | Pattern | Example (platform staging) |
|---|---|---|
| Tunnel | ca-<scope>-pipeline[-staging] (mirrors the FQDN’s left label) | ca-platform-pipeline-staging |
| Access application | ca-<scope>-pipeline-<env> | ca-platform-pipeline-staging |
| Service token | ca-access-<scope>-<env> | ca-access-platform-staging |
One tunnel, one Access application and one service token per (scope, env). Never share a token across environments or across clients — see §7.
3. Agent / config steps
Section titled “3. Agent / config steps”These are dispatchable. They touch the canonical repo and Coolify; they do not touch DNS, mint tokens, or edit the coolify-proxy Traefik configuration.
3.1 Add cloudflared as a service alongside coolify-proxy
Section titled “3.1 Add cloudflared as a service alongside coolify-proxy”Add a cloudflared service on the same host as the target app, attached to the
coolify-proxy network, running tunnel --no-autoupdate run with the §2.5 tunnel token
supplied by env. It needs no ports, no volumes, and no inbound rules — the connector dials
out to Cloudflare’s edge.
3.2 Point the ingress rule at Traefik — the load-bearing invariant
Section titled “3.2 Point the ingress rule at Traefik — the load-bearing invariant”The tunnel’s ingress rule for the hostname MUST resolve to the existing coolify-proxy Traefik, not to the pipeline container:
# cloudflared ingress — through Traefik. `originServerName` makes Traefik's# Host rule match, which is what preserves the path-scope.ingress: - hostname: ca-platform-pipeline-staging.aisolutionhub.co.uk service: https://coolify-proxy:443 originRequest: originServerName: ca-platform-pipeline-staging.aisolutionhub.co.uk - service: http_status:404(Equivalent host-networked form: service: https://127.0.0.1:443 with the connector on
network_mode: host. Prefer the network-attached form above — no host networking.)
Why this is not optional. Traefik’s router rule is
Host(...) && (PathPrefix('/walk') || PathPrefix('/health') || PathPrefix('/extract')).
Everything else on the app — /stage (raw-bytes test-fixture seam) and /producer-run
(the DR-055 manual forced-run surface) — has no router, and that absence is the
exposure guard. A rule pointing at http://ca-cocoindex-platform:8080 bypasses Traefik and
makes both reachable again, and you would have to re-implement the whole path-scope as
Access path policies. Routing through Traefik keeps /walk+/health+/extract-only and
the /stage non-route for free. §5.4 is the test that proves you got this right.
No Traefik change. The router labels in deploy/coolify/docker-compose.*.yaml stay
exactly as they are. The Host rule must keep matching the FQDN — which is why
originServerName is set.
3.3 Inject the Access headers on every app→pipeline call
Section titled “3.3 Inject the Access headers on every app→pipeline call”Four call sites in the canonical repo send app→pipeline requests. (write-back.ts deliberately reimplements the folder-drop nudge locally rather than importing it — a file-ownership convention — which is how the original three-site survey missed it; found + covered S534.) Each gets the two
CF-Access-* headers alongside its existing Authorization: Bearer … — additive, no
replacement:
| Call site | Function | Endpoint | Existing bearer |
|---|---|---|---|
lib/intelligence/pipeline.ts | nudgeCocoindexWalk | POST {COCOINDEX_WORKER_URL}/walk | PIPELINE_TRIGGER_SECRET |
lib/upload/folder-drop.ts | nudgeCorpusRewalk | POST {COCOINDEX_WORKER_URL}/walk | PIPELINE_TRIGGER_SECRET |
lib/extraction/clean-via-worker.ts | cleanViaWorker | POST {COCOINDEX_WORKER_URL}/extract | EXTRACT_API_TOKEN |
lib/edit-intent/write-back.ts | nudgeCorpusRewalk (local reimplementation — see note) | POST {COCOINDEX_WORKER_URL}/walk | PIPELINE_TRIGGER_SECRET |
headers: { Authorization: `Bearer ${pipelineTriggerSecret}`, ...(cfAccessClientId && cfAccessClientSecret ? { 'CF-Access-Client-Id': cfAccessClientId, 'CF-Access-Client-Secret': cfAccessClientSecret, } : {}),}Write it no-op-safe. When CF_ACCESS_CLIENT_ID/CF_ACCESS_CLIENT_SECRET are unset the
headers are simply omitted, and when they are set but Access is not in front of the origin
the headers are ignored. That property is what makes §6 a DNS-only rollback with no app
redeploy. Unlike the bearer, a missing CF_ACCESS_* pair must not short-circuit the
call — the existing unset-bearer guards (warn + skip for the nudges, typed
ExtractEndpointError for /extract) are unchanged.
Ship and deploy this before the parallel-run. Sending the headers early is harmless.
Status (S534, 2026-08-05): implemented at all four sites with no-op-safe guards (empty-string env treated as unset) and per-site tests; riding to
mainthis session. Only the §2.5 owner step (populatingCF_ACCESS_*in Vercel/GitHub) remains before §3.5.
Status (S558, 2026-08-12): prod cutover executed WITHOUT the §3.5 test hostname — the prod host was already dark (its A-record deleted; worker dead pre-{379.3}), so there was no live traffic to parallel-run and precondition 4 was waived by the owner. The pre-staged route/Access-app hostnames carried a
-productionsuffix (the tunnel’s name, not the FQDN) and had to be corrected to the real FQDN on BOTH surfaces — Traefik’s Host rule and the app’sCOCOINDEX_WORKER_URLbind to the FQDN, so a tunnel-name-shaped hostname 503s on the Coolify catch-all (§3.5’s S555 lesson, edge variant). CORRECTED same session: the prod corpus volume was NOT empty (the 7-doc synthetic corpus was staged on the persistent volume) — the §5 verification walk really ingested (7 source_documents rows, op 7cc7d9dd, ~28 claude-opus-4-6 calls) yet landed ZERO pipeline_runs rows. Prod’s run-record webhook fails silently — open defect, id-457. §5.6 on prod is therefore FAILED-pending-id-457, not N/A. (General rule that still holds: an empty/missing corpus makes/walkidle out BEFORE the flow-start webhook, so zero rows after a no-op walk proves nothing.)RESOLVED (S559, 2026-08-12): §5.6 GREEN on prod. Root cause was upstream of the pipeline: Vercel Production’s
SUPABASE_SERVICE_ROLE_KEYwas a legacy JWT and legacy JWT auth is dead on Platform prod (the Supabase↔Vercel integration syncs legacy JWTs, so re-integration could never fix it — sync now OFF, key set manually to thesb_secret_secret key). Proof: full-reprocess walk op090317a7landed ONEpipeline_runsrow transitioningin_progress → completed_with_errorswith full stage_counts (llm_extraction: 28) and derived rows (9 entity_mentions, 10 content_chunks, 10 record_embeddings). The S558 zero-derived-rows was a transient embedding-stage flow death (key disabled mid-walk) whosefailedwebhook the 401 swallowed. Full evidence:tasks/id-457.md(S559 entries); key-naming follow-up: id-458.
Status (S555, 2026-08-12): the three fire-and-forget nudge sites additionally register their fetch chain with
next/server’safter()vialib/runtime/keep-alive.ts. The §5.5 staging proof caught an unregisteredvoid fetchdying silently when the Vercel instance freezes on response — admit logged, zero Access edge events, no walk, no warning.cleanViaWorkeris awaited and was never affected. This defect was latent on the public path too; treat any future fire-and-forget app→pipeline call as requiring the same registration.
3.4 What is deliberately untouched
Section titled “3.4 What is deliberately untouched”| Path | Why it is unaffected |
|---|---|
POST /api/internal/pipeline-runs/record | VPS → Vercel — the opposite direction. It never traverses the pipeline ingress. |
The Coolify scheduled /walk task | Fires on http://127.0.0.1:8080/walk inside the container. Loopback, never public. |
| The Docker healthcheck | bash-builtin loopback GET /health asserting HTTP 200 (id-379 D1, S534 — was a bare TCP connect). Ingress-independent. |
datapath-watch | Reads pipeline_runs rows in Supabase. Never calls /health. |
The bearer gates in server.py | Unchanged. Access is factor one at the edge; the bearer stays factor two at the app. |
3.5 Parallel-run, then cut DNS — [OWNER — Liam] for the DNS steps
Section titled “3.5 Parallel-run, then cut DNS — [OWNER — Liam] for the DNS steps”- Zero Trust → Networks → Tunnels → the tunnel → Public Hostname → add a test
hostname on the same zone (e.g.
ca-platform-pipeline-staging-tunnel.aisolutionhub.co.uk) pointing at the §3.2 service. The production A-record is untouched and still live. On the TEST hostname only, also set Additional application settings → HTTP Settings → HTTP Host Header to the REAL FQDN (found S555):originServerNameonly fixes the TLS handshake — the HTTP Host header still carries the test name, Traefik’s Host rule doesn’t match, and every probe lands on the Coolify catch-all (503 “no available server”). The real-FQDN hostname added in step 4 needs no override — its Host header matches natively. - Add the test hostname to the Access application’s domain list so the same service-token policy covers it.
- Run all of §5 against the test hostname. Everything must be green.
- Cloudflare dashboard → DNS: replace the production A-record for the FQDN with the
Tunnel CNAME (
<tunnel-uuid>.cfargotunnel.com, proxied/orange-cloud) — in practice, move the public hostname in the Tunnel UI onto the real FQDN and let it write the record. - Re-run §5 against the real FQDN. Then remove the test hostname.
3.6 Close the origin — [OWNER — Liam] · VPS firewall
Section titled “3.6 Close the origin — [OWNER — Liam] · VPS firewall”Only after §3.5 step 5 is green: close inbound 443 and 80 at the VPS firewall for the host the FQDN lives on. The origin now has no reachable public port. Verify with §5.1.
Do not close the firewall on a shared server until every FQDN on it has cut over. Server B carries both platform hosts; Server A carries both client hosts. Closing 443 after only one of a server’s two hosts has moved takes the other one offline.
4. Retire the /health monitor (DR-046)
Section titled “4. Retire the /health monitor (DR-046)”DR-046 retires the external HTTP /health uptime ping. There is no bypass policy, no
carve-out, and no replacement pinger. Both real signals were already ingress-independent
and neither changes at cutover:
| Signal | What it proves | Where it lives | Effect of cutover |
|---|---|---|---|
| Docker healthcheck (container liveness + worker health) | /health answers HTTP 200 (503 = crashed worker; the id-379 D1b exit-on-crash then restarts the container) | deploy/coolify/docker-compose.*.yaml → bash-builtin loopback GET /health status probe (id-379 D1, S534), 30s interval | none — pure loopback inside the container |
datapath-watch (the real datapath signal) | ingestion is producing rows | deploy/onprem/monitor/datapath-watch.ts, run by the Vercel cron app/api/cron/datapath-watch (*/15, detectStalls) | none — reads pipeline_runs row arrival in Supabase, never HTTP /health |
Actions:
- If any external uptime monitor is armed against
/healthon a pipeline FQDN, delete it — after cutover it can only produce false alarms. (None is armed in the repo; check the monitoring vendor account, not the codebase.) - Leave
/healthinside the Traefik path rule. Removing it from the router is a separate change with its own blast radius and is out of scope here — behind Access it is not publicly reachable anyway, and it stays useful as the §5.2 probe. - “
/healthreturned 200” is not evidence ingestion works — a green worker thread says nothing about rows landing. That premise is whydatapath-watchexists; it is unchanged by this cutover.
5. Verification
Section titled “5. Verification”Run every check per FQDN, first against the §3.5 test hostname, then against the real one.
$FQDN, $CF_ID, $CF_SECRET, $TRIGGER, $EXTRACT are shell locals — never echoed,
never committed.
5.1 The origin is unreachable without Access
Section titled “5.1 The origin is unreachable without Access”# 1. No Access headers → Access rejects at the edge. Expect 302 (login redirect) or 403.# A 200/202 here means the Access policy is not attached.curl -sS -o /dev/null -w '%{http_code}\n' "https://$FQDN/health"
# 2. Bypass DNS and hit the origin IP directly. AFTER §3.6 expect a connection# timeout/refusal. Anything that answers means the firewall is still open.curl -sS --max-time 10 --resolve "$FQDN:443:87.106.72.59" "https://$FQDN/health" # Server B5.2 Access headers get through
Section titled “5.2 Access headers get through”curl -sS -o /dev/null -w '%{http_code}\n' \ -H "CF-Access-Client-Id: $CF_ID" -H "CF-Access-Client-Secret: $CF_SECRET" \ "https://$FQDN/health" # expect 2005.3 The bearer is still enforced behind Access
Section titled “5.3 The bearer is still enforced behind Access”Access-authenticated but bearer-wrong must still fail — this is the defence-in-depth proof.
# Valid Access headers, WRONG bearer → 401 (Access let it through; the app rejected it)curl -sS -o /dev/null -w '%{http_code}\n' -X POST \ -H "CF-Access-Client-Id: $CF_ID" -H "CF-Access-Client-Secret: $CF_SECRET" \ -H "Authorization: Bearer definitely-not-the-secret" "https://$FQDN/walk"
# Valid Access headers, CORRECT bearer → 202 + requestId (409 if a walk is already running)curl -sS -X POST \ -H "CF-Access-Client-Id: $CF_ID" -H "CF-Access-Client-Secret: $CF_SECRET" \ -H "Authorization: Bearer $TRIGGER" "https://$FQDN/walk"5.4 The Traefik path-scope survived (the §3.2 test)
Section titled “5.4 The Traefik path-scope survived (the §3.2 test)”# Fully Access-authenticated probe of a NON-routed path. Expect 404, or the Coolify# catch-all 503 — never the app. A 202/200 proves cloudflared bypassed Traefik: go# back to §3.2 and fix the ingress rule before cutting DNS.curl -sS -o /dev/null -w '%{http_code}\n' -X POST \ -H "CF-Access-Client-Id: $CF_ID" -H "CF-Access-Client-Secret: $CF_SECRET" \ -H "Authorization: Bearer $TRIGGER" "https://$FQDN/producer-run"curl -sS -o /dev/null -w '%{http_code}\n' -X POST \ -H "CF-Access-Client-Id: $CF_ID" -H "CF-Access-Client-Secret: $CF_SECRET" \ "https://$FQDN/stage"5.5 Every app→pipeline flow is green through the tunnel
Section titled “5.5 Every app→pipeline flow is green through the tunnel”All three §3.3 call sites, end to end, from the deployed app — not curl:
| # | Flow | Trigger | Green looks like |
|---|---|---|---|
| 1 | walk-nudge on feed publish (nudgeCocoindexWalk) | publish a feed with at least one passing article | 202 from /walk; a new pipeline_runs row; no “nudge rejected” warning in logs |
| 2 | folder-drop re-walk (nudgeCorpusRewalk) | upload a file via folder-drop | 202 from /walk; the object is ingested on the resulting walk |
| 3 | /extract pure-cleaner (cleanViaWorker) | run a clean that routes through the worker | 200 + { text, verdict, warnings }; no ExtractEndpointError |
Then confirm the ingress-independent signals are still green: the container healthcheck is
healthy in Coolify, and the datapath-watch cron reports no stall.
5.6 The reverse direction still works
Section titled “5.6 The reverse direction still works”Confirm the pipeline→app webhook landed a pipeline_runs row for the §5.5 walks
(POST /api/internal/pipeline-runs/record). It is VPS→Vercel and should be entirely
unaffected — this check exists to catch a firewall rule that closed egress as well as
ingress in §3.6.
6. Rollback
Section titled “6. Rollback”DNS-level, minutes, no app redeploy (the §3.3 headers are no-op-safe). Unwind in the reverse of the cutover order.
- Re-open inbound 443 + 80 at the VPS firewall for the affected host. Do this first — flipping DNS to a closed origin is a self-inflicted outage.
- Cloudflare dashboard → DNS: replace the Tunnel CNAME with the original A-record
for the FQDN → Server B
87.106.72.59(platform) or Server A88.208.255.178(client), set DNS-only / grey cloud so Access is out of the path. - Wait out the record TTL (keep pipeline records at a low TTL through the cutover window so this is fast).
- Re-run §5.5. The flows should be green on the public path again.
- Leave the tunnel, the Access application and the service token in place — a rollback is not a teardown, and you will want them for the retry.
What keeps working throughout the rollback:
- The bearer never stops gating anything.
PIPELINE_TRIGGER_SECRETon/walkandEXTRACT_API_TOKENon/extractare enforced by the app in every posture — security degrades from two factors to one, never to zero. - The Traefik path-scope, so
/stageand/producer-runstay unrouted either way. - The loopback scheduled
/walk, the container healthcheck, anddatapath-watch— all ingress-independent (§3.4). - The VPS→Vercel
pipeline-runs/recordwebhook. - The other FQDNs. Each host is cut over and rolled back independently.
If a rollback is needed, the app can stay deployed with CF_ACCESS_* set. Only unset
those env vars if you are abandoning the cutover entirely.
7. Per-client note
Section titled “7. Per-client note”Service tokens are issued and stored per client. One Access application and one service token per (client, environment), named per §2.6 — never a shared token across clients or across environments. Two reasons this is structural rather than tidiness:
- DR-045 puts every data-plane agent inside the client’s own isolation boundary. A
client’s
CF_ACCESS_CLIENT_ID/CF_ACCESS_CLIENT_SECRETtherefore live in that client’s own Vercel project, not in the platform project — same posture as their Supabase credentials. - Per-client issuance, storage, rotation and revocation is recurring work that couples to the S440 per-client walk-orchestration gate (assessment §6): once N>1 clients each run a schedule and an LMDB store on the shared Coolify control-plane, the token model and the walk-orchestration model are one decision, not two. Scope them together when GA planning starts.
Escape hatch (DR-046, do not re-derive). For a client contractually requiring that no third party sees ingress cleartext, the preferred alternative is the Vercel Static-IP allowlist — origin-terminated, no Cloudflare in the TLS path — with mTLS as last resort. That is a per-client exception to be exercised on contract terms; it does not re-open the platform default, which is this runbook.
8. Provenance + cross-links
Section titled “8. Provenance + cross-links”- DR-046 — ratified design (S456): Cloudflare
Tunnel default, Static-IP allowlist escape hatch,
/healthmonitor retired, ingress-is-control-plane-only. Sits on DR-042 (compute split), DR-043 (zero-egress definition) and DR-045 (per-client data plane). - DR-086a — the {127.38} qa_sidecar retirement that hard-gates this cutover (§1 precondition 1).
- {127.20} assessment —
specs/id-127-platform-pipeline/notes/id-127.20-private-ingress-assessment.md(authored S454): §1 current public surface, §2 Option A mechanics, §5 recommendation, §6 adjacent gates, §8 Cloudflare/Traefik source confirmations. reference/deployment-architecture.md§4 — the public/private surface table this runbook changes.runbooks/secret-rotation.md— the secret-handling pattern reused here; add the Access service token + tunnel token rows once minted (§2.5 follow-up).runbooks/github-environments.md§2 —Production/Stagingenvironment names (case-sensitive) for the CI-sideCF_ACCESS_*secrets.runbooks/onprem-b1-deploy.md— the Coolify/Traefik ingress mechanics this cutover sits on top of (read its superseded-topology banners first).runbooks/per-client-release-pins.md— the id-45 client domain flip that reorders §1.1 for the two client hosts.