Secret rotation — pipeline + app surfaces
Pipeline secret-rotation runbook (ID-127.19)
Section titled “Pipeline secret-rotation runbook (ID-127.19)”Status: Runbook only. No live rotation, no env mutation, no secret VALUES appear
anywhere below — names and surfaces only. Authored for S436 D1 board ratification.
Generalises the {127.18} CRON_SECRET-split dual-accept pattern
(specs/id-127-platform-pipeline/notes/id-127.18-cron-secret-split-plan.md) into a reusable per-secret
procedure, then applies it to the still-unrotated S424 transcript-leaked staging set.
1. Inventory — rotatable pipeline secrets + surfaces (names only)
Section titled “1. Inventory — rotatable pipeline secrets + surfaces (names only)”Every secret that gates a pipeline door or that the pipeline holds. “Surface” = every
place the value must be set/updated on rotation. Verified against lib/cron-auth.ts,
scripts/cocoindex_pipeline/server.py, scripts/cocoindex_pipeline/flow.py,
deploy/coolify/*.yaml, and deploy/coolify/goose-pilot/.
| # | Secret (env name) | What it gates | Surfaces (where the value lives) | Verifier / consumer (code) |
|---|---|---|---|---|
| 1 | PIPELINE_TRIGGER_SECRET (new, {127.18}) | pipeline↔app boundary — inbound /walk (door 1), inbound pipeline-runs/record webhook (door 2), outbound walk/rewalk nudges (door 4) | 4 cocoindex Coolify apps (compose environment:) + Vercel (Preview/staging + Production scopes) | _walk_handler (server.py), verifyPipelineTriggerAuth (lib/cron-auth.ts) via pipeline-runs/record/route.ts; senders nudgeCocoindexWalk, nudgeCorpusRewalk (write-back.ts/folder-drop.ts), flow.py _emit_pipeline_run_webhook |
| 2 | CRON_SECRET | Vercel Cron scheduler → cron routes (door 3); legacy dual-accept fallback on doors 1/2/4 until {127.18} retire-shared lands | Vercel (all scopes) + 4 cocoindex Coolify apps (compose, until retire-shared) | verifyCronAuth (inline cron routes + verifyCronAuth-backed routes); fallback branch in verifyPipelineTriggerAuth + _walk_handler |
| 3 | EXTRACT_API_TOKEN | dedicated /extract PURE-CLEANER seam bearer (fail-closed) | 4 cocoindex Coolify apps + Vercel (sender scope) | _extract_handler (server.py); sender lib/extraction/clean-via-worker.ts |
| 4 | pipeline-runs webhook bearer | inbound door 2 — not a distinct secret | (same as #1) | flow.py _emit_pipeline_run_webhook sends PIPELINE_TRIGGER_SECRET (else CRON_SECRET); verified by #1. Do not rotate separately — rotating #1 rotates this. |
| 5 | MCP bearer (MCP_BEARER_TOKEN) | goose-pilot → remote MCP Authorization: Bearer … | No static secret. Minted per-run by run-pilot.sh via GoTrue login. Root secret = GOOSE_SERVICE_ACTOR_PASSWORD (see #6). | run-pilot.sh (GoTrue /token?grant_type=password); server verifies the GoTrue JWT. Rotate the actor password (#6), not a token. |
| 6 | GOOSE_SERVICE_ACTOR_PASSWORD (+ GOOSE_SERVICE_ACTOR_EMAIL) | service-actor account the MCP bearer is minted from | goose-pilot Coolify app env + the Supabase auth.users account password | run-pilot.sh login; GoTrue |
| 7 | GOOSE_ANTHROPIC_API_KEY (two leaked values per S424) | Anthropic access for goose-pilot recipe(s) | goose-pilot Coolify app env (docker-compose.yaml) — two distinct actor values | goose runtime |
| 8 | ANTHROPIC_API_KEY / OPENAI_API_KEY | LLM/embedding access for the cocoindex pipeline itself | 4 cocoindex Coolify apps + Vercel | pipeline flow (Python) + Next.js AI routes |
| 9 | SUPABASE_SERVICE_ROLE_KEY | service-role JWT — RLS-bypassing DB access | Supabase project (source of truth) + 4 cocoindex Coolify apps + Vercel (all scopes) + CI secrets | ~275 name-refs across lib/, scripts/, composes |
| 10 | staging DB password (POSTGRES_PASSWORD / Supabase DB password) | Postgres role auth | Supabase project settings (source of truth) + any DATABASE_URL/connection strings + cocoindex composes (POSTGRES_PASSWORD) + CI secrets | pipeline DB connections |
| 11 | Vercel protection-bypass (VERCEL_AUTOMATION_BYPASS_SECRET) | bypass Vercel deployment protection for automated E2E/smoke | Vercel project settings (regenerate = source of truth) + CI env consuming x-vercel-protection-bypass | E2E/smoke callers |
| 12 | Cloudflare Tunnel token (CLOUDFLARE_TOKEN) — one per FQDN (S558, 2026-08-12) | the cloudflared connector’s authorisation to dial out and serve that host’s tunnel — i.e. the private host’s entire ingress path | Cloudflare Zero Trust (source of truth: Networks → Tunnels) + the pipeline app’s Coolify env on the server the tunnel runs on (Server B for the platform pair). Nowhere else — never in Vercel, CI, or the repo | the cloudflared service in deploy/coolify/docker-compose.platform*.yaml. Rotate: re-mint the token in the Zero Trust dashboard → update CLOUDFLARE_TOKEN in Coolify → redeploy the app. Hard cutover; the connector reconnects on restart |
| 13 | Cloudflare Access service-token pair (CF_ACCESS_CLIENT_ID + CF_ACCESS_CLIENT_SECRET) — one pair per (scope, env); ca-access-platform-prod / ca-access-platform-staging exist today, client-scope pairs arrive with the client cutover | edge factor in front of the private pipeline hosts — Access rejects any app→pipeline call without the pair, before the bearer is ever reached | Cloudflare Zero Trust (Access → Service Auth; secret shown once) + Vercel Production and Preview scopes + GH Production/Staging environments + ~/.canonical-secrets/cf-*.env | the four app→pipeline fetches (lib/intelligence/pipeline.ts, lib/upload/folder-drop.ts, lib/extraction/clean-via-worker.ts, lib/edit-intent/write-back.ts). Expiry-driven cadence — Cloudflare defaults to 1 year. Rotate by minting a second token on the same Access policy, updating all surfaces, then deleting the old one (dual-accept: a policy may include two tokens) |
Adjacent-but-out-of-pipeline-scope (rotate on the app cadence, not here):
SENTRY_AUTH_TOKEN, SUPABASE_PUBLISHABLE_KEY (public), NEXT_PUBLIC_* (public).
2. The dual-accept rotation pattern (generalised from {127.18})
Section titled “2. The dual-accept rotation pattern (generalised from {127.18})”{127.18} introduced a new secret name (PIPELINE_TRIGGER_SECRET) alongside a legacy
one (CRON_SECRET), with verifyPipelineTriggerAuth accepting either during the
rollout window. Generalised, the reusable sequence for rotating any single secret’s
value is:
- Introduce-new — set the new value in a second slot the verifier already reads, or (for platform-managed secrets) create a second live credential.
- Dual-accept window — the verifier accepts old OR new; senders still send old. Nothing breaks regardless of deploy order (empty/unset = “not configured”, falls back).
- Cut senders over — update every sender/consumer to send/use the new value; redeploy.
- Verify — run the per-secret smoke (§4) on the new value across every environment.
- Retire-old — remove the old value / revoke the old credential / drop the fallback branch. Only after all surfaces confirmed green.
Which secrets support true dual-accept (zero-downtime) vs need a hard cutover:
| Secret(s) | Mode | Why |
|---|---|---|
SUPABASE_SERVICE_ROLE_KEY (#9) | Dual-accept | Supabase standby/rotation keys — old JWT keeps validating until you revoke the old signing key. Push new everywhere, verify, then revoke old. |
ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOSE_ANTHROPIC_API_KEY (#7/#8) | Dual-accept | Provider consoles allow multiple live keys — create new, deploy, revoke old. |
PIPELINE_TRIGGER_SECRET (#1) | Dual-accept | Verifier already reads a two-slot set (PIPELINE_TRIGGER_SECRET OR CRON_SECRET). Value rotation rides the same two-slot accept. |
CRON_SECRET (#2) | Hard cutover | verifyCronAuth reads one slot. Vercel redeploy is near-instant and a missed cron is idempotent/low-harm → acceptable sub-minute gap. |
EXTRACT_API_TOKEN (#3) | Hard cutover | _extract_handler reads one slot; /extract is on-demand → set new + redeploy sidecar + sender together, brief gap only if a clean is mid-flight. |
| staging DB password (#10) | Hard cutover | A Postgres role has one password. Update connection strings + recycle connections in a low-traffic window (brief connect blip). |
GOOSE_SERVICE_ACTOR_PASSWORD (#6 → MCP bearer #5) | Hard cutover, low-harm | One account password; the MCP bearer is minted per-run, so the next pilot run mints against the new password — only an in-flight run is affected. |
| Vercel protection-bypass (#11) | Hard cutover, low-harm | Vercel regenerates a single value; only automated E2E/smoke consume it — regenerate + update CI secret. |
Hard-cutover secrets are low-blast-radius here (idempotent crons, on-demand extract, per-run MCP token, CI-only bypass) → batch them into one short staging window rather than engineering a two-slot accept for each.
3. Cadence + ownership
Section titled “3. Cadence + ownership”| Secret class | Members | Interval | Owner |
|---|---|---|---|
| App↔pipeline bearers | PIPELINE_TRIGGER_SECRET, CRON_SECRET, EXTRACT_API_TOKEN | 90 days | Platform/pipeline eng (Liam) |
| Provider API keys | ANTHROPIC_API_KEY, OPENAI_API_KEY, both GOOSE_ANTHROPIC_API_KEY | 90 days | Platform eng |
| Supabase service-role key | SUPABASE_SERVICE_ROLE_KEY | 180 days | Platform eng / DB owner |
| Database password | staging DB password (POSTGRES_PASSWORD) | 180 days | Platform eng / DB owner |
| Service-actor password | GOOSE_SERVICE_ACTOR_PASSWORD | 90 days | Platform eng |
| Vercel bypass | VERCEL_AUTOMATION_BYPASS_SECRET | 180 days | Platform eng |
| Cloudflare private-ingress (S558, 2026-08-12) | CLOUDFLARE_TOKEN (per FQDN), CF_ACCESS_CLIENT_ID/CF_ACCESS_CLIENT_SECRET (per scope+env) | Expiry-driven, not interval-driven — the Access service token carries Cloudflare’s chosen duration (default 1 year); diary its expiry and rotate ahead of it. The tunnel token does not expire — rotate on suspected exposure or connector re-provision | Platform eng (Liam) |
Override — immediate rotation on ANY suspected exposure (transcript leak, log leak, repo leak). This override is exactly what triggered the S424 set below.
4. First rotation — the S424 leaked staging set (ordered checklist)
Section titled “4. First rotation — the S424 leaked staging set (ordered checklist)”Rotate the specific STAGING secrets exposed in the S424 transcript. Order = dual-accept (zero-downtime) first, then hard cutovers batched into one short window last.
Precondition (dependency): {127.19} depends on {127.18}. Land {127.18} retire-shared
first (set PIPELINE_TRIGGER_SECRET on staging surfaces, retire the CRON_SECRET
fallback on doors 1/2/4) so that rotating CRON_SECRET below touches only Vercel and
not the 4 cocoindex apps. If {127.18} retire-shared has NOT landed, treat CRON_SECRET
as also a pipeline-app surface (update all 4 cocoindex staging apps too).
Per step: surfaces to update → mode → verification smoke → mark-rotated.
4a. SUPABASE_SERVICE_ROLE_KEY (dual-accept — do first, longest tail)
Section titled “4a. SUPABASE_SERVICE_ROLE_KEY (dual-accept — do first, longest tail)”- Surfaces: Supabase staging project (generate standby key) → platform-staging cocoindex Coolify app → Vercel staging scope → CI secrets.
- Mode: dual-accept via standby key. Push new value to all surfaces, redeploy, THEN revoke old signing key.
- Smoke: a service-role read on staging (e.g. a pipeline run that writes a
pipeline_runsrow) succeeds; Supabase advisors show no auth errors. - Mark: record rotated-on date; revoke old key only after smoke green.
4b. Both GOOSE_ANTHROPIC_API_KEY values (dual-accept)
Section titled “4b. Both GOOSE_ANTHROPIC_API_KEY values (dual-accept)”- Surfaces: Anthropic console (create two new keys) → goose-pilot Coolify app env (both actor slots).
- Mode: dual-accept — deploy new, revoke old two keys in the console.
- Smoke: a goose-pilot run authenticates to Anthropic and completes its recipe.
- Mark: record rotated-on date; revoke old keys after a green pilot run.
4c. Batched hard-cutover window (staging, low-traffic) — 4d–4g together
Section titled “4c. Batched hard-cutover window (staging, low-traffic) — 4d–4g together”Announce a short staging maintenance window; expect sub-minute blips only.
- 4d. staging DB password (
POSTGRES_PASSWORD) — Surfaces: Supabase staging project (reset DB password) → platform-staging cocoindex compose env / connection strings → CI secrets. Mode: hard cutover; recycle connections. Smoke: pipeline connects + a walk run completes. Mark: rotated-on date. - 4e.
CRON_SECRET— Surfaces: Vercel staging scope (+ 4 cocoindex apps ONLY if {127.18} retire-shared not yet landed). Mode: hard cutover; redeploy. Smoke: a Vercel cron route returns 200 with the new bearer and 401 with the old. Mark: rotated-on date. - 4f.
EXTRACT_API_TOKEN— Surfaces: platform-staging cocoindex app + Vercel sender scope. Mode: hard cutover; redeploy sidecar + sender together. Smoke:POST /extractwith new bearer → 200; old bearer → 401. Mark: rotated-on date. - 4g. Vercel protection-bypass (
VERCEL_AUTOMATION_BYPASS_SECRET) — Surfaces: Vercel project settings (regenerate) → CI secret. Mode: hard cutover. Smoke: an automated E2E/smoke hit withx-vercel-protection-bypass: <new>bypasses protection; old value is rejected. Mark: rotated-on date.
4h. Also-leaked service-actor password (if in the S424 set)
Section titled “4h. Also-leaked service-actor password (if in the S424 set)”- Surfaces: Supabase
auth.usersservice-actor account → goose-pilot CoolifyGOOSE_SERVICE_ACTOR_PASSWORD. Mode: hard cutover (per-run mint). Smoke:run-pilot.shmints a fresh MCP bearer and reaches the MCP endpoint. Mark: rotated-on date.
Post-rotation: update the private env-topology table with each rotated-on date, and mark the S424 leaked-set closed in the continuation-prompt-ca-s435 pre-reqs.
Operational caveats — Coolify MCP env_vars reveal scope
Section titled “Operational caveats — Coolify MCP env_vars reveal scope”The Coolify MCP env_vars tool’s per-key reveal does not scope to the requested
key: a reveal call for a single variable (observed S457, during the {127.18}
PIPELINE_TRIGGER_SECRET rollout) returned every app secret in the response,
including unrelated high-value keys (Anthropic / OpenAI / Supabase service-role). The
key filter is a false-scoping promise — do not assume a reveal is narrowed to what you
asked for.
Why it matters at rotation time: revealing one secret to verify a rotation dumps the
whole app’s secret set into the agent/operator transcript — the exact transcript-leak
vector that produced the S424 leaked set this runbook exists to clean up. Treat any
env_vars reveal call as a whole-app secret exposure, not a single-key read.
Mitigation: prefer setting/patching env values blind (write-only) over revealing to verify; where verification is required, verify via a downstream smoke (the §4 per-secret 200/401 probes) rather than reading the value back; scrub any transcript that captured a reveal call.
This is distinct from (a) the S456 env-content hygiene sweep (duplicate/stale env
content, not tool behaviour) and (b) any generic “reveal=true dumps everything”
caveat — this is specifically the per-key filter failing to scope.
Provenance: S457 {127.18} rollout finding, curated under {127.35}
(parent-task-ac predicate, DR-021).
5. Provenance + cross-links
Section titled “5. Provenance + cross-links”Recovered from the S454 exec-127-18 transcript — the {127.19} runbook artefact authored for
the S436 D1 board ratification (generalising the {127.18} CRON_SECRET-split dual-accept
pattern) — and landed into the docs-site here at S458 (10/07/2026).
Operational cross-links: the private env-topology table (add rotated-on columns per §4); the
{127.18} PIPELINE_TRIGGER_SECRET env-matrix row; four-DB topology in
reference/platform-context.md.