Cloud Run Phase 1 handover — Liam-side gcloud operations

RETIRED (S298): Cloud Run is fully decommissioned. The ingestion pipeline now runs on-prem (IONOS VPS + Coolify) — see docs/runbooks/onprem-b1-deploy.md. This runbook and its referenced cloudrun/ manifests are removed from the active deploy path; retained for historical context only.

Spec: docs/audits/kh-production-readiness-phase-1/specs/wp-run-cloud-run-provisioning-spec.md v1.2. Plan: docs/audits/kh-production-readiness-phase-1/plans/wp-run-cloud-run-provisioning-plan.md v1. Track: production-readiness. Owner: Liam (gcloud-authenticated operator).

This runbook documents the Phase 1a + Phase 1b operational tasks that the worktree subagent CANNOT execute (no gcloud credentials in sandbox). Each section enumerates exact commands, expected outputs, and pass/fail criteria.

§1. Pre-flight verification (T1a.* — Liam runs)

Effort: ~20 min total. Deliverable: AC-1 verified.

§1.1 T1a.1 — Verify GCP organisation + projects

Pre-condition: gcloud auth login with the Liam personal Google Workspace account that owns aisolutionhub.co.uk.

1
gcloud organizations list \
2
  --filter='displayName:aisolutionhub.co.uk' \
3
  --format='value(name)'

Expected: organizations/195144871477. Failure: empty output → escalate to main session (org may have been renamed or rotated).

1
gcloud projects describe kh-prod-494815 \
2
  --format='value(projectId,parent.id,parent.type)'

Expected: kh-prod-494815\t195144871477\torganization. Failure: wrong parent → escalate (project may have moved org).

1
gcloud projects describe kh-staging-494815 \
2
  --format='value(projectId,parent.id,parent.type)'

Expected: kh-staging-494815\t195144871477\torganization.

§1.2 T1a.2 — Verify billing wired

1
gcloud beta billing projects describe kh-prod-494815 \
2
  --format='value(billingAccountName,billingEnabled)'

Expected: non-empty billingAccountName, True for billingEnabled.

1
gcloud beta billing projects describe kh-staging-494815 \
2
  --format='value(billingAccountName,billingEnabled)'

Expected: identical billing account name as kh-prod-494815 (same Liam main Google Workspace card per D-RUN-18). Failure (different accounts): non-blocking, but document the delta in deployment.md §5.

§1.3 T1a.3 — Verify APIs enabled + region defaults

1
gcloud services list --project=kh-prod-494815 --enabled \
2
  --filter='config.name:(run.googleapis.com OR cloudscheduler.googleapis.com OR secretmanager.googleapis.com OR artifactregistry.googleapis.com OR cloudbuild.googleapis.com)' \
3
  --format='value(config.name)'

Expected: 5 lines. Failure (<5 lines): auto-fill the missing APIs:

1
gcloud services enable <missing-id> --project=kh-prod-494815

Re-run the verifier. If services enable itself errors → escalate.

Repeat for kh-staging-494815.

Set default region per project (configuration profile pattern):

1
gcloud config configurations create kh-prod-494815 --activate
2
gcloud config set project kh-prod-494815
3
gcloud config set run/region europe-west2
4
gcloud config set compute/region europe-west2
5

6
gcloud config configurations create kh-staging-494815 --activate
7
gcloud config set project kh-staging-494815
8
gcloud config set run/region europe-west2
9
gcloud config set compute/region europe-west2

Switch profiles via gcloud config configurations activate <name>.

Verifier:

1
gcloud config get-value run/region

Expected: europe-west2 per active profile. D-RUN-20 deferred annotation (re-evaluate at first-client-data-handling task) is embedded in each cloudrun/jobs/*.yaml header comment so the deferral is discoverable from the manifests themselves.

Verifies AC-1.

§2. IAM provisioning (T1b.1 — Liam runs)

Effort: ~20 min. Deliverable: 4 runtime SAs + 2 invoker SAs across both projects (6 SAs total).

§2.1 Per-tenant runtime service accounts

Create one runtime SA per tenant per project. Total: 4 SAs.

1
gcloud iam service-accounts create phew-pipeline-sa \
2
  --project=kh-prod-494815 \
3
  --display-name='Phew pipeline runtime SA' \
4
  --description='Cloud Run Job runtime SA for Phew tenant pipeline'
5

6
gcloud iam service-accounts create phew-pipeline-sa \
7
  --project=kh-staging-494815 \
8
  --display-name='Phew pipeline runtime SA' \
9
  --description='Cloud Run Job runtime SA for Phew tenant pipeline'
10

11
gcloud iam service-accounts create kpf-pipeline-sa \
12
  --project=kh-prod-494815 \
13
  --display-name='KPF pipeline runtime SA' \
14
  --description='Cloud Run Job runtime SA for KPF tenant pipeline'
15

16
gcloud iam service-accounts create kpf-pipeline-sa \
17
  --project=kh-staging-494815 \
18
  --display-name='KPF pipeline runtime SA' \
19
  --description='Cloud Run Job runtime SA for KPF tenant pipeline'

§2.2 Bind least-privilege roles to runtime SAs

Per spec §2.11 IAM table:

roles/secretmanager.secretAccessor — scoped to tenant secrets ONLY. This is per-secret-resource, bound during T2.3 (Phase 2). Skip in Phase 1b.
roles/logging.logWriter — project-wide is fine (logs partitioned by job execution).
roles/run.invoker — scoped self-only. Bound per-Job in T1b.3. Skip in Phase 1b.
roles/artifactregistry.reader — project-wide; needed at runtime to pull image (per spec F-16 fix).

Phase 1b binds the two project-wide roles for each runtime SA:

1
for PROJECT in kh-prod-494815 kh-staging-494815; do
2
  for TENANT in phew kpf; do
3
    gcloud projects add-iam-policy-binding "$PROJECT" \
4
      --member="serviceAccount:${TENANT}-pipeline-sa@${PROJECT}.iam.gserviceaccount.com" \
5
      --role=roles/logging.logWriter
6

7
    gcloud projects add-iam-policy-binding "$PROJECT" \
8
      --member="serviceAccount:${TENANT}-pipeline-sa@${PROJECT}.iam.gserviceaccount.com" \
9
      --role=roles/artifactregistry.reader
10
  done
11
done

§2.3 Cloud Scheduler invoker SA (one per project)

Per spec F-15 fix: one invoker SA per project, bound roles/run.invoker per-Job in T1b.3 (not per-SA scope, so we skip global binding here).

1
gcloud iam service-accounts create scheduler-invoker-sa \
2
  --project=kh-prod-494815 \
3
  --display-name='Cloud Scheduler invoker SA'
4

5
gcloud iam service-accounts create scheduler-invoker-sa \
6
  --project=kh-staging-494815 \
7
  --display-name='Cloud Scheduler invoker SA'

§2.4 Project resources — Artifact Registry + Cloud Build staging bucket

Background. Two project-resource provisioning steps that the PRE-ACTIONED audit table does NOT cover and that the first deploy attempt will fail without:

Artifact Registry Docker repo named pipeline per project. Cloud Build’s pack --publish pushes images to europe-west2-docker.pkg.dev/<project>/pipeline/...; if the repo doesn’t exist, the buildpack analyzer fails with NAME_UNKNOWN: Repository "pipeline" not found.
Cloud Build source-staging bucket gs://<project>_cloudbuild per project. gcloud builds submit uploads the source tarball here; if the bucket doesn’t exist (and SA can’t auto-create it under the org’s policies), the call fails with forbidden from accessing the bucket.

§2.4.1 Create Artifact Registry repos

1
gcloud artifacts repositories create pipeline \
2
  --project=kh-prod-494815 \
3
  --repository-format=docker \
4
  --location=europe-west2 \
5
  --description='Cloud Run Job pipeline images (kh-pipeline — single image post-S41 W3 collapse)'
6

7
gcloud artifacts repositories create pipeline \
8
  --project=kh-staging-494815 \
9
  --repository-format=docker \
10
  --location=europe-west2 \
11
  --description='Cloud Run Job pipeline images (kh-pipeline — single image post-S41 W3 collapse)'

§2.4.2 Create Cloud Build staging buckets

1
gcloud storage buckets create gs://kh-prod-494815_cloudbuild \
2
  --project=kh-prod-494815 \
3
  --location=europe-west2 \
4
  --uniform-bucket-level-access
5

6
gcloud storage buckets create gs://kh-staging-494815_cloudbuild \
7
  --project=kh-staging-494815 \
8
  --location=europe-west2 \
9
  --uniform-bucket-level-access

UBLA on so project-level IAM (not legacy ACLs) governs access.

§2.5 Verifier

1
gcloud iam service-accounts list \
2
  --project=kh-prod-494815 \
3
  --filter='email~pipeline-sa OR email~scheduler-invoker' \
4
  --format='value(email)'
5

6
gcloud artifacts repositories list \
7
  --project=kh-prod-494815 --location=europe-west2 \
8
  --format='value(name,format)'
9

10
gcloud storage buckets describe gs://kh-prod-494815_cloudbuild \
11
  --format='value(name,location)'

Expected: 3 SA entries (phew-pipeline-sa, kpf-pipeline-sa, scheduler-invoker-sa); 1 Artifact Registry row (pipeline DOCKER); 1 storage bucket row (kh-prod-494815_cloudbuild EUROPE-WEST2). Failure: re-run §2.1 / §2.3 / §2.4. Repeat for kh-staging-494815.

§3. WIF binding (T1b.2 — Liam runs)

Effort: ~30 min (per-project repeat × 2; provider config is fiddly, spec R-6 likelihood Medium).

Critical: the WIF provider’s --attribute-condition and the iam-policy-binding --member MUST reference the new repo URL ai-solution-hub/knowledge-hub (transferred 29/04/2026 from liam-jons/knowledge-hub). The verbatim quotes below come from plan §2 T1b.2 step 2 lines 327-335.

§3.1 Create WIF pool (per project)

1
gcloud iam workload-identity-pools create gh-actions-pool \
2
  --project=kh-prod-494815 --location=global \
3
  --display-name='GitHub Actions OIDC pool'
4

5
gcloud iam workload-identity-pools create gh-actions-pool \
6
  --project=kh-staging-494815 --location=global \
7
  --display-name='GitHub Actions OIDC pool'

§3.2 Create WIF provider bound to GH Actions OIDC issuer

Verbatim from plan §2 step 2 lines 327-335:

1
gcloud iam workload-identity-pools providers create-oidc gh-actions-provider \
2
  --project=kh-prod-494815 --location=global \
3
  --workload-identity-pool=gh-actions-pool \
4
  --issuer-uri='https://token.actions.githubusercontent.com' \
5
  --attribute-mapping='google.subject=assertion.sub,attribute.repository=assertion.repository,attribute.ref=assertion.ref' \
6
  --attribute-condition='assertion.repository == "ai-solution-hub/knowledge-hub"'

Repeat for kh-staging-494815. The --attribute-condition is repository-pinned to ai-solution-hub/knowledge-hub — the new org URL post-29/04/2026 transfer.

§3.3 Create CI deploy SA (per project)

1
gcloud iam service-accounts create kh-deploy-sa \
2
  --project=kh-prod-494815 \
3
  --display-name='CI deploy identity (WIF)'
4

5
gcloud iam service-accounts create kh-deploy-sa \
6
  --project=kh-staging-494815 \
7
  --display-name='CI deploy identity (WIF)'

Bind the six deploy roles per project:

1
for PROJECT in kh-prod-494815 kh-staging-494815; do
2
  for ROLE in \
3
    roles/run.developer \
4
    roles/cloudbuild.builds.editor \
5
    roles/cloudbuild.builds.builder \
6
    roles/artifactregistry.writer \
7
    roles/iam.serviceAccountUser \
8
    roles/serviceusage.serviceUsageConsumer; do
9
    gcloud projects add-iam-policy-binding "$PROJECT" \
10
      --member="serviceAccount:kh-deploy-sa@${PROJECT}.iam.gserviceaccount.com" \
11
      --role="$ROLE"
12
  done
13
done

Role rationale:

roles/run.developer — deploy Cloud Run Job manifests (gcloud run jobs replace).
roles/cloudbuild.builds.editor — invoke gcloud builds submit.
roles/cloudbuild.builds.builder — required as the build runtime SA. Post-April-2024 GCP projects have NO legacy default Cloud Build SA (<projectnum>@cloudbuild.gserviceaccount.com is no longer auto-created), so the workflow passes --service-account=<deploy-sa> and the deploy SA acts as the build SA. This role grants the logging.logWriter + storage.objectUser + artifactregistry write union the build needs at runtime.
roles/artifactregistry.writer — push image layers (also covered by cloudbuild.builds.builder; kept explicit for least-privilege clarity).
roles/iam.serviceAccountUser — impersonate per-tenant runtime SAs in Job manifests.
roles/serviceusage.serviceUsageConsumer — gcloud builds submit needs serviceusage.services.use to invoke Cloud Build APIs; cloudbuild.builds.editor does NOT include this permission.

§3.4 Bind WIF provider → CI deploy SA

Verbatim from plan §2 step 4 lines 343-349. Substitute <project-number> per project (look up via gcloud projects describe <project> --format='value(projectNumber)').

1
gcloud iam service-accounts add-iam-policy-binding kh-deploy-sa@kh-prod-494815.iam.gserviceaccount.com \
2
  --project=kh-prod-494815 \
3
  --role=roles/iam.workloadIdentityUser \
4
  --member='principalSet://iam.googleapis.com/projects/<kh-prod-494815-project-number>/locations/global/workloadIdentityPools/gh-actions-pool/attribute.repository/ai-solution-hub/knowledge-hub'
5

6
gcloud iam service-accounts add-iam-policy-binding kh-deploy-sa@kh-staging-494815.iam.gserviceaccount.com \
7
  --project=kh-staging-494815 \
8
  --role=roles/iam.workloadIdentityUser \
9
  --member='principalSet://iam.googleapis.com/projects/<kh-staging-494815-project-number>/locations/global/workloadIdentityPools/gh-actions-pool/attribute.repository/ai-solution-hub/knowledge-hub'

The --member principalSet:// URI is the only place the new repo URL ai-solution-hub/knowledge-hub appears in the IAM binding — it must match the --attribute-condition in §3.2 exactly (case-sensitive).

§3.5 Capture provider name + deploy SA email

For each project, capture the full provider resource name. This is the value that will land in GH Actions secrets in §4:

1
gcloud iam workload-identity-pools providers describe gh-actions-provider \
2
  --project=kh-prod-494815 --location=global \
3
  --workload-identity-pool=gh-actions-pool \
4
  --format='value(name)'

Output format: projects/<kh-prod-494815-project-number>/locations/global/workloadIdentityPools/gh-actions-pool/providers/gh-actions-provider

Repeat for kh-staging-494815.

The deploy SA emails are deterministic:

kh-deploy-sa@kh-prod-494815.iam.gserviceaccount.com
kh-deploy-sa@kh-staging-494815.iam.gserviceaccount.com

§3.6 Verifier

1
gcloud iam workload-identity-pools providers describe gh-actions-provider \
2
  --project=kh-prod-494815 --location=global \
3
  --workload-identity-pool=gh-actions-pool \
4
  --format='value(name,oidc.issuerUri)'

Expected: non-empty name; oidc.issuerUri matches https://token.actions.githubusercontent.com.

Repeat for kh-staging-494815.

Risk pointer (R-6): if WIF auth fails on first deploy (§5), re-check the §3.2 --attribute-condition and §3.4 --member strings for ai-solution-hub/knowledge-hub exactness (no trailing slash, no case mismatch).

§4. GitHub Actions secrets (Liam runs)

Effort: ~5 min. Deliverable: GCP_WIF_PROVIDER + GCP_DEPLOY_SA_EMAIL set per environment as Environment variables (vars), not secrets. These are non-sensitive resource paths (WIF provider full name + SA email are public IDs); storing as vars makes them visible in workflow logs which aids debug of WIF binding mismatches. Workflow accesses them via ${{ vars.* }} — secrets.* and vars.* are NOT interchangeable, the workflow will silently read empty if the storage type does not match the access form (per feedback_gha_secrets_vs_vars_silent_empty.md).

Use canonical capital-first env names per docs/runbooks/github-environments.md §2.

§4.1 Set Staging environment variables

1
gh variable set GCP_WIF_PROVIDER \
2
  --env Staging \
3
  --body 'projects/<kh-staging-494815-project-number>/locations/global/workloadIdentityPools/gh-actions-pool/providers/gh-actions-provider'
4

5
gh variable set GCP_DEPLOY_SA_EMAIL \
6
  --env Staging \
7
  --body 'kh-deploy-sa@kh-staging-494815.iam.gserviceaccount.com'

§4.2 Set Production environment variables

1
gh variable set GCP_WIF_PROVIDER \
2
  --env Production \
3
  --body 'projects/<kh-prod-494815-project-number>/locations/global/workloadIdentityPools/gh-actions-pool/providers/gh-actions-provider'
4

5
gh variable set GCP_DEPLOY_SA_EMAIL \
6
  --env Production \
7
  --body 'kh-deploy-sa@kh-prod-494815.iam.gserviceaccount.com'

§4.3 Verifier

1
gh variable list --env Staging
2
gh variable list --env Production

Expected: each environment shows both GCP_WIF_PROVIDER and GCP_DEPLOY_SA_EMAIL.

§5. First deploy (T1b.5 — Liam runs)

Effort: ~30 min (incl. waiting for Cloud Build).

§5.1 Trigger workflow_dispatch

GitHub UI flow:

Repo → Actions → “Cloud Run deploy” workflow.
Click “Run workflow”.
Select branch: production-readiness.
Inputs:
- environment: staging
- fallback-build-mode: buildpack (default; do not change unless R-10 trips during the build step).
Click “Run workflow”.

§5.2 Watch the run

The workflow has 5 stages — monitor each:

Resolve target environment — should map production-readiness ref + staging input → gh-env=Staging, gcp-project=kh-staging-494815.
Authenticate to Google Cloud (WIF) — id-token: write permission required on the job; the action exchanges a GH OIDC token for a GCP access token. R-6 mitigation: if this fails, re-check §3.2 --attribute-condition and §3.4 --member.
Build image via Cloud Build — single kh-pipeline builds via the buildpack (post-S41 W3 collapse). Expect ~6-7 min warm / ~14-15 min cold (first-build, no :latest for cache restore). Workflow passes --service-account=projects/<project>/serviceAccounts/<deploy-sa> (post-April-2024 projects require explicit build SA) and --substitutions=_COMMIT_SHA=$GITHUB_SHA (built-in $COMMIT_SHA only populates for trigger-invoked builds; user-defined _COMMIT_SHA is required for direct gcloud builds submit). cloudbuild.yaml deliberately omits a top-level images: block — pack --publish already pushes images and discards the local Docker daemon copy, so the post-step push that images: triggers would fail with failed to find one or more images.
Deploy tenant manifests — replaces kh-pipeline-phew and kh-pipeline-kpf Jobs with the staging manifests.
Smoke verify (jobs describe) — gcloud run jobs describe returns the job names.

§5.3 Smoke verifications (manual post-workflow)

Per plan T1b.5 step 4 — these verify ACs.

Image list (verifies AC-2):

1
gcloud artifacts docker images list \
2
  europe-west2-docker.pkg.dev/kh-staging-494815/pipeline \
3
  --format='value(IMAGE,DIGEST)'

Expected: row for kh-pipeline:latest + per-build-SHA tag + :cache tag (post-first-cache-bearing build). (Post-S41 W3 collapse — single image.)

Job describe (verifies AC-3 partial — env contract):

1
gcloud run jobs describe kh-pipeline-phew \
2
  --project=kh-staging-494815 \
3
  --region=europe-west2 \
4
  --format='value(spec.template.spec.template.spec.containers[0].env)'

Expected: output includes NEXT_PUBLIC_CLIENT_ID=phew.

Test execution (verifies AC-3 complete):

1
gcloud run jobs execute kh-pipeline-phew \
2
  --project=kh-staging-494815 \
3
  --region=europe-west2 \
4
  --wait

Expected: exit code 0 on a no-op test invocation. The Job exits because no real ingest work is wired yet — Phase 2 lands the actual pipeline. The clean exit verifies the image + manifest + IAM chain.

Verifies AC-2 + AC-3.

§6. Failure-mode triage

§6.1 R-6 — WIF auth fails

Symptom: Failed to fetch ID token or Unable to acquire impersonated credentials in the Authenticate to Google Cloud (WIF) step.

Triage:

Compare §3.2 --attribute-condition against ai-solution-hub/knowledge-hub exactly (case-sensitive, no trailing slash, no whitespace).
Compare §3.4 --member principalSet:// URI against the same string. The attribute.repository/<owner>/<repo> segment must match the --attribute-condition value byte-for-byte.
Verify the project number in the --member URI: gcloud projects describe <project> --format='value(projectNumber)'.
If still failing → fall back to manual deploy from Liam’s gcloud session (per spec R-6 mitigation): gcloud builds submit
- gcloud run jobs replace directly. Document the manual deploy in deployment.md §5.

§6.2 R-10 — Cloud Build fails on bert-score wheel

Symptom: the Build image via Cloud Build step fails on pip install bert-score for kh-pipeline (bert-score now lives in the single image post-S41 W3 collapse).

Triage: trigger a workflow_dispatch run with fallback-build-mode=dockerfile. Note: today this errors with “Dockerfile fallback mode requested but not yet wired” — Phase 1c authors cloudrun/Dockerfile if R-10 trips. Escalate to a follow-up phase 1c task in a fresh session.

§6.3 R-12 — Onboarding tax overflow

Symptom: Phase 1b session runs past the 2h ceiling.

Triage: STOP. Commit progress. Escalate to a Phase 1b-CONTINUED task in a fresh session. Do NOT push past the 2h ceiling per feedback_implementation_workflow.md.

§6.4 Wrong tenant pinned to `NEXT_PUBLIC_CLIENT_ID`

Symptom: the Deploy tenant manifests step exits non-zero with ERROR: <manifest> has NEXT_PUBLIC_CLIENT_ID=X but expected Y.

Triage: the workflow’s per-tenant assertion caught a mis-edit in the manifest YAML. Inspect cloudrun/jobs/<env>-<tenant>.yaml and correct the NEXT_PUBLIC_CLIENT_ID value. R-9 mitigation worked correctly — no GCP-side intervention needed.

§7. Handoff list to Phase 2

What Phase 2 (T2.* per spec §4.1 + plan §4) picks up:

Task	Description	Status
T2.0	Pin `httpx>=0.27,<0.28` + add `socksio` (R-1 mitigation)	shipped pre-S34
T2.1	Provision Secret Manager secrets (full §2.5 list — 11 always-required + 2 optional) on both projects	✅ 11/11 always-required present on both projects (verified S34)
T2.2	Mount the 11 always-required secrets onto each tenant Job	✅ S34 — declarative `--set-secrets` step in `.github/workflows/cloud-run-deploy.yml` (see §9.1 below)
T2.3	Bind `roles/secretmanager.secretAccessor` to per-tenant runtime SAs	✅ S34 — project-scoped (see §9.2 below for rationale)
T2.5	Widen `scripts/kb_pipeline/pipeline_log.py` API per spec §2.9.1	open — Phase 2
T2.6	Forward-wire `start_run` / `complete_run` / `fail_run` callsites in `scripts/ingest.py`, `scripts/ingest_markdown.py`, `scripts/kb_pipeline/pipeline.py`, `scripts/eval-classification.py` (NEW)	open — Phase 2
T2.7	Add `KH_REQUEST_ID` env var read in `scripts/kb_pipeline/config.py`. The Job manifests today set `KH_REQUEST_ID=""` (empty string) as a placeholder so the env var key exists; reader must branch on `os.environ.get('KH_REQUEST_ID') or None` to treat empty-string as unset (Python truthiness — empty string is set-but-empty, not unset).	open — Phase 2

Phase 2 also lands scripts/eval-classification.py (the actual eval entry point — declared in cloudrun/cloudbuild.yaml GOOGLE_ENTRYPOINT today as a forward reference).

Phase 1.5 fold-in: the audit verdicts in docs/audits/kh-production-readiness-phase-1/wp-run-phase-1-5-vercel-audit.md get summarised into docs/runbooks/deployment.md §5 at T4.4. Liam ratifies the 2 PENDING-RATIFY rows (L1 + L2 in §4 of that audit doc) before this WP closes.

§8. Phase 2 close-out (S34 — T2.2 + T2.3 declarative-mount path)

§8.1 T2.2 — declarative `--set-secrets` step in workflow

Spec §2.5 enumerates 11 always-required secrets + 2 optional (SUPABASE_AUTH_EMAIL/PASSWORD — used only by batch-reclassify.ts). S34 wires the 11 always-required mounts via a new step in .github/workflows/cloud-run-deploy.yml placed AFTER the existing Deploy tenant manifests step:

1
- name: Mount Secret Manager secrets (Phase 2 / T2.2)
2
  run: |
3
    SECRETS=$(cat <<'SECRETS_EOF'
4
    ANTHROPIC_API_KEY=ANTHROPIC_API_KEY:latest
5
    OPENAI_API_KEY=OPENAI_API_KEY:latest
6
    SUPABASE_URL=SUPABASE_URL:latest
7
    ...
8
    SECRETS_EOF
9
    )
10
    SECRETS_FLAT=$(echo "$SECRETS" | tr '\n' ',' | sed 's/,$//;s/^,//')
11
    for TENANT in phew kpf; do
12
      JOB_NAME="kh-pipeline-${TENANT}"
13
      gcloud run jobs update "$JOB_NAME" \
14
        --project="$GCP_PROJECT" \
15
        --region=europe-west2 \
16
        --set-secrets="$SECRETS_FLAT"
17
    done

The flag --set-secrets is declarative: each invocation REPLACES the full secret-mount set on the Job. Adding a new secret requires editing the heredoc body once; removing a secret requires deleting its line. Secrets ride at runtime — image build does NOT need them.

Why workflow not manifest: keeping the secret-mount references out of the YAML manifests means new secrets do not require a manifest edit per tenant (4 file edits per addition). The heredoc lives in one place.

NEXT_PUBLIC_CLIENT_ID is not mounted here — it is baked at deploy time per D-RUN-12.1 + F-29 (per-tenant value asserted by the preceding Deploy tenant manifests step’s awk extraction).

§8.2 T2.3 — IAM grant (project-scoped instead of per-secret)

Spec §2.2 originally proposed per-secret-resource bindings (“scoped to tenant secrets ONLY”). S34 grants roles/secretmanager.secretAccessor at project scope to each tenant runtime SA on each project — 4 bindings total instead of 11×2×2 = 44.

Rationale:

Knowledge Hub does not have per-tenant secrets today (e.g. there is no phew-only-ANTHROPIC_API_KEY vs kpf-only-ANTHROPIC_API_KEY). Both tenants need the same 11 always-required secrets, so per-secret bindings would duplicate the same access pattern 11× per tenant.
Granting at project scope means future secrets added under the spec §2.5 list (e.g. the optional SUPABASE_AUTH_EMAIL/PASSWORD pair) are accessible without re-binding.
The blast radius is bounded — tenant runtime SAs cannot create or modify secrets, only read them. Service-account compromise still gates on the SA’s other roles (e.g. run.developer, secretmanager.secretAccessor alone cannot escalate).

If per-tenant secrets ever land (e.g. a per-client API key), narrow the binding at that point.

The 4 bindings shipped:

1
for SA in phew-pipeline-sa kpf-pipeline-sa; do
2
  for PROJ in kh-staging-494815 kh-prod-494815; do
3
    gcloud projects add-iam-policy-binding "$PROJ" \
4
      --member="serviceAccount:${SA}@${PROJ}.iam.gserviceaccount.com" \
5
      --role=roles/secretmanager.secretAccessor \
6
      --condition=None
7
  done
8
done

§8.3 Firstinvoke verification (Hold (a) + Hold (a-prod) close-out)

After the workflow’s first run on each project mounts the secrets:

1
# Staging
2
gcloud run jobs execute kh-pipeline-phew \
3
  --region=europe-west2 \
4
  --project=kh-staging-494815 \
5
  --wait
6
gcloud logging read 'jsonPayload.pipeline_name="ingest"' \
7
  --project=kh-staging-494815 --limit=5
8

9
# Production (after main-branch sync)
10
gcloud run jobs execute kh-pipeline-phew \
11
  --region=europe-west2 \
12
  --project=kh-prod-494815 \
13
  --wait
14
gcloud logging read 'jsonPayload.pipeline_name="ingest"' \
15
  --project=kh-prod-494815 --limit=5

Expected: exit code 0 + ≥1 Cloud Logging entry with jsonPayload.pipeline_name == "ingest" and a correlation ID.

If lib/env-server.ts Zod validation fails fast at boot, that is the “build the thing, forget to turn it on” failure mode — the --set-secrets step did not actually mount everything env-server.ts requires. Re-run gcloud run jobs describe kh-pipeline-phew --format='value(spec.template.spec.template.spec.containers[0].env)' and diff against the spec §2.5 list.

§9. Provenance

Spec version: v1.2 (post-S17 admin amendments — F-P1 NEXT_PUBLIC_CLIENT_ID env-var, F-P2 --region flags, RES.5 fold-down).
Plan version: v1.
GCP infrastructure: PRE-ACTIONED by Liam pre-S16 (org + projects + 5 APIs + initial Secret Manager entries on kh-staging-494815).
Repo URL: ai-solution-hub/knowledge-hub (transferred from liam-jons/knowledge-hub 29/04/2026, GitHub Team plan).
Region: europe-west2 (NL Tier 1) — D-RUN-20 (UK residency) deferred until first-client-data-handling task.

Cloud Run Phase 1 handover — Liam-side gcloud operations

Cloud Run Phase 1 handover — Liam-side gcloud operations

§1. Pre-flight verification (T1a.* — Liam runs)

§1.1 T1a.1 — Verify GCP organisation + projects

§1.2 T1a.2 — Verify billing wired

§1.3 T1a.3 — Verify APIs enabled + region defaults

§2. IAM provisioning (T1b.1 — Liam runs)

§2.1 Per-tenant runtime service accounts

§2.2 Bind least-privilege roles to runtime SAs

§2.3 Cloud Scheduler invoker SA (one per project)

§2.4 Project resources — Artifact Registry + Cloud Build staging bucket

§2.4.1 Create Artifact Registry repos

§2.4.2 Create Cloud Build staging buckets

§2.5 Verifier

§3. WIF binding (T1b.2 — Liam runs)

§3.1 Create WIF pool (per project)

§3.2 Create WIF provider bound to GH Actions OIDC issuer

§3.3 Create CI deploy SA (per project)

§3.4 Bind WIF provider → CI deploy SA

§3.5 Capture provider name + deploy SA email

§3.6 Verifier

§4. GitHub Actions secrets (Liam runs)

§4.1 Set Staging environment variables

§4.2 Set Production environment variables

§4.3 Verifier

§5. First deploy (T1b.5 — Liam runs)

§5.1 Trigger workflow_dispatch

§5.2 Watch the run

§5.3 Smoke verifications (manual post-workflow)

§6. Failure-mode triage

§6.1 R-6 — WIF auth fails

§6.2 R-10 — Cloud Build fails on bert-score wheel

§6.3 R-12 — Onboarding tax overflow

§6.4 Wrong tenant pinned to NEXT_PUBLIC_CLIENT_ID

§7. Handoff list to Phase 2

§8. Phase 2 close-out (S34 — T2.2 + T2.3 declarative-mount path)

§8.1 T2.2 — declarative --set-secrets step in workflow

§8.2 T2.3 — IAM grant (project-scoped instead of per-secret)

§8.3 Firstinvoke verification (Hold (a) + Hold (a-prod) close-out)

§9. Provenance

§6.4 Wrong tenant pinned to `NEXT_PUBLIC_CLIENT_ID`

§8.1 T2.2 — declarative `--set-secrets` step in workflow