CI Runbook

CI Runbook

Status: Initial draft (kh-prod-readiness-S7, WP-G4.1). Owner: prod-readiness track. Audience: Liam + future client-onboarding operator. Pair with: docs/runbooks/github-environments.md (env scopes, secret taxonomy), docs/audits/kh-production-readiness-phase-1/research/01-existing-infra-audit.md §1 (pre-WP-G4 baseline).

---

§1. Purpose

Document the PR-blocking CI workflow (.github/workflows/ci.yml) — what it runs, what it gates, how to keep it green, and how to extend it as later WP-G4.x work-packages add tranches (E2E, MCP eval, security scans, etc.).

This runbook is the operational counterpart to the WP-G4.1 spec.

---

§2. Workflow at a glance

Trigger	Branch behaviour
pull_request	Any base branch (default; no branches: filter)
push to main	Sanity-check after merge — same job runs end-to-end
push to staging	Sanity-check after staging merge/deploy sync — same job runs end-to-end

Draft PRs do not trigger CI to conserve runner minutes; promote to ready-for-review to gate. The pull_request.types filter (opened, reopened, synchronize, ready_for_review) excludes draft creation and re-runs CI when a draft is promoted.

Concurrency: keyed on ${{ github.event_name }} + PR number/ref so push and pull_request runs cannot collide on the same key. New commits supersede stale runs (cancel-in-progress).

The workflow defines 12 jobs. Six are top-level (no needs:): quality-precheck, e2e-smoke, mcp-build, detect-changes, supabase-types-parity, build-gate. The rest chain off those: quality-test + type-drift-parity need quality-precheck; mcp-eval-seed needs [mcp-build, detect-changes]; mcp-eval needs [mcp-build, mcp-eval-seed, detect-changes]; integration needs detect-changes; ci-summary aggregates all of them. Total wall-clock = max(quality job chain, e2e-smoke, mcp-eval matrix, integration).

Push-event real-API gating (ID-91, S323): a lightweight detect-changes job (ID-86 pattern — full before..sha git-diff compare, no 300-file compare-API ceiling) gates integration, mcp-eval-seed and the mcp-eval matrix on push events only: if: github.event_name != 'push' || needs.detect-changes.outputs.relevant == 'true'. Docs/ledger-only pushes skip the real-API lanes with an explicit logged verdict (compared range + changed-file count + matched paths). PR runs are byte-identical to the pre-gate behaviour (non-push short-circuits to relevant=true without checkout; the mcp-eval in-job paths-filter skip flags keep sole control of PR cost). Edge cases fail OPEN to relevant=true (all-zeros/unreachable before, non-push events). detect-changes is wired into ci-summary’s needs + aggregation — a broken gate turns CI red rather than silently skipping forever.

Job	Env scope	Shape	Purpose
quality-precheck	Production	Single job; NODE_OPTIONS=--max-old-space-size=4096 on the build step (S19 WP1.5 — OOM headroom for the larger migration count)	Lint, format, knip, legacy form-template-resolver guard (ID-145 {145.7} — scripts/check-no-legacy-form-template-resolver.ts), build, build-output regression scan, conditional Python pytest (paths-filter; installs pandoc via apt-get first — ID-85.1, B1-image parity so the docx-conversion tests exercise real pandoc, not the tag-stripping fallback)
quality-test	Production	Matrix shard: [1, 2, 3, 4] with fail-fast: false; depends on quality-precheck	bunx vitest run --shard N/4 (PR runs add --changed origin/<base> for diff-scoped tests).
e2e-smoke	Staging	Single job (no continue-on-error — dropped S31 W1(b)); 25 min timeout (S500 cutover headroom); no concurrency group — the lane left integration-staging-e2e at the DR-096 cutover, so smoke jobs from different PRs run fully in parallel	Curated Playwright smoke against a runner-local Supabase stack ({365.5} / DR-096, S500): pinned supabase/setup-cli (CLI 2.109.1) + config patch (scripts/ci/prepare-supabase-ci-config.ts — disables studio/analytics/edge_runtime/realtime; -x alone still pulls images, cli#4088) + supabase start (~64 s measured) + db reset (~28 s; migrations + seed.sql + all declared buckets) + seed:e2e-users. Supabase connection values are the CLI’s public local demo constants (scripts/ci/local-stack.env), asserted against the live stack per run. Build-once + next start (id-128 {128.11}); the staging stale-test-artifact cleanup preflight is gone (fresh stack has no ambient data)
mcp-build	Staging	Single job; uploads mcp-next-build build artefact (1-day retention)	One Staging-env build shared by L1/L3/L4 instead of three rebuilds
detect-changes	none (no secrets)	Single job; ~30-60s (checkout + git diff); push events do the full compare, non-push short-circuits to relevant=true	ID-91 (bl-246) — full before..sha diff against the real-API path set (__tests__/integration/**, helpers, lib/**, app/api/**, types/**, supabase/**, scripts/mcp-eval/**, manifests, ci.yml itself); emits outputs.relevant consumed by the three real-API lanes below
mcp-eval-seed	Staging	Single job; depends on [mcp-build, detect-changes]; push-gated on relevant	Seeds deterministic Q&A corpus (metadata.mcp_eval_seed=true) once before L1/L3/L4 fan-out
mcp-eval	Staging	Matrix layer: [l1, l3, l4] with fail-fast: false; depends on [mcp-build, mcp-eval-seed, detect-changes]; push-gated on relevant; downloads + restores shared build artefact	MCP protocol compliance (L1) + response quality (L3) + functional correctness (L4)
integration	Staging	Single job; hard-gate (no continue-on-error — dropped S29 WP-CI.RES.7 Phase 2 cutover); depends on detect-changes; push-gated on relevant	bun run test:integration against staging branch with real Anthropic + OpenAI calls; 25 min timeout
type-drift-parity	none (no secrets)	Single job; depends on quality-precheck; 10 min timeout	bun run ast-dataflow type-drift-detect --ci — exits non-zero if any new fetcher-only interface is absent from the committed baseline (.type-drift-baseline.json); prevents regression without closing existing 37-interface debt ledger. The ast-dataflow script resolves to the installed @ai-solution-hub/ast-dataflow bin (DR-128, id-377 {377.5}: the tool was extracted to its own public repo and is consumed as a bun git dependency pinned to a SHA; the command is unchanged).
supabase-types-parity	Staging	Single job; 8 min timeout; Supabase CLI pinned (deliberately — an unpinned releases/latest could silently change gen types output; bump on purpose, see the version comment in ci.yml)	ID-47.6 — DB-axis generated-types parity gate. Regenerates supabase gen types typescript --project-id "$PLATFORM_PROJECT_REF" --schema public,api from the live Platform schema and FAILS on any diff against the committed supabase/types/database.types.ts. Diff-and-fail (not regenerate-and-commit) so schema drift must land in a reviewed PR. The MergeDeep override file is separate — the diff is override-blind by construction.
ci-summary	none	Aggregator; if: always(); needs all other jobs; 1 min timeout	OPS-58 — single fixed-name required check. failure is the only red signal; success and skipped both pass, so jobs can be added/removed/sharded without touching the ruleset. Branch-protection ruleset 15785019 requires only CI / ci-summary. Final step posts the status to Vercel (§2.2).
build-gate	none (no secrets)	Single job; no needs: (top-level, always-evaluated); 5 min timeout	bl-200 (S283 retro) — standalone whole-repo bun run typecheck (tsc --noEmit -p tsconfig.ci.json, which includes scripts/ — the base tsconfig.json excludes it and next build never typechecks standalone CLI scripts). Catches a scripts/-only tsc break (e.g. the S283 scripts/ledger-cli.ts regression that sat undetected ~4 sessions). Kept a SEPARATE job with no needs: so a red build-gate is an unambiguous “build is broken” signal that cannot blend into quality-precheck’s other failures. tsconfig.ci.json excludes 18 pre-existing-debt scripts (Bun-runtime globals + untyped Supabase clients) so the gate is green on main; new / newly-clean scripts are gated by default. Wired into ci-summary.

Retired jobs/steps: ledger-mirror-parity was retired under ID-68.35 when the ledgers moved to the private docs-site repo; the task-view clone/provision step that survived inside quality-test went in the ID-165 ordna cutover (the ledger WRITE suites it served were deleted first, in 37042db1). No ci.yml job touches the task ledger any more — tasks are ordna markdown files in the private docs-site (tasks/, see tasks/AGENTS.md there).

Per-step internals inside quality-precheck (the most complex job):

#	Step	Failure mode
1	actions/checkout@v7 (fetch-depth: 0)	infra
2	dorny/paths-filter@v4 — detect Python paths	infra
3	oven-sh/setup-bun@v2 (bun pinned in ci.yml)	infra
4	actions/setup-node@v6 (uses .node-version → 22 LTS)	infra
5	bun install --frozen-lockfile (plain bun install for Dependabot)	dependency drift
6	actions/cache@v4 over .next/cache	infra
7	bun lint (ESLint) — if: !cancelled()	code quality
8	bun run format:check (Prettier) — if: !cancelled()	formatting
9	bun run ci:knip-check — if: !cancelled()	unused-export regression
10	Legacy form-template resolver guard (ID-145 {145.7}) — if: !cancelled()	banned-pattern regression
11	bun run build (Next.js) — NODE_OPTIONS=--max-old-space-size=4096	type errors, env-validation, build
12	bun run test:build (build-output NEXT_PUBLIC_* substitution scan)	client-bundle env regression
13–16	Python setup + pip install + pandoc install (apt-get) + pytest, conditional on scripts/**/*.py / requirements.txt change	Python test failure

Python 3.13 pinned to match requirements.txt minimum; no .python-version exists at repo root, so the workflow is the source of truth.

pandoc is installed via apt-get before the pytest step (ID-85.1). ubuntu-latest runners do not ship pandoc, so without it docx_cell_to_markdown.py falls back to tag stripping and 8 conversion tests fail (6 in test_docx_cell_to_markdown.py, 2 in test_extract_docx_tables.py), gating all 4 quality-test shards. Installing it preserves B1-image parity (the production pipeline image ships pandoc; local dev machines have it) and keeps the tests exercising real conversion — the @requires_pandoc skipif pattern (test_docx_tracked_changes_regression.py) is reserved for genuinely pandoc-optional behaviour tests, not these.

scripts/tests/test_cocoindex_server.py::TestIdleModeBoot::test_idle_mode_boot_returns_clean_worker_stays_healthy self-skips under a sandboxed agent worktree (the real cocoindex Rust engine EPERMs on core.Environment boot) via an engine-availability skipif probe, and runs normally in CI where the engine boots (bl-218).

Order is lint → format → knip → build → test:build so cheap failures surface within ~60 seconds. Build comes before test because a Vitest run that imports a broken module produces noisier output than a clean build error. Vitest itself runs in the downstream quality-test matrix (4-shard split — WP-CI.RES.1 S15).

Migration-replay smoke removed S30 (kh-prod-readiness-S30): the migration-replay.yml workflow + scripts/migration-replay-check.ts were retired. Incremental-migration validation is now canonical via Supabase Auto Branching (preview branches created per-PR replay any new supabase/migrations/** against a clone of prod schema). The staging-protection ruleset’s required-checks list dropped the migration-replay context in the same window — see §3.4.3.

Total runtime target: under 10 minutes wall-clock on a typical PR (quality jobs run in parallel with e2e-smoke + mcp-eval matrix + integration). If it climbs above that, see §7.

2.1 Pre-merge action — bun run format once across the repo

A snapshot of production-readiness at WP-G4.1 capture time (commit c602009b) reports 823 files with Prettier formatting drift when run through bun run format:check. The first PR that lands the new ci.yml will therefore go red on the format step until the repo is brought back into compliance.

Liam-side action before merging WP-G4.1:

bun run format        # rewrites every file to canonical Prettier style
git add -A
git commit -m "chore(format): repo-wide Prettier sweep ahead of WP-G4.1 ci.yml"

This is intentionally not bundled into the WP-G4.1 commit because (a) the WP-G4.1 spec scopes the change to the workflow + runbook + knip baseline; (b) the formatting sweep produces a 800+-file diff that would dwarf the substantive WP-G4.1 review surface.

Alternative: temporarily comment out the Format check (Prettier) step in ci.yml and merge a follow-up PR that reformats and uncomments the step. Strongly preferred to keep them coupled — formatting drift compounds.

2.2 Vercel deploy gate (WP-G4.7)

ci.yml publishes its status to Vercel via the vercel/repository-dispatch/actions/status@v1 action so prod deploys can be gated on it:

Workflow	Check name (Vercel-side)	Posted by
ci.yml (ci-summary job)	Vercel - canonical: ci-summary	Final step of ci-summary, if: always()

cloud-run-deploy.yml is removed (S298 — full Cloud Run teardown), so the old cloud-run-deploy check no longer posts. Operator note: if that check name was opted in as a Vercel Required Check (Settings → Git → Required Checks), it MUST be dropped — otherwise Vercel prod deploys will hang indefinitely waiting for a status that never arrives.

if: always() is critical — without it the action only fires on success and Vercel would only ever see green, defeating the gate. The action reports the actual job conclusion (success / failure / cancelled) on every run.

Vercel-side configuration: the project must opt in to each named check via the Vercel dashboard (Settings → Git → Required Checks). Only checks that have been seen at least once and explicitly required by Vercel become deploy-blockers. New names ship empty until a workflow run posts them.

Why ci.yml and not all workflows: ci.yml is the canonical pass/fail aggregator for code-side correctness. (It previously sat alongside cloud-run-deploy.yml, which gated the Python pipeline image+manifest deploy chain — that workflow is removed at S298; the ingestion pipeline now deploys on-prem via onprem-deploy.yml, which always runs on main/staging pushes with an in-job change-detection gate over the full compare — no workflow-level paths: filter, ID-86 / bl-236 — and does not publish a Vercel gate.) Side workflows (currently: schema-parity.yml, supabase-advisors.yml, taxonomy-sync.yml, compose-parity.yml, token-parity.yml, identity-guard.yml, api-view-coverage.yml, docs-dispatch.yml, staging-reference-refresh.yml, plus the *-nightly.yml lanes) advise but do not deploy — they remain ungated. (task-view-vendor-drift.yml retired with the ID-165 ordna cutover.)

---

§3. Branch protection

Current state (S491): the repo is Pro-tier with rulesets active (§3.4.3). Branch-protection ruleset 15785019 requires only the CI / ci-summary aggregator check (OPS-58) — individual job names are deliberately NOT in the required-checks list, so jobs can be added/removed/sharded without touching the ruleset. §§3.1–3.3 below are retained as historical context from the Free-plan era.

The CI workflow runs and reports its status to every PR even without a branch protection rule. Enforcement — i.e. preventing merge while the check is red — requires a branch protection rule.

3.1 Plan-tier limitation (HISTORICAL — resolved; repo went Pro S18–S19)

This repository is private + on the GitHub Free plan. Branch protection rules on private Free-plan repositories return:

HTTP 403: Upgrade to GitHub Pro, GitHub Team, or GitHub Enterprise
to enable this feature.

What still works without a protection rule:

• The CI / Quality gates check runs on every PR and reports red/green.
• The check appears in the PR conversation timeline.
• A reviewer can voluntarily wait for it before approving.

What does NOT work:

• The Merge button is not disabled when the check is red.
• Force-pushes to main are allowed.
• A reviewer can approve and merge before CI completes.

3.2 When you upgrade (or open-source the repo), apply the rule (HISTORICAL — superseded by the §3.4.3 rulesets; check name is now CI / ci-summary, repo is ai-solution-hub/canonical)

gh api -X PUT repos/ai-solution-hub/canonical/branches/main/protection \
  --input - <<'JSON'
{
  "required_status_checks": {
    "strict": true,
    "contexts": ["CI / Quality gates"]
  },
  "enforce_admins": false,
  "required_pull_request_reviews": null,
  "restrictions": null,
  "allow_force_pushes": false,
  "allow_deletions": false
}
JSON

Notes:

• contexts matches the GHA check display name: <workflow-name> / <job-name> → CI / Quality gates. Verify on a real PR’s checks tab before applying — GitHub occasionally surfaces it as just the job name.
• strict: true forces the PR branch to be up-to-date with main before merge — catches the “passed CI on stale base” failure mode.
• enforce_admins: false lets Liam push hotfixes that bypass the rule; flip to true once the team is bigger than one person.
• required_pull_request_reviews: null — solo developer; flip to { "required_approving_review_count": 1 } when there’s a second reviewer.

Verify after applying:

gh api repos/ai-solution-hub/canonical/branches/main/protection \
  | jq '.required_status_checks.contexts'

(Historical — the classic-protection path was never applied; rulesets won.)

3.3 Alternative: a settings-file ruleset (when Pro is enabled)

Rulesets — newer than classic protection — are also Pro-only on private repos. Same gate; same workaround. Defer until upgrade.

3.4 Canonical merge flow (post-kh-prod-readiness-S22)

The repo went Pro-tier between S18 and S19. Liam configured rulesets via the GitHub UI in S19 and S22 codified the resulting flow as the project’s canonical CI/CD discipline.

3.4.1 Branch topology

feature / worktree branches  ──PR──►  staging  ──PR──►  main
   (short-lived,                       (long-lived,           (long-lived,
    one per WP / agent)                 integration target)    production)

• (Interim note, S500: this section describes the TARGET staging-first model — the id-127/id-128 work moves the repo to it once current issues resolve. In the interim, PRs target main and staging is a deploy-only mirror.)
• feature / worktree branches — short-lived; one per WP or agent; branch from staging (or from a session’s track branch — production-readiness during the active prod-readiness phase). Merge upstream via PR.
• staging — long-lived, deploy-only integration branch (created kh-prod-readiness-S22). Receives PRs from feature / worktree branches. Targets the Platform staging Supabase DB (rbwqewalexrzgxtvcqrh, PLATFORM_PROJECT_REF — a branch of Platform prod; the local-dev + CI target since the staging-first cutover) and the staging Vercel deploy (https://canonical-platform-git-staging-tw-group.vercel.app). Subject to the staging-protection ruleset (id 15785711) — see §3.4.3.
• main — long-lived production branch. Receives PRs only from staging (no direct merges from feature branches outside hotfixes). Targets the Platform prod Supabase project (zjqbrdctesqvouboziae) and the prod Vercel deploy. Subject to the production-protection ruleset (id 15785019 — required check: CI / ci-summary only).

Client prod/staging Supabase projects are per-client and never committed (ID-68) — the full four-DB topology lives in reference/platform-context.md + the local-development runbook.

(Historical: production-readiness was a parallel-track long-lived branch under the staging ruleset during the prod-readiness phase; that phase is closed and the ref retired — staging is the sole integration target.)

3.4.2 Workflow gating

• PR into staging — runs the standard CI workflow (bun run lint, bun run test --changed, bun run build, bun run ci:knip-check). Incremental migration validation is now handled by Supabase Auto Branching (preview branches created per-PR replay any new supabase/migrations/** against a clone of prod schema; failures surface in the Supabase dashboard and PR check). Required-checks match the ruleset bypass-policy (no force-merge by non-admins).
• Merge into staging — triggers the staging deploy (Vercel staging preview promoted to the staging URL) and runs the CI push sanity-check on the resulting staging commit. (The staging-live-mirror.yml workflow no longer exists — staging data strategy is now staging-reference-refresh.yml + deterministic fixtures, WP-CI.RES.7; see the staging-refresh runbook.)
• PR into main — runs the same CI workflow plus production-only gates (advisor lint smoke; e2e smoke). Migration validation is upstream of this gate via Supabase Auto Branching on the source branch’s PR. The production-protection ruleset gates merge.
• Merge into main — triggers the prod deploy.

3.4.3 Active rulesets (verified 02/05/2026)

Ruleset id	Name	Targets	Rules
15785019	production-protection	refs/heads/main	Required status check: CI / ci-summary only (OPS-58 aggregator — see §2 job table). Full matrix: gh api repos/ai-solution-hub/canonical/rulesets/15785019.
15785711	staging-protection	refs/heads/staging (historically also refs/heads/production-readiness during the prod-readiness phase)	deletion, non_fast_forward. Migration-replay smoke removed kh-prod-readiness-S30 (workflow + script deleted entirely; incremental migration validation now canonical via Supabase Auto Branching). Bypass: OrganizationAdmin + RepositoryRole 5 (Liam) always.

Per memory reference_github_rulesets_api: PUT against the rulesets endpoint REPLACES the targeting/rules wholesale — always GET first, mutate, then PUT to avoid losing fields. The ~ALL targeting trap (matching every branch including refs/tags) was the S19 cleanup discovery — explicit refs/heads/<name> patterns avoided.

3.4.4 Cross-references

• Staging data strategy: staging-refresh.md runbook + .github/workflows/staging-reference-refresh.yml (WP-CI.RES.7). (The earlier staging-live-mirror.yml workflow + wp-ci-res2 spec are retired.)
• Track primer (historical): docs/tracks/production-readiness.md.
• Env scoping: docs/runbooks/github-environments.md (Production / Staging GH Environments map onto the corresponding prod / staging Vercel deployments + Supabase projects).
• Local development: docs/runbooks/local-development.md — .env.local points at the staging Supabase branch by default; prod-targeted scripts opt in via --env=prod.

---

§4. Environment scoping (recap)

The quality job declares environment: Production. This is required for ${{ secrets.X }} and ${{ vars.X }} to resolve to env-scoped slots rather than the (empty, post-S4) repo-level scope. See docs/runbooks/github-environments.md §4.1 for the canonical pattern.

Case sensitivity matters: environment: production (lowercase) silently creates a different environment, returning empty secrets at the substitution site. Tests pass; build fails with cryptic Zod errors. Always match Production exactly.

4.1 Secret + variable map (build + test)

Production secret	Used by	Surface
ANTHROPIC_API_KEY	server (lib/env.ts)	required
OPENAI_API_KEY	server	required
SUPABASE_SERVICE_ROLE_KEY	server	required
POSTGRES_PASSWORD	server	required
CRON_SECRET	server	required
SUPABASE_URL	server + mirrored to NEXT_PUBLIC_SUPABASE_URL	required
SUPABASE_PUBLISHABLE_KEY	server + mirrored to NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY	required
SENTRY_AUTH_TOKEN	build (source-map upload)	optional
FIRECRAWL_API_KEY	unused by ci.yml (cron jobs run on Vercel; .optional() in lib/env.ts)	n/a
TAXONOMY_SYNC_CALLBACK_SECRET	unused by ci.yml (taxonomy-sync only)	n/a
GITHUB_SYNC_TOKEN	unused by ci.yml	n/a

Production variable	Used by	Surface
NEXT_PUBLIC_APP_URL	client (lib/env-client.ts)	required
NEXT_PUBLIC_CLIENT_ID	client	required
AI_SUMMARY_MODEL	server	optional override
AI_EMBEDDING_MODEL	server	optional override
AI_EMBEDDING_DIMS	server	optional override
CLASSIFICATION_BATCH_SIZE	server	optional override
SENTRY_ORG	build	optional
SENTRY_PROJECT	build	optional

4.2 Mirrored bare → NEXTPUBLIC* names

The Production environment stores the bare SUPABASE_URL + SUPABASE_PUBLISHABLE_KEY (Vercel→Supabase integration default). lib/env-client.ts requires the NEXT_PUBLIC_* variants for the client schema. The workflow mirrors them inside the env: block of the build + test steps so we don’t have to duplicate the secret in two slots.

4.3 Known gap: NEXT_PUBLIC_SENTRY_DSN

NEXT_PUBLIC_SENTRY_DSN is not present in either Production secrets or variables (verified gh api at the start of WP-G4.1, 2026-04-27). It is .optional() in lib/env-client.ts, so its absence does not break the build — the Sentry SDK simply receives no DSN and silently no-ops on the client.

If/when Sentry browser-side error reporting is required from a CI build, add it via:

gh secret set NEXT_PUBLIC_SENTRY_DSN --env Production --body "<dsn>"

Then add NEXT_PUBLIC_SENTRY_DSN: ${{ secrets.NEXT_PUBLIC_SENTRY_DSN }} to the build step’s env: block. Tracked as a Liam-side action in the WP-G4.1 handoff.

4.4 Dependabot CI parity (full coverage, post-session 238)

GitHub blocks env-scoped + repo-Actions secrets.* from any workflow run triggered by Dependabot (documented behaviour). vars.* continues to resolve. However, GitHub has a separate repo-level Dependabot secret namespace that DOES resolve for Dependabot-triggered runs.

Setup (one-time, applied 14/05/2026):

1. 10 secrets mirrored from .env.local → repo Dependabot namespace via gh secret set <NAME> --app dependabot --body "$VALUE": ANTHROPIC_API_KEY, CRON_SECRET, OPENAI_API_KEY, POSTGRES_PASSWORD, SENTRY_AUTH_TOKEN, SUPABASE_PUBLISHABLE_KEY, SUPABASE_SERVICE_ROLE_KEY, TEST_USER_{1,2,3}_PASSWORD.
2. 13 vars mirrored from Staging env → repo-level via gh variable set <NAME> --body "$VALUE".
3. ci.yml modification: each job’s environment: switched to a conditional expression ${{ github.actor == 'dependabot[bot]' && '' || 'X' }} — empty string drops the env binding for Dependabot so the repo-level Dependabot namespace becomes the resolution path. The 9 prior if: github.actor != 'dependabot[bot]' step- and job-level guards were removed.

Result: Dependabot PRs now run the full CI pipeline (build, 4-shard vitest, e2e-smoke, MCP staging build, MCP eval L1/L3/L4, integration) with staging-flavour secrets. The Turbopack regression on commit 212f89c9 (session 238) would have been caught at PR time.

Operator gotcha — gh secret set --body -

gh secret set --body - writes the literal string "-" as the secret value; the - is not a stdin sentinel. The pipe echo "$X" | gh secret set NAME --body - discards stdin and stores -. Job logs mask the value as *** (non-empty), so the failure mode is silent “Invalid API key” / “Unauthorized” from downstream APIs.

Correct invocations:

# Quoted argument — recommended
gh secret set NAME --app dependabot --body "$VALUE"

# Stdin — OMIT --body entirely
echo -n "$VALUE" | gh secret set NAME --app dependabot

# Bulk from a file
gh secret set -f path/to/values.env --app dependabot

Branch-protection caveat (matrix checks)

The matrix job MCP eval (${{ matrix.layer }}) expands to three required checks MCP eval (l1)/(l3)/(l4). These post normally for Dependabot PRs under the new setup. If Dependabot needs to be temporarily disabled (e.g. while rotating staging keys), branch protection treats the missing matrix checks as “expected, not yet completed” — use admin merge or auto-merge config as a fallback.

Memory cross-references

• feedback_gha_secrets_vs_vars_silent_empty — silent-empty failure class.
• feedback_sdk_init_decouple_from_env_gate — env-gate Zod-schema failure mode.

4.5 Temporary — l4 forced --skip-ai (S509, dead ANTHROPIC_API_KEY)

The mcp-eval matrix’s l4 leg forces --skip-ai until the direct ANTHROPIC_API_KEY repo secret is rotated. The override lives inline in ci.yml next to the layer dispatch (S509, PR #149):

# TEMPORARY (S509): the direct ANTHROPIC_API_KEY repo secret is dead
# (401 authentication_error), so FC-61/62 — the only live-LLM checks
# in l4 — fail on every full-mode run, blocking ci-summary/merge on
# a credential, not a code, defect. Force --skip-ai on l4 until the
# key is rotated (then remove this block; the 61 non-AI checks stay
# live either way). FC-61/62 remain fully exercised by eval-nightly.
if [ "$LAYER" = "l4" ]; then
  SKIP_FLAGS="--skip-ai"
  echo "Mode override: l4 forced --skip-ai (dead ANTHROPIC_API_KEY, S509)"
fi

Why the gate exists. FC-61 (classify_content) and FC-62 (generate_summary) are l4’s only live-LLM checks; both 401 on the dead direct key. Without the override, a credential defect became a merge-blocker via ci-summary (which if: always() aggregates every job). The override keeps the other 61 non-AI l4 checks merge-blocking while deferring the two credential-dependent checks to eval-nightly, where FC-61/62 still run live against a working credential.

Revert on key rotation. When the direct ANTHROPIC_API_KEY repo secret is rotated to a live value, delete the if [ "$LAYER" = "l4" ] block — l4 reverts to honouring the caller’s SKIP_FLAGS (full mode on workflow_dispatch, --skip-ai on PR runs by default per the mcp-evaluation-spec.md cost-zero policy). Leaving the override in place post-rotation silently drops FC-61/62 from the merge gate, which is a regression of the l4 contract.

Scope guardrail. The override is scoped to l4 only. l1 (MCP protocol compliance) and l3 (response quality) run their own dispatch paths and are unaffected; do not widen the if to a matrix-wide skip without a separate decision.

---

§5. Build command — bun run build vs bun run build:vercel

The workflow uses bun run build. Decision rationale:

• package.json defines build as next build (with a prebuild generating the inlined skills bundle), and build:vercel as bun run generate:skills && bun run build:mcp-apps && next build.
• build:vercel includes bun run build:mcp-apps, which compiles the MCP App UIs (Vite single-file builds for Claude Desktop / Claude.ai). Those bundles are deployment artefacts — the source of truth is in mcp-apps/*/dist/** and is regenerated only when those UIs change.
• For a PR-blocking gate, bun run build is sufficient: it exercises the whole Next.js build (env validation, type-check via next build, client/server bundle generation, Sentry upload prep). Building the Vite bundles on every PR adds ~30s with no marginal signal.
• next.config.ts reads process.env.VERCEL_GIT_COMMIT_SHA only as the Sentry release name. Empty value is safe (release tagging falls back to no name; build does not crash). The workflow still passes ${{ github.sha }} so the release tag resolves cleanly.
• Sentry source-map upload is gated on SENTRY_AUTH_TOKEN; when the Production secret is set, source maps upload from CI as well as from Vercel, which is fine (Sentry de-dupes by release + checksum).

If a future regression makes bun run build diverge meaningfully from bun run build:vercel (e.g. an MCP-app build-step starts emitting a required runtime artefact into app/), revisit this.

---

§6. Knip baseline guard

bun run ci:knip-check (calls scripts/check-knip-baseline.ts) runs knip in JSON-reporter mode, sums the per-category counts, and compares them against .knip-baseline.json. CI fails if any current count exceeds the matching baseline.

§6.0 Step independence (S19 WP1.2). Lint, format-check, and knip in the quality-precheck job run with if: '!cancelled()' so each reports independently. The earlier sequencing (default if: success() on every step) hid baseline regressions behind whichever cheaper check failed first — that swallowed the +6 unused-types breach S17 introduced for the entirety of S18 because format-check kept failing first. Build still uses default if: success(): expensive work doesn’t run when checks are already broken. Re-baselining is unchanged from §6.3 — only the visibility of which check failed changed.

6.1 Baseline at WP-G4.1 capture

Captured 2026-04-27 from commit c602009b on branch production-readiness:

Category	Count
dependencies	1
devDependencies	1
exports	53
types	251
unlisted	3
binaries	0
duplicates	0
files	0
enumMembers	0
unresolved	0

Total: 309 findings.

Captured against a clean bun install --frozen-lockfile install (matches CI runner state). Knip’s binaries count rises if node_modules/.bin/ entries are missing — a stale or partial local install can over-report. Always re-baseline against a freshly-installed worktree.

6.2 Why counts, not snapshots

A per-file or per-name snapshot would force a baseline change every time an intentional refactor moved a finding between files. Counts catch the regression we care about (a PR adds a new unused export; one of the counts goes 53 → 54) while remaining stable under benign churn.

The trade-off: a swap (delete one unused export, add a different one) slips through silently. Acceptable cost — the ESLint local/no-unused-imports rule and review process catch the obvious cases.

6.3 Re-baselining (when counts drop)

Reductions are not failures. The script logs a DROP line for any category that came in below baseline so you notice:

Reductions detected — consider re-baselining (see docs/runbooks/ci.md §6):
  exports: 53 -> 47

To lock in the reduction:

bun run knip --reporter json > /tmp/knip.json
# Inspect; pick the new counts; edit .knip-baseline.json.
git add .knip-baseline.json
git commit -m "chore(ci): re-baseline knip after <cleanup description>"

Lower the count, not raise. Raising the baseline to silence a regression is a smell — fix the regression instead.

6.3.1 Tighten or trace — discipline for baseline raises

When raising the baseline IS necessary (cross-track absorb dropping a new entry into the count; spec-deferred broad triage; legitimate new export whose investigation is out-of-session-scope), the discipline is:

Tighten = delete the orphan, wire it to a consumer, OR document why the export is intentionally public-API and add to ignore list.

Trace = raise the baseline by exactly the amount of the breach AND document why in the raising commit’s message so the next maintainer knows the raise was deliberate, not silent. The trace text should answer: (a) what introduced the breach (commit SHA / cross-track merge / spec-deferred WP), (b) why “tighten” wasn’t appropriate this session (e.g. broad-triage scope deferred), (c) the follow-up tracking item if any (backlog row / WP / OPS-N reference).

.knip-baseline.json is counts-only — every raise or ratchet goes in a dedicated commit whose message carries the attribution, and the file’s git history is the archive. Do not inflate the file. (Owner direction S483; the file’s own head note states the same rule at the point of edit. This paragraph previously instructed the opposite — attribution in the capturedFromCommit field — and contradicted it from S483 until S500.)

Default to tighten. Reach for trace only when the breach is sub-three entries AND tightening would expand session scope beyond ratified.

If tracing, prefer a single focused raise (raise only the specific category that breached, not all categories at once) — keeps the next- session signal-to-noise high.

Worked example (kh-prod-readiness-S21 WP3 OPS-50 partial-triage):

• Breach: types: 288 → 289 (+1) after cross-track absorb 55f2c567.
• Tighten attempted: identifying the +1 specific entry would have required full 289-entry triage (spec WP3 option-2 deferred this scope to S22).
• Trace applied: baseline types: 288 → 289 with capturedFromCommit enumerating cross-track-absorb origin + S22 OPS-50.full deferral rationale + the parallel dependencies: 1 carry note for @tailwindcss/typography + ignoreDependencies config bug to investigate.
• Companion tightenings same-commit (delete lib/env.ts + remove madge devDep) shrunk other categories (files 1→0, devDependencies 1→0) so the net signal of the raise is localised rather than across-the-board.

6.4 If knip itself errors

scripts/check-knip-baseline.ts returns exit 1 if knip produced no JSON (i.e. the tool itself crashed, dependency upgrade broke something). Re-run knip locally to reproduce, fix knip config, push.

6.5 Supabase advisor lint guard (WP-G4.6)

.github/workflows/supabase-advisors.yml runs scripts/run-supabase-advisors.ts to fail CI on PR-introduced advisor findings (security + performance) before they reach prod. Same pattern as the knip baseline guard one section up: capture the current acceptable findings as a committed JSON baseline, then fail CI when a run sees a finding not in that baseline.

Triggers: nightly cron at 02:00 UTC (catches Studio / out-of-band drift); push to main / staging when watched paths change (supabase/migrations/**, supabase/seed.sql, the workflow, or the script); workflow_dispatch for ad-hoc re-runs (with an optional capture_baseline input that uploads a fresh baseline JSON artefact). PR runs are deliberately not wired — the advisor reads live prod state, which is unaffected by an unmerged PR; the push-to-main + nightly cadence covers the same signal class.

How the diff works: the script calls GET https://api.supabase.com/v1/projects/{ref}/advisors/security and ?type=performance (Supabase Management API; documented under “Security checks for production” in the platform integrations guide). Each lint is folded into a baseline record keyed on cache_key — Supabase’s unique-per-finding identifier. The diff is set-difference: new findings exit 1, stale baseline entries log a notice and exit 0.

Decision — Management API not direct SQL. The MCP get_advisors tool isn’t available in CI runners. Direct SQL against lint.<func>() was rejected: those functions aren’t part of the Supabase public API and shape can change without notice. The Management API endpoint is documented and stable.

Action contract (S19 codification). The script treats every advisor lint identically regardless of severity (ERROR / WARN / INFO) — the diff is set-difference on cache_key, not severity-weighted. This is intentional: the baseline already encodes accepted deferrals per finding (e.g. _backup_taxonomy_* ERROR-level RLS-disabled), so adding a second severity-aware policy layer would risk drift between CI gating and the deferral register. Outcome matrix:

Outcome	Cause	Exit	CI	Required action
No diff	Live findings exactly match baseline	0	✅ pass	None.
New finding (any level)	Lint present in live but not baseline	1	❌ fail	Fix the underlying issue or re-baseline if the finding is an accepted deferral logged in the roadmap/backlog.
Stale baseline	Lint in baseline but no longer in live (already fixed)	0	✅ + note	Re-baseline opportunistically at next eligible PR — no urgency, never urgent enough to bundle with unrelated work.
Mixed (new + stale)	Both above simultaneously	1	❌ fail	Triage the new finding first; re-baseline as part of the same PR if its closure also clears the stale entry.
Infra failure	SUPABASE_ACCESS_TOKEN missing, API unreachable, baseline JSON malformed	2	❌ fail	Investigate workflow / token / network. Re-run after fix; do not re-baseline blindly.

Re-baseline ownership. PR author for migration-driven changes; track owner (currently prod-readiness) for ad-hoc or notice-only refresh. Never re-baseline to silence a CI failure when the underlying lint hasn’t been either resolved or recorded as an accepted deferral. The §6.3 knip convention applies verbatim here.

Severity escalation. Not currently surfaced as a separate signal — both ERROR and WARN new findings fail-hard. If future operational need demands deferring INFO-level findings (or escalating ERROR to page) the script’s formatRecordLine already prefixes the level, so a threshold flag (--min-level=warn) would be a small extension. Defer that work until a concrete need arises; flat treatment is correct default.

Baseline file: relocated to the private docs-site — ops/supabase-advisor-baseline.json (the workflow resolves the checkout via the resolve-private-docs action and reads it from KH_PRIVATE_DOCS_DIR). The advisor run targets the Platform project via PROJECT_REF: ${{ secrets.PLATFORM_PROJECT_REF }} and --env=auto. (Original capture 2026-04-29 was against the pre-cutover prod project; record counts in this section’s history are point-in-time.)

Known follow-up tranches (S13 verifier-flagged):

• _test_*_broken_auth_user REVOKE. _test_delete_broken_auth_user
  • _test_insert_broken_auth_user are S156 probe helpers exposed SECURITY DEFINER to anon. Migration 20260416102457_pre_squash_reconciliation.sql:104 comments “DO NOT call from production code”. Follow-up: REVOKE EXECUTE FROM anon, authenticated on both, then re-baseline.
• auth_rls_initplan on user_notification_prefs (4 entries). Wrap auth.uid() in (SELECT auth.uid()) in the RLS policies. ~5-line PL/pgSQL diff per advisor recommendation; same fix shape applies to any future RLS-protected table.
• _backup_taxonomy_* lifecycle. Two ERROR-level rls_disabled_in_public findings are on snapshot tables from the financial-taxonomy merge (S188-ish, dated 20260427). RLS-disabled is correct for ops-owned backups. Drop these tables once retention window closes (decision pending; tracked alongside W5 backup/DR runbook in WP-G6.5).

Re-baselining (when live state legitimately changes):

SUPABASE_ACCESS_TOKEN=<pat> PROJECT_REF=<platform-ref> \
  bun run scripts/run-supabase-advisors.ts --env=auto --capture-baseline
# baseline lands in ${KH_PRIVATE_DOCS_DIR}/ops/supabase-advisor-baseline.json
# commit it in the PRIVATE docs-site repo, not this one

Same convention as §6.3 knip — only re-baseline on reduction (fix shipped) or accepted deferral (recorded in roadmap). Never silence a regression you intend to fix.

Required secret: SUPABASE_ACCESS_TOKEN (Production scope; existing — provisioned for WP-G4.5). Runtime budget: under 5 min wall-clock; the workflow’s timeout-minutes: 5 enforces it.

6.6 eval-nightly LLM-tier input (id-389, S506)

eval-nightly.yml carries a workflow_dispatch.inputs.llm_tier choice input — openrouter (default, including the scheduled 03:00 UTC run) or anthropic (direct-key parity opt-in). There is no mock tier on this lane: evals measure model quality, so a canned-mock output would make every score meaningless.

The tier resolves in the first workflow step (Resolve LLM tier), which writes the ANTHROPIC_AUTH_TOKEN / ANTHROPIC_BASE_URL / ANTHROPIC_API_KEY (forced empty) trio to $GITHUB_ENV for the openrouter case — the same Anthropic-skin recipe proven on cocoindex-nightly (S505) and now used by the platform composes (see reference/deployment-architecture.md §2.2). The anthropic case sets nothing and the lane falls through to the direct key.

A minute-1 credential preflight then POSTs a one-token claude-haiku-4-5 ping to the resolved tier’s /v1/messages endpoint and hard-fails on anything other than HTTP 200 — the lane fails before any build spend, not after the eval walk. Required secrets by tier: PRODUCER_AUTH_TOKEN (the OpenRouter key) for openrouter; ANTHROPIC_API_KEY for anthropic.

Note: this lane’s green is currently a known false green (id-379 instance 11). The tier-extension PR changes which credential the lane burns, not its signal quality; the false-green fix remains open.

---

§7. CI optimisation — when the run gets slow

Targets:

• Total wall-clock under 10 minutes for a typical PR.
• Lint + format + knip stages under 90 seconds combined (so cheap failures surface fast).

Levers, in order of impact:

1. Cache bun install. oven-sh/setup-bun@v2 does not cache bun.lock restorations by default. If bun install takes more than ~45s, add actions/cache@v4 over ~/.bun/install/cache keyed on bun.lock hash. (Not yet needed — current baseline is fast enough.)
2. Parallelise jobs. Split lint, format, knip into one fast job; build + test into a second slow job. Trade-off: dependencies install twice. Worth it once the slow job is over 6 minutes.
3. Path filter the slow steps. A docs-only PR doesn’t need a build. Use dorny/paths-filter@v4 (already wired for Python) to gate the build + test steps on changes to app/, components/, lib/, hooks/, contexts/, types/, scripts/, proxy.ts, next.config.ts, package.json, bun.lock.
4. Shard Vitest. --shard <n>/<total> across a matrix of N runners. Reserve for when bun run test (full run on push to main) routinely exceeds 6 minutes.

---

§8. Excluded from v1 — anthropics/claude-code-action

The anthropics/claude-code-action@v1 action (mention-driven Claude assistance + automated PR commentary in GHA) is deferred for this client.

Rationale (per docs/audits/kh-production-readiness-phase-1/research/06-anthropic-managed-agents-fit.md §2.4):

• Per-PR API spend is unbounded for a solo-developer cadence; the action is shaped for multi-developer team workflows.
• Liam’s automation pattern leans on claude.ai client-takeover via Claude Routines for client-side critique + PR review; that path is already free under the existing Claude.ai subscription.
• Re-evaluate when there’s a second developer (or a client team consuming the codebase), at which point per-PR Anthropic-API spend is amortised across more PRs and the value-density flips.

If/when re-enabling: the canonical pattern is a separate workflow file (e.g. .github/workflows/claude-review.yml) triggered by @claude mentions, not a step in ci.yml. Don’t couple the two — that would let a Claude-action infra failure block PR merges.

---

§9. Deliberately-failing PR — verifying enforcement

Pre-merge confidence in this workflow comes from a trial PR after merge to main:

1. Merge this WP-G4.1 PR to main.
2. Open a trivial follow-up PR with one of:
   • A formatting violation (e.g. delete a trailing newline in a .ts file).
   • A tsc violation (e.g. const x: number = 'string' in a non-test file).
   • A new unused export (e.g. export const FOO = 1 in lib/<unused>.ts).
3. Confirm:
   • The CI / ci-summary check runs (plus the per-job checks).
   • The check reports red.
   • PR conversation surfaces the failure inline.
4. Push a fix commit.
5. Confirm the check goes green.
6. Close the trial PR (do not merge — it was just a probe).

Once §3.2 branch protection is enabled (post-Pro upgrade), step 4 ⟶ a “can merge” assertion replaces the manual confirm.

---

§10. Cross-references

• docs/runbooks/github-environments.md — environment-scoping rules, secret taxonomy, multi-tenant fit.
• docs/runbooks/local-development.md §3 — local CLI script env-target conventions (--env=prod opt-in).
• docs/audits/kh-production-readiness-phase-1/research/01-existing-infra-audit.md §1 — pre-WP-G4 baseline; what existed before this workflow.
• docs/audits/kh-production-readiness-phase-1/research/06-anthropic-managed-agents-fit.md §2.4 — claude-code-action exclusion rationale.
• .github/workflows/taxonomy-sync.yml — Production-scoped secret pattern reference.
• GHA: required status checks.
• GHA: workflow security — injection-mitigation patterns informed the env-block structure in §3 of the workflow file.

---

§11. WP-G4.x follow-on items

The WP-G4.1 PR-blocking foundation has been extended substantially since the runbook was first authored. This list tracks shipped vs deferred:

WP	Topic	Status (post-S31)
WP-G4.2	Test environment fan-out (MCP L1/L3/L4 in CI)	Shipped — mcp-eval matrix job in ci.yml (Staging scope; layer matrix; spec wp-g4.4-mcp-eval-ci-spec.md). Build artefact-shared via mcp-build since S30.
WP-G4.3	Playwright E2E (bun run test:e2e)	Shipped — e2e-smoke job (Staging scope; --project=smoke; spec wp-g4.3-e2e-smoke-spec.md). continue-on-error dropped S31 W1(b).
WP-G4.4	Migration drift guard	Superseded — Supabase Auto Branching is now the canonical incremental-migration validator. The local migration-replay workflow + migration-replay-check.ts were retired S30.
WP-G4.5	Bundle size guard	Deferred — next-bundle-analyzer budget + size-delta comment. Not yet wired.
WP-G4.6	Supabase advisor lint guard	Shipped — supabase-advisors.yml (nightly + push-to-main + dispatch). See §6.5 above.
WP-G4.7	Cron contract test	Deferred — periodic cron route smoke against staging.
WP-CI.RES.1	Vitest 4-shard split	Shipped — quality-test matrix (shard: [1,2,3,4], fail-fast: false). Spec wp-ci-res1-vitest-perf-spec.md.
WP-CI.RES.3	Integration-tier in CI	Shipped — integration job (Staging scope, 25 min timeout). continue-on-error dropped S29 (Phase 2 cutover; spec wp-ci-res3-integration-tier-ci-spec.md).
WP-CI.RES.7	Staging data strategy (reference-refresh + fixtures)	Shipped — staging-reference-refresh.yml + __tests__/integration/fixtures/staging-fixture-generator.ts. Spec wp-ci-res7-staging-data-strategy-spec.md.
R-WP17 Wave-B	Type-drift parity CI gate (R-WP21 option c)	Shipped S10 WP1 — type-drift-parity job in ci.yml. Runs bun run ast-dataflow type-drift-detect --ci; fails on any new fetcher-only interface absent from .type-drift-baseline.json. 37-interface baseline locked at S9 close-out. Spec docs/specs/id-16-ast-dataflow-tool/type-safety-pipeline/PRODUCT.md D-19; decision decision-OPS-T1.md. Since id-377 {377.5} (DR-128) the ast-dataflow tool is consumed from the standalone public ai-solution-hub/ast-dataflow repo via a pinned bun git dependency; the script name and gate behaviour are unchanged.

Future additions land as separate workflow files where they need different trigger semantics (cron, schedule, repository_dispatch) or as additional jobs in ci.yml where they share the per-PR cadence.