Local Development Runbook

Local Development Runbook

⚠️ SUPERSEDED (S436, 2026-07-02): since the staging-first cutover, .env.local targets Platform staging rbwqewalexrzgxtvcqrh (PLATFORM_PROJECT_REF) as the local + CI DB — NOT the client refs the body still shows. Four DBs: Platform prod zjqbrdctesqvouboziae · Platform staging rbwqewalexrzgxtvcqrh (local+CI) · client prod rovrymhhffssilaftdwd · client staging turayklvaunphgbgscat. Prod CLI opts in via --env=prod; always cat supabase/.temp/project-ref before any db push. See reference/deployment-architecture.md §2.

Status: Draft (kh-prod-readiness-S6 W1, 27/04/2026). Audience: Liam + future contributors running CLI scripts locally against Knowledge Hub. Owner: prod-readiness track. Pair with: docs/audits/kh-production-readiness-phase-1/specs/wp-s5.2-cli-script-audit-spec-v1.md (the canonical per-script classification table) and docs/runbooks/staging-refresh.md (when staging needs a rebuild).

---

§1. Overview

Post-WP-S5.2, local development hits staging by default. Since the staging-first cutover the default target is Platform staging rbwqewalexrzgxtvcqrh (PLATFORM_PROJECT_REF) — the local + CI DB — NOT the client staging branch turayklvaunphgbgscat this doc’s older examples show. Prod-targeted CLI scripts opt-in via either (a) an --env=prod flag (where implemented per spec D-23 top-10) or (b) an explicit env override at invocation time (SUPABASE_URL=<prod-url> ... bun run scripts/X.ts).

The full topology is four DBs: Platform prod zjqbrdctesqvouboziae · Platform staging rbwqewalexrzgxtvcqrh (local + CI target) · client prod rovrymhhffssilaftdwd · client staging turayklvaunphgbgscat; each client also runs its own prod + staging project. This is the inverse of the pre-flip state where .env.local pointed at prod and there was no built-in way to easily target staging; the staging-first cutover further moved the default off the client refs onto the Platform staging DB. Always cat supabase/.temp/project-ref before any db push — a stale ref lands the push on the wrong DB (see §5.3 / §6.3).

Why default-staging?

• Cheaper experimentation: a typo against staging is recoverable; the same typo against prod corrupts the canonical KB.
• E2E test users belong on staging only.
• Enables proper preview-deploy testing: Vercel Preview deploys hit staging; local CLI work on the same data is now consistent.
• Matches the discipline established by WP-G3.x staging-DB work (per STATUS.md §4).

When to hit prod

• You need to query the canonical KB (bun run kb-search).
• You need to ingest real client content (python3 scripts/ingest.py).
• You need to backfill or re-classify prod data (any scripts/backfill-* or scripts/batch-* script — see §3 table).
• You need to run an eval that compares against the prod gold standard.

---

§2. .env.local canonical values

2.1 Pre-flip state (before WP-S5.2 impl pass)

# Supabase — pointed at PROD
SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co
NEXT_PUBLIC_SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co
SUPABASE_PUBLISHABLE_KEY=sb_publishable_N1G97m0Ew1aEl7xrP59oEQ_P_CoRWcC
NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY=sb_publishable_N1G97m0Ew1aEl7xrP59oEQ_P_CoRWcC
SUPABASE_SERVICE_ROLE_KEY=<prod service-role JWT>
POSTGRES_PASSWORD=<prod DB password>

# Upstream APIs — same in both states
ANTHROPIC_API_KEY=<prod>
OPENAI_API_KEY=<prod>
FIRECRAWL_API_KEY=<prod>

# Cron secret — currently the prod value
CRON_SECRET=<prod cron secret>

# App URL
NEXT_PUBLIC_APP_URL=https://knowledge-hub-seven-kappa.vercel.app

# Client ID — the S196 incident guard (must be set for BRANDING.organisationName)
NEXT_PUBLIC_CLIENT_ID=phew

# Test users (E2E)
TEST_USER_1_EMAIL=admin@example.com
TEST_USER_1_PASSWORD=<test pw>
# (TEST_USER_2 / 3 similar)

2.2 Post-flip state (target — what Liam manually edits to)

Source: docs/audits/kh-production-readiness-phase-1/research/09-vercel-supabase-env-mapping-spec.md §4 (Vercel Preview target column).

# Supabase — pointed at STAGING (changed)
SUPABASE_URL=https://turayklvaunphgbgscat.supabase.co
NEXT_PUBLIC_SUPABASE_URL=https://turayklvaunphgbgscat.supabase.co
SUPABASE_PUBLISHABLE_KEY=sb_publishable_T_9lvCgslAlyw2gCCpVbKQ_NZWmym5R
NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY=sb_publishable_T_9lvCgslAlyw2gCCpVbKQ_NZWmym5R
SUPABASE_SERVICE_ROLE_KEY=<staging service-role JWT — from staging branch dashboard>
POSTGRES_PASSWORD=<staging DB password — Supabase auto-generates per branch>

# Upstream APIs — KEEP PROD VALUES (same usage either side)
ANTHROPIC_API_KEY=<prod>
OPENAI_API_KEY=<prod>
FIRECRAWL_API_KEY=<prod>

# Cron secret — STAGING value (different from prod, per Vercel Preview env target)
CRON_SECRET=<staging cron secret>

# App URL — per D-14 (research/09 §11.2)
NEXT_PUBLIC_APP_URL=https://knowledge-hub-git-staging-tw-group.vercel.app

# Client ID — UNCHANGED (staging serves same client)
NEXT_PUBLIC_CLIENT_ID=phew

# Test users (E2E) — UNCHANGED (synthetic emails work on either env)
TEST_USER_1_EMAIL=admin@example.com
TEST_USER_1_PASSWORD=<test pw>

2.3 What changed and why

Variable	Pre-flip	Post-flip	Reason
SUPABASE_URL	rovrymhhffssilaftdwd	turayklvaunphgbgscat	Default-staging move.
NEXT_PUBLIC_SUPABASE_URL	same	same	Same project as SUPABASE_URL.
SUPABASE_PUBLISHABLE_KEY	prod publishable	staging publishable	Per-project.
NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY	same as server	same as server	Both client and server read the same publishable key.
SUPABASE_SERVICE_ROLE_KEY	prod service-role	staging service-role	Per-project.
POSTGRES_PASSWORD	prod DB password	staging DB password	Supabase auto-generates per branch.
CRON_SECRET	prod	staging	Per-env per research/09 §4.3 (rate-limit isolation).
NEXT_PUBLIC_APP_URL	prod URL	staging Vercel git-branch URL	Per D-14 — never use prod URL on non-prod env.
Other API keys	prod	prod	Identical usage either side.
Test-user vars	unchanged	unchanged	Synthetic creds; works on either env once test users seeded.

2.4 Sourcing the staging values

Value	Where to find it
Staging URL	https://turayklvaunphgbgscat.supabase.co (constant).
Staging publishable key	Supabase dashboard → project turayklvaunphgbgscat → Settings → API → “publishable key” / via mcp__supabase__get_publishable_keys{project_id:"turayklvaunphgbgscat"}. Already known: sb_publishable_T_9lvCgslAlyw2gCCpVbKQ_NZWmym5R.
Staging service-role key	Supabase dashboard → project turayklvaunphgbgscat → Settings → API → service_role JWT (display only — copy once).
Staging POSTGRES_PASSWORD	Supabase dashboard → project turayklvaunphgbgscat → Settings → Database → “Connection string” tab — the password is in the URI.
Staging CRON_SECRET	Vercel dashboard → project knowledge-hub → Settings → Environment Variables → filter target=Preview → CRON_SECRET value.

2.5 KH_CLIENT_NAME_DENYLIST (client-name identity guard)

The CI identity-guard workflow (identity-guard.yml) and the denylist-consuming generator scripts (generate-classification-prompt-taxonomy.ts, generate-purge-path-inventory.ts) consume the same env var/secret. (The task-view patch-server — formerly the other consumer, via ID-90 {90.8} invariants 28–35 — is retired with the ID-165 ordna cutover.) The value is the FULL canonical denylist JSON — never a bare or comma-separated name list:

{
  "tokens": [
    { "value": "<client-name>", "case_insensitive": true, "class": "client" }
  ],
  "exclusion_patterns": []
}

Shape rules (strict JSON):

• tokens[] — required, non-empty. Each entry needs value (non-empty string) + case_insensitive (boolean); class is optional metadata.
• exclusion_patterns[] — optional array of { pattern } objects, applied by the CI identity-guard as carve-outs.
• Legacy comma-separated values are never comma-split (T-1; the bl-244 root cause was a JSON secret silently comma-split into garbage tokens).

Set-but-invalid is a loud error: the identity-guard workflow fails fatally when the secret is empty or fails the tokens[] shape check (jq gate in identity-guard.yml). Misconfiguration never silently disables the guard.

Locally the var has no always-on consumer since the patch-server retirement — set it in KH .env.local (single-quoted on one line: KH_CLIENT_NAME_DENYLIST='{"tokens":[...]}'; Bun auto-loads .env.local) only when running the generator scripts above; otherwise leave it unset.

The canonical denylist copy lives in THIS (private docs-site) repo at ops/identity-denylist.json; the GitHub secret KH_CLIENT_NAME_DENYLIST is synced manually from it on change (one rotation point). Never commit or echo token values — docs and reports use placeholders.

---

§3. Hitting prod from local

Two patterns. Choose per-script intent and frequency.

3.1 Pattern A — --env=prod flag (top-10 scripts only)

For scripts where Liam invokes prod most often, the script implements a --env=prod flag (per spec §7.1 / §7.2). When set, the flag asserts that SUPABASE_URL already contains the prod project ref. Liam still provides the prod credentials at invocation time:

# Standard invocation — runs against staging by default
bun run kb-search "memory solutions for LLM agents"

# Prod invocation
SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  bun run kb-search "memory solutions for LLM agents" --env=prod

Top-10 scripts implementing an explicit --env flag (per spec D-23 + Liam ruling — DECIDED v1.1; impl pending W3b/W3c):

1. scripts/kb-search.ts (--env=prod; the headline always-prod script)
2. scripts/ingest.py (--env=prod; highest-risk prod write)
3. scripts/batch-generate-summaries.ts (--env=prod)
4. scripts/eval-classification.ts (--env=prod)
5. scripts/eval-entity-classification.ts (--env=prod)
6. scripts/eval-search.ts (--env=prod)
7. scripts/eval-holder-rule-ts.ts (--env=prod)
8. scripts/normalise-entities.ts (--env=prod)
9. scripts/import_bid_library.py (--env=prod)
10. scripts/wipe-bid-responses.ts (--env=staging — D-22; staging is the safe-default; prod-wipe is the footgun, requires positive --env=prod opt-in + confirm prompt)

The remaining 41 always-prod scripts use Pattern B (documented invocation only) per D-21=(c). Future options F-1/F-2/F-3 for batch flag adoption captured in spec §9 D-21 — deferred to S7+ for Liam ruling.

3.2 Pattern B — explicit env override (no flag)

For all other always-prod scripts, override the env at invocation:

# Single-shot
SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  POSTGRES_PASSWORD=$KH_PROD_POSTGRES_PASSWORD \
  bun run scripts/<script>.ts <args>

# Or, source a prod-env file before invoking
source ~/.config/kb-prod-creds.env
bun run scripts/<script>.ts <args>
unset SUPABASE_URL SUPABASE_SERVICE_ROLE_KEY POSTGRES_PASSWORD

The .config/kb-prod-creds.env is outside the repo and contains only prod values — never committed. Sample template:

# ~/.config/kb-prod-creds.env (NOT in repo)
export SUPABASE_URL="https://rovrymhhffssilaftdwd.supabase.co"
export NEXT_PUBLIC_SUPABASE_URL="$SUPABASE_URL"
export SUPABASE_SERVICE_ROLE_KEY="<prod service-role>"
export SUPABASE_PUBLISHABLE_KEY="<prod publishable>"
export NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY="$SUPABASE_PUBLISHABLE_KEY"
export POSTGRES_PASSWORD="<prod DB password>"
export CRON_SECRET="<prod cron secret>"

When sourced, prod env vars temporarily shadow .env.local for processes spawned in that shell. Always unset after to avoid later commands leaking into prod.

3.3 The full per-script classification

See docs/audits/kh-production-readiness-phase-1/specs/wp-s5.2-cli-script-audit-spec-v1.md §6 for the canonical per-script table (56 rows + 14 lib/test rows). Summary:

Bucket	Behaviour post-flip
Always-prod (51 scripts)	Read env. Default = staging (will fail or return empty). Use Pattern A or B.
Env-agnostic (5 scripts)	Read env. Default = staging — works as intended. Hit prod only when explicitly needed via Pattern B.
Always-staging (2 scripts)	Read env. Default = staging — correct. Never invoke against prod.
Library / test (14 files)	No env read at runtime (mocked or library-only).

---

§4. Common script invocations (post-flip)

4.1 Knowledge base search (most-used)

# Default: staging (empty result expected)
bun run kb-search "test query"

# Prod (the canonical invocation):
SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  bun run kb-search "test query" --env=prod

# Or with sourced creds:
source ~/.config/kb-prod-creds.env
bun run kb-search "test query" --env=prod

4.2 Ingest a URL into prod

# Highest-risk script. Always invoke with explicit prod-pin:
SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  ANTHROPIC_API_KEY=$KH_PROD_ANTHROPIC_KEY \
  OPENAI_API_KEY=$KH_PROD_OPENAI_KEY \
  PYTHONUNBUFFERED=1 \
  python3 scripts/ingest.py --env=prod https://example.com/article

# Dry-run version (no DB write):
SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  python3 scripts/ingest.py --dry-run https://example.com/article

Note (v1.1): Per Liam D-20=α ruling, the Python pipeline now reads .env.local only (no .env fallback — .env being phased out of dev-side state). Post-impl, local Python scripts inherit the same env-source as TS scripts. The production Python pipeline deploys on-prem (IONOS VPS + Coolify via onprem-deploy.yml; the earlier WP-RUN.1 Railway adoption is superseded) and reads its env from the Coolify-managed compose environment.

4.3 Seed E2E test users (always-staging)

# Default post-flip: hits staging — correct intent.
bun run scripts/seed-e2e-users.ts

# Verify-only mode:
bun run scripts/seed-e2e-users.ts --check

If you see a “this is not staging” warning, double-check .env.local points at turayklvaunphgbgscat. The script must NEVER hit prod.

4.4 Batch generate summaries (always-prod)

SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  ANTHROPIC_API_KEY=$KH_PROD_ANTHROPIC_KEY \
  bun run scripts/batch-generate-summaries.ts --limit 20 --env=prod

4.5 Run an eval (always-prod)

# Classification eval (cached mode — reads prod KB, no API calls):
SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  bun run eval:classification --cached --env=prod

# Search eval (also reads prod):
SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  OPENAI_API_KEY=$KH_PROD_OPENAI_KEY \
  bun run eval:search --env=prod

4.6 Backfill Q&A entities (always-prod, Python)

# Always-prod via env override:
SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  ANTHROPIC_API_KEY=$KH_PROD_ANTHROPIC_KEY \
  python3 scripts/backfill_entities_qa.py

4.7 Generate codebase stats with DB access (env-agnostic)

# Default: env-agnostic; --db reads whichever DB is linked.
bun run scripts/generate-codebase-stats.ts --db

# Stats vary per env (acceptable; output overwrites the file each
# run). If you want prod stats specifically:
SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  bun run scripts/generate-codebase-stats.ts --db

4.8 Wipe bid responses (env-agnostic, HIGH FOOTGUN)

# Default post-flip: hits staging.
bun run scripts/wipe-bid-responses.ts --env=staging

# To wipe prod (rare; require manual confirmation):
SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  bun run scripts/wipe-bid-responses.ts --env=prod

NEVER invoke this without an explicit --env flag — pre-impl-pass the script does not assert env, so any shell-level prod env exports shadow .env.local and the script wipes prod.

4.9 Sync plugin taxonomy (always-prod — touches prod DB)

SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  bun run sync:taxonomy --env=prod

4.10 Snapshot prod content state (always-prod, used pre-re-ingest)

SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  bun run scripts/snapshot-content-state.ts --env=prod

4.11 Seed admin-dedup E2E fixtures (always-staging)

# Default: timestamp-based runId
bun run scripts/seed-admin-dedup-fixtures.ts

# Named runId for repeatable manual sessions
bun run scripts/seed-admin-dedup-fixtures.ts --tag=manual-2026-05-05

# Cleanup by tag
bun run scripts/seed-admin-dedup-fixtures.ts --tag=manual-2026-05-05 --cleanup --yes

# Nuke every e2e_dedup_fixture_run_id row (interactive prompt)
bun run scripts/seed-admin-dedup-fixtures.ts --cleanup-all

# Dry-run preview before destructive cleanup
bun run scripts/seed-admin-dedup-fixtures.ts --tag=manual-2026-05-05 --cleanup --dry-run

Seeds 26 rows (6 §1.7 queue pairs + 7 §1.9 near-dup pairs) tagged with metadata.e2e_dedup_fixture_run_id. Used by the worker-scoped Playwright fixture (e2e/fixtures/admin-dedup-fixture.ts) and for manual UI walks against staging. Design + locked decisions in docs/audits/s213b-admin-dedup-fixtures-design.md §9.1-§9.6.

---

§5. Verifying current CLI link

The Supabase CLI maintains a separate state in supabase/.temp/project-ref — this is what supabase db push, supabase migration new, and supabase gen types read.

5.1 Check the current link

cat supabase/.temp/project-ref

Possible outputs:

• turayklvaunphgbgscat → CLI is linked to staging.
• rovrymhhffssilaftdwd → CLI is linked to prod.
• empty / file missing → CLI not linked; first supabase link will initialise it.

5.2 Confirm migration state matches

# Staging count (should be 35 post-WP-G3.1'):
mcp__supabase__list_migrations{project_id:"turayklvaunphgbgscat"}

# Prod count:
mcp__supabase__list_migrations{project_id:"rovrymhhffssilaftdwd"}

Drift between these counts indicates staging needs a refresh — see staging-refresh.md for the refresh ritual.

5.3 Pipeline run audit / ingest_source distribution

Run after any S207 WP-A4 backfill, post-cutover sanity checks, or whenever investigating provenance for a cohort of content_items. The first query is the typed-column post-WP-A4 distribution; the second is the granular per-history-row breakdown that lives on content_history.metadata->>'ingest_source' (written by every v1 trigger emit — see data-entry-points.md Appendix F for the canonical 11-value vocabulary). The third query is the residual-NULL guard; expect <5% of content_items.ingest_source to be NULL post-backfill, otherwise re-run 20260428180945_backfill_ingest_source.sql or check whether new INSERT-time wire sites are skipping the typed column.

-- Distribution of ingest_source across content_items (post-WP-A4)
SELECT ingest_source, COUNT(*) FROM content_items GROUP BY 1 ORDER BY 2 DESC;

-- Distribution of ingest_source on v1 content_history rows (granular observability)
SELECT
  metadata->>'ingest_source' AS ingest_src,
  COUNT(*) AS v1_count
FROM content_history
WHERE version = 1
GROUP BY 1
ORDER BY 2 DESC;

-- Residual NULL ingest_source rows (should be <5% of total post-backfill)
SELECT COUNT(*) AS null_count,
       (SELECT COUNT(*) FROM content_items) AS total
FROM content_items
WHERE ingest_source IS NULL;

Run via mcp__supabase__execute_sql{project_id:"rovrymhhffssilaftdwd", query:"…"} for prod or substitute turayklvaunphgbgscat for staging. The two distributions should agree (same value set, similar relative ordering); a mismatch points to a wire site that writes content_items.ingest_source but not the corresponding content_history.metadata->>'ingest_source' (or vice versa).

---

§6. Switching env contexts

6.1 Relink CLI to prod (for prod migration push)

# Use the canonical CLI path — never `supabase` from PATH (zsh may pick up
# a dev version):
/opt/homebrew/bin/supabase link --project-ref rovrymhhffssilaftdwd

# Verify
cat supabase/.temp/project-ref
# → rovrymhhffssilaftdwd

# Now push migrations:
/opt/homebrew/bin/supabase db push

6.2 Relink CLI to staging

/opt/homebrew/bin/supabase link --project-ref turayklvaunphgbgscat
cat supabase/.temp/project-ref
# → turayklvaunphgbgscat

6.3 Stale .temp/project-ref warning

Per CLAUDE.md gotcha “CLI .temp/project-ref can silently go stale post-env-flip”: always cat the file before any db push. A stale ref means db push lands on the wrong project (looks like silent-fail on the intended project; actually a successful push to the wrong one).

6.4 CLI scripts vs CLI tool

The Supabase CLI tool (/opt/homebrew/bin/supabase) and the CLI scripts (scripts/**/*.{ts,py}) read different env sources:

Surface	Reads	Honours flip
CLI tool (supabase db push)	.temp/project-ref + supabase/config.toml	NO — independent of .env.local
CLI scripts (bun run kb-search)	.env.local first, then .env, plus shell exports	YES
Python scripts (python3 scripts/X.py)	.env.local (via kb_pipeline/config.py post-D-20=α impl) plus shell exports	YES
On-prem pipeline (Coolify, onprem-deploy.yml)	Coolify-managed compose env vars (Railway superseded)	n/a — server env independent of local dotenv

This is why the §6.1 / §6.2 commands relink the CLI tool but do NOT move the scripts — flips and relinks are independent.

(v1.1 update: pre-impl, Python read .env only — see §7.5 footgun. Post-D-20=α impl, Python and TS share .env.local.)

Footnote on Pattern B-prime Python scripts (3 of 11): Three Python scripts pre-load .env.local independently before importing kb_pipeline.config: backfill_chunks_stage2.py:31, backfill_entities_qa.py:28, ingest_stage2_markdown.py:36. Per spec §4.3 Pattern B-prime — they already hit .env.local first; the D-20=α refactor brings the remaining 6 transitive consumers in line.

---

§7. Known footguns

7.1 Bun fetch HTTP 204 sandbox hang

CLAUDE.md gotcha: supabase-js .update()/.insert()/.upsert()/.delete() without .select() returns 204, which Bun hangs on in the sandbox. Applies to staging writes too (post-flip). Run any supabase-writing script with dangerouslyDisableSandbox: true from a Claude Code session. Production (Vercel) is unaffected.

7.2 Sandbox ALL_PROXY breaks anthropic SDK httpx

Sandboxed Bash injects ALL_PROXY=socks5h://.... The Anthropic Python SDK uses httpx which rejects the socks5h scheme without socksio installed. Result: SyncHttpxClientWrapper._state AttributeError. Run Python scripts that hit Anthropic via dangerouslyDisableSandbox: true.

7.3 .env.local gitignored — worktree subagents see different baseline

.env.local is in .gitignore so worktree subagents (e.g. agents spawned via isolation: "worktree") do NOT inherit Liam’s local file. They see a clean repo with no .env.local. Implication: tests in worktree subagents that depend on env vars MUST patch them in test setup, NEVER rely on .env.local being present.

7.4 Stale RETIRED_PROJECT_REF in backfill-content-history-v1.ts — RESOLVED (script deleted)

Per spec §5.1, this script’s safety guard refused to run against rovrymhhffssilaftdwd — which is current client prod — making it unrunnable. Resolved by deletion: the script was removed in the S393 jscpd-dedup sweep (canonical commit 7b9c31e6); RETIRED_PROJECT_REF no longer exists anywhere in the repo. Project refs are now env-sourced via scripts/lib/project-refs.ts (PLATFORM_PROJECT_REF / STAGING_PROJECT_REF / PROD_PROJECT_REF) — no client ref is hardcoded in tracked source (ID-68).

7.5 Python pipeline reads .env not .env.local — RESOLVED + .env retired

Pre-impl: kb_pipeline/config.py:14 ENV_PATH = os.path.join(PROJECT_ROOT, ".env"). After the flip, .env and .env.local would diverge.

Post-D-20=α impl + .env retirement (S6 W3 close-out 27/04/2026): ENV_PATH switched to .env.local only. .env file deleted from both prod-readiness and main worktrees. .env.local is now the single source of truth for both TS and Python pipelines.

Server implication: the production Python pipeline runs on-prem (IONOS VPS + Coolify — Railway superseded); server-side env vars shadow file-based env via standard os.environ precedence, so no code change was needed in config.py for the hosted path.

7.6 CLI .temp/project-ref stale post-flip

Per CLAUDE.md and §6.3 above. Always cat before any db push.

7.7 seed-e2e-users.ts against prod (pre-flip risk)

If you run this script BEFORE the manual flip, env still points at prod — and it would create test users in prod auth.users. Not recoverable without a manual auth.admin.deleteUser() call. Always verify .env.local before running.

7.8 wipe-bid-responses.ts against prod via shell-export shadow

If you sourced ~/.config/kb-prod-creds.env for an unrelated prod script and forgot to unset, then ran bun run scripts/wipe-bid-responses.ts, the wipe lands on prod. Pre-impl-pass the script does NOT assert env; the only protection is shell hygiene. After D-22 ruling lands the script gains an explicit --env=staging|prod flag.

7.9 CRON_SECRET drift

Staging CRON_SECRET differs from prod (per research/09 §4.3). Cron-poking scripts (e.g. local invocations of /api/cron/* endpoints) need the staging value to authenticate against staging. If you see “401 unauthorised” hitting a local cron route, check your .env.local CRON_SECRET matches Vercel Preview’s value.

7.10 NEXT_PUBLIC_CLIENT_ID must be set

Per CLAUDE.md gotcha: missing this var → BRANDING falls back to “Knowledge Hub” → corrupts holder derivation in classifier scripts. Both .env.local and any sourced prod env file must include NEXT_PUBLIC_CLIENT_ID=phew.

7.11 Intent workspace provisioning + parent .env.local drift

Intent (by Augment) workspaces are git worktrees under ~/intent/workspaces/<name>/canonical, seeded by scripts/provision-worktree.sh (node_modules copy-on-write clone, .venv symlink, .env.local + other .worktreeinclude copies, surgical .gitnexus seed).

node_modules is cloned, not symlinked (changed 26/07/2026). A symlinked node_modules breaks package resolution — realpath escapes the worktree, so bun/next resolve against the source root. It is now provisioned with an APFS copy-on-write clone (cp -c -R): a real directory, ~1 min for ~1.8 GB / 120k files, and near-zero disk until the trees diverge. .cache is no longer provisioned at all (its consumer retired with the old task-view ledger).

How the wiring works. Intent executes a workspace “setup script” by writing the stored script body to a temp file and running it with no positional arguments — context arrives only via env vars ($MAIN_CHECKOUT, $WORKTREE_PATH, $BRANCH_NAME, $SOURCE_BRANCH). Since the id-164 fix, provision-worktree.sh falls back to $WORKTREE_PATH/$MAIN_CHECKOUT when $1/$2 are absent, so it works under Intent as-is. Two wiring rules:

1. Keep Intent’s stored setup script a thin one-liner — exec bash "$MAIN_CHECKOUT/scripts/provision-worktree.sh" "$WORKTREE_PATH" "$MAIN_CHECKOUT" — never a pasted copy of the script body. A pasted copy drifts from the repo; a pre-fix pasted body (which still required $1) is exactly what left the attached-review workspace with no .env.local/node_modules (the script died on its first line, silently, in a background terminal). The repo-level default in <main-checkout>/.intent/config.json (setupScript key) already holds this one-liner; the Intent UI’s saved “provision-worktree” library script should match it.
2. Verify every fresh workspace with the three checks: ls -ld node_modules (a real directory — a symlink means a pre-26/07/2026 provisioner and package resolution will escape the worktree), .env.local present with the expected staging ref, and python3 -m pytest --version. Task-ledger access needs no per-worktree provisioning (ID-165 ordna cutover): tasks are plain markdown in the private docs-site — cat "$KH_PRIVATE_DOCS_DIR/tasks/id-<N>.md" (see tasks/AGENTS.md there).

Parent .env.local drift caveat. Provisioning copies .env.local from the main checkout at creation time; the copy never tracks its source. After any rotation or cutover edit to the main checkout’s .env.local, live workspaces keep the stale value until you re-copy the file (or re-run scripts/provision-worktree.sh <workspace-path> — the file copy overwrites cleanly). The drift is two-deep at present: ~/Developer/canonical (the main worktree Intent clones from) itself carries a manual copy of ~/Documents/development/canonical/.env.local, and its node_modules/.cache are symlinks into that older checkout — so bun install must run in the tree that owns the real node_modules (currently ~/Documents/development/canonical), and .env.local edits made there must be propagated forward to ~/Developer/canonical before they reach new workspaces.

7.12 “Another next dev server is already running” — the per-directory lock

Next 16 takes a lock at <distDir>/dev/lock and refuses a second next dev for the same directory even on a different port. Several agents sharing one checkout therefore collide on the lock, not the port — so assigning ports (-p, portless) does not fix it on its own. Verified 26/07/2026.

• Different worktrees — nothing to do. Separate directories, separate locks. This is the case for every Intent workspace.

• Same checkout — give each session its own build root:

  NEXT_DIST_DIR=.next-1 bun dev   # slots .next-1 .. .next-4

  next.config.ts reads NEXT_DIST_DIR (default .next), so CI, Vercel and single-session local dev are untouched.

Use only the registered slot names. .next-1...next-4 are pre-declared in tsconfig.json’s include. Next matches those strings literally — a glob such as .next-*/types/**/*.ts does not satisfy the check (tested). Any unregistered dist dir makes Next rewrite and reformat tsconfig.json on startup, dirtying the tree for every parallel session. Need a fifth slot? Add both its types and dev/types entries to include first.

Stale servers are the other half of this: a crashed or forgotten next dev keeps holding .next/dev/lock. Find and clear it with

ps -eo pid,etime,command | grep "[n]ext-server"

Named URLs (optional). portless (Vercel Labs) wraps the dev command, assigns a free port via PORT, and proxies a stable .localhost name; in a git worktree it prefixes the branch name automatically (wt-probe.canonical.localhost). It solves ports and URL discovery, not the lock — the two are complementary.

# proxy on a high port avoids the sudo that binding 443/80 needs
portless proxy start --no-tls --port 1355
portless run next dev        # -> http://<worktree>.canonical.localhost:1355
portless list                # active routes

Caveats: requires Node 24+ per its engines (runs fine under our pinned 22 in practice; install it with a Node 24+ npm). The default HTTPS mode binds 443 and adds a local CA to the system trust store — both need sudo, so prefer the unprivileged form above for agent use.

---

§7A. The local Supabase stack doubles as the CI E2E substrate (DR-096, S500)

Local development normally targets the HOSTED Platform staging DB via .env.local (§1) — but the repo’s supabase/config.toml also drives a full local stack (supabase start / supabase db reset), and since the {365.5} cutover that same local stack IS the substrate for both CI E2E lanes (ci.yml e2e-smoke + e2e-nightly.yml shards). Consequences worth knowing locally:

• db reset is the only idempotent provisioning primitive. supabase start applies migrations + seed.sql + the declared [storage.buckets.*] only on a FRESH Docker volume; on a cached volume it silently skips. db reset replays all of it unconditionally. (Measured in CI: start ~64 s, reset ~28 s on 16 GB runners.)
• Exclusions are config-driven, not flag-driven. supabase start -x <svc> does NOT prevent image pulls — one-shot setup jobs gate on config.toml enabled, not the exclude list (supabase/cli#4088), and an invalid exclude name (e.g. the renamed inbucket) warns-and-continues. CI therefore rewrites the config on the runner via scripts/ci/prepare-supabase-ci-config.ts (studio/analytics/edge_runtime/realtime → enabled = false). The base file keeps them enabled for local dev — never commit the patched output.
• Local demo credentials are public constants, committed at scripts/ci/local-stack.env (legacy demo JWTs — deterministic per pinned CLI version; CI asserts the running stack agrees each run). They grant access only to a loopback stack.
• Config-shape caveat: the local stack runs the BASE config blocks — [remotes.*] are inert locally, so no signup hook and base-file TOTP (false), same posture as every branch mode (S497 retro).
• CLI pin: CI pins the Supabase CLI (2.109.1 — the DR-095 floor). A local version far ahead of the pin can behave differently from CI; check supabase --version when chasing a CI-only stack difference.

§8. Cross-references

• Spec: docs/audits/kh-production-readiness-phase-1/specs/wp-s5.2-cli-script-audit-spec-v1.md — canonical per-script table.
• Env-var matrix: docs/audits/kh-production-readiness-phase-1/research/09-vercel-supabase-env-mapping-spec.md §4 (Local-dev / Vercel Preview / Vercel Prod).
• Staging refresh: docs/runbooks/staging-refresh.md.
• GitHub Environments: docs/runbooks/github-environments.md.
• CLAUDE.md gotchas section — Supabase + General + Testing + Plugin / MCP.
• Project memory references:
  • feedback_branding_client_id_env.md (S196 incident: missing NEXT_PUBLIC_CLIENT_ID corrupted prod entity_mention rows).
  • feedback_classifier_eval_nondeterminism.md (destructive classifyContent({force:true}) mode unsuitable for prod).

---

§9. Changelog

• 26/07/2026 (S497) — v1.5: New §7.12 — Next 16’s per-directory dev lock (<distDir>/dev/lock) is what blocks parallel dev servers, not the port; NEXT_DIST_DIR slots .next-1...next-4 (registered literally in tsconfig.json — globs do not satisfy Next’s check), stale-server recovery, and portless as the optional named-URL layer. §7.11 corrected: node_modules is now an APFS copy-on-write clone, not a symlink (a symlink breaks package resolution); .cache no longer provisioned.
• 21/07/2026 (id-164 {164.3}) — v1.4: New §7.11 Intent workspace provisioning — Intent runs setup scripts argless (env vars only), so provision-worktree.sh gained an $WORKTREE_PATH/$MAIN_CHECKOUT fallback; thin-wrapper wiring rule, four-check verification list, and the parent .env.local drift caveat (copies never track their source; two-deep parent chain via ~/Developer/canonical).
• 09/06/2026 (kh bl-270) — v1.3: New §2.5 KH_CLIENT_NAME_DENYLIST — canonical denylist JSON shape (tokens[].{value,case_insensitive,class} + exclusion_patterns[]), the {90.8} strict-JSON parser’s invariant-35 behaviour (set-but-invalid ⇒ HTTP 500 on every patch-server mutation), and the two local options (canonical JSON in .env.local, or unset). Pointer to ops/identity-denylist.json as the one rotation point.
• 28/04/2026 (S207 WP-A4 Phase 4) — v1.2: New §5.3 “Pipeline run audit / ingest_source distribution” — three SQL queries for verifying typed-column vs content_history.metadata->>'ingest_source' parity post-backfill, and a residual-NULL guard. Resolves WP3 verifier F-2 / plan 4.2-AC4. Pairs with data-entry-points.md Appendix G (canonical pipeline_runs.pipeline_name values) added in the same commit.
• 27/04/2026 (kh-prod-readiness-S6 W3a) — v1.1: verifier findings
  • Liam D-19 to D-23 rulings folded in. Key changes: §3.1 top-10 list reframed as “explicit --env flag” (was “--env=prod” — wipe-bid-responses uses --env=staging); §4.2 Python ingest now reads .env.local post D-20=α impl; §6.4 dual-surface table extended with Railway-hosted Python row + footnote on 3 Pattern B-prime pre-loading scripts; §7.5 marked RESOLVED post-D-20=α with Railway implication; new sub-spec §9 references runbook §7’s full 10-footgun list rather than the original 5.
• 27/04/2026 (kh-prod-readiness-S6 W1) — v1: Initial draft. Documents the post-flip staging-default contract and the two opt-in patterns for hitting prod. Pending: WP-S5.2 impl pass (per-script --env=prod flag implementation per spec D-23 list) + Liam manual .env.local flip.