Production E2E Read-Only Runbook

Production E2E Read-Only Runbook

Status: Draft v1 (handover-grade). Audience: Liam + future client-onboarding operator. Owner: prod-readiness track. Pair with: docs/runbooks/staging-refresh.md §5 (the staging-side happy path) and docs/runbooks/local-development.md §3 (env-var override patterns).

---

§1. Purpose

This runbook documents the procedure to safely run a curated subset of the Knowledge Hub E2E and integration test suites against the production Supabase project (rovrymhhffssilaftdwd). It exists to mitigate three specific risks during client handover:

1. Production-only environment drift — staging is a Supabase branch of prod, but env-var differences (CRON_SECRET, app URL, Sentry DSN, OAuth redirects) can hide bugs that only surface against prod credentials. A read-only smoke pass against prod gives early warning before clients hit the bug.
2. Schema parity verification post-cutover — after a staging refresh (per docs/runbooks/staging-refresh.md) or a prod migration push, run the same read-only specs against both targets and diff the results. Divergence indicates either a stale staging branch or a prod migration that didn’t apply cleanly.
3. Pre-handover smoke verification — before handing the system over to a client, prove that the read paths render real production data correctly (correct branding, real entity counts, real domain coverage, real workspaces). Staging fixtures are synthetic; prod data may surface issues fixtures mask.

This runbook is NOT a substitute for the staging E2E run. The full suite must always run green against staging first. Prod read-only is an additional verification gate, not a replacement.

What this runbook deliberately excludes

• Any test that creates, updates, or deletes content (content_items, workspaces, bid_*, feed_*, entity_*, notifications, read_marks, source_documents, oauth_grants, governance_review).
• Any test that mutates user state (role changes, profile edits, magic-link emails, OAuth approvals, sign-ups).
• Any test that triggers AI work (digest generation, draft streaming, classification, embedding generation) — these incur cost and emit Sentry/PostHog events even when read-only-adjacent.
• Any test that uses the worker-scoped workerData fixture from e2e/fixtures/test-data-fixture.ts — that fixture seeds 12+ content items, 4 workspaces, bid questions, responses, notifications, read marks, intelligence articles, and feed sources on every worker setup and CASCADEs them on teardown. Running it against prod would corrupt the canonical KB.

---

§2. Scope: what counts as “read-only”

A test is classified safe to run against prod if and only if every single statement in the test body, all helpers it imports, every fixture it depends on, and every API endpoint it hits satisfies all of the following:

2.1 Hard requirements

1. No DB writes. No INSERT, UPDATE, UPSERT, DELETE, or RPC call to a function that writes (e.g. record_pipeline_run, apply_diff, approve_change).
2. No supabase-js write methods. No .insert(), .update(), .upsert(), .delete(), or chained .throwOnError() on a write.
3. No HTTP write methods. No page.request.post(), page.request.put(), page.request.patch(), or page.request.delete().
4. No file uploads. No setInputFiles(), no <input type="file"> interaction. File uploads create source_documents rows and Supabase Storage objects.
5. No MCP tool that writes. Per docs/generated/mcp-inventory.md, 16 of 57 MCP tools write and 2 are destructive — none of these may be invoked. Read-only tools (e.g. search_knowledge_base, get_item, coverage_overview) are safe.
6. No worker-scoped fixture. The workerData fixture from e2e/fixtures/test-data-fixture.ts writes 12 content_items, 4 workspaces, 4 bid_questions, 2 bid_responses, 2 notifications, 2 read_marks, entity mentions, entity relationships, intelligence feed sources, and feed articles on EVERY worker setup. The teardown block deletes them but a crashed worker leaves orphans. Any spec importing from '../fixtures' (the combined fixture) inherits this write footprint and is automatically NOT safe.
7. No write-button clicks. Even when no DB call is made directly in the test body, clicking buttons like “Verify”, “Flag”, “Submit”, “Approve”, “Save”, “Generate”, “Apply”, “Create” triggers an API write. Specs that click these buttons are NOT safe regardless of whether the test body itself touches the DB.
8. No intentional Sentry capture. Tests that throw on purpose to exercise error boundaries pollute the prod Sentry project with synthetic events. Confirm the spec doesn’t throw new Error() inside page contexts that get captured.
9. No global-setup / global-teardown writes. The Playwright global-teardown.ts deletes any rows where title LIKE '[E2E-%' or title LIKE '[E2E Test]%'. Against prod this only removes rows the prod data accidentally happens to match — but the safe approach is to disable global-teardown for prod runs (see §5.3).

2.2 Soft requirements (caveat-not-blocker)

• Sign-in via signInWithPassword. Calls supabase.auth.signInWithPassword do not write to public schema, but they DO write to auth.refresh_tokens / auth.sessions. These rows are bounded (each test user has at most one active session) and do not pollute application data — acceptable with the caveat documented in §3.2.
• Read-only RPC calls. RPCs like get_user_display_names, get_guide_content, get_guide_coverage, search_chunks_v2 perform no writes — safe.
• page.goto() to authenticated pages. Server-rendered pages may call analytics endpoints (PostHog page view events). PostHog is per-env; prod page views during a smoke run will appear in the prod PostHog project. Treat as expected operational telemetry, not as a write-side-effect to gate on.

---

§3. Inventory of safe tests

The classification below was produced by reading every spec file under e2e/tests/ and __tests__/integration/, then auditing each helper and fixture transitively. Test names map to file paths.

3.1 E2E specs — Safe to run against prod

These specs perform read-only navigation, render assertions, and assertions on data that already exists in the target DB. They do NOT import the worker-scoped workerData fixture for any write, do not click write buttons, and do not call write APIs.

Spec file	What it asserts	Safety rationale
e2e/tests/coverage-page.spec.ts	/coverage dashboard renders with summary cards, expandable domain sections, gap badges, navigation links to /browse, refresh button, mobile layout.	Pure page render assertions on production data. Refresh button only re-fetches; no write. Auth-only — no workerData.
e2e/tests/role-gating.spec.ts	Viewer cannot see Review/Team/Quality Review nav links. Editor sees Review but no system admin. Admin sees all.	Three role-pages render with no clicks beyond a single sidebar nav button which is a URL change. No DB writes.
e2e/tests/settings.spec.ts	/settings heading, sidebar nav, navigation between sections via URL params, profile/integrations/team section content.	One header click navigates to settings; rest is page.goto() with section query params + visibility assertions. No mutations.
e2e/tests/workspaces.spec.ts	/workspaces page shows Bids card and Sales Proposals “coming soon” card, navigation to /bid works, viewer sees same.	All assertions are visibility or href attribute checks. One click is bidsCard.click() which navigates to /bid (read).
e2e/tests/provenance.spec.ts	/provenance admin access, tab navigation, deep-linking, /activity redirects, command palette opens.	Admin reads + viewer/editor see AccessDenied + Cmd+K opens dialog. No DB writes. Note: command palette test uses keyboard input + click on dialog item.
e2e/tests/provenance-pipeline-audit.spec.ts	/provenance?tab=pipeline-health and ?tab=audit show time-range buttons, kind filter (when data exists), audit feed, “Export PDF” button + date inputs.	Pure visibility assertions; no clicks beyond beforeEach page navigation. No workerData.
e2e/tests/provenance-audit-export.spec.ts	Clicking “Export PDF” on the Audit tab triggers a download with the correct filename pattern.	The export endpoint is a SELECT-only read from verification_history. No DB write. Download stays on disk in e2e/test-results/.
e2e/tests/guide-pages.spec.ts	/guide redirects to /coverage?tab=guides, guide detail page loads with metadata, table of contents, back navigation, nonexistent guide shows error.	Pure navigation + render assertions. No DB writes.
e2e/tests/auth.spec.ts (unauthenticated only)	The baseTest.describe('Authentication — unauthenticated access') block: /login rendering, email validation, method selection back-and-forth.	Unauthenticated journey; magic-link click NOT included (sends email — see §3.2).
e2e/tests/digest-page.spec.ts (read-only subset)	Page loads with “Change Reports” heading, mode selector tabs (Period/Daily/Custom) work as URL toggles, past-reports section visibility check, custom domain/keyword filter badge rendering.	Subset only — exclude the “Generate Report” click test (it triggers AI work + DB write — see §3.3).

3.2 E2E specs — Safe with caveats

These specs are safe but require an explicit operator decision to accept the side effect documented in the rationale.

Spec file	Side effect	Cleanup expectation
e2e/tests/auth.spec.ts (authenticated session block)	Calls signInWithPassword for admin/editor/viewer test users (writes to auth.refresh_tokens, auth.sessions); the “Sign out” test invalidates the active session. Magic-link test (shows magic link confirmation after choosing magic link) calls auth.signInWithOtp which sends a real email to user@example.co.uk — the test only asserts the UI transition but the email IS dispatched.	Required setup: create prod-handover-readonly@<client-domain> test users in prod auth.users via bun run scripts/seed-e2e-users.ts against the prod project. DO NOT use the existing test.user1@test-kb-aish.co.uk accounts — those are test-domain emails seeded into the staging branch only. Cleanup: invalidate refresh tokens via auth.admin.signOut(<user-id>) after the run.
e2e/tests/dashboard.spec.ts (read-only subset)	Tests in this file rely on workerData for the bid card test and content health unhealthy indicators — those subtests MUST be excluded. The hero search test, attention section visibility, compliance section, reorientation, recent activity, viewer role, mobile layout, and the WarningsBanner negative test all only require existing prod data.	Use --grep to select only the safe subset. Excluded subtests: active bids section shows seeded bid card, active bids card links to bid detail page, quick stats strip shows unhealthy content indicators (depends on stale/expired worker fixture).
e2e/tests/intelligence-workflow.spec.ts (passive subset)	The “intelligence card appears on /workspaces page” subtest is pure read. Most others depend on workerData.intelligenceWorkspaceId (worker-seeded workspace). The viewer role-gating subtest reads articles without flag buttons — safe IF prod has any intelligence workspace at all.	Use --grep "intelligence card appears" for a true read-only subset. The viewer-flag-button test requires a known-existing workspace UUID hard-coded into the test invocation, NOT the worker-seeded one.
__tests__/integration/get-user-display-names.integration.test.ts	Pure read of the get_user_display_names RPC. Calls getTestUserId('admin') which uses auth.admin.listUsers() (read-only, but service-role).	None — the RPC has no write side. Caveat is documenting that the test reads the prod admin user list (audit log entry).
__tests__/integration/display-name-routes.integration.test.ts	Calls cacheAllTestUserSessions() which performs three signInWithPassword calls (admin/editor/viewer); these write to auth.sessions/auth.refresh_tokens. Routes themselves only POST to /api/users/display-names and GET /api/content-owners/stats — both read-only at the route handler level.	Same as auth.spec.ts caveat row — use the prod-handover-readonly@ test-user pattern + auth.admin.signOut() post-run.

3.3 NOT safe to run against prod

These specs MUST NOT be invoked against the production Supabase project. The “Why not” column maps each rejection to the §2.1 hard requirement it violates.

Spec file	Why not
e2e/tests/bid-pipeline.spec.ts	§2.1.6, §2.1.7 — POSTs /api/bids to create bid workspaces. Inserts into workspaces.
e2e/tests/bid-draft-stream.spec.ts	§2.1.6, §2.1.7, §2.1.5 — Triggers SSE draft-stream which writes bid_responses + bid_response_history, calls AI (cost).
e2e/tests/bid-export.spec.ts	§2.1.6 — Imports createTestBid and createExportReadyBid, both of which insert into workspaces and bid_responses.
e2e/tests/bid-questions.spec.ts	§2.1.6, §2.1.7 — Worker-fixture bid + edit-question buttons that PATCH bid_questions.
e2e/tests/bid-session.spec.ts	§2.1.6 — Depends on workerData.bidId (worker-seeded bid in drafting state). Session page reads its data from worker-fixture rows that don’t exist in prod; tests would fail with 404 anyway.
e2e/tests/browse-cards.spec.ts	§2.1.6 — Imports the combined '../fixtures' test which transitively wires up workerData, triggering worker-scoped seeds + teardown.
e2e/tests/browse-search.spec.ts	§2.1.6 — Same as browse-cards: combined-fixture import triggers workerData seed/teardown.
e2e/tests/content-creation.spec.ts	§2.1.1, §2.1.2 — Creates content items via /item/new form submit. Inserts into content_items + may call AI for classification.
e2e/tests/content-ingestion-upload.spec.ts	§2.1.1, §2.1.4 — Uploads files to /api/upload. Writes content_items, source_documents, Supabase Storage objects.
e2e/tests/content-ingestion-url.spec.ts	§2.1.1, §2.1.5 — POSTs to /api/ingest/url. Writes content_items, source_documents, embeddings; calls Firecrawl + OpenAI (cost).
e2e/tests/dashboard.spec.ts (full)	§2.1.6 — Includes subtests that require workerData (active bid card, unhealthy content indicators); see §3.2 for the safe subset.
e2e/tests/digest-page.spec.ts (full)	§2.1.5 (the “Generate Report” subtest only) — clicking Generate triggers digest generation via AI (cost, latency) and writes to digest_reports. Other subtests in this file are read-only — see §3.1 for the subset.
e2e/tests/document-diff.spec.ts	§2.1.6, §2.1.1 — Inserts source_documents and source_document_diffs rows in beforeEach. Apply/Dismiss buttons mutate change_proposals.
e2e/tests/entity-filters.spec.ts	§2.1.6 — Combined-fixture import triggers workerData seed/teardown.
e2e/tests/governance-review.spec.ts	§2.1.7 — Verify/Flag/Skip buttons all POST to /api/governance/review/* endpoints, mutating governance_review_log and content_history.
e2e/tests/intelligence-workflow.spec.ts (full)	§2.1.6, §2.1.7 — Worker-seeded intelligence workspace + Flag-as-irrelevant button writes to feed_articles.flagged_status + intelligence_flags. Subset of read-only tests is in §3.2.
e2e/tests/item-detail.spec.ts	§2.1.6 — All tests use workerData.articleId/qaPairId/staleItemId/expiredItemId/... directly. Items don’t exist in prod by those UUIDs.
e2e/tests/layer-suggestion.spec.ts	§2.1.1 — Creates content items end-to-end via /item/new form to trigger the banner.
e2e/tests/mcp-invocation.spec.ts	§2.1.5, §2.1.1 — Seeds a sentinel content item via createTestItem() (insert), then calls MCP tools requiring an OAuth grant (write to auth.oauth_grants via Supabase Auth).
e2e/tests/oauth-consent-flow.spec.ts	§2.1.6, §2.1.7 — Drives a real OAuth flow; clicks “Approve” which writes auth.oauth_grants, calls /api/oauth/decision, /api/oauth/revoke. Provisions OAuth client in beforeAll (admin write).
e2e/tests/provenance-per-item.spec.ts	§2.1.6 — Uses workerData.articleId for the “valid content item” subtest. The lookup-form-empty-state and not-found subtests are read-only but not worth the partition; treat the whole spec as worker-dependent.
e2e/tests/qa-library.spec.ts	§2.1.1 — beforeAll calls createTestQAPair() three times (insert into content_items); afterAll deletes them — but a prod insert leaves a window of data leakage even if cleanup succeeds.
e2e/tests/role-write-enforcement.spec.ts	§2.1.3, §2.1.6 — POSTs/PATCHes against /api/items, /api/bids, /api/upload to verify viewer cannot write. Even though it ASSERTS the writes are rejected, the API call still hits the endpoint and may produce Sentry “forbidden access” telemetry.
e2e/tests/settings-mutations.spec.ts	§2.1.3, §2.1.7 — Submits invite forms, role-change dropdowns, governance config edits.
e2e/tests/si-prompt-refinement.spec.ts	§2.1.5 — Mocks the analyse/preview API responses with page.route() BUT clicks “Apply” which calls the real /api/intelligence/.../prompts/[id] PATCH endpoint to create a new prompt version. Even with mocked previews, the apply path mutates intelligence_prompt_versions.
e2e/tests/si-starter-pack-seeding.spec.ts	§2.1.6 — Seeds a Procurement starter pack which inserts 4 feed_sources rows + creates an intelligence workspace.
e2e/tests/wave1-cert-renew.spec.ts	§2.1.6 — Worker-fixture certifications (workerData.certificationId etc.). Even the supplier-toggle subtest requires the worker-seeded compliance section data.
e2e/tests/wave1-dashboard-expiry.spec.ts	§2.1.6 — Worker-fixture certifications + compliance section.
e2e/tests/wave1-guide-sections.spec.ts	§2.1.1 — Creates content items via /item/new to trigger the guide section banner.
e2e/tests/wave1-item-detail-dates.spec.ts	§2.1.6 — Uses workerData.expiredItemId (worker-seeded item with temporal references).
__tests__/integration/admin-users.integration.test.ts	§2.1.1 — Uses auth.admin.createUser() to seed test users for role tests. Direct auth.users write.
__tests__/integration/bid-library-ingest/parity.integration.test.ts	§2.1.1 — Inserts into bid_library_items for parity comparison.
__tests__/integration/certification-bridge-flow.integration.test.ts	§2.1.1 — Inserts entity_mentions + entity_relationships to verify bridge.
__tests__/integration/golden-path-real-db.integration.test.ts	§2.1.1 — Full golden path: classify + insert content_items + entities + temporal references.
__tests__/integration/intelligence-golden-path.integration.test.ts	§2.1.1 — Inserts feed_sources, feed_articles, content_items.
__tests__/integration/items-patch-publication-status.integration.test.ts	§2.1.1 — PATCHes content_items.publication_status.
__tests__/integration/publication-status-migration.integration.test.ts	§2.1.1 — Verifies migration; inserts test content_items.
__tests__/integration/publication-status-trigger.integration.test.ts	§2.1.1 — Triggers PG triggers via insert/update on content_items.
__tests__/integration/qa-editor-create-post-populates-answer-standard.integration.test.ts	§2.1.1, §2.1.3 — POSTs new q_a_pair items via /api/items.
__tests__/integration/qa-editor-chunk-parity-and-regen.integration.test.ts	§2.1.1 — POST + PATCH q_a_pair + regenerate chunks + embeddings.
__tests__/integration/qa-editor-patch-content-shape-reconciliation.integration.test.ts	§2.1.3 — PATCHes existing q_a_pair items (no insert but mutates).
__tests__/integration/review-cadence-lifecycle.integration.test.ts	§2.1.1 — Inserts content_items with various review cadences to test lifecycle.
__tests__/integration/si-google-news-dedup.integration.test.ts	§2.1.1 — Inserts feed_articles to test dedup logic.
__tests__/integration/supersession-filter.integration.test.ts	§2.1.1 — Inserts content_items with parent_id chains.
scripts/seed-e2e-users.ts	OUT OF SCOPE — script writes to auth.users and user_roles. Per its own header comment (“ALWAYS-STAGING. … NEVER invoke against prod”), this script must never target the prod project. The handover read-only run uses pre-seeded prod-handover-readonly@ users instead (see §3.2).

3.4 Mock-only integration tests (run anywhere safely)

These tests use createMockSupabaseClient() and never touch a real DB. They can run against any env without effect — but running them against prod credentials gains nothing because they don’t read prod either. Listed for completeness:

• __tests__/integration/classification-entity-certification-flow.test.ts
• __tests__/integration/content-to-guide.test.ts
• __tests__/integration/golden-path-e2e.test.ts

Skip these in the prod handover run — there’s no prod-vs-staging signal.

---

§4. Pre-flight checklist

Before invoking any test against prod, complete every item below. The checklist exists so the post-incident “did we forget X” question has a definitive answer.

4.1 Environment

• Prod DB snapshot timestamp recorded. Note the most recent Supabase backup timestamp via the Supabase dashboard (rovrymhhffssilaftdwd → Database → Backups). Record it in your session log — this is the rollback floor if anything goes wrong.
• .env.local does NOT silently shadow your overrides. Per docs/runbooks/local-development.md §7.6, the Supabase CLI’s .temp/project-ref may be stale. Verify with cat supabase/.temp/project-ref. CLI scripts read .env.local first, so always invoke prod commands with the env override prefix pattern (see §5.1).
• No leaked prod env in shell. Run env | grep -i supabase — if SUPABASE_URL is exported and points at prod, you have a shell-level export that will shadow .env.local for any subsequent test command. unset it before the run if you want a controlled invocation.
• prod-handover-readonly@ test users exist in prod. Run bun run scripts/seed-e2e-users.ts --check against the prod project (per the script’s --check flag) to confirm. If they don’t exist, see §4.5.
• Disable Sentry capture for the test process. Set NEXT_PUBLIC_SENTRY_DSN="" and SENTRY_DSN="" in the invocation environment. Test renders that fall back to the error boundary (e.g. a 404 page) WILL otherwise capture a synthetic event in the prod Sentry project.

4.2 Operational

• On-call notified. Post in the Knowledge Hub Slack channel: “Running prod E2E read-only smoke at HH:MM. Expected duration N minutes.” Even a read-only run will increase prod request volume; on-call needs to know it’s not a real incident.
• Freeze window respected. Do NOT run during a client demo window or scheduled cron-job window (see Vercel cron schedule in vercel.json). The cron jobs may write to the same tables your reads sample.
• No concurrent client activity. Verify in PostHog dashboard (or Vercel analytics) that there’s no active user session in the last 5 minutes. Reading prod while a client edits is fine from a correctness POV but introduces analytical noise that makes debugging harder if the test catches a bug.

4.3 Test-suite preconditions

• Staging E2E suite is GREEN. Run bun run test:e2e against staging FIRST. A red staging run before a prod read-only run means you can’t distinguish staging-only flakes from prod-specific issues. The whole point of prod read-only is to catch deltas; you need a known-good baseline.
• Staging integration suite is GREEN. Run bun run test:integration against staging. Same rationale.

4.4 Network

• VPN connected if required. If your client onboarding runbook requires VPN-on for prod admin operations, confirm the tunnel is active. (For the current Knowledge Hub deployment this is not required — included as a placeholder for future client deployments.)

4.5 Test user provisioning (one-time per prod project)

If prod-handover-readonly@ users don’t yet exist in the prod project, provision them once via a temporary modification:

# 1. Confirm you're targeting prod (NOT staging)
echo $SUPABASE_URL  # MUST show rovrymhhffssilaftdwd.supabase.co

# 2. Edit scripts/seed-e2e-users.ts temporarily to use these emails
#    (do NOT commit this change):
#       prod-handover-readonly-1@<client-domain>
#       prod-handover-readonly-2@<client-domain>
#       prod-handover-readonly-3@<client-domain>
#    AND comment out the script's "this is staging?" assertion.

# 3. Run with explicit prod env:
SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  bun run scripts/seed-e2e-users.ts

# 4. Revert the script edit. NEVER commit the prod-targeted version.

# 5. Set the matching passwords in your invocation env:
export PROD_TEST_USER_1_EMAIL=prod-handover-readonly-1@<client-domain>
export PROD_TEST_USER_1_PASSWORD=<generated-pw>
# (TEST_USER_2/3 similarly)

Why prefixed emails — using prod-handover-readonly@ rather than test.user1@test-kb-aish.co.uk:

1. The test-kb-aish.co.uk domain is reserved for the staging branch’s E2E users; mixing them across envs causes auth confusion if a future migration cross-references emails.
2. The prefix makes it immediately obvious in the prod auth.users table that these accounts exist for handover validation, not for real human use — easy to filter for cleanup.
3. The prod auth hook hook_restrict_signup_to_phew_domain (per docs/runbooks/staging-refresh.md §4.3) only restricts unauthenticated signups; service-role provisioning bypasses it, but using a non-client-domain email is a defensive layer.

---

§5. Invocation steps

5.1 Single-spec invocation pattern (preferred)

Run one safe spec at a time so a failure doesn’t cascade. Use the explicit-env-override pattern from docs/runbooks/local-development.md §3.2:

# Single E2E spec against prod, read-only:
SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  NEXT_PUBLIC_SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_PUBLISHABLE_KEY=$KH_PROD_PUBLISHABLE_KEY \
  NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY=$KH_PROD_PUBLISHABLE_KEY \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  TEST_USER_1_EMAIL=$PROD_TEST_USER_1_EMAIL \
  TEST_USER_1_PASSWORD=$PROD_TEST_USER_1_PASSWORD \
  TEST_USER_2_EMAIL=$PROD_TEST_USER_2_EMAIL \
  TEST_USER_2_PASSWORD=$PROD_TEST_USER_2_PASSWORD \
  TEST_USER_3_EMAIL=$PROD_TEST_USER_3_EMAIL \
  TEST_USER_3_PASSWORD=$PROD_TEST_USER_3_PASSWORD \
  PLAYWRIGHT_BASE_URL=https://knowledge-hub-seven-kappa.vercel.app \
  NEXT_PUBLIC_SENTRY_DSN="" \
  SENTRY_DSN="" \
  bunx playwright test e2e/tests/coverage-page.spec.ts \
    --project chromium-desktop

Notes on the invocation:

• PLAYWRIGHT_BASE_URL points at the prod deployment, not localhost. The default playwright.config.ts webServer block boots a local bun dev server when PLAYWRIGHT_BASE_URL is not set. For prod read-only we want the deployed prod app, NOT a local dev server — that’s the whole point of catching deployment-environment drift.
• --project chromium-desktop runs only one viewport. The mobile project doubles every test’s writes (each browser project triggers its own auth.setup.ts sign-in). For a smoke run, desktop is enough.
• Single spec. Do NOT pass a glob. The grep-filtering pattern in §5.2 is for running multiple safe specs but with explicit subset control.
• NEXT_PUBLIC_SENTRY_DSN="" suppresses Sentry capture on the client; SENTRY_DSN="" suppresses it on the Playwright test process itself. Both required.

5.2 Multi-spec invocation with grep filter

If you trust a curated list of specs (per §3.1), run them via Playwright’s file-pattern argument:

# Run all §3.1 safe E2E specs:
SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  NEXT_PUBLIC_SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_PUBLISHABLE_KEY=$KH_PROD_PUBLISHABLE_KEY \
  NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY=$KH_PROD_PUBLISHABLE_KEY \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  TEST_USER_1_EMAIL=$PROD_TEST_USER_1_EMAIL \
  TEST_USER_1_PASSWORD=$PROD_TEST_USER_1_PASSWORD \
  TEST_USER_2_EMAIL=$PROD_TEST_USER_2_EMAIL \
  TEST_USER_2_PASSWORD=$PROD_TEST_USER_2_PASSWORD \
  TEST_USER_3_EMAIL=$PROD_TEST_USER_3_EMAIL \
  TEST_USER_3_PASSWORD=$PROD_TEST_USER_3_PASSWORD \
  PLAYWRIGHT_BASE_URL=https://knowledge-hub-seven-kappa.vercel.app \
  NEXT_PUBLIC_SENTRY_DSN="" \
  SENTRY_DSN="" \
  bunx playwright test \
    e2e/tests/coverage-page.spec.ts \
    e2e/tests/role-gating.spec.ts \
    e2e/tests/settings.spec.ts \
    e2e/tests/workspaces.spec.ts \
    e2e/tests/provenance.spec.ts \
    e2e/tests/provenance-pipeline-audit.spec.ts \
    e2e/tests/provenance-audit-export.spec.ts \
    e2e/tests/guide-pages.spec.ts \
    --project chromium-desktop

For specs with subset filtering (§3.2 caveats), use --grep:

# Dashboard read-only subset (excludes worker-fixture-dependent tests):
bunx playwright test e2e/tests/dashboard.spec.ts \
  --grep "hero search|attention section|compliance status|reorientation|recent activity|viewer role|mobile layout|warnings banner is hidden" \
  --project chromium-desktop

5.3 Disable global-teardown for the prod run

Even though the global-teardown.ts only deletes rows where title LIKE '[E2E-%' (which prod data shouldn’t contain), the safe default is to disable it entirely for prod runs. Either:

Option A — env-flag override (preferred, requires no edit):

The teardown script can be conditionally skipped by setting an env var that gates the cleanup block. Add this to the invocation:

# Prepend to the invocation envs above:
E2E_SKIP_GLOBAL_TEARDOWN=true \
  SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co ...

Note: at the time of writing this runbook the E2E_SKIP_GLOBAL_TEARDOWN env var is NOT yet honoured by e2e/global-teardown.ts. The variable name is reserved here so that when the teardown is hardened (a small WP-S8h follow-up), the runbook already documents the contract.

Option B — temporary edit (until Option A lands):

Comment out the body of globalTeardown() in e2e/global-teardown.ts before the run. Revert immediately after. Do NOT commit the gutted version.

5.4 Integration test invocation

For the safe integration tests in §3.1 / §3.2:

SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  NEXT_PUBLIC_SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  SUPABASE_PUBLISHABLE_KEY=$KH_PROD_PUBLISHABLE_KEY \
  NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY=$KH_PROD_PUBLISHABLE_KEY \
  SUPABASE_SERVICE_ROLE_KEY=$KH_PROD_SERVICE_ROLE_KEY \
  TEST_USER_1_PASSWORD=$PROD_TEST_USER_1_PASSWORD \
  TEST_USER_2_PASSWORD=$PROD_TEST_USER_2_PASSWORD \
  TEST_USER_3_PASSWORD=$PROD_TEST_USER_3_PASSWORD \
  bun run test:integration \
    __tests__/integration/get-user-display-names.integration.test.ts \
    __tests__/integration/display-name-routes.integration.test.ts

The integration test runner (vitest.integration.config.ts) does NOT boot a Playwright browser, so PLAYWRIGHT_BASE_URL is not needed. Routes under test are mounted in-process via next/headers-mocked cookie shims (see __tests__/integration/helpers/auth-session.ts).

---

§6. Rollback expectations

If a “safe” test surprises us by mutating prod state (e.g. a regression introduces a write into a previously read-only path, or a misclassified spec slipped past §3 review), follow this checklist:

6.1 Forensic capture (do this first, before remediation)

1. Stop the test runner immediately. Ctrl-C the Playwright / Vitest process. Don’t let further mutations queue up.
2. Capture the current auth.users count, content_items count, workspaces count, bid_responses count, and pipeline_runs count. Use mcp__supabase__execute_sql against the prod project:

   SELECT 'auth_users' AS table, count(*) AS rows FROM auth.users
   UNION ALL SELECT 'content_items', count(*) FROM content_items
   UNION ALL SELECT 'workspaces', count(*) FROM workspaces
   UNION ALL SELECT 'bid_responses', count(*) FROM bid_responses
   UNION ALL SELECT 'pipeline_runs', count(*) FROM pipeline_runs
   UNION ALL SELECT 'content_history', count(*) FROM content_history
   UNION ALL SELECT 'governance_review_log', count(*) FROM governance_review_log;

3. Diff against the snapshot from §4.1 pre-flight. Any non-zero delta on a non-auth_users row is a real mutation that needs rollback. (The auth_users count may legitimately drift by 1-3 from sign-in session writes per §3.2 caveat.)
4. Save the Playwright HTML report. Screenshot the test results page from playwright-report/index.html — the trace files are the primary forensic record.

6.2 Identifying the culprit

Use the timeline:

• The pre-flight snapshot timestamp from §4.1.
• The test runner kick-off timestamp from your shell history.
• The current timestamp.

Query Supabase for inserts/updates within that window:

SELECT id, created_at, created_by, title, content_type
FROM content_items
WHERE created_at BETWEEN $kickoff_ts AND $now_ts
ORDER BY created_at DESC;

The created_by column should map to one of the prod-handover-readonly-N@ user UUIDs (per §4.5 setup) or the service account UUID (a0000000-0000-4000-8000-000000000001) — that’s how you trace back to the test that wrote it.

6.3 Revert path

For each row found in §6.2:

1. Verify it’s safe to delete — confirm via timestamp + creator that it’s a test-run artifact, not a coincidental client write.
2. Soft-delete first if the table supports it. Most app tables have archived_at; set that and re-query downstream views to confirm nothing breaks before hard-deleting.
3. Hard-delete in FK-safe order. For content_items deletions:
   • Delete from read_marks WHERE content_item_id = $id
   • Delete from content_item_workspaces WHERE content_item_id = $id
   • Delete from entity_mentions WHERE content_item_id = $id
   • Delete from entity_relationships WHERE source_item_id = $id
   • Delete from content_history WHERE content_item_id = $id
   • Delete from content_chunks WHERE content_item_id = $id
   • Finally, delete from content_items WHERE id = $id
4. Re-snapshot and verify counts are back to baseline.

If counts diverge by >10 rows, escalate — do not attempt mass deletion manually. Restore from the §4.1 snapshot via the Supabase dashboard backup UI. The current Knowledge Hub deployment is small enough (< 1000 content_items) that point-in-time restore is fast.

6.4 Incident comms template stub

Subject: Prod E2E read-only run side effect — N rows mutated

What happened:
- Ran §X.Y safe-classified test against prod at HH:MM.
- Test wrote N rows to <table> (expected: 0).
- Affected user: <prod-handover-readonly-N@…>.

Impact:
- <Brief: did this affect a client view? cron job? other?>
- No client data was deleted. Test-created rows were removed at HH:MM.

Root cause:
- <The spec's classification was wrong, or>
- <A regression introduced a write into a previously read-only path.>

Remediation:
- Test removed from prod-safe list (PR #NNN) pending re-classification.
- Counts restored to baseline (verified at HH:MM via re-snapshot).

Follow-up:
- Update §3 of docs/handover/prod-e2e-readonly.md to reflect new classification.
- File guard test in __tests__/build/ that asserts the spec is excluded.

---

§7. Post-run verification

After every prod read-only run, run these checks. They take 5 minutes total and catch the failure modes that don’t show up in the test report (silent state pollution, cron-job collisions, security regressions).

7.1 Supabase advisor check

mcp__supabase__get_advisors{project_id:"rovrymhhffssilaftdwd", type:"security"}
mcp__supabase__get_advisors{project_id:"rovrymhhffssilaftdwd", type:"performance"}

Compare against the pre-flight snapshot (§4.1). New advisors (particularly security ones) may indicate the test created a row that RLS doesn’t cover correctly, or invoked a code path that exposed a table without the right policy.

7.2 pipeline_runs table check

SELECT id, kind, status, started_at, finished_at, items_created
FROM pipeline_runs
WHERE started_at > $pre_flight_ts
ORDER BY started_at DESC
LIMIT 20;

A safe read-only run should produce ZERO new pipeline_runs rows. If new rows appear:

• kind = cron → not your fault; a scheduled cron fired during the window. Confirm via vercel.json cron schedule.
• kind = manual_ingest / classification / etc. → a test triggered a write path. Map the row’s started_at back to the test runner timeline and identify the culprit spec.

7.3 content_history row-count check

SELECT count(*) FROM content_history
WHERE created_at > $pre_flight_ts;

content_history is the audit log for content_items mutations. A safe read-only run produces ZERO new rows. Any non-zero count means content was mutated (insert, update, or governance status change) — investigate immediately.

7.4 auth.audit_log_entries check (optional, deeper)

For very paranoid runs, query Supabase Auth’s audit log:

SELECT id, payload, created_at, ip_address
FROM auth.audit_log_entries
WHERE created_at > $pre_flight_ts
ORDER BY created_at DESC;

Expect: one row per signInWithPassword call (one per role per spec file), plus one row per signOut if the auth.spec.ts sign-out test ran. Anything else (user creation, email change, password reset) is unexpected.

7.5 Sentry & PostHog spot-check

Open the prod Sentry project and confirm no new issues appeared in the test run window. PostHog page-view events are expected (one per page.goto()); error events are not.

---

§8. Known caveats and gotchas

8.1 workerData fixture is lazy but transitively imported

e2e/fixtures/index.ts extends e2e/fixtures/test-data-fixture.ts to compose authenticatedPage + editorPage + viewerPage + workerData into a single test object. Playwright fixtures are lazy: workerData only runs when a test destructures it (async ({ workerData }) => …). Specs that import from '../fixtures' but never reference workerData in their test bodies do NOT trigger the worker-scoped seed/teardown.

Verification step before running any spec against prod:

grep -E "(workerData|prefix)" e2e/tests/<spec>.spec.ts

If the only matches are JSDoc comments (per e2e/tests/workspaces.spec.ts:10) and not in test-body destructures, the spec is safe per this dimension. If the test body contains async ({ ..., workerData }) — even in a single subtest — the worker fixture runs for the whole worker and writes the full 12-item + 4-workspace + bid + intelligence-articles seed.

The e2e/fixtures/auth.ts file provides an auth-only fixture with no workerData — future read-only-safe specs should import from '../fixtures/auth' instead of '../fixtures' to make the contract explicit, but the current '../fixtures' import is harmless when no test body uses workerData.

8.2 Playwright webServer boots bun dev by default

Per playwright.config.ts:59-64, when PLAYWRIGHT_BASE_URL is not set, Playwright spawns bun dev on port 3000 and runs tests against localhost. This means a naive invocation runs against your local dev server connected to whatever DB .env.local points at — which is staging post-WP-S5.2. Always set PLAYWRIGHT_BASE_URL explicitly when targeting prod (per §5.1). Forgetting this turns a “prod read-only” run into “local dev server pointed at prod DB” — which adds extra failure modes (compilation errors, dev-only overlays, HMR injecting test data).

8.3 process.env.NEXT_PUBLIC_E2E = 'true' in playwright.config.ts

Per playwright.config.ts:6, the config sets NEXT_PUBLIC_E2E=true unconditionally. This disables some non-essential overlays (CopilotKit Web Inspector etc.). Against prod this is fine — the overlay is client-side-only and the prod build doesn’t ship dev overlays anyway. But know that the test runner is running with this signal asserted.

8.4 auth.setup.ts runs THREE sign-ins per project per run

The setup project signs in as admin, editor, AND viewer regardless of which spec requires which role. So even a single-spec run that only uses viewerPage triggers three signInWithPassword calls. Each is a write to auth.refresh_tokens / auth.sessions. Stay below Supabase auth’s rate limit (~30 req/5min per IP) by not running the suite twice in quick succession.

8.5 Conditional if (visible) fallbacks silently pass on empty DBs

Per CLAUDE.md gotcha and feedback_e2e_conditional_false_pass.md, some older specs use if (await x.isVisible().catch(() => false)) patterns that silently pass when the asserted element is missing. Against prod with real data populated this isn’t a problem — the assertions will fire honestly. But if you’re running against a freshly-cloned prod backup that’s missing recent data, those specs may report PASS when the absence-of-data should have failed them. Prefer hard await expect(x).toBeVisible() assertions; the §3.1 inventory already excludes the worst offenders.

8.6 The S196 BRANDING incident pattern

Per feedback_branding_client_id_env.md, missing NEXT_PUBLIC_CLIENT_ID causes BRANDING to fall back to “Knowledge Hub” — and the classifier script that read holder-derived strings then poisoned the prod entity_mentions table. Against the prod app deployment this is fine (Vercel sets the env). But if you ever proxy a local dev build at prod, double-check NEXT_PUBLIC_CLIENT_ID=phew is set in your invocation env.

8.7 Bun fetch HTTP 204 sandbox hang

Per CLAUDE.md gotcha, supabase-js .update()/.insert()/.upsert()/.delete() without .select() returns 204 which Bun hangs on in the Claude Code sandbox. This affects integration tests if they ever fall back to a write path (which the §3 safe set forbids). If a test hangs at exactly 30s on a supabase call, this is the cause — kill the process and re-confirm the spec is in the safe set.

8.8 Prod cron jobs may collide with read assertions

The Vercel cron schedule (per vercel.json) writes pipeline_runs, feed_articles, content_history rows on a schedule. If a cron runs DURING your read-only window, the post-run §7.2 check will show non-zero new pipeline_runs rows — that’s NOT a test bug, it’s expected cron activity. Cross-reference timestamps before escalating.

8.9 Worker-scoped fixture cleanup is best-effort

Even when a spec correctly uses workerData, the teardown block at e2e/fixtures/test-data-fixture.ts:505-552 runs only if the worker exits cleanly. A crashed Playwright worker (e.g. browser process OOM) leaves orphan rows behind. This is the reason the §3.3 list excludes all worker-using specs from prod — the cleanup contract is too brittle for the prod KB.

---

§9. Cross-references

• Staging refresh procedure: docs/runbooks/staging-refresh.md (the canonical reset path for the staging branch).
• Local dev env-override patterns: docs/runbooks/local-development.md §3 (Pattern A --env=prod flag, Pattern B explicit env override).
• GitHub Environments setup: docs/runbooks/github-environments.md (Vercel env-var mapping and case-sensitive Production/Staging environments).
• CI runbook: docs/runbooks/ci.md (the CI-side read-only guarantees and how this runbook differs).
• CLAUDE.md “E2E / Playwright” gotchas: browser install, mobile viewports, auth timing, conditional fallbacks.
• Spec backing: WP-S8g follow-up captured in docs/audits/kh-production-readiness-phase-1/STATUS.md (S10 Wave 1).
• Project memory references:
  • feedback_e2e_no_workarounds.md — E2E tests must validate real behaviour; no weakened assertions.
  • feedback_e2e_conditional_false_pass.md — conditional fallbacks silently pass; prefer hard expects.
  • feedback_integration_test_location.md — *.integration.test.ts must live under __tests__/integration/**.
  • feedback_test_runners_split.md — bun run test excludes integration; use bun run test:integration.

---

§10. Changelog

• 28/04/2026 (kh-prod-readiness-S10 W1) — v1: Initial draft. WP-S8g follow-up. Classifies all 39 E2E specs and 21 integration tests by mutation footprint. Documents pre-flight checklist, invocation patterns (single-spec preferred), forensic rollback procedure, post-run verification queries, and 9 known caveats. The E2E_SKIP_GLOBAL_TEARDOWN env-var contract is reserved here for the WP-S8h teardown hardening follow-up.