ID-128 PLAN — E2E nightly lane restructure (run #9 fix)

PLAN — E2E nightly lane restructure

Authored as {128.4} PLAN by the Task Planner (S421), formalising the ratified design scratchpad/id-128-e2e-restructure-design.md (e2e-redesign Plan agent, grounded in playwright-best-practices ci-cd/performance/global-setup). Liam has ratified the design: target = LOCAL prod build (not deployed Vercel); ephemeral Supabase branch deferred to the last slice. This document captures the design faithfully and decomposes it into the six ordered implementation Subtasks {128.6}–{128.11} (slices a–f).

Scope & provenance. This PLAN restructures the E2E nightly lane only — the e2e-desktop job in .github/workflows/e2e-nightly.yml, created earlier under {128.1} (PR #72, S406). It is a focused follow-on to the broader Task ID-128 (“wire orphaned integration/e2e suites into CI”), whose evidence basis is the S403 coverage audit (specs/id-128-test-ci-integration/S403-coverage-audit.md). The audit-derived subtasks {128.1}–{128.5} are a separate concern; this lane-stabilisation work is the run #9 remediation and lands as new sibling subtasks {128.6}–{128.11}.

---

0. Context / Problem — nightly run #9 diagnosis (taken as given)

Nightly run #9 of the full chromium-desktop project: 116 failed, 60 did-not-run, timeout-truncated (50-min budget exhausted before the suite completed).

Root cause (ratified): playwright.config.ts webServer.command: 'bun dev' (next dev --turbopack = compile-on-first-hit) combined with --workers=3 on a 2-core GitHub runner. The first navigation to each uncompiled route blocks on JIT compilation; with three workers hammering cold routes in parallel, the per-navigation budget (navigationTimeout: 15_000) is exceeded en masse → a 196 × page.goto: Timeout 15000ms storm → ~660 cascade failures → the 50-min --global-timeout=3000000 budget is exhausted so ~60 specs never run.

Two compounding factors behind the residual failures:

1. Ambient-data coupling. ~96 of the failures are specs asserting against ambient staging content (e.g. a browse-search relevance assertion expecting score > 0.9 that returned 0 rows; an admin-dedup spec looking for a “Resolve near-duplicate pair” that no longer existed). These break whenever staging data drifts — independent of the compile storm.
2. Cross-track DDL churn. Run #9 overlapped an interactive human session on ID-130 that was mutating api.* views on the shared staging DB. The integration-staging-e2e concurrency group did not protect against it because that human session was not a CI job in the group.

Current evidence in the tree (grounded against HEAD):

Symptom source	File:line	Current state
next dev server (compile-on-first-hit)	playwright.config.ts:123	command: 'bun dev'
Smoke-tuned caps overridden by magic CLI flags	e2e-nightly.yml:160-168	--global-timeout=3000000 --max-failures=0 --workers=3 --retries=1
Config caps (env-unaware)	playwright.config.ts:67-68	globalTimeout/maxFailures keyed only on CI
Worker seed prefix (not shard-aware)	e2e/fixtures/test-data-fixture.ts:109	`[E2E-W${workerInfo.workerIndex}]`
NEXT_PUBLIC_E2E set at runner-process level only	playwright.config.ts:6	process.env.NEXT_PUBLIC_E2E = 'true'
Bid-* temporarily excluded (id-130 coupling)	e2e-nightly.yml:159, playwright.config.ts:30-38	E2E_EXCLUDE_BID: "1"

---

1. Pre-ratification empirical verification (Playwright external-library APIs)

Per the OQ-3 / Q-EX2 forcing function, the Playwright APIs this PLAN mandates were import-and-call verified against the pinned version before this spec was returned.

• Date: 25/06/2026
• Pinned version: @playwright/test ^1.60.0 (package.json); installed 1.60.0.

Symbol path	Result	Evidence (pinned tree)
FullConfig.shard → null | { total: number; current: number } (one-based)	PRESENT	node_modules/playwright/types/test.d.ts:2097 (config option: :1653)
['blob'] reporter (['blob'] | ['blob', BlobReporterOptions])	PRESENT	test.d.ts:38
playwright merge-reports CLI (“merge multiple blob reports (for sharded tests) into a single report”)	PRESENT	node_modules/.bin/playwright merge-reports --help
--shard=<current>/<total> CLI flag (one-based)	PRESENT	shard config option, test.d.ts:1637-1662

workerInfo.config.shard (used by the shard-aware seed prefix) resolves to the verified FullConfig.shard shape — shard.current is valid and one-based. No ABSENT / SIGNATURE_DRIFT / BEHAVIOUR_DRIFT.

---

2. Code-intelligence orientation

gitnexus_query({query: 'e2e playwright nightly test data seeding fixture worker prefix', repo: 'canonical'}) returned no execution flows (processes: [], process_symbols: []) — verbatim. This is expected and honest: the mandated changes live in (a) Playwright config objects (defineConfig({…})), (b) a worker-scoped fixture closure (test.extend({ workerData: [...] }) in test-data-fixture.ts — the seed-prefix is a const inside the closure, not a graph-indexed named symbol), and (c) workflow YAML. None are call-graph nodes.

gitnexus_context({name: 'globalTeardown', repo: 'canonical', file_path: 'e2e/global-teardown.ts'}) confirms teardown fan-out: globalTeardown (test.d.ts global-teardown.ts:72) calls cleanupContentItemsByTitlePrefix + cleanupContentItemsByDedupFixtureTag — the prefix-sweep path that the shard-aware prefix (slice b) must remain compatible with (the [E2E-S{n}-W{w}] form still begins [E2E- so a broad [E2E-% prefix sweep keeps catching it; verify the teardown match is not pinned to the exact [E2E-W literal).

ast-dataflow / ccc fallback not required — the change set is fully enumerated above by direct file:line reads, not symbol discovery.

---

3. Ratified decisions (7)

1. Serve a LOCAL PRODUCTION BUILD (bun run build → next start), not bun dev. Build ONCE in a fan-in job, ship .next as an artifact to each shard.
2. Shard N=4 across a matrix, workers=2 per shard (8-way). Each shard runs its own setup project + next start.
3. Fail-fast: per-shard --max-failures=4 + per-shard --global-timeout ≈ 18 min + job timeout-minutes: 20. Drive the caps from ENV in the config so smoke keeps its 5 / 12 min defaults.
4. Shard-aware seed prefix (workerInfo.config.shard.current) + make ambient-data specs self-seeding (test-philosophy.md §2.1). End-state: an ephemeral Supabase branch per run.
5. Nightly hits the LOCAL prod build on the runner, NOT the deployed Vercel staging URL. Deployed Vercel cannot bake NEXT_PUBLIC_E2E and cannot pin to the run’s commit (staging is deploy-only and lags main) — both disqualifying. Keep the deployed URL as an opt-in PLAYWRIGHT_BASE_URL for occasional “is prod up” checks only.
6. Single shared playwright.config.ts; every change is a no-op for smoke by default (env-gated). The PR-blocking smoke gate is untouched day one.
7. Six ordered slices {128.a}–{128.f}, highest-leverage / lowest-risk first. The prod-build switch alone (slice a) likely clears most of run #9.

---

4. Per-slice plan (concrete deltas)

{128.6} (slice a) — Prod-build switch, single runner [no deps]

The highest-leverage, lowest-risk change: it kills the 196 page.goto timeouts + the ~660 cascade by serving pre-compiled routes, and touches one workflow file + one config line. No sharding yet.

• playwright.config.ts:123 — env-drive the web-server command: command: process.env.PLAYWRIGHT_WEB_SERVER_CMD ?? 'bun dev' (smoke/local unchanged; nightly sets 'bun run start' = next start).
• e2e-nightly.yml — add a Build (Next.js prod) step BEFORE the Playwright run step. Its env: block mirrors the quality job’s build step (ci.yml:126-153): NODE_OPTIONS, the server build secrets, and all NEXT_PUBLIC_* staging vars (ci.yml:140-144), plus NEXT_PUBLIC_E2E: "true". Step runs bun run build.
• e2e-nightly.yml run step — add PLAYWRIGHT_WEB_SERVER_CMD: "bun run start" to the existing run-step env: (the server env block already present feeds next start, which validates env at module load via lib/env-server.ts). The run line otherwise unchanged for this slice (single runner).

★ LOAD-BEARING (verbatim): NEXT_PUBLIC_E2E is client-inlined (lib/env-client.ts:114, literal process.env.NEXT_PUBLIC_E2E accessor) → it is BAKED AT next build, not at next start. So NEXT_PUBLIC_E2E=true + every NEXT_PUBLIC_* staging var MUST be exported in the BUILD step (mirror ci.yml:140-144). If set only at start, the test-mode overlay / integration-suppression (e.g. CopilotKit disable) won’t be in the served bundle. This is the one thing that silently breaks if missed. Note playwright.config.ts:6 still sets NEXT_PUBLIC_E2E for the runner process — harmless, but it does NOT affect the pre-built served bundle.

{128.7} (slice b) — Shard ×4 + build-once artifact + shard-aware seed prefix [deps: a]

Sharding (separate machines) is the real parallelism; a 2-core runner caps in-machine at ~2. ~40 specs / 4 ≈ 10/shard ≈ 12 min/shard; +4 build +2 merge ⇒ ~18 min wall vs 50.

• e2e-nightly.yml → 3 jobs:
  • build — run bun run build (full Staging build env incl NEXT_PUBLIC_E2E=true); tar --exclude='.next/cache' -czf /tmp/e2e-next-build.tar.gz .next; upload-artifact (retention-days: 1, if-no-files-found: error). Reuse the mcp-build tar pattern (ci.yml:525-534); cache .next/cache like the quality job (ci.yml:90-100). ~3–4 min.
  • e2e-shard — needs: [build]; strategy: { fail-fast: false, matrix: { shardIndex: [1,2,3,4], shardTotal: [4] } }. Download + untar the artifact. Run bunx playwright test --project=chromium-desktop --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --workers=2 --retries=1. web-server = bun run start. Each shard’s setup project authenticates the four users and writes e2e/.auth/*.json locally; the destructive sign-out auth.spec lands in ONE shard, isolated by TEST_USER_4.
  • merge-reports — needs: [e2e-shard], if: ${{ !cancelled() }}. Download all blob artifacts; bunx playwright merge-reports --reporter html ./all-blob-reports; upload the aggregated HTML. Lane pass/fail = AND of the shards (any matrix-job failure fails the workflow).
• playwright.config.ts reporter — env-gate the blob reporter so smoke is unaffected: reporter: process.env.PW_BLOB_REPORT ? [['blob']] : (process.env.CI ? [['html'],['list'],['github']] : [['html'],['list']]). The shard job sets PW_BLOB_REPORT=1.
• test-data-fixture.ts:109 — shard-aware seed prefix:

  const shard = workerInfo.config.shard; // FullConfig.shard: null | { current, total }
  const prefix = shard
    ? `[E2E-S${shard.current}-W${workerInfo.workerIndex}]`
    : `[E2E-W${workerInfo.workerIndex}]`; // null → unchanged for smoke/local

  The same ${prefix} flows through the stale-data cleanup (:215) and the worker teardown (:602); extend the orphan sweep / teardown match (global-setup.ts step 0 + global-teardown.ts prefix sweep) to keep catching the [E2E-S… form (broad [E2E-%).

★ LOAD-BEARING (verbatim): the shard seed-prefix collision — test-data-fixture.ts:109 uses [E2E-W${workerIndex}]; each shard restarts workerIndex at 0, so shard-1-W0 and shard-2-W0 both seed [E2E-W0] → COLLISION on the shared staging DB. The shard component (workerInfo.config.shard.current) is what makes the per-shard datasets disjoint. null shard (smoke/local) → prefix unchanged.

{128.8} (slice c) — Fail-fast budgets, env-driven [deps: b]

Single source of truth in the config, env-overridable; drop the magic CLI flags.

• playwright.config.ts:67-68:

  globalTimeout: process.env.PW_GLOBAL_TIMEOUT_MS
    ? Number(process.env.PW_GLOBAL_TIMEOUT_MS)
    : (process.env.CI ? 12 * 60_000 : undefined),
  maxFailures: process.env.PW_MAX_FAILURES
    ? Number(process.env.PW_MAX_FAILURES)
    : (process.env.CI ? 5 : 0),

• e2e-nightly.yml shard run-step env: — set PW_MAX_FAILURES: "4" and PW_GLOBAL_TIMEOUT_MS: "1080000" (18 min). Remove --global-timeout=3000000 --max-failures=0 from the run line (now config-driven). Job timeout-minutes: 20 per shard.
• A sick server bails a shard in ~2 min (max-failures 4); up to ~16 across shards before all bail — but on a healthy run shards are disjoint so coverage stays full. Smoke env-unset → keeps 5 / 12 min.

{128.9} (slice d) — Data-determinism: self-seeding specs [deps: b]

The real run #9 residual triage, folded in. Per-spec parallelizable.

• Re-point the ~96 ambient-data failures at worker-seeded fixtures (the prefix-scoped rows the workerData fixture seeds), never ambient staging content. test-philosophy.md §2.1.
• Replace soft guards (if (await x.isVisible().catch(() => false))) with HARD assertions.
• Named run #9 examples to re-point: the browse-search score > 0.9 assertion → assert against the seeded embedded items (precomputedEmbeddings / EMBEDDING_ITEM_INDICES, test-data-fixture.ts:199-210); the admin-dedup “Resolve near-duplicate pair” → assert against the seeded admin-dedup fixture rows (admin-dedup-fixture-helpers).
• Depends on b because the re-pointed specs must use the shard-unique [E2E-S… prefix so seeded data stays disjoint when sharded.

{128.10} (slice e) — Ephemeral Supabase branch per run [deps: d]

Heaviest, last. Ratify cost/time with Liam FIRST (design Risk #4).

• create_branch → apply migrations → bun run seed:e2e-users and seed application_types / users / roles (the fixture resolves application_types.key → id at test-data-fixture.ts:226-242; application_types.id is gen_random_uuid(), NOT stable across branches) → run the sharded suite against the branch URL → always() delete_branch.
• Fully decouples nightly from cross-track DDL churn on shared staging; lets nightly leave the integration-staging-e2e concurrency group.
• Depends on d: a fresh branch has NO ambient data, so every data-asserting spec must already be self-seeding (slice d) or it breaks on the empty branch.

{128.11} (slice f) — Fast-follow: smoke migration + E2E_EXCLUDE_BID removal [deps: b]

Off the critical path; do after the nightly is stable.

• Migrate the e2e-smoke job (ci.yml:392-429) onto the same build-once + next start path (PLAYWRIGHT_WEB_SERVER_CMD=bun run start + a build step with NEXT_PUBLIC_E2E=true). Depends on b (the build-once + artifact pattern it reuses).
• Remove E2E_EXCLUDE_BID: "1" (e2e-nightly.yml:159) + the E2E_EXCLUDE_BID branch (playwright.config.ts:30-38) once the api.* views (api.bid_questions) regenerate. CROSS-TASK COORDINATION with id-130 {130.9} — this is NOT a sibling dependency (it lives in another Task); it is a coordination note only and must not be encoded as a Subtask dep.

---

5. Smoke-gate safety contract (every change no-ops for smoke by default)

Lever	Nightly	Smoke / local (default)
webServer.command	PLAYWRIGHT_WEB_SERVER_CMD='bun run start'	unset → 'bun dev' (unchanged)
Reporter	PW_BLOB_REPORT=1 → blob	unset → html/list/github (CI) / html/list (local)
maxFailures	PW_MAX_FAILURES=4	unset → CI ? 5 : 0
globalTimeout	PW_GLOBAL_TIMEOUT_MS=1080000	unset → CI ? 12 min : undefined
Seed prefix	shard set → [E2E-S{n}-W{w}]	shard null → [E2E-W{w}] (unchanged)
Workers / retries	--workers=2 --retries=1 (matrix)	config CI ? 1 : 3 / CI ? 2 : 0

The PR-blocking e2e-smoke gate (ci.yml) is untouched day one. Slice f is the deliberate, deferred migration of smoke onto the shared prod-build path.

---

6. Risks & open questions (for Liam)

1. Build-env completeness. next start needs the server env at START and every NEXT_PUBLIC_* (incl. NEXT_PUBLIC_E2E) at BUILD. The env block must appear in BOTH steps. lib/env-server.ts validates at module load → fails fast if a server var is missing.
2. N=4 / workers=2 are starting points. Tune after the first green (maybe N=3, or workers=3).
3. Concurrent shared-user auth. 4 users × 4 shards authenticate near-simultaneously — check Supabase signIn rate limits. Dissolves once slice e (ephemeral branch) lands.
4. Ephemeral branch (e) cost/time + it must seed application_types / users / roles. Ratify vs the short-term DDL-freeze window before building slice e.
5. Confirm LOCAL prod build over deployed Vercel for the nightly (design recommends local; Liam has ratified).
6. mcp-invocation.spec vs mcp-apps. Plain bun run build does NOT bundle mcp-apps (nor does bun dev today) → parity holds. Flag the pre-existing gap during slice d.

---

7. Subtask map & dependency graph

Task 128 already carries integer-id subtasks 1–5 (the S403-audit work: {128.1} e2e-nightly lane + {128.2} eval specs SHIPPED; {128.3}–{128.5} pending). The next free id is 6, so the six run #9 slices land as {128.6}–{128.11}. Slice letters a–f map to ids 6–11.

{128.6}  a  prod-build switch        deps: []          ← first, dependency-free (mandated)
{128.7}  b  shard + artifact + prefix deps: [6]
{128.8}  c  fail-fast budgets         deps: [7]
{128.9}  d  data-determinism          deps: [7]         ← parallel branch off b
{128.10} e  ephemeral Supabase branch deps: [9]         ← gated on Liam cost/time ratification
{128.11} f  fast-follow (smoke + bid) deps: [7]         ← cross-task coordination w/ id-130 {130.9}

All dependencies are sibling-only (every dep is another {128.x}). The id-130 {130.9} coupling (slice f) is a coordination note in details, not a dependency.

---

8. Key files & lines

• playwright.config.ts:6 (NEXT_PUBLIC_E2E runner-process set), :30-38 (E2E_EXCLUDE_BID branch), :67-68 (caps), :122-127 (webServer).
• .github/workflows/e2e-nightly.yml:159 (E2E_EXCLUDE_BID), :160-168 (run line), :106-168 (run step env).
• .github/workflows/ci.yml:90-100 (.next/cache cache), :126-153 (build env block), :140-144 (NEXT_PUBLIC_* mirror), :392-429 (e2e-smoke run step), :525-534 (mcp-build tar + upload-artifact pattern).
• e2e/fixtures/test-data-fixture.ts:109 (seed prefix), :199-210 (precomputed embeddings), :215 + :602 (prefix cleanup/teardown), :226-242 (application_types resolve).
• e2e/global-setup.ts (step 0 orphan sweep), e2e/global-teardown.ts:72 (globalTeardown → cleanupContentItemsByTitlePrefix).
• lib/env-client.ts:114 (NEXT_PUBLIC_E2E literal accessor — the client-inline point).

---

9. Authoring calls (where the design needed a decision)

1. Subtask ids 6–11, not “a–f”. TM-shape Subtask id is an integer local to the parent; task 128 already uses 1–5. The design’s letter labels avoided collision with the integer-numbered audit subtasks; mapped to ids 6–11 (add-subtasks keeps explicit ids).
2. Env var named PLAYWRIGHT_WEB_SERVER_CMD (underscored), for consistency with the existing PLAYWRIGHT_WEB_SERVER_PORT in playwright.config.ts:124. The design wrote PLAYWRIGHT_WEBSERVER_CMD; I standardised on the underscored form. The Executor should use PLAYWRIGHT_WEB_SERVER_CMD throughout.
3. Spec dir slug. The Orchestrator brief named id-128-e2e-nightly-restructure; a pre-existing id-128-test-ci-integration/ holds the S403 audit. I honoured the brief’s slug and cross-referenced S403; flagged for the Orchestrator to reconcile if a single per-Task dir is preferred.
4. Slice d depends on b (not a). The self-seeding re-points must use the shard-unique [E2E-S… prefix to stay disjoint under sharding, so d follows b. (a alone would suffice technically for single-runner, but the design sequences d after b.)
5. Slice f depends on b (not a). The smoke migration reuses the build-once + artifact pattern that is fully realised in b.