S423 Whole-System Investigation — e2e / pipeline / deploy

S423 Whole-System Investigation — e2e ↔ pipeline ↔ deploy

Scope note on inputs. Only D1 arrived as a fully-researched dimension with adversarial verification (3 load-bearing claims, all confirmed/partly with minor corrections). D2, D3, D4 arrived as placeholders ("answer": "test", one stub claim "a" ruled unverifiable). Sections 2–4 below are therefore grounded by my own direct reads in this repo (cited file:line) and flagged at lower confidence than D1 — they are a first-pass orientation, not verified findings. Treat D1 as decision-grade; treat 2–4 as “here is what the code says, confirm before acting.”

---

TL;DR (the decisions that matter)

• Do NOT converge e2e provisioning onto the CocoIndex pipeline /walk. The pipeline writes the same tables e2e asserts on, but produces them non-deterministically (Anthropic claude-opus-4-6 extraction, default sampling, no seed — extraction.py:71,1042-1053,656) and only via a running Python worker over an authored corpus. e2e needs fast, byte-deterministic, worker-free fixtures. Two lanes by design — converge them on schema shape, not execution. (D1, high confidence.)
• The convergence the strategic framing asks for already exists as a different lane: the ~42 TS cocoindex integration specs ARE the pipeline-as-whole-system test (real /walk, real rows). Don’t recreate it inside e2e. (D1, high.)
• The one unification worth putting to Liam is a “golden snapshot”: run /walk once over the synthetic corpus, freeze content_items+embeddings to a versioned seed, replay it deterministically in e2e → real pipeline output, no worker, no token burn. Cost: a regen step on model-tier bumps. (D1.)
• e2e target today = LOCAL next start prod build + STAGING Supabase DB (playwright.config.ts:128-137, e2e-nightly.yml:231-260). {128.10} moves the data target to an ephemeral branch per run (empty DB, no worker), which structurally forbids pipeline-backed e2e — reinforcing the two-lane decision. (D1 + my reads, high.)
• NEXT_PUBLIC_E2E='true' is set unconditionally but has ZERO app-code consumers (playwright.config.ts:6; only echoed in lib/env-client.ts:72,114; repo-wide grep finds no gate in app/ lib/ components/ proxy.ts). The config comment claims it “disables overlays / third-party integrations” — that behaviour does not exist in current code. Either a dead flag or silently-removed gating. Flag for D2/D4 follow-up. (My read, medium confidence — worth a deliberate grep before acting.)

---

1. Pipeline ↔ e2e content (D1) — should provisioning converge?

Answer: No, not for the e2e lane. Keep two provisioning models; converge on schema, not execution. (Confidence: high — D1 fully verified.)

Question	Finding	Evidence
Does the pipeline write the tables e2e asserts on?	Yes — content_items, content_chunks, embeddings, entity_mentions, pipeline_runs, forms-path form_templates	server.py:415-490; cocoindex-write-model.md:99-112,187-201
Is that output deterministic?	No — live Anthropic claude-opus-4-6, default sampling, no seed; “fencing is non-deterministic per call”	extraction.py:71,1042-1053,656
(verifier correction)	Embeddings text-embedding-3-large are ~deterministic; the extraction stage alone carries the non-determinism. Conclusion unchanged.	verdict #1 correction
Does e2e run a worker / authored corpus?	No worker anywhere in e2e; folder-drop route is stubbed (“worker is not present in CI”)	content-ingestion-folder-drop.spec.ts:10-12,38-46; e2e-nightly.yml (no service block)
Does /walk require both?	Yes — worker _walk_handler + a named 400 if corpus dir missing	server.py:450,500-513; lib/upload/folder-drop.ts:262
Is seed-synthetic-corpus.ts an e2e content fixture?	No — structure only (workspaces + domain_metadata + form_questions + walk manifest); explicitly does NOT seed content/responses	seed-synthetic-corpus.ts:15-21,96-104 (verdict #3 confirms)
Does e2e seed content_items?	Yes, directly via service-key inserts (bypassing pipeline); its “q_a_pairs” are content_items rows with content_type='q_a_pair', never the corpus q_a_pairs table	test-data-fixture.ts:145-149; test-data.ts:84,96 (verdict #3 correction)

Verifier nuance that sharpens, not weakens, the call: the corpus is cheaply stageable in-CI via POST /stage (server.py:266), so the single binding constraint is the worker’s availability, not corpus authoring. And {128.9} is a PENDING PLAN slice (deps [7]), so “already commits” is forward-looking — but the pattern it formalises is already in code (test-data-fixture.ts self-seeds 12 content_items + 4 workspaces via prefix-scoped inserts). Both facts reinforce separation.

Recommendation (D1, verbatim-priority): (1) proceed with {128.9} self-seeding SQL fixtures + HARD asserts — required anyway for {128.10}’s empty ephemeral branch; (2) formalise the two-lane model in spec; (3) add a schema-conformance guard that the e2e content_items fixture is column-shape-conformant to /walk output (op_id nullable, source_document_id, content_type, embedding dims) so it can’t silently drift; (4) put the golden snapshot to Liam.

---

2. Staging-first + NEXT_PUBLIC_E2E (D2) — does it change the e2e target?

Answer (my reads — D2 finding was a placeholder; confidence MEDIUM, confirm before acting): The e2e target is split: app server = local next start production build, data = remote STAGING Supabase. NEXT_PUBLIC_E2E does not currently change any runtime target or behaviour.

Aspect	What the code says	Evidence
App server in nightly	Local next start prod build (precompiled, avoids dev compile-on-hit timeout storm)	playwright.config.ts:128-137 (PLAYWRIGHT_WEB_SERVER_CMD)
Base URL	PLAYWRIGHT_BASE_URL ?? http://localhost:3000	playwright.config.ts:76
Data target	Remote Staging Supabase (server env at boot)	e2e-nightly.yml:231-260 (SUPABASE_URL = vars.SUPABASE_URL)
{128.10} change	Data target → ephemeral branch per run (create→migrate→seed→run→delete), leaves the integration-staging-e2e concurrency group	PLAN.md {128.10}
NEXT_PUBLIC_E2E	Set to 'true' unconditionally in playwright config; validated in env-client; no consumer in app/ lib/ components/ proxy.ts	playwright.config.ts:6; lib/env-client.ts:72,114; repo-wide grep = 3 hits only

The decision-useful flag: NEXT_PUBLIC_E2E is effectively dead. Its config comment (“disables non-essential overlays and third-party integrations that interfere with test interactions”, playwright.config.ts:4-5) describes behaviour that no longer exists in the code (grep finds zero isE2E/NEXT_PUBLIC_E2E gates outside config+env-client). So “staging-first + NEXT_PUBLIC_E2E” does not change the e2e target — the target is set by PLAYWRIGHT_BASE_URL (server) + SUPABASE_URL (data). If something e2e relies on that flag to suppress overlays, that reliance is currently a no-op and a flake source. Confirm with a deliberate grep + a check of git history for removed gates before acting.

---

3. Test users (D3)

Answer (my reads — D3 was a placeholder; confidence MEDIUM): Test users are pre-seeded into the persistent staging DB by a dedicated script and consumed by credentials in CI env — they are NOT created per-run today, but {128.10} makes seeding part of the ephemeral-branch lifecycle.

Aspect	Finding	Evidence
Seeder	scripts/seed-e2e-users.ts exists (~28KB); bun run seed:e2e-users	e2e-nightly.yml:245-246; ls scripts/seed-e2e-users.ts
Roles	admin / editor / viewer (user1/2/3) + TEST_USER_4 = dedicated sign-out user (S420)	test-data.ts:405-422; auth-session.ts:12; e2e-nightly.yml:256-260
Domain	test.user{1,2,3}@test-kb-aish.co.uk	test-data.ts:410,414,418
Setup coupling	The setup project authenticates all four; if any creds are missing, setup fails and every spec in the shard is skipped	e2e-nightly.yml:256-258; playwright.config.ts setup project
Fixture dependency on missing users	admin-dedup helper hard-errors with “Run bun run seed:e2e-users”	admin-dedup-fixture-helpers.ts:135
{128.10} change	branch must seed seed:e2e-users + application_types/users/roles because application_types.id is gen_random_uuid() — NOT stable across branches; fixture resolves key→id at test-data-fixture.ts:226-242	PLAN.md {128.10}

Decision-useful risk: the “setup fails → whole shard silently skipped” coupling is a conditional-false-pass-adjacent failure mode (a missing secret yields green-ish skips, not red). This is exactly the antipattern test-philosophy §2.1 warns about, applied at the auth layer rather than the data layer. Worth a guard that converts “setup auth failed” into a hard CI failure.

---

4. Step-back / over-engineering + CI-infra bugs (D4)

Answer (my reads — D4 was a placeholder; confidence LOW-MEDIUM, first-pass only):

Over-engineering check — the genuine risk is the OPPOSITE direction. The strategic pull is to “unify” e2e onto the centralised pipeline. D1 shows that would be the over-engineering: slow (120s polls/fixture), token-burning, flaky (model-drift breaks score>0.9 and exact near-duplicate asserts), and impossible in the {128.10} no-worker branch (D1 risksIfWrong; extraction.py:71). The right step-back is to STOP at two lanes and resist re-plumbing. The only unification that pays off is the read-only golden snapshot (§1.4).

CI-infra observations worth triage (not yet verified as bugs):

Observation	Why it matters	Evidence
NEXT_PUBLIC_E2E dead flag	Config promises overlay/third-party suppression that doesn’t exist → potential flake source / misleading comment	playwright.config.ts:4-6; repo grep
E2E_EXCLUDE_BID="1" temporary exclusion	All bid-* specs excluded pending id-130 {130.9} api.* view regen; cross-task coupling, easy to forget to remove	e2e-nightly.yml (E2E_EXCLUDE_BID)
Setup-auth → silent shard skip	Missing test-user secret skips a whole shard green-ish (false-pass-adjacent)	e2e-nightly.yml:256-258
always() delete_branch (planned)	{128.10} correctly uses always() cleanup; verify it fires on cancel/timeout, not just failure	PLAN.md {128.10}
Orphan content_history cleanup step	Nightly already runs cleanup:orphaned-content-history on staging post-run — staging accretes test debris (the very reason {128.10} moves to ephemeral branches)	e2e-nightly.yml:288-296

This section needs a real D4 pass — the placeholder gave nothing and the items above are surface-level. Do not treat as exhaustive.

---

5. Recommended sequence (act on this)

Ordered by leverage ÷ risk. Items 1–4 are D1-grade (high confidence); 5–7 are medium/low (verify first).

#	Action	Leverage	Risk if skipped	Confidence
1	Proceed with {128.9} as written: re-point the ~96 ambient-data specs to self-seeding worker-scoped SQL fixtures (test-data-fixture.ts), HARD asserts, shard-unique [E2E-S… prefix. Prereq for {128.10}.	HIGH — unblocks the empty-branch end-state	Ephemeral branch breaks on empty DB	HIGH
2	Formalise the two-lane model in the id-128 spec: (a) fast deterministic e2e = SQL fixtures + precomputed embeddings + ephemeral branch + NO worker; (b) pipeline-as-system nightly = the existing ~42 cocoindex TS integration specs seeded by synthetic-corpus structure + real /walk. State explicitly: lane (b) IS the convergence; do not rebuild it in (a).	HIGH — kills the recurring “unify onto pipeline” pull	Repeated re-litigation; wasted plumbing	HIGH
3	Add a schema-conformance guard test: assert the e2e content_items fixture is column-shape-conformant to /walk output (op_id nullable, source_document_id, content_type, embedding dims).	MED-HIGH — stops silent fixture↔pipeline drift	Bespoke fixture drifts from real contract	HIGH
4	Put the golden snapshot to Liam (see §6 Q1). If approved: run /walk once over the synthetic corpus on staging, freeze content_items+embeddings to a versioned seed (generalise embeddings.json), e2e replays it.	HIGH — real pipeline content, deterministic, worker-free	— (opt-in)	HIGH
5	Resolve the NEXT_PUBLIC_E2E dead flag: grep history for removed gates; either restore the overlay/third-party suppression it promises or delete the flag + misleading comment.	MED — removes a latent flake/confusion source	Silent no-op flag misleads future work	MED
6	Convert setup-auth failure into a hard CI failure (not a silent shard skip) — guard against missing TEST_USER_* secrets producing green-ish runs.	MED — closes an auth-layer false-pass	Missing secret hides a red shard	MED
7	Schedule the E2E_EXCLUDE_BID removal as a tracked fast-follow on id-130 {130.9} ({128.11}), so bid specs re-enter the gate the moment api.* views regen.	LOW-MED — prevents permanent silent exclusion	bid specs stay dark indefinitely	MED

Do NOT do: force e2e onto live /walk; add a cocoindex worker to the e2e CI lane; seed content_items/q_a_pairs from the pipeline for e2e; reseed synthetic-corpus to include content. (All refuted by D1; would re-introduce §2.1 antipattern or break {128.10}.)

---

6. Open questions for Liam (genuine decisions only)

1. Golden snapshot — worth it? Freeze one real /walk output of the synthetic corpus to a versioned seed that e2e replays, so e2e asserts against real pipeline content while staying deterministic and worker-free. Cost: a regeneration step whenever the extraction/embedding model tier bumps (extraction.py:71). Yes/no?

2. Schema-conformance guard — adopt? A test that fails if the bespoke e2e content_items fixture diverges (columns / embedding dims) from what /walk writes. Cheap insurance against drift. Yes/no?

3. Cocoindex nightly provisioning — canonicalise? Make synthetic-corpus structure + real /walk the canonical provisioning for the {128.3} cocoindex integration lane (replacing any drifting ambient staging data there), kept explicitly separate from e2e. Yes/no?

4. {128.10} ephemeral-branch cost ratification (PLAN flags “ratify cost/time FIRST”): approve branch-per-run (create→migrate→seed→run→always() delete) to decouple nightly from shared-staging DDL churn? This is the change that ends staging test-debris accumulation. Go/no-go?