ID-104 {104.4} PLAN — bottom-up agent-eval engine

Date: 15/06/2026 Tier: Full chain ({104.1} RESEARCH ratified S356 → {104.2} PRODUCT ratified S356 → {104.3} TECH ratified → this PLAN). Authored by a FRESH Planner instance (Q-PLANNER-2 / B4 — NOT the {104.1/2/3} Planner) against the ratified {104.2} PRODUCT.md (B-INV-1..25, groups A–I) + {104.3} TECH.md (T1..T25, 1:1 §M map). British English throughout. DD/MM/YYYY.

Decomposition source of truth: TECH §M (25 B-INV ↔ 25 T-changes). Every implementation Subtask below maps to a contiguous T-change set; the §M 1:1 mapping is preserved (no orphan T-change, no Subtask without a T-change anchor). Subtask ids start at {104.5} ({104.1/2/3} = ratified spec chain, {104.4} = this PLAN).

This PLAN does not re-litigate behaviour or mechanics (settled S356 + ratified TECH). It sequences the T-changes into independently-testable, single-pass-committable implementation Subtasks, assigns sibling-only dependencies, flags parallel-safe vs serial groups, and surfaces the contract-first ID-71 unblock milestone + the functional-correctness.ts shared-file serialisation flag.

Context — code-intelligence orientation (cited verbatim, re-verified 15/06/2026)

Per the binding orientation rule, the surface this PLAN decomposes was re-verified by direct file reads + bun scripts/ast-dataflow-cli.ts anchors + grep + the ccc search fallback before authoring. The knowledge-hub GitNexus index reports stale (“never” indexed; a refresh was running but had not completed at authoring time), so per RESEARCH §1.5 the conclusive evidence is direct source reads + grep/SQL sweeps, stated explicitly per finding:

Greenfield confirmation (grep, conclusive). grep -rniE 'recordAiCall|record_ai_call' --include=*.ts --include=*.tsx --include=*.py --include=*.sql → ZERO hits. grep for create table … (eval_touchpoints|eval_runs|eval_baselines| ai_call_events|touchpoint) over supabase/migrations/*.sql → ZERO hits. ls app/admin/refinement/ → does not exist (only app/admin/content-dedup/ present). ls .github/workflows/eval-nightly.yml → does not exist (existing scheduled lanes: staging-reference-refresh.yml, supabase-advisors.yml, taxonomy-sync.yml only). Verdict: lib/eval/contract.ts, the registry, the four tables, recordAiCall(), the central runner, the nightly lane, the graduation metric, and the /admin/refinement route are all GREENFIELD — matching RESEARCH §1.5 / PRODUCT / TECH §Context.
ccc search fallback attempted, daemon errored (NOT an absence signal). ccc search "recordAiCall AI call cost capture" raised DaemonStartError (cocoindex cli.py:116/:572 traceback — a local environment issue per RESEARCH §1.5). The greenfield disclaimer is therefore applied only AFTER both grep (zero hits) AND the ccc fallback (errored, inconclusive) were run — grep is conclusive on its own.
lib/eval/baseline.ts CURRENT shape (direct read). saveBaseline writes JSON to __tests__/fixtures/eval-baselines/${suiteName}.baseline.json (:36-54); checkRegression (:65-105) implements min (currentValue < threshold.min → fail) + max_drop ((baselineValue - currentValue) > threshold.max_drop → fail) — both one-sided (improvement-asymmetry, F17, to port intact); evalPassed (:116-126) returns result.passed on no-baseline (first run). lib/eval/types.ts:EvalBaseline.thresholds (:42) is Record<string, { min?: number; max_drop?: number }> — no severity, no variance_band, no touchpoint/contract/kind/grounding_shape fields; no AgentEvalContract. Confirms the T8/T11 port-semantics-rebuild-storage split. Four baseline JSON files exist today (classification, entity-classification, search, summarisation).
scripts/mcp-eval/ CURRENT structure (direct ls). protocol-compliance.ts (~23KB, L1), response-quality.ts (~33KB, L3), functional-correctness.ts (~84KB, L4), plus fixtures.ts / seed-data.ts / seed-fixtures.ts. Confirms the T23 reuse-as-suites surface (96 checks = 42 L1 + 17 L3 + 37 L4); the check logic is wrapped UNCHANGED.
Seven legacy scripts/eval-*.ts entries (direct ls). eval-classification, eval-entity-classification, eval-holder-rule-ts, eval-procurement-drafting, eval-search, eval-summarisation, eval-tag-morphology-adoption — each its own entry point; no central dispatcher (T9 is genuinely new). package.json confirms eval:* per-suite scripts (:51-56) + eval:all chaining them sequentially, and test:mcp-eval/ :rq/:fc + seed:mcp-eval (:36-42).
ci.yml mcp-eval matrix (direct read :711-915). The mcp-eval matrix job (l1→test:mcp-eval, l3→:rq, l4→:fc) depends on [mcp-build, mcp-eval-seed, detect-changes], is push-events-only + paths-filtered (real-API gate ID-91/bl-246), Staging-env-scoped, with a skip-flag policy (--skip-ai/--skip-search). There is NO scheduled/nightly lane today — confirms B-INV-13/T13 is greenfield, and the PR gate STAYS this matrix (now routed through the runner+contract).
Auth + safe-access anchors (grep). lib/auth.ts:86 getAuthorisedClient returns { success }; :124 authFailureResponse. lib/supabase/safe.ts:91 sb<T>() / :160 tryQuery<T>(). lib/anthropic.ts:54 estimateCost(model, usage) reads input_tokens/output_tokens/cache_creation_input_tokens/cache_read_input_tokens (the existing token-usage surface recordAiCall() consumes — T14, no new external API).
Precedents (grep/ls). Admin page+API precedent: app/admin/content-dedup/page.tsx + app/api/admin/content-dedup/.../route.ts (all authFailureResponse-gated). Grep-guard forcing-function precedent for T16: __tests__/mcp/mcp-fixture-sync.test.ts + __tests__/validation/eval-fixture-sync.test.ts. RLS helper get_user_role() present in migrations. lib/query/{query-keys,fetchers}.ts is the TanStack Query home for T17.
Migration discipline (supabase/CLAUDE.md, canonical). First action on a worktree: supabase link --project-ref turayklvaunphgbgscat (staging); regen via supabase gen types typescript --project-id rovrymhhffssilaftdwd --schema public > supabase/types/database.types.ts (NEVER hand-edit). supabase db push runs FOREGROUND (interactive prompt; hangs background shells). RLS role-based via get_user_role().

Line anchors are indicative, not exact — they locate the symbol at authoring time and may drift; the symbol name + file path is the durable anchor. The Executor resolves by symbol, not line, and runs bun run gitnexus:analyze before symbol-modifying work once the index refresh completes.

Architecture decisions (sequencing rationale)

Contract-first, standalone, zero-dep — {104.5} is the ID-71 unblock milestone. T1/T2 (lib/eval/contract.ts + Zod schema + the four field unions + OutcomeSignal) has NO dependency on any migration or other module and is HARD-UPSTREAM of ID-71 Wave 2/3. It is the FIRST Subtask, deliberately atomic, so the Orchestrator can cherry-pick it early to unblock ID-71 (ID-71 TECH §OQ-2 / §X#2) independently of the rest of ID-104.
Migrations serialise by ordering (M1→M2→M3→M4). Four new tables; eval_runs (M2) FKs eval_touchpoints (M1); eval_baselines+audit (M3) and ai_call_events (M4) reference touchpoints. They are authored as a single migration concern-grouped Subtask run FOREGROUND via db push on staging first — NOT parallelised (a single migration lane on one DB; concurrent db push would collide). Type regen lands in the same Subtask.
Lib modules fan out parallel AFTER contract + their backing table. Once {104.5} (contract) and {104.6} (migrations) land, the leaf lib modules own disjoint files and are parallel-safe: severity.ts (T7), regression.ts (T8), registry.ts (T3/4/5), record-ai-call.ts (T14/16), baseline-store.ts (T11/12), graduation.ts (T18/19).
The runner is the integration point — serial, late. scripts/eval-runner.ts (T9/T10) composes registry + severity + regression + baseline-store + writes eval_runs; it depends on all of them. It ALSO touches scripts/mcp-eval/functional-correctness.ts for suite registration (T23) — a shared file with the concurrent ID-71 {71.14} rename — so the runner + T23 Subtask carries an explicit shared-file contention note for the Orchestrator to serialise cherry-picks.
Surfaces + nightly + rollup fan out after their backing module. /admin/refinement page + 4 API routes (T20/21/22) need the registry; cost-tab-stub.tsx rollup (T17) needs ai_call_events (M4); eval-nightly.yml (T13) needs the runner.
Deferred-with-anchor + ID-71 satisfaction are coordination/verify Subtasks, not new organs. T24 (record the deferred organs + present-but-empty patterns/proposals endpoints) and T25 (verify ID-71 imports the contract; no eval internal duplicated) are lightweight and land last.
Test discipline baked into every Subtask testStrategy (behaviour-first per reference/test-philosophy.md; bun run test NEVER bun test; KH quality bars — semantic tokens, auth.success+authFailureResponse(auth), sb()/tryQuery(), no barrel re-exports, TanStack Query, UK English).

Wave / parallel-safety plan

Wave	Subtasks	Parallel-safe within wave?	Why
W0 — contract-first	{104.5}	n/a (single)	HARD-UPSTREAM of ID-71; cherry-pick EARLY. Zero deps.
W1 — migrations	{104.6}	n/a (single)	One DB lane; M1→M4 serial; `db push` FOREGROUND staging-first; regen types.
W2 — leaf lib modules	{104.7} severity, {104.8} regression, {104.9} registry, {104.10} record-ai-call, {104.11} baseline-store, {104.12} graduation	YES — disjoint files	Each owns one new `lib/eval/*.ts` + its own test; all depend only on {104.5}+{104.6}.
W3 — runner (integration)	{104.13} eval-runner + exit codes, {104.14} register L1/L3/L4 + legacy suites	SERIAL ({104.14} after {104.13}); both flag shared `functional-correctness.ts`	Runner composes all W2 modules; T23 registration touches the ID-71-`{71.14}`-shared file.
W4 — surfaces / rollup / nightly	{104.15} cost rollup, {104.16} /admin/refinement page+API, {104.17} nightly lane, {104.18} graduation-surface wiring	MOSTLY parallel ({104.17} after {104.13}; {104.16} after {104.9}; {104.15} after {104.6}; {104.18} after {104.12}+{104.16})	Disjoint files (component / route tree / workflow).
W5 — coordination + verify	{104.19} deferred-organs anchor + present-but-empty endpoints, {104.20} ID-71 consumption verify	parallel	Doc/coordination + import-verify; no shared files.

Contract-first unblock note (load-bearing): {104.5} (lib/eval/contract.ts) is the ONLY Subtask ID-71 Wave 2/3 is blocked on. The Orchestrator should dispatch + cherry-pick {104.5} FIRST, ahead of the rest of ID-104, so ID-71 unblocks in parallel with W1–W5. ID-71’s lib/eval/types.ts placeholder is a schedule-slip fallback only — NOT the intent (ID-71 TECH §OQ-2).

Shared-file serialisation flag (load-bearing): scripts/mcp-eval/functional-correctness.ts is touched by ID-104 {104.13}/{104.14} (suite registration) AND by the concurrent ID-71 {71.14} rename. The Orchestrator MUST serialise cherry-picks of these against {71.14} (cherry-pick one, rebase the other) — they cannot land in parallel without conflict. Both ID-104 Subtasks carry this note in details.

Risks and mitigations

Risk	Impact	Mitigation
`functional-correctness.ts` cherry-pick collision with ID-71 {71.14}	High	Serialise cherry-picks; flagged in {104.13}/{104.14} `details`; Orchestrator-owned ordering.
DB-backed baseline cutover (file-JSON → rows) leaves dual paths	Medium	{104.11} seeds rows from the 4 existing JSON fixtures (one-shot bootstrap); file path removed only after every suite re-pointed + nightly green; ported `checkRegression` math unit-tested against legacy outputs on identical inputs (parity proof).
Zero-egress regression (Raindrop / graduation metric POSTs client data)	High	Network-assertion gating tests in {104.12}/{104.16} (Workshop empty `writeKey`; no `raindrop.ai` POST path; metric computed on-platform).
`recordAiCall()` instrumentation drift	Medium	{104.10} ships the grep-guard test (`mcp-fixture-sync.test.ts` shape); ID-71 M38 extends it.
New admin API route leaks via `proxy.ts` publicRoutes	High	{104.16} deliberately does NOT add routes to `publicRoutes`; test asserts non-admin redirect/reject via `authFailureResponse(auth)`.
Nightly real-API spend (`--all` burns Anthropic on L3/L4)	Medium	{104.17} off-peak schedule, staging-targeted, `--touchpoint` scope flag + existing skip-flag policy; watch spend during soak.
GitNexus index stale at authoring	Low	Orientation grounded in direct reads + grep (conclusive); Executor runs `gitnexus:analyze` before symbol-modifying work once refresh completes; resolves by symbol not line.

Open questions for Liam

The big decisions are SETTLED (S356); TECH raised one non-blocking OQ (file_sha256 nullable column — default taken: include nullable). PRODUCT raised two (outcome-signal enum = win|fail|loop|refusal — now RATIFIED into TECH §Contract; patterns/proposals endpoints present-but-empty — now RATIFIED into T22/T24). No PLAN-level open questions are blocking. The decomposition uses 16 implementation Subtasks ({104.5}–{104.20}), well under the 25-Subtask soft ceiling, all sibling-only deps (no cross-Task Subtask dep surfaced — the ID-71 coupling stays at Task level per TECH §X#1).

Implementation Subtask summary (full TM-shape records handed to the Orchestrator)

id	title	T-changes / B-INV	deps	wave
5	Canonical `AgentEvalContract` module + Zod (ID-71 unblock)	T1,T2 / B-INV-1,2	—	W0
6	Four eval migrations M1–M4 + type regen (staging)	T3-tbl,T11-tbl,T12-tbl,T15-tbl / B-INV-3,11,12,15	5	W1
7	`lib/eval/severity.ts` — severity→exit disposition	T6,T7 / B-INV-6,7	5	W2
8	`lib/eval/regression.ts` — ported `variance_band` math	T8 / B-INV-8	5	W2
9	`lib/eval/registry.ts` — register/get/list + version + gate	T3,T4,T5 / B-INV-3,4,5	5,6	W2
10	`lib/eval/record-ai-call.ts` + grep-guard	T14,T16 / B-INV-14,16	5,6	W2
11	`lib/eval/baseline-store.ts` — DB baseline + promote/history/compare + audit	T11,T12 / B-INV-11,12	5,6	W2
12	`lib/eval/graduation.ts` — in-house metric + zero-egress assertion	T18,T19 / B-INV-18,19	6	W2
13	`scripts/eval-runner.ts` — central dispatch + exit 0/1/2	T9,T10 / B-INV-9,10	7,8,9,11	W3
14	Register 96 L1/L3/L4 + 7 legacy suites as touchpoints (bootstrap)	T23 / B-INV-23	9,13	W3
15	`cost-tab-stub.tsx` rollup over `ai_call_events`	T17 / B-INV-17	6 (10 pref.)	W4
16	`/admin/refinement` page + 4 curl-able API routes	T20,T21,T22 / B-INV-20,21,22	9	W4
17	`eval-nightly.yml` scheduled `--all` lane	T13 / B-INV-13	13	W4
18	Wire graduation metric into runner + `/admin/refinement`	T19-surface / B-INV-19	12,16	W4
19	Deferred-organs register + present-but-empty `patterns`/`proposals`	T24 / B-INV-24	16	W5
20	Verify ID-71 imports contract; no eval internal duplicated	T25 / B-INV-25	5	W5

Coverage check vs §M: T1–T25 all assigned (T25 = no-code-change verify Subtask {104.20}; the table-creation halves of T3/T11/T12/T15 fold into migration Subtask {104.6}, their helper halves into the matching lib Subtask). No orphan T-change; no Subtask without a T-change anchor. The PR gate stays the existing mcp-eval matrix (routed through the runner

contract in {104.14}); the nightly lane is {104.17}.

*End of {104.4} PLAN. {104.5} (lib/eval/contract.ts) is the contract-first ID-71 unblock milestone — dispatch + cherry-pick FIRST. Migrations {104.6} serialise (one db push lane, staging-first). W2 leaf lib modules are parallel-safe (disjoint files). The runner {104.13}

suite-registration {104.14} flag the shared scripts/mcp-eval/functional-correctness.ts for cherry-pick serialisation against ID-71 {71.14}. 16 impl Subtasks, all sibling-only deps, under the 25-Subtask soft ceiling.*