ID-104 {104.3} TECH — bottom-up agent-eval engine

ID-104 {104.3} TECH — bottom-up agent-eval engine

Status: {104.3} TECH spec. Authored by a FRESH Planner instance (Q-PLANNER-2 / B4 — NOT the {104.1} RESEARCH / {104.2} PRODUCT Planner) against the ratified predecessors: {104.1} RESEARCH.md and {104.2} PRODUCT.md (B-INV-1..25, groups A–I; RESEARCH + PRODUCT both ratified by Liam, S356). Every Proposed change below maps 1:1 to a numbered B-INV via the orphan-check table in §M — the format ID-71 TECH §M established and the Checker + the new eval-runner self-eval verify. British English throughout. DD/MM/YYYY dates.

This spec fixes the mechanics; it does not re-litigate behaviour. Settled context (do NOT re-open, per S356): bottom-up rebuild (not lift-and-extend); canonical AgentEvalContract in lib/eval/contract.ts with seven mandatory fields + optional graduation_metric; 4-tier block|warn|info|infra severity; single central eval-runner (exit 0/1/2); DB-backed baseline registry porting checkRegression semantics; metrics.ts reused as-is; recordAiCall() single capture point with outcome-signal enum win|fail|loop|refusal; in-house graduation metric (never Raindrop cloud); three layered surfaces (Workshop-local + create-skill viewer + KH /admin/refinement stub); L1/L3/L4 reused as suites; scope-phased HITL (substrate IN, organs DEFERRED with the present-but-empty patterns/proposals stubs).

This TECH is the HARD-UPSTREAM gate that unblocks ID-71 Wave 2/3. The canonical AgentEvalContract finalised in §Contract is the type ID-71’s M38 guard imports (ID-71 TECH §OQ-2). ID-104 is sequenced FIRST.

---

Context

ID-104 builds the bottom-up agent-eval substrate platform-direction.md’s two anchor principles stand on: eval-everything (every AI touchpoint born-evaluable) and progressive-trust graduation (auto-apply earned per-workflow on an in-house metric). It owns the canonical AgentEvalContract, the touchpoint registry, the 4-tier severity + variance_band model, recordAiCall() cost/signal capture, the central eval-runner, the DB-backed baseline lifecycle, the in-house graduation metric, the nightly lane, and the KH-owned /admin/refinement stub-spine. Behaviour is fixed in {104.2} PRODUCT; this spec fixes how each invariant lands. The full reuse-vs-rebuild verdicts are settled in RESEARCH §3 and restated here as concrete architecture, not re-litigated.

Code-intelligence orientation (cited verbatim, re-verified 15/06/2026 — not paraphrased)

Per the binding orientation rule, the symbols this spec mandates be touched were queried before drafting. The knowledge-hub GitNexus index is partially stale for this surface (several saveBaseline incoming-call UIDs returned byte-garble in RESEARCH/PRODUCT, and the query below returned mostly unrelated Start → cost/quality processes), so per RESEARCH §1.5 the conclusive evidence is the clean symbol records + direct source reads + grep/SQL sweeps. Which tool gave which finding is stated explicitly:

• gitnexus_query({query:'eval runner baseline severity touchpoint cost tracking recordAiCall registry', repo:'knowledge-hub'}) returned mostly noise (cost/quality Start → processes) but its definitions block cleanly surfaced the real eval surface: Function:lib/eval/baseline.ts:saveBaseline (:35-53) + :evalPassed (:115-125); Function:components/provenance/cost-tab-stub.tsx:CostTabStub (:21-128) + :fetchCostAggregate (:27-72); and Function:scripts/quality-gate.ts:severityFor (:323-333) — the content quality-gate severity mapper, a DISTINCT surface, NOT the AI-eval layer (do NOT conflate — RESEARCH §3 flagged this).
• gitnexus_context({name:'CostTabStub', repo:'knowledge-hub'}) returned ambiguous (3 candidates: components/provenance/cost-tab-stub.tsx:21, app/provenance/provenance-content.tsx:82 ×2). The component is the canonical interim slot.
• Direct read of lib/eval/baseline.ts confirms saveBaseline writes JSON to __tests__/fixtures/eval-baselines/${suiteName}.baseline.json (:16,:52); checkRegression (:65-105) implements min (currentValue < min → fail) + max_drop ((baseline - current) > max_drop → fail) — both one-sided (the improvement-asymmetry to port intact); evalPassed (:116-126) returns true on no-baseline (first run).
• Direct read of lib/eval/types.ts confirms EvalBaseline.thresholds is Record<string, { min?: number; max_drop?: number }> only (:42) — no severity, no variance_band, no touchpoint/contract/kind/grounding_shape fields. EvalResult (:28-35) carries suite_name/metrics/passed/failures. There is no AgentEvalContract.
• Direct read of lib/eval/metrics.ts confirms it is pure + stateless (precision/recall/ f1Score/accuracy/rougeL/rouge1/mrr/ndcgAtK/precisionAtK — no I/O). Reused as-is (RESEARCH §3).
• Direct read of components/provenance/cost-tab-stub.tsx confirms CostTabStub (labelled “Interim — Wave B”) reads pipeline_runs.cost over 30 days via tryQuery (:34-41) — it reads the pipeline cost aggregate; there is no per-touchpoint recordAiCall() persistence behind it. It is the natural host slot for the rollup (B-INV-17). Uses createClient + tryQuery + logBestEffortWarn — the KH safe-access pattern.
• scripts/eval-search.ts (grep of the main() pattern) confirms the per-suite entry: imports saveBaseline/checkRegression (:29-30), async function main() (:255), checkRegression(baseline, metrics) (:419), saveBaseline(SUITE_NAME, …) (:432), and process.exit(1) at :119/:284/:290/:297/:308/:491/:497. It uses exit(1) for BOTH quality-fail AND infra-error — there is no 0/1/2 split today; the deterministic exit disposition (B-INV-10) is genuinely new. Seven sibling scripts/eval-*.ts exist (classification, entity-classification, holder-rule-ts, procurement-drafting, search, summarisation, tag-morphology-adoption) — each its own entry point; no central dispatcher.
• package.json (grep) confirms eval:* per-suite scripts (:51-56) and the test:mcp-eval / :rq / :fc + seed:mcp-eval scripts (:36-42); eval:all chains the per-suite scripts sequentially.
• .github/workflows/ci.yml (grep) confirms the mcp-eval matrix job (l1→test:mcp-eval, l3→:rq, l4→:fc, :888-890), gated on mcp-eval-seed (:649,:709) + detect-changes (:745), push-events-only / paths-filtered (:777, real-API gate ID-91/bl-246). There is NO scheduled/nightly lane today — the nightly full-set lane (B-INV-13) is greenfield; only staging-reference-refresh.yml + supabase-advisors.yml are non-CI scheduled lanes.
• SQL sweep (grep create-table over supabase/migrations/*.sql): ZERO ai_call/cost_track/cost_event/eval_result/eval_run/eval_baseline/touchpoint/ agent_eval tables. Python + TS + SQL sweep for recordAiCall|record_ai_call: ZERO hits. (gitnexus + ast-dataflow do not cover SQL/Python — grep is the conclusive tool here.) The ccc search fallback errored in RESEARCH/PRODUCT (local cocoindex daemon — environment issue, not an absence signal); grep + GitNexus definitions are conclusive on their own.
• Filesystem checks: app/admin/refinement does NOT exist (only app/admin/content-dedup is present) — the stub route is greenfield. .claude/skills/create-skill/eval-viewer/viewer.html is PRESENT (the skills-surface viewer, B-INV-20-ii, reused as-is).

Orientation verdict (matches RESEARCH §1.5 / PRODUCT): the touchpoint registry, the canonical AgentEvalContract, the central eval-runner, the 4-tier severity model, the DB-backed baseline store, recordAiCall() + its cost/signal tables, the nightly lane, the graduation metric, and the /admin/refinement route are GREENFIELD. The adjacent existing artefacts are: the file-JSON baseline lifecycle (baseline.ts — port the regression semantics, rebuild the persistence), the pure metrics.ts (reuse as-is), the labelled-interim cost-tab-stub.tsx (fold the rollup behind it), and the CI-wired L1/L3/L4 mcp-eval suite (reuse as the suite layer). Tool/query catalogues: .gitnexus/CLAUDE.md, .ast-dataflow/CLAUDE.md.

Line anchors are indicative, not exact. Cited line numbers locate the symbol at authoring time and may drift; the symbol name + file path is the durable anchor. The Executor resolves by symbol, not line.

Empirical verification (Q-EX2 — external-library symbols only)

ID-104 introduces no new external-library API surface: the engine is built on the already-pinned @anthropic-ai/sdk (token-usage capture path, already in use by lib/anthropic.ts:estimateCost), the already-pinned @supabase/supabase-js (DB I/O via sb()/tryQuery()), and stdlib/framework built-ins (Next.js App Router, fs/path). Raindrop Workshop is a local OTel viewer process (localhost:5899), not an imported KH dependency — KH POSTs nothing to it on the client-data path (B-INV-21). No import-and-call verification block is required (the Q-EX2 forcing function scopes to external-library symbols a spec mandates calling — none are introduced here). The token-usage fields recordAiCall() reads off the Anthropic response (usage.input_tokens / usage.output_tokens / usage.cache_*) are an existing, exercised surface (lib/anthropic.ts), not a new claim.

---

Proposed changes

Each change is keyed T<N> to the B-INV it satisfies (full table in §M). Changes are grouped by area (A–I, mirroring PRODUCT’s invariant groups); the §M table is the orphan-check of record.

Area A — AgentEvalContract + touchpoint registry (B-INV-1..5)

• T1 canonical contract module. Declare AgentEvalContract (the type in §Contract) in a NEW ID-104-owned module lib/eval/contract.ts — the single source of truth. NOT cribbed into lib/eval/types.ts (that file keeps the legacy EvalBaseline/EvalResult/gold-item shapes; the contract is its own module). ID-71’s M38 guard imports AgentEvalContract from @/lib/eval/contract (direct file import — no barrel re-export). Co-locate the four field unions (TouchpointKind, GroundingShape, SeverityTier, plus the OutcomeSignal enum used by T14) so ID-71 and the registry import one module. (B-INV-1.)
• T2 seven mandatory fields + optional graduation_metric. lib/eval/contract.ts exports the exact seven-field shape (touchpoint_id, kind, owner, suite_name, grounding_shape, severity_on_fail, variance_band) + optional graduation_metric?: string, with the ratified union types. A Zod schema agentEvalContractSchema (z.infer ≡ the interface) is co-exported so registry writes validate at the boundary. ID-104 MAY add further bottom-up fields (e.g. file_sha256? for git-backed touchpoints — RESEARCH §2b) but the seven are REQUIRED and unrenamed. (B-INV-2.)
• T3 registry-of-record table. NEW migration table eval_touchpoints (§Migration M1): touchpoint_id PK (unique), kind, owner, suite_name, grounding_shape, severity_on_fail, variance_band, graduation_metric (nullable), contract_version int, registry_version int, file_sha256 (nullable), timestamps. A registry helper lib/eval/registry.ts (registerTouchpoint / getTouchpoint / listTouchpoints) writes via sb()/tryQuery(); a duplicate touchpoint_id insert is REJECTED (PK conflict surfaced as a clear error, not swallowed). (B-INV-3.)
• T4 registration-as-gate. The eval-runner (T9) resolves each touchpoint from eval_touchpoints BEFORE dispatch; an eval-relevant touchpoint with no registry row is REJECTED with an explicit not registered: <id> signal (exit 2 disposition, T10). This is the substrate ID-71 B-INV-13’s born-evaluable guard enforces against. (B-INV-4.)
• T5 version advance on contract change. registerTouchpoint bumps the per-touchpoint contract_version when any contract field changes (computed by comparing the incoming contract hash to the stored row) and advances the table-level registry_version; the /admin/refinement listing + the version-history endpoint (T22) read it. (B-INV-5.)

Area B — severity + variance model (B-INV-6..8)

• T6 canonical 4-tier severity. SeverityTier = 'block'|'warn'|'info'|'infra' lives in lib/eval/contract.ts; every contract carries exactly one. infra is reserved for transient-provider failures (Anthropic 529 / timeout / 503 — phase-3 §6.6 porting source) and is NEVER counted as a quality regression. Supersedes the historic phase-3 3-tier draft (block|warn|info, which lib/eval/types.ts never shipped). (B-INV-6.)
• T7 severity → exit disposition. A NEW lib/eval/severity.ts disposition(results) maps the worst severity across a run to an exit class: block-fail → gate fail; warn/info → recorded-but-pass; infra → recorded as infrastructure noise, does not fail the gate. The eval-runner (T9/T10) consumes it. (B-INV-7.)
• T8 variance_band regression tolerance. Port checkRegression (baseline.ts:65-105) semantics into lib/eval/regression.ts keyed by touchpoint + contract variance_band (default 0.02): a drop within ±variance_band is NOT a regression; a drop beyond it (in the worsening direction) IS; an improvement of any size passes (improvement-asymmetry, F17 — implemented one-sided per the ported min/max_drop math, NEVER Math.abs(...)). (B-INV-8.)

Area C — central eval-runner, baseline lifecycle, nightly lane (B-INV-9..13)

• T9 single central eval-runner. NEW scripts/eval-runner.ts: reads eval_touchpoints (T3), dispatches each touchpoint’s suite_name (the L1/L3/L4 suites + the legacy per-suite callees, registered as touchpoints — T23), applies the severity model (T7) + regression math (T8), and writes results UNIFORMLY to eval_runs (§Migration M2). Replaces the per-suite scripts/eval-*.ts main() → saveBaseline pattern (no central dispatcher exists today — orientation-confirmed). The seven legacy eval-*.ts scripts become thin callees the runner invokes (their check logic unchanged); a --touchpoint <id> / --all flag selects scope. (B-INV-9.)
• T10 deterministic exit 0/1/2. scripts/eval-runner.ts exits 0 (all gating checks pass), 1 (≥1 block-severity regression — quality gate fail), 2 (runner/infra error — could not complete, including an unregistered touchpoint per T4, an Anthropic 529 surfaced as infra with no other failures still exits 0, but a runner crash / DB-unreachable exits 2). Distinct from the legacy uniform exit(1); a CI job branches on the three codes. (B-INV-10.)
• T11 DB-backed baseline store + ported regression math. NEW migration table eval_baselines (§Migration M3): per-touchpoint (touchpoint_id, metrics jsonb, thresholds jsonb, registry_version, promoted_by, promoted_at), replacing the flat __tests__/fixtures/eval-baselines/*.json. lib/eval/baseline-store.ts exposes loadBaseline(touchpointId) / promoteBaseline(...) reading from the DB via sb(). The PORTED checkRegression math (T8) runs against the store; the file-JSON saveBaseline/ loadBaseline in lib/eval/baseline.ts are retained only until the seven legacy suites are re-pointed (Executor migrates fixtures into rows; the file path is then dead). (B-INV-11.)
• T12 promote / history / compare + audit. lib/eval/baseline-store.ts exposes promoteBaseline(touchpointId, result, actor) (writes a new eval_baselines row + an eval_baseline_audit row recording who/when/which registry_version — §Migration M3), baselineHistory(touchpointId) (ordered prior baselines), compareBaselines(a, b) (metric deltas). Names mirror the historic phase-4 promoteBaseline/baselineHistory/ compareBaselines hooks (porting the lifecycle shape, rebuilding the storage). (B-INV-12.)
• T13 nightly lane. NEW workflow .github/workflows/eval-nightly.yml (schedule: cron, daily) runs bun run scripts/eval-runner.ts --all against the FULL registered touchpoint set, records every result + severity disposition into eval_runs, and is DISTINCT from the push-gated mcp-eval matrix (which stays a subset, paths-filtered). It targets the staging Supabase branch (real-API gate posture mirrors mcp-eval-seed); failures surface on the /admin/refinement surface, not as a hard PR block. (B-INV-13.)

Area D — recordAiCall() cost + signal capture + rollup (B-INV-14..17)

• T14 single capture point recordAiCall(). NEW lib/eval/record-ai-call.ts exporting recordAiCall(args) — the ONE capture point. Writes (via sb()/tryQuery()): touchpoint_id, model + tier, input_tokens/output_tokens/cache_* (off the Anthropic usage object — existing surface, lib/anthropic.ts), cost_usd (computed via the existing estimateCost path), and outcome_signal = the RATIFIED enum win | fail | loop | refusal (sized for exactly this set, extensible via the enum). Greenfield (no persistence today). (B-INV-14.)
• T15 persisted substrate, zero egress. NEW migration table ai_call_events (§Migration M4): per-call rows keyed by touchpoint_id, with the outcome_signal enum column, RLS so rows are tenant-safe + admin-readable, written on-platform only (no off-platform egress of captured data — the cost/signal data never leaves KH/Supabase). A network assertion in the test suite confirms zero off-platform egress. (B-INV-15.)
• T16 guardable presence. recordAiCall( is a greppable literal; a guard test (__tests__/eval/record-ai-call-guard.test.ts) detects an instrumented touchpoint that omits the call (ID-71’s M38 extends this pattern — the same forcing-function shape as mcp-fixture-sync.test.ts). (B-INV-16.)
• T17 rollup into cost-tab-stub.tsx. Re-point components/provenance/cost-tab-stub.tsx fetchCostAggregate from the interim pipeline_runs.cost read to a real aggregate over ai_call_events (T15) keyed by touchpoint_id (TanStack Query fetcher in lib/query/ per the components/CLAUDE.md rule; semantic tokens only; keep the “Interim — Wave B” banner until the aggregate is wired, then update the copy). NO fourth surface is spawned — the persistence folds behind the existing slot. (B-INV-17.)

Area E — in-house graduation metric (B-INV-18..19)

• T18 in-house graduation metric. NEW lib/eval/graduation.ts computes the per-workflow quality metric (WS-5 progressive-trust) from ai_call_events (T15) + eval_runs (T9) on KH’s own infrastructure — NO client-data trajectory POSTed to Raindrop cloud (a network assertion confirms zero Raindrop-cloud egress). Directly satisfies ID-71 B-INV-15. (B-INV-18.)
• T19 contract-addressable metric. A contract MAY declare graduation_metric?: string (T2); where present, the runner + /admin/refinement surface report the touchpoint’s current value against it (lib/eval/graduation.ts:metricFor(touchpointId)). This PRODUCT computes the metric; auto-apply on it is the DEFERRED follow-up (T24). (B-INV-19.)

Area F — three layered observability surfaces (B-INV-20..22)

• T20 three layered surfaces, composed by kind. (i) Raindrop Workshop local viewer (localhost:5899, OTel-fed) for agents + goose recipes — runtime traces, LOCAL only; (ii) create-skill’s .claude/skills/create-skill/eval-viewer/viewer.html for skills (present, reused as-is); (iii) the KH-owned /admin/refinement stub (T22) as the single registry-of-record spine across ALL kinds, deep-linking to (i)/(ii). Layered by kind, not redundant; the KH surface is the ONLY registry-of-record. (B-INV-20.)
• T21 Raindrop Workshop-local only; hosted excluded. Raindrop runs locally with empty writeKey (⇒ cloud POST is a no-op, OTel-fed); hosted raindrop.ai is PERMANENTLY excluded from the client-data path (HR-5 / WS-14 blocker). A network assertion confirms Workshop with empty writeKey performs zero trace egress; no code path POSTs client-derived data to raindrop.ai. (B-INV-21.)
• T22 KH /admin/refinement stub-spine. NEW admin-only route app/admin/refinement/page.tsx (Warm Meridian semantic tokens only; UK English; WCAG 2.1 AA — colour never the sole signal), gated via getAuthorisedClient() → auth.success → authFailureResponse(auth). Lists registered touchpoints + registry_version + per-touchpoint unprocessed-signal count, PLUS FOUR per-touchpoint API endpoints an operator can curl (phase-4 AC#12 REVISED F-10 shape): GET /api/refinement/touchpoints/[id]/signals, /patterns, /proposals, /version-history — each admin-gated via authFailureResponse(auth). patterns + proposals ship PRESENT-BUT-EMPTY (they back the DEFERRED organs, T24 — empty 200 with a stable shape, NOT 404/absent), so the follow-up only fills them. New API routes are NOT added to proxy.ts publicRoutes (admin-only, must redirect to /login for non-admins). The full cross-touchpoint dashboard is a NAMED follow-up, not launch. (B-INV-22.)

Area G — L1/L3/L4 reuse as the suite layer (B-INV-23)

• T23 register the 96 CI-wired checks as touchpoints. The shipped scripts/mcp-eval/ suite — L1 protocol-compliance (42 checks), L3 response-quality (17), L4 functional-correctness (37) = 96 CI-wired checks — is REGISTERED as touchpoints in eval_touchpoints (T3) with a bound AgentEvalContract (suite_name: 'l1'|'l3'|'l4', severity_on_fail + variance_band attached). Their pass/fail routes through eval-runner (T9) + the contract, NOT the file-JSON saveBaseline path. The check logic in protocol-compliance.ts/response-quality.ts/ functional-correctness.ts is UNCHANGED — only the orchestration around it (registry + contract + runner) is rebuilt. A seed migration / registerTouchpoint bootstrap script registers the three suites + the seven legacy eval-*.ts suites at first run. (B-INV-23.)

Area H — scope-phased HITL boundary (B-INV-24)

• T24 substrate IN; organs DEFERRED-with-anchor. IN scope NOW (T1–T19): registry, contract, severity|variance, recordAiCall() capture, baseline lifecycle, graduation metric. DEFERRED to a NAMED follow-up Task gated on signal volume: the cross-touchpoint pattern detector, the parallel A/B runner against gold standards, the auto-rollback registry. Recorded in ID-104’s “deferred, gated on signal volume” register with a back-pointer to B-INV-24 (the Orchestrator opens the follow-up Task; this is a ledger coordination note, §X). The patterns/proposals endpoints (T22) ship present-but-empty as the deferral anchor — NOT a silent orphan. (B-INV-24.)

Area I — ID-71 born-evaluable consumption satisfaction (B-INV-25)

• T25 no-code-change — satisfied by T1/T2/T3/T4/T6/T18/T19/T16. ID-71’s M38/M13/M14/M15/M40 guards read: the canonical contract (T1/T2), the registry + registration gate (T3/T4), the severity model (T6), the grounding-shape field (T2), the graduation metric (T18/T19), and the recordAiCall() guard surface (T16). ID-71 imports AgentEvalContract from @/lib/eval/contract and reads all consumption fields with NO adaptation; no eval-engine internal is duplicated in ID-71. This is a satisfied-by-X record (the satisfaction is the composition of T1..T19), not a new code change. (B-INV-25.)

---

The canonical AgentEvalContract definition (HARD-UPSTREAM gate for ID-71)

This is the finalised, ID-104-owned canonical definition that lands in lib/eval/contract.ts (NOT lib/eval/types.ts). ID-71’s M38 guard imports it directly. The seven mandatory fields + optional graduation_metric are frozen by S356; OutcomeSignal (used by recordAiCall(), T14) is co-located in the same module.

// lib/eval/contract.ts — OWNED + canonically defined by ID-104 (bottom-up).
// Single source of truth; ID-71 imports `AgentEvalContract` from here (no barrel re-export).

export type TouchpointKind =
  | 'tool'
  | 'prompt'
  | 'skill'
  | 'inline'
  | 'agent_recipe';

export type GroundingShape =
  | 'structured_output'
  | 'forced_tool_strict'
  | 'citations'
  | 'n/a'; // B-INV-2 / ID-71 B-INV-35

export type SeverityTier =
  | 'block'    // fails the gate (non-zero exit)
  | 'warn'     // recorded + surfaced, does not fail the gate
  | 'info'     // recorded only
  | 'infra';   // transient-provider failure (Anthropic 529 / timeout) — NOT a quality regression

/** Outcome signal captured by recordAiCall() — RATIFIED enum (sized for exactly this set,
 *  extensible by adding members). */
export type OutcomeSignal = 'win' | 'fail' | 'loop' | 'refusal';

export interface AgentEvalContract {
  touchpoint_id: string;        // stable id: tool name | prompt name | skill | recipe slug
  kind: TouchpointKind;
  owner: string;                // touchpoint registry: owner of record
  suite_name: string;           // the eval suite this touchpoint runs under (e.g. 'l1' | 'l3' | 'l4')
  grounding_shape: GroundingShape;
  severity_on_fail: SeverityTier;
  variance_band: number;        // per-touchpoint regression tolerance (default 0.02)
  graduation_metric?: string;   // B-INV-19: in-house WS-5 auto-apply metric (optional, contract-addressable)
}

A co-exported Zod schema agentEvalContractSchema (with z.infer<typeof agentEvalContractSchema> structurally equal to AgentEvalContract) validates registry writes at the boundary (T2).

---

Migration plan (staging-first; supabase/CLAUDE.md DDL discipline)

All DDL via supabase migration new + foreground db push (NEVER MCP execute_sql/ apply_migration for DDL; db push prompts interactively — run foreground). A worktree Executor’s FIRST action is supabase link --project-ref turayklvaunphgbgscat (staging — worktrees inherit no link state). Apply to staging first; the prod push is a GATED operator step. After the schema change, regen types: supabase gen types typescript --project-id rovrymhhffssilaftdwd --schema public > supabase/types/database.types.ts (NEVER hand-edit). RLS is role-based via get_user_role(). No public.*() SECURITY DEFINER functions are introduced (the engine reads/writes via per-user sb() clients), so no REVOKE EXECUTE … FROM anon grants are required — but if a registry/rollup RPC is added during implementation, it MUST carry SET search_path = public, extensions + an explicit REVOKE … FROM anon.

Four new tables (one migration may carry several; group by concern):

#	Table	Purpose	Key columns	RLS
M1	eval_touchpoints	registry-of-record (T3/T5/T23)	touchpoint_id PK, kind, owner, suite_name, grounding_shape, severity_on_fail, variance_band, graduation_metric, contract_version, registry_version, file_sha256	admin read/write; get_user_role()
M2	eval_runs	uniform run results (T9/T10/T13)	id, touchpoint_id FK, metrics jsonb, passed, severity_disposition, exit_class, run_at, source (nightly|ci|manual)	admin read; writer = service/runner
M3	eval_baselines + eval_baseline_audit	DB-backed baseline + lifecycle (T11/T12)	baselines: (touchpoint_id, metrics jsonb, thresholds jsonb, registry_version, promoted_by, promoted_at); audit: (touchpoint_id, action, actor, registry_version, at)	admin read; promote = admin write
M4	ai_call_events	recordAiCall() cost + signal (T14/T15/T17)	id, touchpoint_id, model, tier, input_tokens, output_tokens, cache_*, cost_usd, outcome_signal (enum win|fail|loop|refusal), created_at	tenant-safe + admin read; never egresses off-platform

outcome_signal is a Postgres enum sized for exactly the ratified four (win|fail|loop|refusal), extensible via ALTER TYPE … ADD VALUE in a future migration. The cost-tab-stub.tsx rollup (T17) reads an aggregate over ai_call_events.

---

Testing and validation

Each B-INV maps to a concrete check. bun run test is the regression gate (NEVER bun test); the MCP eval L4 layer (scripts/mcp-eval/functional-correctness.ts) remains the primary MCP-tool verifier; behaviour-first per reference/test-philosophy.md. The eval-runner itself is verified by a registered self-eval touchpoint.

• B-INV-1/2/5 (contract + version): unit test imports AgentEvalContract from @/lib/eval/contract, asserts the seven fields + graduation_metric optional with the ratified union types; agentEvalContractSchema rejects a missing/renamed field; a contract change advances contract_version + registry_version.
• B-INV-3/4 (registry + gate): a duplicate touchpoint_id insert is rejected; the runner rejects an unregistered touchpoint with not registered: <id> and exit class 2.
• B-INV-6/7/8 (severity + variance): a block fail → gate fail; warn/info/infra → pass-with-recorded; a within-band drop passes, a beyond-band drop fails, an improvement of any size passes (improvement-asymmetry); an Anthropic 529 classified infra is NOT a regression.
• B-INV-9/10 (runner + exit): every touchpoint dispatches through eval-runner; the three conditions map deterministically to exit 0/1/2; a CI job branches on the code.
• B-INV-11/12 (baseline store + lifecycle): a baseline resolves from eval_baselines (not flat JSON); ported regression math matches the legacy checkRegression outputs on the same inputs; promoteBaseline writes an audit row; baselineHistory/compareBaselines return prior baselines + deltas.
• B-INV-13 (nightly lane): eval-nightly.yml runs --all on schedule against the full registered set; results land in eval_runs + surface on /admin/refinement.
• B-INV-14/15/16/17 (cost + signal): an instrumented call writes an ai_call_events row keyed by touchpoint_id with one of the four outcome_signal values; rows survive restart; a network assertion confirms zero off-platform egress; the guard test fails an instrumented touchpoint omitting recordAiCall(; cost-tab-stub.tsx reads a real ai_call_events aggregate (no fourth surface).
• B-INV-18/19 (graduation): the metric is produced by lib/eval/graduation.ts from on-platform data; a network assertion confirms zero Raindrop-cloud egress; a touchpoint declaring graduation_metric reports its current value via runner + surface.
• B-INV-20/21/22 (surfaces): each surface is present + scoped to its kind; /admin/refinement is the single registry-of-record; a non-admin request is rejected via authFailureResponse(auth); an admin sees the listing + version + signal counts and can curl the four endpoints; patterns/proposals return empty-200 (present-but-empty); a network assertion confirms Raindrop Workshop (empty writeKey) zero egress + no raindrop.ai POST path; semantic tokens only, colour never the sole signal (WCAG 2.1 AA).
• B-INV-23 (L1/L3/L4 reuse): each suite is a registered touchpoint with a bound contract routing through the runner; a diff confirms the check logic in the three mcp-eval files is unchanged from the shipped suite.
• B-INV-24 (scope-phased HITL): substrate invariants (T1–T19) ship; the three deferred organs are recorded as a named follow-up with a signal-volume gating condition + back-pointer; patterns/proposals endpoints present-but-empty (not absent, not built).
• B-INV-25 (ID-71 consumption): ID-71’s guard imports the ID-104 contract and reads all consumption fields; no eval-engine internal is duplicated in ID-71.

---

Deploy + nightly-CI-lane mechanics

• Nightly lane (.github/workflows/eval-nightly.yml, B-INV-13/T13). on: schedule: - cron (daily, off-peak UTC) + workflow_dispatch (manual). One job: supabase link staging → bun install → bun run scripts/eval-runner.ts --all. The runner exits 0/1/2 (T10); the job records the exit + uploads the eval_runs summary as an artefact. Mirrors the mcp-eval-seed real-API gate posture (push/schedule only; staging Anthropic + Supabase secrets; concurrency group to avoid overlap). It is NOT PR-blocking (the push-gated mcp-eval matrix stays the PR gate); a nightly block regression surfaces on /admin/refinement for operator triage.
• PR-gate wiring (ci.yml). No new PR-blocking job at launch — the existing mcp-eval (l1/l3/l4) matrix continues to gate MCP-tool quality, now routing through eval-runner + the contract (T23) rather than the legacy saveBaseline path. The eval-runner --touchpoint scope flag lets a future paths-filtered PR subset run only affected touchpoints.
• Type regen + schema parity. After M1–M4 land on staging + (gated) prod, regen database.types.ts; the schema-parity side-workflow asserts prod↔staging parity; the task-view-vendor-drift reminder does not apply (no ledger-schema change).

---

Risks and mitigations

• GitNexus index partial-staleness on the eval surface. Several saveBaseline incoming-call UIDs returned byte-garble (RESEARCH §1.5). Mitigation: this spec’s orientation is grounded in direct source reads + grep/SQL sweeps (conclusive), not the stale graph edges; the Executor runs bun run gitnexus:analyze before symbol-modifying work and resolves by symbol, not line.
• DB-backed baseline cutover (file-JSON → rows). Re-pointing the seven legacy eval-*.ts suites + the L1/L3/L4 suites from __tests__/fixtures/eval-baselines/*.json to eval_baselines rows risks a window where both paths exist. Mitigation: a one-shot fixture-import bootstrap (T11/T23) seeds rows from the existing JSON; the file path is removed only after every suite is re-pointed and the nightly lane is green; the ported checkRegression math is unit-tested against the legacy outputs on identical inputs (parity proof).
• Zero-egress assertions (Raindrop + graduation metric). A regression could silently POST client-derived data off-platform. Mitigation: B-INV-15/18/21 network assertions in the test suite (Workshop empty writeKey; no raindrop.ai POST path; graduation metric computed on-platform) — these are gating tests, not advisory.
• recordAiCall() instrumentation drift. An instrumented touchpoint could omit the capture call. Mitigation: the T16 grep-guard (the mcp-fixture-sync.test.ts forcing-function shape) fails CI on an un-instrumented touchpoint; ID-71’s M38 extends it.
• /admin/refinement proxy allowlist. A new admin API route accidentally added to proxy.ts publicRoutes would expose registry data. Mitigation: the four endpoints are admin-only via authFailureResponse(auth) and are deliberately NOT in publicRoutes (non-admins redirect to /login); a test asserts a non-admin request is rejected.
• ID-71 hard-upstream coupling. ID-71 Wave 2/3 blocks until lib/eval/contract.ts exists. Mitigation: T1/T2 (the contract module) is the first slice; ID-71’s lib/eval/types.ts placeholder is a last-resort schedule fallback only (NOT intent — ID-71 TECH §OQ-2).
• Nightly real-API cost. --all against the full registered set burns real Anthropic on the L3/L4 suites nightly. Mitigation: off-peak schedule; staging-targeted; the runner’s --touchpoint scope flag + the existing --skip-ai/--skip-search skip-flag policy bound cost; watch spend during the soak.

---

Cross-Task coordination intents {#X}

Orchestrator-applied ledger/coordination notes (sibling-only Subtask dep constraint honoured — these are cross-Task, so they stay at Task level, NOT bent into Subtask deps):

1. ID-71 (AI tooling, spec_needed/in-flight) ← ID-104 HARD-UPSTREAM. lib/eval/contract.ts (T1/T2) is the type ID-71’s M38 guard imports; ID-104 is sequenced BEFORE ID-71 Wave 2/3 (ID-71 TECH §OQ-2 / §X#2). ID-71 CONSUMES the contract only and MUST NOT implement eval-runner / severity / recordAiCall() / the contract. ID-104 also answers ID-71’s three §X#2a-c deliverable requirements: (2a) the three named surfaces (T20/T21/T22); (2b) inherit-or-scope phase-4 HITL = SCOPED-phased (T24); (2c) L1/L3/L4 = REUSED as suites (T23).
2. Deferred HITL organs follow-up (T24 / B-INV-24) — NAMED, gated on signal volume. The Orchestrator opens a follow-up Task for the cross-touchpoint pattern detector + parallel A/B runner + auto-rollback registry, gated on ai_call_events accumulating signal; the patterns/proposals endpoints (T22) are its present-but-empty anchor. Recorded in the “deferred, gated on signal volume” register with a back-pointer to B-INV-24.
3. cost-tab-stub.tsx rollup (T17). Re-pointing the interim pipeline_runs.cost read to ai_call_events touches a provenance component; coordinate with any in-flight provenance work so the slot is not double-owned.

---

B-INV → Proposed-change mapping (orphan-check) {#M}

1:1 mapping; every B-INV has a change (or an explicit “no code change — satisfied by X”). No B-INV without a change; no change without a B-INV.

B-INV	Proposed change	Notes
B-INV-1	T1	canonical AgentEvalContract in lib/eval/contract.ts (single source of truth)
B-INV-2	T2	seven mandatory fields + optional graduation_metric; Zod schema; ratified unions
B-INV-3	T3	eval_touchpoints registry-of-record (M1); duplicate touchpoint_id rejected
B-INV-4	T4	registration-as-gate; unregistered touchpoint rejected (exit 2)
B-INV-5	T5	contract_version/registry_version advance on contract change
B-INV-6	T6	canonical 4-tier block|warn|info|infra; infra ≠ regression
B-INV-7	T7	lib/eval/severity.ts maps severity → exit disposition
B-INV-8	T8	ported checkRegression semantics; variance_band default 0.02; improvement-asymmetry
B-INV-9	T9	single central scripts/eval-runner.ts; replaces per-suite main()+saveBaseline
B-INV-10	T10	deterministic exit 0/1/2 (quality-fail vs infra-error distinct)
B-INV-11	T11	DB-backed eval_baselines (M3); ported regression math; flat JSON retired
B-INV-12	T12	promoteBaseline/baselineHistory/compareBaselines + eval_baseline_audit
B-INV-13	T13	eval-nightly.yml cron lane runs --all registered set; distinct from PR gate
B-INV-14	T14	recordAiCall() single capture; outcome_signal enum win|fail|loop|refusal
B-INV-15	T15	ai_call_events (M4) persisted, tenant-safe, zero off-platform egress
B-INV-16	T16	greppable recordAiCall( guard test (ID-71 M38 extends)
B-INV-17	T17	rollup folds into cost-tab-stub.tsx; no fourth surface
B-INV-18	T18	in-house lib/eval/graduation.ts; zero Raindrop-cloud egress assertion
B-INV-19	T19	contract-addressable graduation_metric; runner/surface report value
B-INV-20	T20	three surfaces layered by kind; KH /admin/refinement = single registry-of-record
B-INV-21	T21	Raindrop Workshop-local (empty writeKey, zero egress); hosted raindrop.ai excluded
B-INV-22	T22	/admin/refinement stub-spine; four curl-able endpoints; patterns/proposals present-but-empty
B-INV-23	T23	96 L1/L3/L4 checks REUSED as touchpoints; check logic unchanged
B-INV-24	T24	substrate IN; pattern-detector/A-B-runner/auto-rollback DEFERRED-with-anchor (§X#2)
B-INV-25	T25	no code change — satisfied by T1/T2/T3/T4/T6/T18/T19/T16 (ID-71 imports contract, no duplication)

Coverage: all 25 invariants mapped; no orphan invariants, no orphan changes. ONE “no code change — satisfied by X” entry (B-INV-25, composed of T1..T19). Net-new tables: M1 eval_touchpoints, M2 eval_runs, M3 eval_baselines+eval_baseline_audit, M4 ai_call_events. The 1:1 B-INV↔change coverage is preserved across the ratified PRODUCT.

---

Open questions for Liam ratification

The big decisions are SETTLED (S356); only genuinely new TECH-level questions are raised. One, and it is non-blocking (a sensible default is stated and used):

1. (T2 / file_sha256 bottom-up field) — RESEARCH §2b notes git-backed touchpoints (skills/ prompts) benefit from a file_sha256 on the registry row to detect drift (the deferred pattern-detector’s eventual input). This spec includes file_sha256 as an OPTIONAL, nullable registry column (M1) — present-but-unenforced at launch (no producer populates it until the deferred organs land). Confirm this is the right launch posture (column present so the follow-up needs no migration), versus omitting it until the follow-up. Default taken: include it nullable — adding a column later is a migration; a present-nullable column is free and avoids a future schema change on the registry-of-record table. No other OQs.

---

End of {104.3} TECH. The canonical AgentEvalContract (§Contract, lib/eval/contract.ts) is the HARD-UPSTREAM gate that unblocks ID-71 Wave 2/3 (ID-71 TECH §OQ-2). 25 B-INV → 25 T-changes, 1:1 in §M (one “no code change — satisfied by X”: B-INV-25). Four new tables (M1 eval_touchpoints, M2 eval_runs, M3 eval_baselines+audit, M4 ai_call_events with the ratified win|fail|loop|refusal enum), staging-first per supabase/CLAUDE.md. Nightly lane eval-nightly.yml runs the full registered set; the PR gate stays the mcp-eval matrix (now routed through eval-runner+contract). One non-blocking OQ (file_sha256 nullable column — default taken). Feeds {104.4} PLAN decomposition.