ID-104 {104.1} RESEARCH — bottom-up agent-eval engine

ID-104 {104.1} RESEARCH — bottom-up agent-eval engine (RESEARCH-light)

Date: 15/06/2026 Tier: RESEARCH-light. The historic Phase-3/4/5 ai-eval specs were verdicted near-implementation-ready, so this pass does not re-derive the eval domain. Its job is to GROUND three deliberate decisions (2a observability surfaces, 2b HITL inherit-vs-scope, 2c L1/L3/L4 reuse-vs-rebuild) with a bottom-up lens and hand Liam ratifiable recommendations. Owns (canonical, bottom-up): the per-agent AgentEvalContract, the severity|variance model, the touchpoint registry, recordAiCall() cost tracking, eval-runner, the nightly lane, the Claude-as-judge tool-description rubric. Hard-upstream of: ID-71 Wave 2/3 (ID-71 consumes the contract; it must not implement it — ID-71 TECH §OQ-2, §X#2).

---

0. The bottom-up law (non-negotiable framing)

Per Liam’s steer (S354 OQ-2, 15/06/2026), ID-104 is a bottom-up rebuild, NOT a lift-and-extend. Liam explicitly did not ratify reuse+extend. The agent-eval layer chosen now MUST be the long-term one. ID-104’s stale ledger description (now corrected) once read “lift from specs/ai-eval … extending lib/eval/baseline.ts” — that framing is REJECTED and is not inherited here. Every legacy layer below (lib/eval/baseline.ts lifecycle, the callTool harness, the severity model, the L1/L3/L4 suite, the phase-4 HITL loop) is treated as a reference artefact for a deliberate reuse-vs-rebuild call, never as an assumed baseline. The default posture is clean-sheet; reuse must be earned by an explicit “this artefact already expresses the bottom-up shape we’d design anyway” argument.

This RESEARCH grounds the engine in platform-direction.md’s two anchor principles: eval-everything (every AI touchpoint is born-evaluable) and progressive-trust graduation (agents propose; auto-apply is earned per-workflow on an in-house quality metric). ID-104 is the substrate both principles stand on.

---

1. Code-intelligence orientation (cited, not paraphrased)

Orientation tools run against the knowledge-hub repo before authoring.

1.1 gitnexus_query({query: 'eval runner baseline severity touchpoint cost tracking'})

Top processes returned were Start → EstimateCost (lib/anthropic.ts:estimateCost :53-66), checkResponseQuality (lib/ai/quality-check.ts:186-242), and runDeterministicChecks (lib/ai/quality-check.ts:56-100). Standalone definitions surfaced the real eval surface:

• Function:lib/eval/baseline.ts:saveBaseline (:35-53) and Function:lib/eval/baseline.ts:evalPassed (:115-125) — the file-JSON baseline lifecycle.
• Interface:lib/eval/types.ts:ItemScoreBase (:20-24) — the per-item score shape.
• Function:scripts/quality-gate.ts:severityFor (:323-333) — an existing severity mapper, but for the content quality-gate, NOT the AI-eval layer (distinct surface; note for 2c boundary).
• Function:components/provenance/cost-tab-stub.tsx:fetchCostAggregate (:27-72) and CostTabStub (:21-128) — a labelled-interim (“Interim — Wave B”) cost tab UI, which reads an aggregate but has no persistence layer behind it.

1.2 gitnexus_context on the baseline lifecycle

context({name: 'saveBaseline'}) shows incoming CALLS from scripts/eval-procurement-drafting.ts:main, scripts/eval-search.ts:main, scripts/eval-summarisation.ts:main (plus index-corrupted entries for the canonical eval-items / load-env helpers). context({name: 'evalPassed'}) shows it calls checkRegression and is called by nothing indexed — i.e. the regression gate is wired through the per-suite main() scripts, not a central runner. There is no eval-runner symbol and no central dispatcher — each scripts/eval-*.ts is its own entry point.

1.3 Direct source read — lib/eval/ (5 files, ~17KB total)

• baseline.ts — loadBaseline / saveBaseline / checkRegression / evalPassed. Baselines are JSON files in __tests__/fixtures/eval-baselines/, keyed by suite_name. Threshold model today is { min?, max_drop? } only — no severity, no variance_band in the shipped EvalBaseline.thresholds.
• types.ts — EvalResult, EvalBaseline, RegressionResult, plus per-suite gold-item shapes. No AgentEvalContract, no touchpoint/kind/grounding_shape, no severity enum.
• metrics.ts — pure, stateless metric functions (precision/recall/F1, ROUGE-L/1, MRR, nDCG@k, P@k). Clean and reusable; no I/O.
• reporter.ts, fixtures.ts — reporting + fixture loaders.

1.4 SQL / Python sweep (gitnexus & ast-dataflow do not cover these)

grep over supabase/migrations/*.sql: zero ai_call / cost_track / eval_result / eval_run / touchpoint tables. grep over scripts/**/*.py: no eval/cost persistence. So recordAiCall() cost tracking has no DB substrate today — it is fully greenfield.

1.5 Greenfield confirmation for recordAiCall (ccc fallback noted)

grep -rniE 'recordAiCall|record_ai_call' over *.ts/*.py/*.sql returns zero hits. The ccc search fallback was attempted but the local cocoindex daemon errored (environment issue, not an absence signal); the grep + gitnexus evidence is conclusive on its own. recordAiCall(), the touchpoint registry, the central eval-runner, the nightly lane, and the persisted cost/eval tables are all GREENFIELD. The only adjacent existing artefacts are: the file-JSON baseline lifecycle (§1.3), the pure metrics, the labelled-interim cost-tab-stub.tsx (no persistence), and the CI-wired mcp-eval L1/L3/L4 suite (§4).

---

2. The three deliberate decisions for Liam ratification

2a — Name the eval-result / observability UI surface(s) explicitly

Three named surfaces compose into one coherent observability story; none is left implicit, and hosted raindrop.ai stays EXCLUDED (HR-5 client-data blocker; B-INV-15 forbids the graduation metric ever leaving for Raindrop cloud).

Surface	Scope (what it observes)	Hosting	Status today
Raindrop Workshop local viewer (localhost:5899, OTel-fed)	Agents + goose recipes (runtime traces)	MIT, LOCAL dev only; empty writeKey; no egress	Not wired in repo (no Raindrop refs found)
create-skill eval-viewer/viewer.html	Skills (per-skill eval review)	In-repo, static HTML (.claude/skills/create-skill/eval-viewer/viewer.html, confirmed present, ~45KB)	Exists today
KH-owned ID-104 results surface (/admin/refinement stub, inherited concept)	Touchpoints: registry listing + per-touchpoint signal/pattern/proposal/rollback	KH Next.js, admin-only, authFailureResponse()	Greenfield (concept inherited from phase-4 AC#12)

How they compose: the three are layered by touchpoint kind, not redundant. Raindrop Workshop is the agent/recipe runtime lens (the only one that traces a live goose run); create-skill’s viewer is the skill-authoring lens (per-skill, dev-time); the KH-owned surface is the registry-of-record lens — the one place that lists every registered touchpoint, its AgentEvalContract, its registry version, and its unprocessed-signal count across ALL kinds. The KH surface is the spine (it owns the contract + registry); the other two are kind-specific viewers it can deep-link to. The legacy cost-tab-stub.tsx (labelled-interim) is the natural host slot for the recordAiCall() cost rollup once the persistence layer lands — fold it in rather than spawn a fourth surface.

Recommendation (2a): Adopt all three, layered as above. The KH-owned surface is ID-104’s minimal deliverable: a stub page (Warm Meridian primitives only) listing registered touchpoints + registry version + unprocessed-signal count, plus per-touchpoint API endpoints (signals / patterns / proposals / version-history) an operator can curl — exactly the phase-4 AC#12 (REVISED, F-10) shape. The full cross-touchpoint dashboard is a named follow-up, not launch scope. Raindrop Workshop stays LOCAL-only with the egress assertion; raindrop.ai hosted stays EXCLUDED.

2b — Inherit-or-explicitly-scope the phase-4 HITL refinement layer

The historic phase-4 loop is: signal aggregation → pattern detection → Claude-proposed prompt diffs → A/B against gold standards → human approve → rollback registry, generalised across all touchpoints (ai-eval-phase4-hitl-refinement-spec.md §1.2). ID-71 narrowed this to just propose + publication-gate + the B-INV-38 forcing-function guard. This is the biggest scope lever in ID-104.

	Inherit full phase-4 loop	Scope to propose-only (ID-71 narrowing)
Delivers	Auto pattern→proposal→A/B→approve→rollback, all touchpoints	Propose-only; human authors the diff; gate on publish
Cost	Large: pattern detector, A/B runner, rollback registry, /admin/refinement full dashboard, SHA-256 drift detection	Small: contract + registry + severity model + the propose hook
Risk	A/B + auto-rollback infra is heavy and unproven at v1; over-builds before the graduation metric has data to act on	Defers the auto-refinement promise; manual loop only
Graduation fit	Auto-apply needs some of this (the metric + rollback)	The graduation metric (B-INV-15) is computed here regardless; auto-apply earned later

Recommendation (2b): SCOPE, do not inherit wholesale — phased. Build the substrate the loop needs (touchpoint registry with file_sha256 for git-backed touchpoints, recordAiCall() signal capture, the severity|variance model, the baseline-lifecycle hooks promote/history/compare, and the in-house graduation metric per B-INV-15) now, because the bottom-up engine and progressive-trust graduation both require them. Defer the heavy auto-refinement organs (cross-touchpoint pattern detector, parallel A/B runner, auto-rollback) to a named follow-up Task, gated on the registry actually accumulating signal. Rationale: A/B + auto-apply with no signal corpus is premature; the bottom-up substrate (registry + contract + metric) is what unblocks ID-71 and is genuinely long-term. This is an explicit scope decision with a named deferral, not a silent orphan — the phase-4 organs are written into ID-104’s “deferred, gated on signal volume” register.

2c — L1/L3/L4 reuse-vs-rebuild (DELIBERATE)

The shipped suite (scripts/mcp-eval/, CI-wired in ci.yml as the mcp-eval (l1|l3|l4) Staging matrix):

• L1 Protocol Compliance — protocol-compliance.ts (~786 LOC, 42 checks).
• L3 Response Quality — response-quality.ts (~1127 LOC, 17 checks).
• L4 Functional Correctness — functional-correctness.ts (~3107 LOC, 37 checks, live DB).

It is live, green, and load-bearing (it caught the S77 search_for_bid_response search_path regression). The skip-flag/seed/matrix CI policy is mature (WP-G4.4).

Recommendation (2c): REUSE as the touchpoint suite layer; REBUILD only the contract/registry/severity wiring above it. The L1/L3/L4 checks are exactly the “per-touchpoint eval suite” the AgentEvalContract.suite_name field points at — rebuilding ~5000 LOC of working, CI-wired, regression-proven checks would be pure value-destruction and violates “the approach chosen now must be long-term” (this suite is the long-term suite for MCP tools). What ID-104 rebuilds is the layer around them: register each L1/L3/L4 suite as touchpoints in the new registry, attach severity_on_fail + variance_band from the new model, and route their pass/fail through eval-runner + the contract — not the file-JSON saveBaseline path. Bottom-up justification: the suite is a clean, well-shaped artefact that already expresses the touchpoint-level eval the bottom-up design would author anyway; the orchestration around it (registry, contract, central runner, cost tracking) is what was never designed bottom-up and is the actual ID-104 deliverable. Reuse here is earned, not assumed.

---

3. Bottom-up reuse-vs-rebuild grounding per legacy surface

Legacy surface	Verdict	Clean-sheet reasoning
lib/eval/baseline.ts lifecycle (file-JSON, __tests__/fixtures/eval-baselines/)	REBUILD the storage/lifecycle; KEEP the regression semantics	The checkRegression math (min / max_drop, improvement-asymmetry per phase-3 F17) is correct and worth carrying. But file-JSON baselines keyed by suite_name cannot express a per-touchpoint registry, an owner-of-record, file_sha256 drift, or the contract fields. The bottom-up engine needs a DB-backed registry + baseline store (greenfield migration), not flat JSON. Port the regression semantics; rebuild the persistence.
callTool / per-suite main() harness (scripts/eval-*.ts, each its own entry)	REBUILD as a central eval-runner	Today there is no central runner — each suite script calls saveBaseline independently (gitnexus confirms no dispatcher symbol). Bottom-up requires one eval-runner that reads the registry, dispatches each touchpoint’s suite_name, applies the severity model, and writes results uniformly. The per-suite check logic (incl. L1/L3/L4) is reused as callees; the runner is new.
Severity model	REBUILD as the canonical 4-tier model; the phase-3 draft validates the shape	Phase-3 already prototyped severity?: 'block'|'warn'|'info' + variance_band as extensions to the threshold map — but never shipped them into EvalBaseline (confirmed: shipped types.ts has neither). ID-104 owns the canonical severity_on_fail: 'block'|'warn'|'info'|'infra' (note the added infra tier for Anthropic 529 / transient failures, per phase-3 §6.6) + variance_band: number on the contract, not bolted onto a baseline file. The phase-3 draft is corroborating evidence the shape is right; the implementation is clean-sheet on the contract. Distinct from scripts/quality-gate.ts:severityFor (content-quality-gate, unrelated surface — do not conflate).
metrics.ts pure functions	REUSE as-is	Stateless, dependency-free, correct. The bottom-up engine would author the same functions. Direct reuse, earned.
recordAiCall() + cost tables	BUILD greenfield	Zero existing persistence (§1.4-1.5). New migration for an ai_call / cost-event table; fold the rollup into the labelled-interim cost-tab-stub.tsx slot.
mcp-eval L1/L3/L4	REUSE as suite layer (see 2c)	—

---

4. The AgentEvalContract consumption surface (ID-104 owns the canonical definition)

ID-71’s guard (M38) reads this shape; ID-104 owns and may refine it. Recorded here as the bottom-up anchor for {104.2} PRODUCT / {104.3} TECH:

interface AgentEvalContract {
  touchpoint_id: string;            // stable id: tool name | prompt name | skill | recipe slug
  kind: 'tool' | 'prompt' | 'skill' | 'inline' | 'agent_recipe';
  owner: string;                    // touchpoint registry: owner of record
  suite_name: string;               // the eval suite this touchpoint runs under (e.g. L1/L3/L4)
  grounding_shape: 'structured_output' | 'forced_tool_strict' | 'citations' | 'n/a'; // B-INV-35
  severity_on_fail: 'block' | 'warn' | 'info' | 'infra';  // ID-104 model (note: 'infra' tier)
  variance_band: number;            // ID-104 model; tolerance before regression (phase-3 default 0.02)
}
// graduation_metric?: string;      // B-INV-15/19 — in-house WS-5 auto-apply metric (optional)

This MUST be defined in ID-104 (e.g. lib/eval/contract.ts), NOT cribbed into lib/eval/types.ts as ID-71’s last-resort placeholder (that fallback is a schedule-slip hedge, explicitly not the intent — ID-71 TECH §OQ-2). ID-104 is sequenced first.

---

5. Open questions for Liam ratification

1. (2a) Observability surfaces — Ratify the three-surface, layered-by-kind model: Raindrop Workshop (LOCAL, agents/recipes) + create-skill viewer (skills) + KH-owned /admin/refinement stub-spine (touchpoints, registry-of-record), raindrop.ai hosted EXCLUDED. Is the KH-owned stub (registry listing + per-touchpoint curl-able endpoints) the right minimal launch surface, with the full dashboard as a named follow-up?
2. (2b) HITL refinement — Ratify SCOPE-phased: build the substrate (registry + contract + severity model + recordAiCall signal capture + baseline-lifecycle hooks + in-house graduation metric) now; defer the auto-refinement organs (pattern detector, A/B runner, auto-rollback) to a named follow-up Task gated on signal volume. Or does Liam want the full phase-4 loop inherited at v1?
3. (2c) L1/L3/L4 — Ratify REUSE as the suite layer (register the 42+17+37 CI-wired checks as touchpoints under the new contract/runner), rebuilding only the registry/contract/severity orchestration around them. Confirm no rebuild of the working check logic.
4. (severity tiers) — Confirm the 4-tier 'block' | 'warn' | 'info' | 'infra' model (adding infra for transient-provider failures per phase-3 §6.6) as canonical, superseding the phase-3 3-tier draft.
5. (persistence boundary) — Confirm the bottom-up engine moves baselines from file-JSON (__tests__/fixtures/eval-baselines/) to a DB-backed touchpoint registry + baseline store (new migration), porting the checkRegression semantics but rebuilding storage.
6. (cost-surface host) — Confirm recordAiCall() cost rollup folds into the existing labelled-interim cost-tab-stub.tsx slot rather than a new surface.

---

6. Pointers (for {104.2} PRODUCT / {104.3} TECH)

• ID-71 TECH §OQ-2 (bottom-up steer + consumption shape), §X#2a-c (deliverable requirements). ID-71 PRODUCT HC-5 / B-INV-13/14/15/35/38/40 (born-evaluable invariants).
• ID-71 headless-requirement-refinement.md HR-5 (Raindrop Workshop LOCAL; hosted excluded).
• Historic reference (NOT a lift baseline): specs/ai-eval/ai-eval-phase3-regression- infrastructure-spec.md (severity/variance draft, baseline lifecycle), …phase4-hitl- refinement-spec.md (§1.2 loop, AC#12 stub ~:1136-1152), …phase5-full-coverage-spec.md, mcp-evaluation-spec.md §1 + impl note (L1/L3/L4 = 42+17+37, CI-wired).
• In-repo: lib/eval/{baseline,types,metrics,reporter,fixtures}.ts, scripts/eval-*.ts, scripts/mcp-eval/{protocol-compliance,response-quality,functional-correctness}.ts, .claude/skills/create-skill/eval-viewer/viewer.html, components/provenance/cost-tab-stub.tsx.