ID-104 {104.2} PRODUCT — bottom-up agent-eval engine

ID-104 {104.2} PRODUCT — bottom-up agent-eval engine

Date: 15/06/2026 Tier: Full chain ({104.1} RESEARCH ratified S356 → this PRODUCT → {104.3} TECH → {104.4} PLAN). Owns (canonical, bottom-up): the per-touchpoint AgentEvalContract, the touchpoint registry, the 4-tier severity + variance_band model, recordAiCall() signal/cost capture, the central eval-runner, the baseline-lifecycle hooks, the in-house graduation metric, the nightly lane, and the three layered eval/observability surfaces. Hard-upstream of: ID-71 Wave 2/3 — ID-71 consumes this contract via its M38 guard; it must not implement it (ID-71 TECH §OQ-2). ID-104 is sequenced first.

This PRODUCT turns the three ratified RESEARCH decisions (2a surfaces, 2b scope-phased HITL, 2c L1/L3/L4 reuse) plus the ratified technical defaults into numbered, testable behaviour invariants (B-INV) with explicit Pass/Fail. The big decisions are SETTLED in {104.1} RESEARCH §2 and Liam’s S356 ratification — they are restated here as invariants, not re-litigated.

---

Problem

Knowledge Hub’s platform direction stands on two principles (reference/platform-direction.md §AI-tooling direction): eval-everything — every AI touchpoint is born-evaluable — and progressive-trust graduation — agents propose, humans gate writes at launch, and auto-apply is earned per-workflow on an in-house quality metric. ID-104 is the substrate both principles stand on. Without it, ID-71’s born-evaluable waves (B-INV-13/14/15/38/40) cannot ship: there is no contract to declare, no registry to register against, and no metric to graduate on.

Why bottom-up (the non-negotiable framing)

Per Liam’s S354 steer (carried in {104.1} RESEARCH §0), ID-104 is a bottom-up rebuild, NOT a lift-and-extend. The agent-eval layer chosen now must be the long-term one. Every legacy artefact is a reuse-vs-rebuild reference, never an assumed baseline; reuse is earned by an explicit “this artefact already expresses the shape we’d design anyway” argument.

Code-intelligence orientation (cited, re-verified — not paraphrased)

The RESEARCH §1 orientation was re-verified against the knowledge-hub GitNexus index before authoring (the index reports partially stale — several saveBaseline incoming-call UIDs returned corrupted byte-garble entries; the conclusive evidence below is the clean symbol records plus direct source reads, consistent with RESEARCH §1.5’s note):

• gitnexus_query({query:'eval runner baseline severity touchpoint cost', repo:'knowledge-hub'}) surfaced lib/eval/baseline.ts:saveBaseline (:35-53), evalPassed (:115-125), lib/eval/types.ts:EvalResult/ItemScoreBase, scripts/quality-gate.ts:severityFor (:323-333 — the content quality-gate severity mapper, a DISTINCT surface; not the AI-eval layer — do not conflate), and components/provenance/cost-tab-stub.tsx:CostTabStub (:21-128) / fetchCostAggregate (:27-72).
• gitnexus_context({name:'saveBaseline', repo:'knowledge-hub'}) confirms incoming CALLS from scripts/eval-procurement-drafting.ts:main, eval-search.ts:main, eval-summarisation.ts:main and no central dispatcher (no eval-runner symbol) — each scripts/eval-*.ts is its own entry point.
• gitnexus_context({name:'CostTabStub', repo:'knowledge-hub'}) confirms it has no incoming callers and only calls its own fetchCostAggregate — a labelled-interim (“Interim — Wave B”) UI reading an aggregate with no persistence layer behind it.
• Direct read of lib/eval/types.ts confirms EvalBaseline.thresholds is Record<string, { min?: number; max_drop?: number }> only — no severity, no variance_band, no touchpoint/contract fields. lib/eval/metrics.ts is pure and stateless (precision/recall/F1, ROUGE, MRR, nDCG@k — no I/O).
• SQL/Python sweep (RESEARCH §1.4, re-confirmed): zero ai_call / cost_track / eval_result / eval_run / touchpoint tables in supabase/migrations/*.sql; zero recordAiCall hits in *.ts/*.py/*.sql. The ccc search fallback errored (local cocoindex daemon — environment issue, not an absence signal); grep + GitNexus are conclusive. recordAiCall(), the touchpoint registry, the central eval-runner, the nightly lane, and the persisted cost/eval tables are all GREENFIELD.

Orientation verdict: the touchpoint registry, contract, central runner, severity model, cost capture, persisted stores, and graduation metric are greenfield; the adjacent existing artefacts are the file-JSON baseline lifecycle (regression semantics worth porting), the pure metrics.ts, the labelled-interim cost-tab-stub.tsx, and the CI-wired L1/L3/L4 mcp-eval suite (reused as the suite layer per 2c).

---

Goals / Non-goals

Goals (in scope NOW): the touchpoint registry + AgentEvalContract; the 4-tier severity

• variance_band model; recordAiCall() signal/cost capture; baseline-lifecycle hooks (promote / history / compare); the in-house graduation metric; the central eval-runner (exit 0/1/2) registering the reused L1/L3/L4 suite as touchpoints; the nightly lane; the three layered observability surfaces (with the KH-owned /admin/refinement stub-spine as ID-104’s minimal owned deliverable); the cost rollup folded into cost-tab-stub.tsx.

Non-goals (DEFERRED to a NAMED follow-up Task, gated on signal volume — see B-INV-23): the cross-touchpoint pattern detector, the parallel A/B runner against gold standards, and the auto-rollback registry. These are recorded as a deferred-with-anchor boundary, NOT a silent orphan.

Out of scope permanently: hosted raindrop.ai on the client-data path (HR-5 / WS-14 blocker — B-INV-21 forbids it); rebuilding the L1/L3/L4 check logic (2c — only the orchestration around it is rebuilt); a fourth observability surface (the cost rollup folds into the existing interim cost tab — B-INV-19).

---

Behavior

Numbered, testable invariants. Each has an explicit Pass / Fail. Grouped: A. Contract + registry; B. Severity + variance; C. Runner + lifecycle + nightly lane; D. Cost capture + rollup; E. Graduation metric; F. Observability surfaces (2a); G. L1/L3/L4 reuse (2c); H. Scope-phased HITL boundary (2b); I. ID-71 consumption satisfaction.

A. The AgentEvalContract + touchpoint registry

1. (B-INV-1) ID-104 owns the canonical AgentEvalContract definition, defined bottom-up. The contract is declared in a dedicated ID-104-owned module (e.g. lib/eval/contract.ts), NOT cribbed into lib/eval/types.ts as ID-71’s last-resort placeholder (ID-71 TECH §OQ-2 — that fallback is a schedule-slip hedge, explicitly not the intent). Pass: the canonical contract type lives in an ID-104-owned eval module and is the single source of truth; ID-71 imports it. Fail: the contract is defined in ID-71’s surface, or duplicated as a provisional placeholder that ships alongside the canonical one.

2. (B-INV-2) The contract carries the seven mandatory consumption fields ID-71’s M38 guard reads, in the ratified shape. Every AgentEvalContract declares: touchpoint_id: string; kind: 'tool'|'prompt'|'skill'|'inline'|'agent_recipe'; owner: string; suite_name: string; grounding_shape: 'structured_output'|'forced_tool_strict'|'citations'|'n/a'; severity_on_fail: 'block'|'warn'|'info'|'infra'; variance_band: number. ID-104 MAY add further bottom-up fields, but these seven MUST be present and readable. Pass: a contract instance exposes all seven fields with the ratified types; ID-71’s M38 guard reads them without adaptation. Fail: any of the seven is absent, renamed, or retyped such that ID-71’s guard cannot read it.

3. (B-INV-3) The registry is the registry-of-record for every AI touchpoint. A persisted touchpoint registry lists every registered touchpoint keyed by touchpoint_id, with its owner of record, its bound AgentEvalContract, and a registry version. A touchpoint_id is unique within the registry. Pass: every registered touchpoint resolves to exactly one registry row carrying its owner + contract + version; a duplicate touchpoint_id registration is rejected. Fail: a touchpoint exists without a registry row, two rows share a touchpoint_id, or a row lacks an owner or bound contract.

4. (B-INV-4) Registration is the gate, not optional metadata. A touchpoint that is eval-relevant (any tool / prompt / skill / inline / agent_recipe) cannot be considered shipped until it is registered with a bound contract. This is the substrate ID-71’s B-INV-13 born-evaluable guard enforces against. Pass: an attempt to operate an unregistered eval-relevant touchpoint through the runner is rejected with a clear “not registered” signal. Fail: an unregistered touchpoint runs silently with no contract.

5. (B-INV-5) The registry records a registry version that advances on contract change. The registry exposes a registry version (and per-touchpoint contract version) so consumers and the observability surface can detect drift. Pass: changing a touchpoint’s contract advances its recorded version; the surface and API reflect the new version. Fail: a contract change leaves the version unchanged or the version is absent.

B. Severity + variance model

6. (B-INV-6) Severity is the canonical 4-tier model block | warn | info | infra. Every contract declares exactly one severity_on_fail from this set. The infra tier is reserved for transient-provider failures (e.g. Anthropic 529 / timeout) and is distinct from a genuine quality regression (per RESEARCH technical defaults / phase-3 §6.6). This supersedes the historic phase-3 3-tier draft. Pass: every contract carries one of the four tiers; an infra-classified failure is not counted as a quality regression. Fail: a contract carries a tier outside the four, or a transient-provider failure is mis-scored as a block/warn quality regression.

7. (B-INV-7) Severity determines the runner’s exit disposition. A block failure fails the gate (non-zero exit); warn and info are recorded and surfaced but do not fail the gate; infra failures are recorded as infrastructure noise and do not fail the gate on their own. Pass: a block failure yields a failing runner exit while warn/info/infra failures yield a passing-with-recorded-findings exit. Fail: a warn/info/infra failure fails the gate, or a block failure passes the gate.

8. (B-INV-8) variance_band is the per-touchpoint regression tolerance. A metric movement within ±variance_band of the baseline is NOT a regression; a drop beyond variance_band (in the worsening direction) IS. Improvement is never a regression (improvement-asymmetry, carried from phase-3 F17). The default band is 0.02 unless the contract overrides it. Pass: a metric drop inside the band passes regression, a drop beyond it fails, and an improvement of any size passes. Fail: a within-band drop fails, a beyond-band drop passes, or an improvement is flagged as a regression.

C. The central eval-runner, baseline lifecycle, and nightly lane

9. (B-INV-9) A single central eval-runner dispatches every touchpoint. The runner reads the registry, dispatches each touchpoint’s suite_name, applies the severity model, and writes results uniformly — replacing the per-suite scripts/eval-*.ts main() pattern where each script called saveBaseline independently (no central dispatcher exists today — gitnexus-confirmed). Pass: every touchpoint’s eval is dispatched through the one runner and results are written uniformly. Fail: a touchpoint’s eval bypasses the runner or writes results via an ad-hoc per-suite path.

10. (B-INV-10) The runner exits 0 / 1 / 2 deterministically. Exit 0 = all gating checks pass; exit 1 = at least one block-severity regression (quality gate fail); exit 2 = runner/infrastructure error (could not complete — distinct from a quality fail). Pass: the three conditions map to the three exit codes deterministically and a CI job can branch on them. Fail: a quality fail and an infra error return the same code, or the codes are non-deterministic.

11. (B-INV-11) Baselines are persisted in a DB-backed touchpoint store, not flat JSON. The bottom-up engine moves baseline persistence from file-JSON (__tests__/fixtures/eval-baselines/) to a DB-backed per-touchpoint baseline store, while porting the checkRegression semantics (min / max_drop / improvement-asymmetry) intact. Pass: a touchpoint’s baseline resolves from the DB-backed store and regression math matches the ported semantics. Fail: baselines still resolve from flat JSON keyed by suite_name, or the ported regression math diverges from the legacy semantics.

12. (B-INV-12) Baseline lifecycle is promote / history / compare, with an audit trail. An operator can promote a current result to the touchpoint’s baseline, view its baseline history, and compare two baselines. Each promotion is auditable (who/when/which registry version). Pass: promote advances the baseline and records an audit row; history lists prior baselines; compare returns the metric deltas between two. Fail: a promotion is unaudited, history is unavailable, or compare is absent.

13. (B-INV-13) A nightly lane runs the full registered touchpoint set. A scheduled nightly run dispatches every registered touchpoint through the runner and records results + severity dispositions for the observability surface; it is distinct from the PR-blocking CI gate (which may run a subset). Pass: the nightly lane runs the full registered set on schedule and its results are visible on the surface. Fail: the nightly lane runs only a subset of registered touchpoints, or its results are not recorded.

D. recordAiCall() cost + signal capture, and the rollup

14. (B-INV-14) recordAiCall() is the single capture point for AI-call signal + cost. Every instrumented AI touchpoint call records, via recordAiCall(), at minimum: the touchpoint_id, the model/tier, token usage, computed cost, and an outcome signal (e.g. success / refusal / loop / error class). This is greenfield (no persistence exists today). Pass: an instrumented call writes a cost+signal record keyed by touchpoint_id. Fail: an AI call at an instrumented touchpoint records no cost/signal, or records it un-keyed to a touchpoint.

15. (B-INV-15) Cost capture has a persisted substrate. recordAiCall() writes to a persisted cost/signal store (new migration); the store is per-tenant-safe and never egresses client data off-platform. Pass: records persist and survive process restart; a network assertion confirms zero off-platform egress of the captured data. Fail: records are in-memory only, or any captured client-derived data egresses off-platform.

16. (B-INV-16) recordAiCall() presence is enforceable as a guard surface. The capture point is greppable/guardable so a tooling-change guard (ID-71’s M38 extends this) can assert instrumented touchpoints actually call it. Pass: a guard test can detect an instrumented touchpoint that omits recordAiCall(). Fail: an instrumented touchpoint can omit capture with no guard signal.

17. (B-INV-17) Captured cost rolls up into the existing cost-tab-stub.tsx slot. The recordAiCall() cost data surfaces through the labelled-interim components/provenance/cost-tab-stub.tsx (which today reads an aggregate with no persistence behind it) — folding the persistence behind the existing slot rather than spawning a fourth surface. Pass: the cost tab reads real persisted aggregates from the recordAiCall() store. Fail: a new parallel cost surface is created, or the cost tab still reads a stub with no persistence.

E. The in-house graduation metric

18. (B-INV-18) The graduation metric is computed in-house, never via Raindrop cloud. The per-workflow quality metric that earns auto-apply (WS-5 progressive-trust graduation) is computed on ID-104’s own infrastructure from the captured signals/evals; no client-data trajectory is POSTed to Raindrop cloud. Pass: the metric is produced by the in-house harness and a network assertion confirms zero client-data egress to Raindrop cloud. Fail: the metric depends on Raindrop cloud, or client-derived trajectories egress to Raindrop cloud. (Directly satisfies ID-71 B-INV-15.)

19. (B-INV-19) The graduation metric is contract-addressable. A contract MAY declare an optional graduation_metric?: string naming the in-house metric a touchpoint graduates on; where present, the runner/surface can report a touchpoint’s current value against it. Pass: a touchpoint declaring graduation_metric reports its current value through the surface/runner. Fail: graduation_metric is declared but unreadable, or auto-apply is granted with no metric basis. (Note: this PRODUCT computes the metric; auto-apply on it is the deferred follow-up per B-INV-23.)

F. The three layered observability surfaces (2a — ratified)

20. (B-INV-20) Three layered surfaces compose by touchpoint kind; none is implicit. The observability story is (i) Raindrop Workshop local viewer (localhost:5899, OTel-fed) for agents + goose recipes (runtime traces); (ii) create-skill’s eval-viewer/viewer.html for skills (per-skill, dev-time); (iii) the KH-owned /admin/refinement stub as the registry-of-record spine across ALL touchpoint kinds. They are layered by kind, not redundant; the KH surface owns the contract + registry and deep-links to the kind-specific viewers. Pass: each surface is present and scoped to its kind; the KH surface is the single registry-of-record. Fail: a surface is dropped, two surfaces claim the registry-of-record role, or the layering is left implicit.

21. (B-INV-21) Raindrop is Workshop-local only; hosted raindrop.ai is excluded from the client-data path. Raindrop Workshop runs locally (MIT, empty writeKey ⇒ cloud POST is a no-op, OTel-fed); hosted raindrop.ai is permanently excluded from the client-data path (HR-5 / WS-14 blocker). Pass: a network assertion confirms Raindrop Workshop with empty writeKey performs zero trace egress; no code path POSTs client-derived data to raindrop.ai. Fail: any client-derived trace egresses to Raindrop cloud, or a hosted Raindrop integration sits on the client-data path.

22. (B-INV-22) The KH-owned /admin/refinement stub is the minimal launch surface, admin- only. ID-104’s owned launch deliverable is a stub page (Warm Meridian semantic tokens only; UK English) listing registered touchpoints + registry version + per-touchpoint unprocessed-signal count, plus per-touchpoint API endpoints an operator can curl: signals / patterns / proposals / version-history (the phase-4 AC#12 REVISED F-10 shape). It is admin-only, gated via authFailureResponse(auth) on auth.success. The full cross-touchpoint dashboard is a NAMED follow-up, not launch. Pass: a non-admin request is rejected via the auth-failure helper; an admin sees the registry listing + version + signal counts and can curl the four per-touchpoint endpoints; colour is never the sole signal (WCAG 2.1 AA). Fail: the stub is reachable by non-admins, omits any of the four endpoints, uses raw colour tokens, or attempts the full dashboard at launch.

G. L1/L3/L4 reuse as the suite layer (2c — ratified)

23. (B-INV-23) The 96 CI-wired L1/L3/L4 checks are REUSED as touchpoint suites, not rebuilt. The shipped scripts/mcp-eval/ suite — L1 protocol-compliance (42 checks), L3 response-quality (17), L4 functional-correctness (37) = 96 CI-wired checks — is registered as touchpoints under the new contract/runner with severity_on_fail + variance_band attached; its pass/fail routes through eval-runner + the contract, NOT the file-JSON saveBaseline path. The check logic is reused unchanged. Pass: each L1/L3/L4 suite is a registered touchpoint with a bound contract and routes through the central runner; the check logic is unchanged from the shipped suite. Fail: any L1/L3/L4 check logic is rewritten, or the suite still routes through the legacy file-JSON baseline path.

H. Scope-phased HITL refinement boundary (2b — ratified, deferred-with-anchor)

24. (B-INV-24) The HITL substrate is IN scope; the auto-refinement organs are DEFERRED with an explicit anchor. IN scope NOW (the substrate the loop needs): the touchpoint registry, AgentEvalContract, the severity|variance model, recordAiCall() signal capture, the baseline-lifecycle hooks (promote/history/compare), and the in-house graduation metric. DEFERRED to a NAMED follow-up Task gated on signal volume: the cross-touchpoint pattern detector, the parallel A/B runner against gold standards, and the auto-rollback registry. This is an explicit scope decision with a named deferral, NOT a silent orphan — the deferred organs are recorded in ID-104’s “deferred, gated on signal volume” register with a pointer back to this invariant. Pass: the substrate invariants (1–19) ship and the three deferred organs are recorded as a named follow-up with a gating condition (signal volume) and a back-pointer. Fail: a deferred organ is built at v1, OR a deferred organ is dropped with no named follow-up anchor (silent orphan).

I. ID-71 born-evaluable consumption satisfaction

25. (B-INV-25) ID-104’s contract + registry satisfy ID-71’s HC-5 born-evaluable consumption. The contract (B-INV-1/2), registry (B-INV-3/4), severity model (B-INV-6), grounding-shape field (B-INV-2), graduation metric (B-INV-18/19), and the recordAiCall() guard surface (B-INV-16) collectively provide everything ID-71’s M38/M13/ M14/M15/M40 guards read — so ID-71 can ship born-evaluable without implementing any eval-engine internals. Pass: ID-71’s guard imports the ID-104 contract and reads all consumption fields; no eval-engine internal is duplicated in ID-71. Fail: ID-71 must re-implement any contract/registry/severity/metric surface to ship born-evaluable. (See the explicit mapping below.)

---

ID-71 HC-5 consumption mapping (which ID-104 B-INV satisfies which ID-71 need)

ID-71 invariant	What ID-71 needs	Satisfied by ID-104 B-INV
HC-5 / B-INV-13 — every touchpoint ships an ID-104 eval contract; the guard rejects a contract-less touchpoint	A canonical contract type + a registry to register against + registration-as-gate	B-INV-1 (canonical contract), B-INV-2 (seven fields), B-INV-3/4 (registry + registration gate)
B-INV-14 — fleet evals authored locally (Raindrop Workshop), committed into KH’s L3/L4 + baseline harness, KH-owned format	KH-owned suite/baseline harness + Workshop-local (not Raindrop-proprietary)	B-INV-23 (L3/L4 reused as KH-owned suites under the runner), B-INV-11/12 (KH baseline harness), B-INV-20/21 (Workshop-local)
B-INV-15 — WS-5 graduation metric computed in-house, never via Raindrop cloud, zero client-data egress	An in-house metric + an egress-zero assertion	B-INV-18 (in-house metric, egress assertion), B-INV-19 (contract-addressable), B-INV-21 (Workshop zero-egress)
B-INV-35 — every touchpoint declares exactly one grounding shape	A grounding_shape field on the contract	B-INV-2 (grounding_shape: 'structured_output'|'forced_tool_strict'|'citations'|'n/a')
B-INV-38 — a tooling change forces a skill + eval/fixture update, enforced by the recordAiCall() grep-guard pattern	A greppable recordAiCall() capture point the guard extends	B-INV-16 (recordAiCall() guardable presence), B-INV-4 (registration gate)
B-INV-40 — schema enforcement is not the quality gate; the ID-104 eval is	An eval that runs in addition to the schema constraint, gating on severity	B-INV-6/7 (severity gate), B-INV-9/10 (runner is the gate), B-INV-23 (per-touchpoint suite)

---

Open questions for Liam ratification

The big decisions are SETTLED (S356). Only genuinely new questions surfaced during PRODUCT authoring are raised here.

1. (B-INV-14 outcome-signal taxonomy) — recordAiCall() captures an outcome signal (success / refusal / loop / error class). RESEARCH HR-5 notes Raindrop’s Signals taxonomy (win / fail / loop / refusal-rate) is a design reference only. Confirm the in-house outcome-signal enum to capture at launch — adopt the Raindrop-derived four (win/fail/loop/refusal) as the bottom-up starting set, or a narrower KH-specific set? This is a {104.3} TECH detail but worth a one-line steer so the persistence schema is sized right.

2. (B-INV-22 endpoint surface for deferred organs) — the /admin/refinement stub exposes per-touchpoint signals / patterns / proposals / version-history endpoints. patterns and proposals correspond to the DEFERRED organs (B-INV-24). Confirm these two endpoints ship at launch as empty-but-present stubs (so the surface shape is stable and the follow-up only fills them), rather than being absent until the follow-up — the phase-4 AC#12 shape implies present-but-empty, which is the assumed reading here.