AI Evaluation Phase 3: Regression Infrastructure Spec

AI Evaluation Phase 3: Regression Infrastructure Spec

Status: Draft pending adversarial review Author: Claude Code (S154 WP4) Date: 08/04/2026 Roadmap: §3.3.1 through §3.3.5 Parent strategy: docs/audits/ai-eval/ai-touchpoint-map-and-improvement-strategy.md §8 Phase 3 Parent gap analysis: docs/audits/ai-eval/ai-eval-gap-analysis-s149.md Phase 4 dependency consumer: docs/specs/ai-eval-phase4-hitl-refinement-spec.md §3 Dependencies

Review status: Updated 08/04/2026 per S155 WP1.2 adversarial review (docs/audits/s155-spec-review-ai-eval-phase3-regression-infrastructure.md). Verdict: PASS WITH NOTES → findings applied (4 HIGH, 9 MED, 9 LOW, 3 NIT). Phase 3 implementation may now begin.

---

1. Overview

Phase 3 of the AI evaluation pathway is the regression-detection layer. Phases 1 and 2 produced the baselines (lib/eval/baseline.ts

• five eval scripts). Phase 3 makes those baselines load-bearing — every model change, taxonomy change, data-model change, and skill file edit triggers an automated check that the relevant baseline has not regressed beyond a defined tolerance.

Phase 4 (HITL refinement) is built on top of Phase 3. The Phase 4 spec already on main (docs/specs/ai-eval-phase4-hitl-refinement-spec .md §3 Dependencies) explicitly relies on Phase 3 for: (a) signal collection from eval failures, (b) statistical machinery for the A/B testing path, (c) the regression threshold model, and (d) the baseline lifecycle. This spec must deliver each of those contracts.

1.1 Where Phase 3 fits

Phase 1 (Foundation) ─► Phase 2 (Skill conversion) ─► Phase 3 (Regression infra)
                                                       │
                                                       ├─► Phase 4 (HITL refinement)
                                                       └─► Phase 5 (Full coverage)

Phase 3 sits between the static “we have baselines” state and the dynamic “the baselines actually catch problems automatically” state. Without Phase 3, every regression has to be caught by hand on the day it happens, which means most regressions are not caught at all.

1.2 Items in scope

#	Item	Roadmap
3.3.1	Per-touchpoint eval automation	Should
3.3.2	Taxonomy change impact detection	Should
3.3.3	Model change protocol	Should
3.3.4	CI gates for eval suites	Should
3.3.5	Data-model + workspace-creation change impact detection	Should

1.3 What Phase 3 is NOT

• Not Phase 4 HITL refinement — Phase 3 detects regressions; Phase 4 proposes fixes. Separate specs, separate workstreams.
• Not Phase 5 full coverage — Phase 3 only automates the eval suites that already exist. New eval suites (digest, summarisation skill conversion, MCP tool description audit) are Phase 5.
• Not the entity classification tightening (§1.5) — that’s the workstream that uses Phase 3’s regression detection to prove its iterations are honest. Phase 3 delivers the harness; §1.5 uses it.

---

2. Motivation

Concrete risks Phase 3 mitigates, each grounded in something that has happened or is about to happen on the Knowledge Hub roadmap:

M-1. Silent model upgrades. Anthropic ships a new default model periodically (Claude Sonnet 4.5 → 4.6). The default model used by getModelForTier('default') shifts under code that imports it. There is no regression check today. A model change that quietly degrades classification precision by 5pp would land without anyone noticing until a customer complains. Phase 3.3.3 closes this.

M-2. Taxonomy changes. The taxonomy admin UI lets editors add or modify domains and subtopics. Each change forces re-classification of new content under the new vocabulary. There is no automatic regression check that the classification eval still passes after a taxonomy edit. A poorly-named subtopic could push the eval below its threshold and nobody would know. Phase 3.3.2 closes this.

M-3. Data-model migrations. Adding a new column to content_items or changing how entity_mentions is keyed could subtly break the classification pipeline (e.g. by changing what classifyContent reads or writes). The pipeline-parity guard catches some of this, but the semantic effect on eval metrics is invisible. Phase 3.3.5 closes this.

M-4. Workspace provisioning regressions. When a new workspace is created (admin UI flow), the first batch-classification on that workspace’s content uses whatever skill file is currently on disk. If a skill edit is in flight when a new workspace is provisioned, the new workspace gets the in-flight version. Phase 3.3.5 also closes this — specifically the workspace-creation analog of M-3.

M-5. Pre-merge regression. Today an engineer can land a skill file edit, a filter expansion, or a model parameter change without running any eval. The change ships, and the precision number is whatever it ends up being. Phase 3.3.1 + 3.3.4 close this with pre-merge eval automation and CI gating.

---

3. Dependencies

3.1 Hard external dependencies (must exist before Phase 3 lands)

• A CI pipeline — Knowledge Hub currently has zero GitHub Actions workflows (.github/workflows/ does not exist). Phase 3 cannot CI-gate anything until CI exists. This spec commits to Phase 3 building the minimal CI foundation itself, rather than waiting on §3.6. See §4.3.
• lib/eval/baseline.ts — already exists. Provides loadBaseline, saveBaseline, checkRegression. Phase 3 extends but does not replace this.
• The five eval scripts — already exist (scripts/eval-{classification, entity-classification, summarisation, search, bid-drafting}.ts). Phase 3 wraps them with automation rather than rewriting them.

3.2 Soft dependencies

• §3.6 MCP Eval Layer 5 GitHub Actions CI — orthogonal but overlapping. Phase 3 ships its own minimal CI; §3.6 adds the MCP eval to that CI. Whichever lands first creates the foundation; the other extends it.
• §9.7 Structured logging — Phase 3’s observability would benefit from structured logging, but does not block on it. Plain console output is acceptable for Phase 3 v1.
• §1.5 Entity classification tightening — uses Phase 3’s regression detection as its iteration loop. Not a dependency on Phase 3 (Phase 3 can land without §1.5), but Phase 3 should validate that its API can support §1.5’s iteration protocol.
• Silent-failure prevention spec Phase 5 (docs/specs/silent- failure-prevention-spec.md §5.7) — also waits on CI. Phase 3’s CI foundation unblocks it.
• Phase 5 full coverage spec (docs/specs/ai-eval-phase5-full- coverage-spec.md) — soft dependency only (F18). The Phase 5 spec text says “Eval integrated into Phase 3 regression detection (if Phase 3 has landed)”, which means Phase 5 can ship independently of Phase 3 and hook into Phase 3 retroactively. Phase 3’s CI foundation is reusable by Phase 5 but not a hard prerequisite.

3.3 Phase 4 dependency contracts (Phase 3 must expose these)

Per docs/specs/ai-eval-phase4-hitl-refinement-spec.md §3 Dependencies, Phase 3 must expose:

1. Eval regression scanner contract — a callable that, given a touchpoint identifier, returns the most recent regression result (or null if no regression). Phase 4’s EvalRegressionScanner SignalSource adapter consumes this.
2. Statistical machinery for A/B testing — the ability to run the same eval suite against two prompt variants and produce a per-metric delta with a confidence indicator. Phase 4’s A/B evaluator builds on this.
3. Per-suite regression threshold model — Phase 4 needs to know what counts as a regression. Phase 3 must define and expose this.
4. Baseline lifecycle hooks — Phase 4 needs a way to “promote a new prompt version → save its eval result as the new baseline” atomically. Phase 3 must provide a clean API for this.

---

4. Architecture

4.1 Tier model

Different eval suites have different latency and cost profiles, so they slot into different automation tiers:

Tier	Latency	Cost	Trigger	Eval suites
Pre-merge (PR check)	<2 min (target — see note below)	$0	Every PR + push to main	bun run test, bun lint, fixture-count guards
Pre-merge AI-light	<5 min	<$0.10	PRs touching lib/ai/** or scripts/eval-* (paths: filter — see §4.3.2)	Search eval (no LLM calls), classification eval --cached
Scheduled nightly	<60 min	~$5.50–$8	Cron 02:00 UTC (incl. weekends — Q1)	All five live eval suites including eval:entity --live --validate
Manual / on-demand	varies	varies	gh workflow run	Full Phase 5 coverage when it lands

Pre-merge <2 min budget is aspirational and unverified (F10). No session has measured the wall-clock for bun run test + bun lint + bun run knip on a fresh checkout. With ~9131 tests at S154 baseline, parallelised Vitest plus ESLint plus knip is plausibly 2–4 minutes but has not been measured. Phase 3.0 must begin with a wall-clock measurement (see §5.1 Tasks) and the budget must be re-stated if the measurement exceeds 2 minutes — either by raising the target or splitting the suite. Do not gate PRs on <2 min until the measurement exists.

Entity --live --validate latency is DISPUTED (F2). Two estimates disagree by an order of magnitude:

• Rate-limit math (~3 min): scripts/eval-entity-classification.ts:741 rate-limits at 1 req/sec. 95 items × ~1.5s/item ≈ ~2.5 min wall-clock.
• Empirical (~25 min): docs/specs/entity-classification-prompt-tightening-spec.md and docs/reference/two-pass-validation-results.md both report ~25 min for the two-pass run as actually measured during S151 WP3.

Both cannot be right. The entity classification spec’s diagnostic question D-Q1 will empirically resolve this. Until then, Phase 3 must budget for both outcomes:

• The <60 min nightly budget is sized for the 25-minute entity case (the safer, larger figure), so the nightly cron is unaffected regardless of which way D-Q1 resolves.
• If D-Q1 confirms ~3 min, add a Phase 3.x follow-up to evaluate whether entity --live --validate could in fact run on the AI-light pre-merge tier for PRs touching lib/ai/skills/classification.md or __tests__/fixtures/entity-eval-*. That would shorten the §1.5 entity prompt-tightening feedback loop dramatically. Track this as conditional follow-up; do not commit to it in Phase 3.0–3.1.
• If D-Q1 confirms ~25 min, the existing “entity is nightly only” framing stands as-is.

The entity classification --live --validate run cannot block PR merge under either resolution (latency too high in the empirical case, cost still meaningful in the rate-limit case). It runs nightly and its results are reported in a daily summary; regressions trigger an alert but do not block existing PRs.

4.2 Regression threshold model

Extends the existing lib/eval/baseline.ts { min, max_drop } model with two new fields:

export interface RegressionThreshold {
  /** Absolute minimum — current value must be >= min (existing) */
  min?: number;
  /** Maximum allowed drop from baseline — (baseline - current) <= max_drop (existing) */
  max_drop?: number;
  /** NEW: Severity tier — controls whether a breach hard-fails CI or just warns */
  severity?: 'block' | 'warn' | 'info';
  /** NEW: Tolerance for inherent LLM variance — a breach within this band is reported as warn, not block */
  variance_band?: number;
}

Defaults: severity: 'warn', variance_band: 0.02 (2pp). Hard-fail gating is opt-in per metric so Phase 3 can roll out as warn-only first.

Per-metric override (F22): the 2pp default applies only when a baseline file does not specify its own variance_band for that metric. Each baseline file’s threshold map MAY override variance_band per metric — for example, entity precision (which has historically moved by ~1pp between runs) should set a tighter band than the domain_accuracy metric (currently ~0.97, where 2pp is fine). The runner reads the per-metric value first and falls back to the global default only when absent.

Improvement asymmetry (F17). variance_band is intentionally asymmetric. A current value above baseline by more than variance_band is reported as an improvement, never as a breach. Implementers must NOT code this as Math.abs(current - baseline) > variance_band. The existing checkRegression at lib/eval/baseline.ts:65-105 already gets this right (it only checks currentValue < min and (baseline - current) > max_drop); the threshold extension preserves that one-sided behaviour. The runner’s exit code for the “every metric improved by more than variance_band” case is 0 with an “improvement detected” log line in the summary (Q-MISS-5).

Renamed/missing metrics (F19). If a baseline threshold key has no matching metric in the current run (e.g. a metric was renamed from subtopic_accuracy to taxonomy_subtopic_accuracy), the runner MUST emit a loud warning rather than silently treating the missing value as zero. The existing checkRegression at lib/eval/baseline.ts:71 silently treats missing metrics as zero — Phase 3 must change this so the runner detects renames/missing metrics and surfaces them as a distinct category in the summary (infra failure / data shape change, not a regression). Per S155 WP1.2 audit Q-MISS-3.

4.3 CI foundation (Phase 3.0 — the prerequisite)

Phase 3 ships the first CI workflows for the Knowledge Hub repo. Two new files under .github/workflows/:

.github/workflows/ci.yml — runs on PR + push to main:

• bun install --frozen-lockfile
• bun run test (the full Vitest suite, ~9131 tests at S154 baseline — wall-clock budget unverified; see §4.1 note on the <2 min target and the F10 measurement task in §5.1)
• bun lint
• bun run knip (warn-only — knip false positives are tracked under roadmap §14.7)

.github/workflows/eval-nightly.yml — runs on cron 0 2 * * * (includes weekends, per Q1):

• bun install --frozen-lockfile
• Each of the five eval scripts is invoked individually (not via bun run eval:all — see F6 note below) so that one suite’s failure does not short-circuit the rest:
  • bun run eval:classification
  • bun run eval:summarisation
  • bun run eval:search
  • bun run eval:bid-drafting (skipped until §1.4 lands real bid data — see Q3)
  • bun run scripts/eval-entity-classification.ts --live --validate --confirm --save-baseline (the entity live run, invoked directly rather than via the EVAL_ENTITY=1 Vitest wrapper at __tests__/eval/entity-classification-eval.test.ts:1-15 — see F25 note below)
• Each invocation runs inside a set +e (or per-step continue-on-error: true) block so a failure in one suite does NOT short-circuit the others (F3 — see §6.5 below).
• Posts a single consolidated summary comment to a dedicated tracking issue at the end, regardless of which suites passed or failed, including a per-suite status table and a “cost burned: $X” line.
• The summary explicitly distinguishes between (a) real regressions, (b) infra failures (infra severity tier — Anthropic 529, OpenAI 503, etc., see §6.6), and (c) renamed/missing metrics (see §4.2).

F6 — eval:all is NOT used by the nightly workflow. The package.json script eval:all chains the five suites with &&, which short-circuits on the first failure and (worse) already includes eval:entity (which is itself the EVAL_ENTITY=1 Vitest wrapper). Calling eval:all plus the explicit scripts/eval-entity-classification.ts line would (a) double-run the entity eval and (b) lose the second-and-subsequent suites’ results on any earlier failure. The nightly workflow therefore calls each script individually as listed above. eval:all may continue to exist as a local-developer convenience but the CI workflow does not invoke it.

F25 — entity eval invocation layer. package.json’s eval:entity script is EVAL_ENTITY=1 bun run test __tests__/eval/entity-classification-eval.test.ts — a Vitest wrapper gated on the env var. The nightly workflow bypasses the wrapper and calls scripts/eval-entity-classification.ts --live --validate --confirm --save-baseline directly so the runner gets a clean exit code without the Vitest reporter layer in between. The Vitest wrapper remains in place for local-developer ergonomics (bun run eval:entity in a terminal is friendlier than the long script invocation).

Bun version pin (CRITICAL — F4, CLAUDE.md gotcha). Both workflows MUST pin bun-version: 1.3.4 in setup-bun@v2 (not “latest”). Per CLAUDE.md “Bun fetch hangs on HTTP 204 through sandbox proxy” gotcha, the issue is pinned to Bun 1.3.4 specifically; CI auto-upgrading bun is a real regression vector even though the GitHub Actions runner itself is not sandboxed. Until the Bun upstream bug is fixed, pin the version explicitly.

These two workflow files are also the foundation for §3.6 (MCP Eval Layer 5 CI integration), the silent-failure prevention spec Phase 5, and any future CI-gated work. Phase 3 owns the foundation; subsequent specs extend it.

4.3.1 GitHub Actions hygiene checklist (F4 — must include)

Both ci.yml and eval-nightly.yml MUST include the following keys. The implementer is free to write the YAML in their preferred style, but every entry below is mandatory unless explicitly noted as “optional”:

Key	Required	Notes
name:	yes	Human-readable workflow name.
on: pull_request: + push: (ci.yml only)	yes	ci.yml runs on every PR + push to main.
on: schedule: cron: '0 2 * * *' (eval-nightly.yml only)	yes	02:00 UTC nightly, includes weekends per Q1.
on: workflow_dispatch:	yes (both)	Allows Liam to trigger ad-hoc runs without waiting for cron. Eval-nightly especially.
permissions:	yes	Set explicit minimal permissions per job. Default contents: read; issues: write only for the eval-nightly summary-posting job. NEVER use permissions: write-all.
concurrency: group: + cancel-in-progress:	yes	Group by workflow + ref so duplicate PR pushes cancel earlier in-flight runs of ci.yml. For eval-nightly.yml, group by workflow only and set cancel-in-progress: false so a cron tick that overlaps a manual run does not abort it.
timeout-minutes: (per job)	yes	ci.yml jobs ≤ 10 min. eval-nightly.yml jobs ≤ 90 min (covers the 25-min entity case + buffer + retries). Without this a stuck Anthropic call could hang the runner for the GH Actions default 6 hours and burn the cost cap.
setup-bun@v2 with bun-version: 1.3.4	yes	Pin per CLAUDE.md gotcha.
setup-node@v4 (only if needed)	optional	Vitest does not require Node, only Bun. Skip unless an action explicitly needs Node.
Required secrets	yes	See §4.3.2 below.
actions/checkout@v4	yes	Standard.
actions/cache for ~/.bun/install/cache	optional	Speeds up re-runs but not load-bearing for v1.
if: failure() summary-poster step	yes (eval-nightly.yml only)	Per F3, the summary MUST be posted even on partial failure. Pair with continue-on-error: true on the per-suite steps.

4.3.2 Required secrets (F4, F15)

The following GitHub Actions repository secrets must be set before either workflow runs successfully. The implementer should add a checklist item to the Phase 3.0 PR description so Liam confirms each secret exists before merging.

Secret	Used by	Notes
ANTHROPIC_API_KEY	classification, entity, summarisation, bid-drafting evals	Anthropic SDK reads ANTHROPIC_API_KEY from env.
OPENAI_API_KEY	search eval (embeddings), classification eval (embedding lookups)	text-embedding-3-large via OpenAI.
SUPABASE_SECRET_KEY	every eval script (DB writes for --save-baseline, even --cached mode reads classifications from the DB)	Service-role key. F15 — --cached mode is NOT credential-free; CI still needs this secret.
SUPABASE_URL	Python eval scripts (none in Phase 3, but kept for parity)	eu-west-2 Supabase project URL.
NEXT_PUBLIC_SUPABASE_URL	TypeScript supabase-js client used by eval scripts	Same value as SUPABASE_URL.
NEXT_PUBLIC_SUPABASE_ANON_KEY	TypeScript supabase-js client	Public-side key; safe to expose to runner env.
AI_SUMMARY_MODEL	(optional) summarisation eval if pinning model under test	Defaults to claude-sonnet-4-6 per CLAUDE.md. Set explicitly only if Phase 3.3.3 is mid-protocol.

GitHub Actions secrets are masked in workflow logs by default; the spec relies on this default behaviour and does not introduce any additional masking layer.

4.3.3 Path filter for the AI-light tier (F14)

The §4.1 “Pre-merge AI-light” tier triggers only on PRs that touch lib/ai/** or scripts/eval-*.ts. GitHub Actions implements this via a paths: filter on the pull_request event:

on:
  pull_request:
    paths:
      - 'lib/ai/**'
      - 'scripts/eval-*.ts'
      - 'lib/eval/**'
      - '__tests__/fixtures/eval-baselines/**'

This MAY live in ci.yml as a separate job (eval-light: job with its own paths: block at the workflow level is not possible in GH Actions — the workflow paths: filter applies to ALL jobs in the file). For job-level path filtering, the implementer should use the dorny/paths-filter action OR put the AI-light job in a separate workflow file (ai-light.yml). The latter is simpler and recommended for v1.

4.4 Baseline lifecycle

Three new operations:

1. promoteBaseline(suiteName, runId) — atomically marks a specific eval run as the new baseline. Records the previous baseline in a git-auditable history. Used by Phase 4 when an approved A/B winner should become the new reference.

   F7 — runId source and history/ mechanism (clarified):

   • runId is generated by the runner, not user-supplied. Format: {ISO8601-UTC-timestamp}-{git-sha-short} — e.g. 20260408T020134Z-3e82dbca. The runner emits this ID at the start of every invocation (CI cron, manual, or local) and it appears in the run’s summary output. To promote a specific past run, the user copies the runId from that run’s summary. The runner does NOT persist a database of past runs; runIds are recoverable from the GitHub Actions UI run-page URL or the local terminal output.
   • history/ directory: prior baselines are written to __tests__/fixtures/eval-baselines/history/{suiteName}/{runId}.baseline.json by promoteBaseline BEFORE the current baseline file is overwritten. The history directory is checked into git (small JSON files; total volume is bounded by the number of deliberate promotions, which is O(months) not O(days)).
   • Atomic write (F9 — see §6.7 below): promoteBaseline writes the new baseline to a *.tmp sibling file then renames it into place, so a crash mid-write cannot leave the file half-written.
   • Git history is the secondary record: the diff between the old and new __tests__/fixtures/eval-baselines/{suite}.baseline.json is captured by the commit that lands the promotion. The history/ directory exists for runs that were promoted but never committed (rare — A/B testing iterations during Phase 4 experimentation).
   • CLI invocation: bun run scripts/eval-runner.ts promote --suite <name> --run-id <runId>.

2. baselineHistory(suiteName) — returns the chronological list of baselines for a suite. Reads the __tests__/fixtures/eval-baselines/history/{suiteName}/ directory AND walks git log for the canonical baseline file. The history/ directory takes precedence for any runId that exists there; git history is the fallback.

3. compareBaselines(suiteName, baselineA, baselineB) — produces a per-metric delta between two historical baselines. Used by Phase 4 for A/B path 2 (gold-standard A/B).

4.5 Statistical confidence

For variance estimation, Phase 3 uses a pragmatic approach: each nightly eval run is treated as one sample. Two consecutive runs that agree within variance_band are considered stable. A run that disagrees by more than variance_band triggers a one-shot re-run; if the re-run agrees with the second run, the first is discarded as noise. This is not statistically rigorous but is sufficient for the “is this a real regression or LLM jitter” question Phase 3 needs to answer.

---

5. Per-item sections

5.1 §3.3.1 Per-touchpoint eval automation

Goal: Every existing eval suite runs automatically on a defined trigger, without an engineer having to remember to run it.

Current state: All five eval scripts are run-by-hand only.

Proposed design:

• The pre-merge ci.yml workflow runs bun run test (which includes the Vitest eval wrappers gated by EVAL_*=1 env vars — Phase 3 sets these for the cached/no-LLM-cost eval paths only).
• The nightly eval-nightly.yml workflow runs the full bun run eval:all plus the explicit entity classification --live run.
• A new helper script scripts/eval-runner.ts wraps the individual eval scripts with shared scaffolding: timing, baseline-load, regression-check, output formatting, exit code based on threshold severity.
• The runner exits 0 on no-regression, exits 1 on severity: 'block' regression, and exits 0 with a warning footer on severity: 'warn' regression. Exit code drives CI gate behaviour.

Tasks:

1. Measure baseline CI runtime first (F10). Before committing the <2 min pre-merge budget, run bun install --frozen-lockfile && bun run test && bun lint && bun run knip on a fresh clone with no warm caches and record the wall-clock as metadata in the Phase 3.0 PR description. If the measurement exceeds 2 minutes, raise the §4.1 / §6.3 budgets to match (don’t pretend the budget is met).
2. Add .github/workflows/ci.yml with the lint + test + knip steps, including all hygiene keys from §4.3.1 (permissions:, concurrency:, timeout-minutes:, bun-version: 1.3.4).
3. Add .github/workflows/eval-nightly.yml with the cron schedule and the per-suite continue-on-error pattern from §6.6, plus the summary-poster job with if: always().
4. (Optional, recommended for AI-light tier) Add .github/workflows/ai-light.yml with paths: filter per §4.3.3.
5. Write scripts/eval-runner.ts (~150 lines, or up to ~250 lines accounting for the F8 infra-failure handling, F19 missing-metric detection, F9 atomic write helper, and the promote subcommand from §4.4).
6. Verify each existing eval script can be invoked from the runner without modification. Adjust scripts for clean exit codes if needed.
7. Document the runner usage in CLAUDE.md Commands table.

Acceptance:

• A new PR opened against main triggers ci.yml and reports pass/fail.
• The nightly cron runs eval-nightly.yml once; the result is visible in the GitHub Actions UI.
• A deliberate regression (manually edit a baseline file to make the current run fall below min) is correctly detected and reported.

Effort: 3-4h.

5.2 §3.3.2 Taxonomy change impact detection

Goal: A taxonomy change (new domain, new subtopic, renamed subtopic) automatically triggers the classification eval and reports any regression.

Current state: Taxonomy changes ship via the admin UI with no eval check. The bun run sync:taxonomy command regenerates the classification prompt but does not run any eval.

Proposed design:

A new GitHub Actions workflow .github/workflows/taxonomy-change- check.yml triggered by changes to lib/validation/schemas.ts (the taxonomy constants source) OR by manual gh workflow run invocation after a DB-side taxonomy edit.

The workflow:

1. Runs bun run sync:taxonomy to ensure the classification prompt reflects the current taxonomy.
2. Runs bun run eval:classification (cached mode, fast).
3. Reports the result via the same regression-check pipeline as §3.3.1.
4. If the eval was last run more than 24 hours ago, also enqueues a nightly --live re-run.

For DB-side taxonomy edits (admin UI), the spec proposes a small Supabase Edge Function or Vercel API route hook that POSTs to a GitHub Actions workflow_dispatch endpoint after the edit. This is optional polish; the manual gh workflow run path is sufficient for v1.

Tasks:

1. Add the workflow file.
2. Add a script scripts/check-taxonomy-eval-impact.ts that wraps the sync-then-eval flow.
3. Document the manual invocation path in docs/operations/taxonomy-change-runbook.md (existing doc).
4. (Optional) Add the post-edit hook from the admin UI.

Acceptance:

• Changing a taxonomy constant in lib/validation/schemas.ts and pushing to a PR triggers the workflow.
• A taxonomy change that drops classification eval below threshold is reported as a regression.
• The runbook documents the manual invocation and the expected output.

Effort: 2-3h.

5.3 §3.3.3 Model change protocol

Goal: Swapping the default LLM model is a documented, tooled procedure that runs the relevant eval suites before and after.

Current state: Model defaults are set via getModelForTier() in lib/anthropic.ts. Changing the default is a one-line edit with no mandatory pre/post measurement.

Proposed design:

Documented procedure at docs/operations/model-change-protocol.md:

1. Capture baseline: before any change, run bun run scripts/eval- runner.ts --suite all --tag pre-model-change-{date}. Saves a tagged baseline copy to a new file in __tests__/fixtures/eval-baselines/history/.
2. Apply change: edit lib/anthropic.ts to swap the model.
3. Re-measure: run bun run scripts/eval-runner.ts --suite all --compare-tag pre-model-change-{date}. Outputs a diff against the captured baseline.
4. Decide: accept (commit the change + the new baseline) or reject (revert the change).
5. Document: add an entry to docs/operations/model-change-log.md recording the date, model versions, and per-suite delta.

The eval-runner.ts from §3.3.1 grows two new flags: --tag <name> to capture a tagged baseline copy, and --compare-tag <name> to diff against a previously-tagged copy.

Tasks:

1. Write the protocol document.
2. Extend eval-runner.ts with --tag and --compare-tag flags.
3. Create the empty model-change-log.md and history/ directory.
4. Add a CLAUDE.md note pointing at the protocol.

Acceptance:

• The protocol document is committed and discoverable from CLAUDE.md.
• The runner supports tag/compare-tag.
• A dry-run of the full protocol completes successfully.

Effort: 2-3h.

5.4 §3.3.4 CI gates for eval suites

Goal: PRs that regress an eval suite beyond the threshold cannot merge to main without explicit override.

Current state: No CI gates exist (no CI). Phase 3.0 establishes CI; this item adds the merge gating.

Proposed design:

GitHub branch protection rule on main:

• Require status checks to pass: ci.yml
• Require status checks to pass: eval-nightly.yml (last successful run within the past 36 hours, NOT a per-PR check — the nightly result acts as a “the latest baseline is healthy” gate)

The pre-merge ci.yml only runs cheap evals. The nightly eval- nightly.yml runs everything. A PR that breaks the cheap evals is caught immediately; a PR that breaks the expensive evals is caught the next morning. This is a tradeoff: faster merges, slightly delayed catch on the expensive cases.

The threshold model from §4.2 controls per-metric severity:

• severity: 'block' metrics fail the PR check.
• severity: 'warn' metrics emit a warning comment but pass the PR check.
• severity: 'info' metrics are reported in the summary but ignored by the gate.

Default: classification subtopic accuracy and search MRR are severity: 'block'. Everything else starts at severity: 'warn' and graduates to 'block' after a sustained period of stability.

Tasks:

1. Configure branch protection on main (requires Liam to do this in the GitHub UI; the spec documents the desired settings rather than automating them).
2. Set per-metric severity in each baseline file’s threshold map.
3. Wire the runner’s exit code to the severity tier (already designed in §3.3.1).

Acceptance:

• Branch protection blocks a PR that fails ci.yml.
• A PR with a deliberate severity: 'block' regression cannot merge.
• A PR with a severity: 'warn' regression merges but emits a warning comment.

Effort: 1-2h (mostly documentation; the gating itself is configuration).

5.5 §3.3.5 Data-model + workspace-creation change impact detection

Goal: Schema migrations and workspace provisioning flows trigger relevant eval re-runs to catch silent regressions.

Current state: Migrations land via supabase db push with no eval check. New workspaces are created via the admin UI with no post-creation classification check.

Proposed design:

Two distinct flows.

Flow A — schema migration impact:

Add a step to the existing pipeline-parity guard (__tests__/validation/pipeline-parity.test.ts) or a new __tests__/validation/migration-eval-impact.test.ts that:

1. Detects a new migration file in supabase/migrations/.
2. Reads the migration’s diff (touched tables + columns).
3. If the diff touches content_items, entity_mentions, feed_articles, or any other classification-input table, surfaces a warning to run the relevant eval suites manually before merge.

This is a soft check — it doesn’t run the eval automatically (too slow), it just makes the requirement visible. The manual run is via the §3.3.1 runner.

Flow B — workspace creation impact: REMOVED FROM PHASE 3 (F5).

The original Flow B proposal (a post-creation hook in POST /api/intelligence/workspaces that runs eval against a sample of newly-classified content, plus an admin notification UI element) is product code, not CI infrastructure. It does not belong in this spec. See §9.1.1 for the rationale and onward placement.

Tasks:

1. Write the migration-impact guard test.
2. Document Flow A in docs/operations/migration-impact-runbook.md (or extend an existing migration runbook).

Acceptance:

• A new migration touching content_items triggers the warning in test output.
• The runbook documents the manual eval-rerun procedure and the expected output.

Effort: 1-2h (down from 3-4h after F5 removed Flow B).

---

6. Cross-cutting concerns

6.1 Observability

• Each eval run logs its trigger (PR, cron, manual), timestamp, and per-metric result via plain console. Phase 3 v1 does not depend on §9.7 structured logging.
• The nightly workflow posts a summary comment to a tracking issue rather than emailing or paging — keeps the noise level low.
• Failed runs link to the GitHub Actions log URL for debugging.

6.2 Cost

Cost figures verified against scripts/eval-entity-classification.ts:227-248 cost estimator + lib/ai/pricing.ts:32-34 per S155 WP1.2 audit F1.

Per-run LLM cost (Sonnet 4.6 rates: input $3/M, output $15/M):

Suite	Items	Per-item tokens	Per-item $	Per-run $ (single pass)	Per-run $ (two-pass)
classification (live)	91	~3000 in / ~600 out	~$0.018	~$1.64	~$3.28
entity (live, two-pass)	95	~3000 in / ~600 out	~$0.018	~$1.71	~$3.42
summarisation (live)	8–15	varies	varies	~$0.50	n/a
search	0 LLM calls	n/a	$0	$0	n/a
bid-drafting	0 (no real baseline; deferred until §1.4)	n/a	$0	$0	n/a
Nightly total					~$5.50

• Pre-merge ci.yml runs cost $0 — no LLM calls in the test suite.
• Nightly eval-nightly.yml costs ~$5.50 per run with current per-suite item counts (classification 91 + entity 95 two-pass + summarisation small set + search 0 + bid drafting deferred). With a buffer for prompt growth and retry cost, budget ~$7–8 per night, or $210–240/month at daily cadence. This is meaningful but acceptable for the visibility Phase 3 provides. If summarisation gold-standard count grows (S145 baseline only has a small set), the per-night figure scales linearly with item count.
• §3.3.2 taxonomy-change checks run only on relevant PRs; cost is negligible (~$1.64 per affected PR — single classification pass).
• §3.3.3 model-change protocol is rare (a few times per year); cost per invocation is one full eval pass (~$5.50–$8 once buffered).
• §3.3.5 workspace-creation hook is moved out of Phase 3 (see §9.1.1 Out-of-scope, F5). If a future spec adds it back, cost would be small-sample classification (~$0.50).
• Cost tripwire (Q-MISS-1): if multiple eval workflows can fire on the same day (nightly + manual + ad-hoc model-change), the daily cost can spike. Phase 3 does not implement an automated tripwire in v1; the implementer should set an Anthropic workspace-level spend alert at $20/day as a manual check. Track adding an automated budget guard as a Phase 3.x follow-up.

6.3 Latency

• Pre-merge: <2 min target (unverified — F10). Must be measured on a fresh checkout in Phase 3.0 before being committed as a hard budget. See §4.1 note.
• Nightly: <60 min. Sized for the upper-bound entity case (~25 min empirical) plus the four other suites and headroom for retries. If D-Q1 of the entity tightening spec confirms the rate-limit-math estimate (~3 min), the budget could be tightened to ~30 min — but Phase 3 ships with the 60 min figure to be safe under either resolution. See §4.1 latency-disputed note (F2 + F11).
• Per-PR taxonomy check: <5 min.
• Workspace-creation hook: removed from Phase 3 scope (F5 — moved to §9.1.1 Out-of-scope).

6.4 Sandbox / environment assumptions

GitHub Actions runners do not have a Claude Code sandbox, so the Bun-fetch-204 gotcha does not apply there. Anthropic API key + Supabase credentials must be set as GitHub Actions secrets.

6.5 Flakiness handling

• The variance_band field on the threshold model lets metrics tolerate normal LLM variance.
• A failing nightly run that disagrees with the most recent passing run by less than variance_band triggers a one-shot re-run before alerting.
• A persistent failure across two consecutive nightly runs is a real regression and alerts.

Pre-merge tier determinism (clarification — addresses §5.4 of the S155 audit). The pre-merge AI-light tier (classification eval --cached, search eval) has no LLM jitter risk because it reads existing classifications from the DB rather than making fresh calls. The runner’s --cached mode is therefore deterministic modulo DB state, and pre-merge runs are not subject to the variance_band tolerance — a pre-merge regression is a regression. Only the nightly tier needs flakiness handling.

Test-suite green prerequisite (F21). Phase 3 ships only after bun run test is reliably green on a fresh clone — i.e. the test suite itself must not be flaky. Otherwise the gating mechanism gets discredited within the first week. CLAUDE.md already documents two known fragilities (plugin-taxonomy-consistency.test.ts failing on worktree branches, mcp-fixture-sync.test.ts breaking on structural changes); both are accounted for and are not blockers — but no new flaky tests may land between Phase 3 spec ratification and Phase 3.0 deployment. This is a precondition, not a deliverable, but is listed as AC #11 in §10 below.

6.6 Partial nightly failure handling (F3)

The nightly workflow MUST tolerate per-suite failures without short-circuiting. Concretely:

• Each per-suite invocation in eval-nightly.yml runs with continue-on-error: true (or wraps the script in a set +e; ...; rc=$?; set -e block that captures the exit code into a per-suite variable).
• A final “summary poster” job runs after all per-suite jobs with if: always(). It collects each suite’s status (success, failure, or skipped) and the cost-burned total (read from per-suite stdout, which the runner emits).
• The summary comment on the tracking issue contains:
  • Run ID + timestamp.
  • Per-suite status table (suite name, exit code, cost burned, brief error if failed, link to the suite’s own log section).
  • Total cost burned across all suites that ran (including failed suites that completed an Anthropic call before failing).
  • “Real regressions detected” section (only entries where a metric breach was determined; infra failures and shape changes are listed separately).
  • “Infra failures” section (see §6.6 below).
  • “Shape changes” section (renamed/missing metrics, see §4.2).
• --save-baseline is attempted ONLY for suites that exited cleanly with no severity: 'block' regression. A partially failed nightly run does NOT promote any baselines.
• The workflow itself reports failure to the GitHub Actions UI iff any suite either (a) hit a severity: 'block' regression OR (b) failed for a non-infra reason. Pure infra failures (Anthropic 529 / OpenAI 503 / network) report success so the daily-run-within-36h branch-protection gate (§5.4) is not tripped by upstream provider outages.

6.7 Upstream LLM provider outage handling (F8)

The S154 sub-agent failure that triggered this spec’s drafting was itself an Anthropic HTTP 529 (“Overloaded”). Phase 3’s nightly runs will hit the same failure mode periodically. The spec must address it explicitly:

• Retry with exponential backoff at the SDK layer. The Anthropic TypeScript SDK already retries 5xx + 529 with backoff by default. Verify the eval scripts don’t disable this. Default retry count is 2 — Phase 3 raises this to 4 for the eval scripts via SDK options.
• After exhausting retries, the suite exits with a labelled error ({ ok: false, reason: 'upstream_unavailable', provider: 'anthropic', http_status: 529 }) rather than treating the failure as a regression.
• severity: 'infra' tier is added to the existing 'block' | 'warn' | 'info' triplet. Infra failures appear in a dedicated “Infra failures” section of the nightly summary, do NOT trigger a block-severity alert, and do NOT cause the workflow to report failure to the GH Actions UI (so the 36h freshness gate from §5.4 is not tripped). The runner emits an exit code of 2 (distinct from 0 no-regression and 1 real regression) for the infra-failure case.
• Same handling for OpenAI 503 (used by search eval embeddings).
• Same handling for transient Supabase failures (e.g. --cached mode being unable to read classifications from the DB) — labelled provider: 'supabase'.

The threshold model’s severity enum is therefore extended:

severity?: 'block' | 'warn' | 'info' | 'infra';

'infra' is set programmatically by the runner when it traps an upstream-unavailable error; it cannot be configured per-metric in a baseline file (it’s a runtime category, not a per-metric policy).

6.8 Baseline file corruption protection (F9)

saveBaseline and loadBaseline in lib/eval/baseline.ts are currently exposed to mid-write corruption. If a --save-baseline write is interrupted (CI runner OOM, Vercel SIGTERM, git race), the JSON file can end up partially written and loadBaseline:22-29’s JSON.parse(raw) throws on the next CI run.

Phase 3 hardens both functions:

• Atomic write in saveBaseline: write the new JSON to a sibling *.tmp file in the same directory, then fs.renameSync to the canonical path. POSIX rename is atomic on the same filesystem, so a crash mid-write either leaves the old file in place or replaces it cleanly with the new file — never a half-written file.
• try/catch + shape validation in loadBaseline: parse errors return null and emit a loud warning (via logBestEffortWarn or console.error), rather than throwing. The runner treats a null baseline as “no baseline yet” — same as a fresh suite that has never been run.
• Shape validation runs against a Zod schema that mirrors the TypeScript Baseline interface, including the new severity and variance_band extensions. A baseline file whose shape doesn’t match the schema is treated as corrupted (warn + null).

These changes are backward compatible — every existing baseline file will continue to validate.

---

7. Implementation phasing within Phase 3

To de-risk the rollout, Phase 3 ships in five sub-phases:

Sub-phase	Scope	Effort
3.0 CI foundation	The two workflow files (§4.3) + branch protection config	1-2h
3.1 Per-touchpoint automation	§3.3.1 runner, eval-runner.ts, runner integration into the workflows	3-4h
3.2 Threshold model + warn-only gating	§4.2 schema extension, default thresholds set to warn, no merge blocking yet	2h
3.3 Model + taxonomy + data-model protocols	§3.3.3, §3.3.2, §3.3.5 documentation + tooling	5-6h
3.4 Hard merge gating	Promote selected metrics to severity: 'block', enable branch protection	1-2h (mostly waiting for confidence that warns are stable)

Total Phase 3 effort: 12-16h across the sub-phases. Sub-phase 3.0 is the unblock for §3.6, silent-failure Phase 5, and §1.5’s iteration loop, so it should land first regardless of the sequencing of the other sub-phases.

---

8. Test strategy

How is Phase 3 itself tested?

• Unit tests: __tests__/lib/eval/baseline.test.ts (existing) + new tests for the threshold model extension (severity, variance_band).
• Integration test: __tests__/integration/eval-runner.test.ts — invokes the runner against a fixture baseline and asserts the exit code matches the expected severity tier.
• Meta-test: __tests__/integration/regression-detection.test.ts — deliberately corrupts a baseline file to simulate a regression, runs the runner, asserts the corruption is detected.
• CI smoke test: the first ci.yml run on a real PR is the acceptance test for §3.0 — there’s no way to test “does GitHub Actions run our workflow correctly” except by running it.

---

9. Out of scope

• Phase 4 HITL refinement (separate spec, already on main).
• Phase 5 full coverage (separate spec, already on main).
• §3.1 two-pass cost vs quality measurement — tracked separately in roadmap §3.1.
• §3.6 MCP Eval Layer 5 CI — Phase 3.0 lays the CI foundation that §3.6 will extend, but the MCP eval integration itself is §3.6’s job.
• Per-touchpoint cost tracking infrastructure — that’s §3.5.5 in Phase 5.
• Database migrations — Phase 3 adds no new DB tables. Baselines remain JSON files in __tests__/fixtures/eval-baselines/.
• The entity classification tightening itself — that’s §1.5. Phase 3 provides the harness; §1.5 uses it.
• Email / Slack alerts — Phase 3 v1 uses GitHub Actions tracking issues for alerting. Push channels are a Phase 5 cross-cutting concern.
• Workspace-creation eval hook (formerly §3.3.5 Flow B) — moved out per S155 WP1.2 F5. See §9.1.1 below for the rationale and where the work lives now.
• Automated cost tripwire — the Anthropic workspace spend > $X per day alert is a manual check in Phase 3 v1 (see §6.2). Adding an automated tripwire is a Phase 3.x follow-up.
• Branch-protection automation — Phase 3 documents the desired branch-protection settings in docs/operations/branch-protection.md but does not automate them via the GitHub API. Liam configures the rules manually via the GitHub UI per §5.4 + AC #8.

---

9.1 Items moved out by S155 WP1.2 review (flagged for future work)

This sub-section captures items that the spec originally proposed but that were moved out during S155 WP1.2 review. They are NOT part of Phase 3 and MUST NOT be implemented as part of Phase 3 work.

9.1.1 Workspace-creation eval hook (originally §3.3.5 Flow B) — F5

Original proposal: Add a post-creation hook in POST /api/intelligence/workspaces that, once a new workspace has its first batch of classified content, runs the classification eval against a small sample of that content, compares the per-domain accuracy against the global baseline, and surfaces a warning to admins if the new workspace’s accuracy is materially lower. Plus an admin notification UI element.

Why moved: Workspace-creation hook is product code (an API route addition + UI element), not CI infrastructure. It does not belong in a CI/regression-infrastructure spec. Including it would conflate two separate workstreams and inflate Phase 3’s effort budget without meaningful integration with the rest of Phase 3.

Where it lives now: The Sector Intelligence hardening workstream is the natural home — workspaces are an SI concept, and the SI hardening plan already covers post-provisioning quality checks. Track adding a “workspace-creation classification quality check” spec under SI hardening, NOT under Phase 3. Cross-reference from the SI hardening parent spec when it’s drafted.

What Phase 3 still owns from §3.3.5: Flow A (schema migration impact detection via the pipeline-parity guard test) remains in scope. Flow B is the only piece that moved out.

9.1.2 Branch-protection bypass policy (F20)

Decision: GitHub branch protection allows admin override for emergency hotfixes. The Phase 3 spec does NOT prescribe a specific bypass policy; the operational rule is:

• Admin override is permitted for hotfixes that cannot wait for the next eval cycle.
• The PR description MUST record the reason for the bypass and link to the failing eval (or note “infra failure” if applicable).
• Bypassed PRs are reviewed weekly to confirm the bypass was legitimate.

This policy is documented in docs/operations/branch-protection.md alongside the protection rules themselves; it is NOT an implementation task for Phase 3 beyond the documentation note.

9.1.3 In-flight PR vs nightly baseline update (Q-MISS-2)

Question: What happens to an in-flight PR when nightly saves a new (stricter) baseline at 02:00 UTC? The PR opened at 09:00 now has to clear a higher bar.

Decision: This is acceptable behaviour. A stricter baseline is the system’s most accurate current view of “no regression”; an in-flight PR that suddenly fails is signalling that it would have regressed against the new reality. The PR author should rebase and re-evaluate. No special handling is required in Phase 3.

---

10. Acceptance criteria

The Phase 3 spec is implemented when:

1. .github/workflows/ci.yml and eval-nightly.yml exist and run on the appropriate triggers without errors. The first PR opened after Phase 3.0 ships triggers ci.yml and the run appears in the GitHub Actions UI with a green status. The first scheduled cron tick of eval-nightly.yml produces a tracking-issue comment with the per-suite status table from §6.6.
2. scripts/eval-runner.ts wraps all five eval scripts with shared scaffolding and exit-code semantics:
   • 0 = no regression
   • 1 = real regression at severity: 'block'
   • 2 = infra failure (per §6.7)
   • Warn-severity regressions exit 0 with a warning footer.
3. The threshold model extension is in lib/eval/baseline.ts, with severity and variance_band fields, the 'infra' severity tier extension from §6.7, and defaults set per the spec (severity: 'warn', variance_band: 0.02).
4. All five existing eval suites have updated threshold maps with appropriate severity settings (block / warn / info per metric). Verified by greppable severity: field in each __tests__/fixtures/eval-baselines/*.baseline.json.
5. Model change protocol dry-run produces a per-suite delta table with no breaches. Concretely: bun run scripts/eval-runner.ts --suite all --tag pre-test --compare-tag pre-test exits 0, prints a table listing each suite with delta: 0 for every metric, and creates a __tests__/fixtures/eval-baselines/history/ entry. Failure = runner exits non-zero, table missing, or any metric reports a delta exceeding variance_band.
6. Taxonomy change workflow dry-run produces the expected output. Concretely: triggering taxonomy-change-check.yml via gh workflow run with no actual taxonomy edit produces a workflow run that (a) completes inside the §6.3 latency budget, (b) emits a “no regression” verdict via the runner, and (c) posts a comment on the tracking issue summarising the result. Failure = workflow errors, summary missing, or runner reports a regression on a no-op change.
7. A deliberate regression (intentionally corrupted baseline) is correctly detected by the runner and surfaces in CI as a non-zero exit code and a labelled entry in the nightly summary.
8. Branch protection on main is configured per §5.4 (requires Liam to do this manually in the GitHub UI; documented in docs/operations/branch-protection.md). Verified by opening a deliberately-failing PR (e.g. baseline edit that introduces a severity: 'block' regression) and confirming GitHub blocks the merge button (F16). CI cannot self-test branch-protection gating; Liam must perform this verification step manually.
9. Phase 3 exposes the four contracts named in Phase 4 §3.1 (F12 — testable at Phase 3 ship time; previously phrased as “Phase 4’s contracts are satisfied” which couldn’t be tested before Phase 4 exists). Concretely:
   • lib/eval/baseline.ts exports a RegressionThreshold type including the { min, max_drop, severity, variance_band } schema extension.
   • scripts/eval-runner.ts exports / exposes a callable that, given a touchpoint identifier, returns the most recent regression result (or null) — the EvalRegressionScanner contract from Phase 4 §3.1.
   • The runner supports --suite <name> and --compare-tag <tag> for the A/B testing path Phase 4 §3.1 names.
   • promoteBaseline is callable from the runner CLI (promote subcommand) per §4.4 — the baseline-lifecycle hook Phase 4 §3.1 names. This AC tests Phase 3’s API surface, not Phase 4’s behaviour.
10. The runner is documented in CLAUDE.md Commands table.
11. Test-suite green prerequisite (F21). bun run test runs cleanly on a fresh clone of main (modulo the two known CLAUDE.md fragilities), with no flaky tests added between Phase 3 spec ratification and Phase 3.0 deployment. Verified by three consecutive green runs on a fresh clone.
12. Cost cap not exceeded — the first 7 nightly runs report a total Anthropic spend within ±20% of the §6.2 estimate ($210–240 × 7/30 ≈ $49–56 over a week). Spend outside this range triggers a re-estimation pass before Phase 3.4 (hard merge gating) ships.

---

11. References

• docs/audits/ai-eval/ai-eval-gap-analysis-s149.md — Phase 3 scoping context (note: many S149 gaps in the doc are now closed — C1, H2, H3, H4 partial — the spec only addresses forward-looking Phase 3 items).
• docs/audits/ai-eval/ai-touchpoint-map-and-improvement-strategy.md §8 Phase 3 — strategic framing.
• docs/specs/ai-eval-phase2-classification-skill-spec.md — structure / tone reference.
• docs/specs/ai-eval-phase4-hitl-refinement-spec.md §3 Dependencies — the contracts Phase 3 must expose.
• docs/specs/ai-eval-phase5-full-coverage-spec.md — the downstream consumer of Phase 3’s CI foundation.
• docs/specs/mcp-evaluation-spec.md — the MCP eval spec referenced by §3.6 (Layer 5 CI integration).
• docs/specs/silent-failure-prevention-spec.md §5.7 — Phase 5 deferred until CI exists; same dependency.
• docs/specs/entity-classification-prompt-tightening-spec.md — the §1.5 workstream that consumes Phase 3’s regression detection.
• lib/eval/baseline.ts, lib/eval/metrics.ts, lib/eval/types.ts — the eval infrastructure Phase 3 extends.
• scripts/eval-{classification,entity-classification,summarisation, search,bid-drafting}.ts — the five existing eval scripts.
• lib/anthropic.ts getModelForTier() — the model dispatch Phase 3.3.3 protects against silent upgrades.
• lib/validation/schemas.ts — the taxonomy constants Phase 3.3.2 watches.
• docs/operations/taxonomy-change-runbook.md — the operational doc Phase 3.3.2 extends.
• package.json eval:* scripts — the existing manual eval entry points the runner wraps.

---

12. Open questions for adversarial review

Status update (S155 WP1.2): Q1, Q2, Q3, Q5, Q6, Q7 resolved per the audit recommendations. Q4 carried into Phase 3.0 with caveat (pre-merge budget already unverified per F10; Q4 adds ~3 min so the budget needs re-measurement before Q4 is enabled).

Q1. RESOLVED. Should the nightly eval workflow run on weekends? Decision: yes. Regressions don’t take the weekend off, and the cost is the same. Implemented in §4.3 cron schedule.

Q2. RESOLVED. Should the CI gate require the nightly run to be within 24 hours, or 36 hours? Decision: 36 hours, to give a buffer for the nightly run to complete and for an engineer to investigate before the “latest baseline is stale” gate triggers. Implemented in §5.4 branch protection.

Q3. RESOLVED. Should eval:bid-drafting run nightly even though it has no real baseline? Decision: no — skip until §1.4 (real bid data) lands. Adding bid drafting before that is burning cost for no signal. Implemented in §4.3 (bid-drafting line is commented out).

Q4. CARRIED — depends on F10 measurement. Should Phase 3.0 (CI foundation) also include a bun build step? Original recommendation: yes — catches Next.js build errors that the test suite misses. Adds ~3 minutes to the pre-merge run. Open dependency: the pre-merge budget itself is unverified (F10). Phase 3.0 must measure the test + lint + knip wall-clock first; if the measurement is already at or above 2 minutes, adding bun build is not viable on the pre-merge tier and bun build should run on a separate path-filtered job instead. Track as Phase 3.0 follow-up; do not commit either way until F10 measurement is in.

Q5. RESOLVED — moved out of scope per F5. The workspace-creation hook (originally §3.3.5 Flow B) is removed from Phase 3 entirely. The hard-block-vs-warning question is no longer Phase 3’s to answer. See §9.1.1 for placement under SI hardening.

Q6. RESOLVED. Should the model change protocol (§3.3.3) require manual approval after the diff, or auto-accept if the deltas are within tolerance? Decision: manual approval, always. Model swaps are infrequent and high-stakes. Implemented in §5.3 protocol step 4.

Q7. RESOLVED. Variance band default of 2pp — too lenient or too tight? Decision: start at 2pp default with per-metric override allowed (F22). Tighten after 4-6 weeks of nightly data shows the actual run-to-run variance is smaller. Per-metric override is explicitly part of the threshold model per §4.2.

12.1 New questions surfaced by S155 WP1.2

Q8. Should Phase 3 add an automated cost tripwire (hard stop if daily Anthropic spend exceeds $X)? Recommendation: not in v1; rely on the manual workspace-level alert per §6.2. Revisit if the actual spend over the first 30 days deviates materially from the §6.2 estimate.

Q9 (D-Q1 dependency). What is the actual wall-clock for entity --live --validate? Resolves the dispute between rate-limit math (~3 min) and empirical (~25 min). Both possibilities are budgeted in §4.1. The entity classification spec’s D-Q1 will produce the empirical answer; Phase 3 inherits that result automatically.

Q10. Should the nightly summary be posted to a tracking issue or to a dedicated eval-status channel/page? Recommendation: v1 uses GitHub Issues for the lowest-friction implementation; revisit if the noise level becomes a problem.

---

End of spec.