AI Evaluation Phase 5: Full Coverage Spec

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Review status: Updated 08/04/2026 per S155 WP1.4 adversarial review (docs/audits/s155-spec-review-ai-eval-phase5-full-coverage.md). Original verdict: FAIL (2 BLOCKERs, 5 HIGH, 10 MEDIUM, 9 LOW, 2 NIT). All 28 findings have been applied to this spec in the S155 edit pass (see §0 “Review findings applied” below). After the edits the spec is cleared for implementation, subject to the surviving dependency on Phase 4 for adapter tasks (F2, made explicit per-task) and subject to the W6 evaluator approach having been switched from a non-existent Python skill to a Claude-as-judge rubric (F1). Re-verification expected at first implementation touch.

Goal: Extend the skill/eval pattern established in Phases 1-4 to every remaining AI touchpoint in the platform: bid drafting, summarisation, digest generation, output mode standardisation, cost tracking, and MCP tool descriptions. Close the gap between “we measure classification quality” and “we measure every AI interaction that matters”.

Motivation: By the end of Phase 2, the platform has evaluation baselines for classification (domain/subtopic), entity extraction, summarisation (ROUGE, low quality), and search (MRR/P@K). By the end of Phase 3, those baselines regress-detect automatically. By the end of Phase 4, flag signals and A/B tests close the loop on touchpoints with gold standards. Phase 5 is what remains: touchpoints that either have no gold standard, have no quality rubric, have no cost visibility, or have no eval at all.

Specifically, at the start of Phase 5 the following touchpoints are still uncovered:

Touchpoint	What exists	What is missing
Bid drafting	Eval runner + 24 synthetic fixtures	Real bid data, quality rubric, baseline
Summarisation	Skill-free hardcoded prompt; eval runner with low ROUGE thresholds	Skill conversion, stronger rubric, BERTScore
Digest generation	Hardcoded programmatic prompt; NO eval	Gold standard, eval runner, skill conversion
Output mode usage	5 different strategies across 18 touchpoints	Standardisation policy + migration guide
Cost tracking	Ad-hoc `response.usage` logging; no aggregation	Shared wrapper, per-touchpoint persistence, trend dashboard
MCP tool descriptions	41 tool descriptions + L4 functional correctness metric	Claude-as-judge rubric audit + systematic improvements (per S155 WP1.4 F1 — original `tool-design` skill reference was stale)

Phase 5 is the “full coverage” phase because it ensures no AI touchpoint ships without baseline measurement, cost visibility, and quality rubric.

Parent documents:

docs/audits/ai-eval/ai-touchpoint-map-and-improvement-strategy.md §8 Phase 5
docs/audits/ai-eval/ai-eval-gap-analysis-s149.md §2 Phase 5
docs/reference/product-roadmap.md §3.5

0. Review findings applied (S155 WP1.4)

Source audit: docs/audits/s155-spec-review-ai-eval-phase5-full-coverage.md. Every finding from the audit’s §15 Findings table is addressed here. The original verdict was FAIL — two BLOCKERs and five HIGHs — but after applying the resolutions below, the spec is implementable subject to the per-task Phase 4 dependency markers (F2) and the W6 rewrite (F1).

BLOCKERs

F1 — W6 tool-design skill does not exist. Original spec (line 242) asserted .claude/skills/context-engineering-collection/skills/tool-design/ contained a ToolDescriptionEvaluator Python class with five scoring dimensions. Verified not to exist. Resolution (option B from audit §16): W6 is rewritten in §5.6 below to use a hand-rolled Claude-as-judge rubric against the five dimensions (clarity, parameter documentation, example quality, edge case coverage, error handling hints), scored 1-5 per dimension, aggregated into a per-tool score. No Python ToolDescriptionEvaluator dependency. The scoring happens via a new scripts/eval-mcp-tool-descriptions.ts that reads the 41 tool definitions from docs/generated/mcp-inventory.md and calls Claude via getModelForTier('analysis'). Output format: JSON with per-tool scores and per-dimension feedback. Baseline stored at __tests__/fixtures/eval-baselines/mcp-tool-descriptions.baseline.json.

F2 — Phase 4 adapter dependency contradicts independence claim. Original spec line 77-78 said “Phase 5 does NOT need to wait for Phase 4” but tasks 5.1.8, 5.2.9, 5.3.7, 5.6.8 all create lib/refinement/adapters/*.ts — a directory Phase 4 creates (Phase 4 spec line 376). Resolution: line 77-78 corrected to “Phase 5 eval/skill tasks (5.1.1-5.1.7, 5.2.1-5.2.8, 5.3.1-5.3.6, 5.6.1-5.6.7) do NOT wait for Phase 4. Phase 5 adapter tasks (5.1.8, 5.2.9, 5.3.7, 5.6.8) DO depend on Phase 4 Tasks 4.1 and 4.2 being merged.” Each adapter task in §5 has been marked with an explicit **Depends on:** Phase 4 Tasks 4.1 + 4.2 line. See §3.6 for the full adapter-dependency table.

HIGH findings

F3 — BERTScore already implemented. Task 5.2.5 (“Implement lib/eval/bertscore.ts, 2h”) duplicates scripts/compute-bertscore.py which is already wired into scripts/eval-summarisation.ts via the --bertscore flag (lines 23, 309-352). Resolution: Task 5.2.5 struck. Replaced with “verify the existing scripts/compute-bertscore.py integration meets W2 needs; if needed, extract a TypeScript wrapper lib/eval/bertscore.ts that shells out to the Python script. Effort: 30 min verification + optional 1h TS wrapper if Tier-1 BERTScore is needed by non-eval code.”

F4 — Digest schema already extracted. Task 5.3.1 (“Extract digest schema to lib/validation/digest-schema.ts, 1h”) is redundant — the schema lives at lib/validation/ai-schemas.ts:51-62 as DigestResponseSchema, imported by lib/ai/digest.ts:10. Resolution: Task 5.3.1 rewritten as “verify DigestResponseSchema in lib/validation/ai-schemas.ts:51-62 is the canonical evaluable contract, and document it as such in the eval runner header. Effort: 15 min.”

F5 — claude-sonnet-4-6 pricing already added. Spec line 660 asserted the model was “Missing” from lib/ai/pricing.ts (S149 L3). Verified already present at lib/ai/pricing.ts:32-37. Resolution: §5.5 “Missing” claim struck. Task 5.5.1 rewritten as “add getPricing(model) helper with fallback for unrecognised models (return zero + log warning via logBestEffortWarn). Add unit test for the fallback case. Effort: 30 min.”

F6 — Vague acceptance criteria. §10 used “raised thresholds”, “no regression, ideally an improvement”, “every AI call site wrapped” without numeric targets or enumerations. Resolution: §10 rewritten with concrete numeric targets and Appendix B enumeration as the source of truth for “every AI call site”. Specific rewrites:

W2 “raised thresholds” → “ROUGE-L thresholds: executive ≥0.30, detailed ≥0.25 (documented against the current summarisation.baseline.json values 0.361/0.254 — F26 note: current baseline essentially equals the proposed floor, so this is a confirmation of current state not an improvement target; W2 may need re-scoping to 0.40/0.35 if headroom is genuinely wanted)”.
W3 “baseline saved” → “all 5 metrics (theme coherence, coverage, structural compliance, narrative quality, faithfulness) within their first-run bands; LLM-as-judge calibration correlation ≥0.6 with human reference scores on a 10-item stratified sample”.
W5 “every AI call site wrapped” → “Every call site enumerated in Appendix B is wrapped. Enumeration is updated in this commit to add missing call sites from lib/intelligence/article-summariser.ts, lib/intelligence/guide-generator.ts, lib/intelligence/feed-poller.ts (F17). Enforcement: a unit test at __tests__/lib/ai/cost-tracking-coverage.test.ts asserts every grep match for anthropic.messages.create in lib/ai/**, lib/intelligence/**, app/api/** has a matching recordAiCall call site within 10 lines.”
W6 “no regression, ideally an improvement” → “Baseline score ≥ first-run baseline (no regression). If the baseline improves during the audit, record it; improvement is not a ship gate.”

F7 — Effort estimates underestimate by ~2x. §7 said “20-26 hours”; per-task sum was ~53h. Resolution: §7 updated to “53h total across 8-12 sessions (revised upward per F7 from original 20-26h / 5-6 sessions claim)”. Per- workstream: W1 9h post-§1.4, W2 11.5h (minus 2h from F3 BERTScore strike = 9.5h), W3 10h (minus 0.75h from F4 digest schema strike = 9.25h), W4 5h, W5 10.25h, W6 7.5h (plus ~2h for Claude-as-judge rewrite per F1 = 9.5h). Revised total: ~52h.

MEDIUM findings

F8 — W4 output mode policy contradicts parent audit. Parent audit (docs/audits/ai-eval/ai-touchpoint-map-and-improvement-strategy.md §2.5 lines 215-218) recommends Structured Outputs; Phase 5 W4 §3.3 proposed tool_use (forced) without acknowledgement. Resolution: §3.3 rewritten to adopt Structured Outputs as default for new touchpoints, with tool_use (forced) as a documented fallback for models/tasks where Structured Outputs is not yet supported. The parent audit’s recommendation is cited explicitly.

F9 — W1 acceptance criterion premature. Acceptance bullet 1 required “runs end-to-end against real bid data” but tasks 5.1.3-5.1.4 (Zod schema + evaluator) are doable without §1.4. Resolution: W1 split into W1-pre (rubric schema + evaluator + dry-run fixture template + synthetic-fixture unit tests) and W1-post (real-data run + calibration + first baseline, gated on §1.4). Only W1-pre ships in this Phase 5 scope; W1-post is tracked as a §1.4 follow-up.

F10 — W5 Bun fetch 204 sandbox gotcha not mentioned. recordAiCall() writes to ai_call_costs on every invocation. Local dev/test loops will hang without sandbox bypass. Resolution: §6 and W5 §5.5 both updated with a CLAUDE.md gotcha reference: “All scripts invoking recordAiCall() from a sandboxed Bun environment must run with dangerouslyDisableSandbox: true. Production (Vercel) is unaffected.”

F11 — W5 sb() / tryQuery() choice unspecified. recordAiCall() writes to Supabase but didn’t say which helper. Resolution: W5 §5.5 specifies tryQuery() from @/lib/supabase/safe so a write failure returns a Result rather than throwing (cost tracking is best-effort; failure must not break the caller’s AI invocation). On Result = Err, call logBestEffortWarn ('ai_cost_tracking.record_ai_call.failed', msg, { err }).

F12 — W5 missing correlation_id column. ai_call_costs schema didn’t include correlation_id (nullable) which §9.7 structured logging will add to activity_history. Resolution: W5 schema updated to include correlation_id TEXT NULL so a future join to request-level timelines is trivial. Cheap to add now; backfill pain avoided.

F13 — W5 /admin/ai-costs missing project conventions. Task 5.5.6 didn’t mention TanStack Query, Warm Meridian tokens, or getAuthorisedClient() discriminated-union check. Resolution: W5 task 5.5.6 updated to require: (a) TanStack Query with keys in lib/query/query-keys.ts and fetchers in lib/query/fetchers.ts (no SWR or raw fetch); (b) Warm Meridian semantic tokens per docs/design/warm-meridian-implementation-spec.md (no raw Tailwind colours); (c) const auth = await getAuthorisedClient(['admin']) with if (!auth.success) return authFailureResponse(auth) pattern per CLAUDE.md.

F14 — W3 NLI/coverage underestimated. Tasks 5.3.3 (3h) and 5.3.4 (2h) underestimated sentence-level NLI between input items and narrative_summary. Resolution: Both task estimates bumped to 5h and 3h respectively (total W3 adds 3h, reflected in F7 re-cost).

F15 — Phase 3 tier placement unspecified. Spec didn’t say which Phase 3 tier each new eval runs in. Resolution: Added per-workstream tier assignments in §3.1:

W1 bid drafting: manual on-demand (Opus is too expensive for scheduled)
W2 summarisation: scheduled nightly (Haiku/Sonnet, already in nightly)
W3 digest: scheduled nightly (Haiku)
W4 output mode: pre-merge AI-light (static check, no AI call)
W5 cost tracking: pre-merge (unit test only, no AI call)
W6 MCP tool descriptions: manual on-demand (60-tool × 5-dimension Sonnet audit is ~$3 per run)

F16 — Severity / variance_band defaults unspecified. Phase 3 §4.2 extended baseline schema with severity and variance_band; Phase 5 must specify per-metric defaults. Resolution: Added a per-workstream severity table in §4.2 shared interfaces with default severity: 'warn' for all new metrics (human-interpretation required before blocking) and variance_band: ±5% for LLM-judge metrics, ±2% for deterministic metrics (BERTScore, ROUGE).

F17 — Cost tracking coverage matrix incomplete. Appendix B omitted call sites in lib/intelligence/article-summariser.ts, guide-generator.ts, feed-poller.ts. Resolution: Appendix B updated to include all three. The enforcement test from F6 guarantees completeness going forward.

LOW findings (documented resolutions)

F18-F20 — Stale line counts (bid-writing.md, uk-procurement.md, eval-summarisation.ts). Resolution: All three updated inline in §5.1 and §5.2 text to current values: 617, 335, 721 respectively.

F21 — “18 touchpoints” count error. Parent audit has 17 (8+2+3+1+1+2). Resolution: Corrected to 17 in §3.3 and Appendix B.

F22 — W6 plugin audit bonus scope. Phase 5.6e was “if time allows”. Resolution: Promoted to its own Phase 5.6f task with explicit acceptance, or demoted to a follow-up item tracked under §11 References. Chose the latter for scope discipline; added to the “Open follow-ups” list in §11.

F23 — Digest gold-standard size 10 unjustified. Resolution: Added one-sentence note in §5.3 explaining 10 is a pragmatic lower bound (5 weekly digests × 2 time windows) stratified by item count (5 per bucket) and that statistical power is explicitly deferred until the calibration step in W3 task 5.3.5.

F24 — No 529 retry/backoff strategy. Resolution: §6.2 references Phase 3’s retry/backoff (once Phase 3 lands) and notes that the nightly workflow must retry once with 30-second backoff before marking the eval as completed_with_errors.

F25 — No ai_call_costs retention policy. Resolution: Added §5.5 note: “ai_call_costs rows older than 365 days are archived to a nightly ai_call_costs_archive table (or simply deleted if the admin dashboard is the only consumer). Retention owner: the next admin-tooling housekeeping session; tracked in §11 References.”

F26 — W2 ROUGE proposed thresholds = current baseline. Current summarisation.baseline.json metrics are 0.361/0.254; proposed floor 0.30/0.25 is below current. Not an improvement. Resolution: W2 §5.2b revised — if W2 wants to raise the bar it should target 0.40/0.35 (+10pp above current), otherwise ship the skill conversion as a no-quality-change refactor and leave the thresholds at their current measured values.

NIT findings

F27 — No §Open questions section. Resolution: Added §12 Open questions at the end of the spec with the real ambiguities (F9-fall-through, W4 Structured Outputs scope, digest variance band defaults, W6 Claude-as-judge vs external evaluator, cost retention, dashboard auth scope).

F28 — W6 “1-5 scale” unverifiable. The Claude-as-judge rewrite (F1) resolves this — the scale is now explicitly defined in the spec rather than inherited from an external skill.

1. Overview

Phase 5 delivers six workstreams, one per roadmap item:

Workstream	Roadmap	Priority
W1 — Bid drafting rubric	§3.5.1	Could (blocked on §1.4)
W2 — Summarisation skill conversion	§3.5.2	Could
W3 — Digest evaluation	§3.5.3	Could
W4 — Output mode standardisation	§3.5.4	Could
W5 — Cost tracking across touchpoints	§3.5.5	Could
W6 — MCP tool description audit	§3.5.6	Could

Why “Could” priority? The roadmap classifies these as “Could” because the highest-stakes touchpoints (classification, entity extraction, SI relevance scoring) already have evals. Phase 5 is quality-of-life improvement: regressions to these touchpoints would be slower to detect but not catastrophic in the same way a classification regression would be.

Dependencies: Each workstream has its own prerequisite. See §3 and per-workstream sections.

Relationship to Phase 4: Phase 5 workstreams integrate with Phase 4’s refinement infrastructure. Specifically:

W2-W3 add new GoldStandardEvaluator instances to Phase 4’s adapter registry.
W4’s output mode standardisation feeds into Phase 4’s A/B testing (variant A and variant B must speak the same output mode).
W5 is a hard dependency for Phase 4’s observability — without per- touchpoint cost tracking, the refinement loop’s cost is invisible.
W6 is conceptually a Phase 4 application — auditing MCP tool descriptions is a specific case of “use an eval to drive prompt improvement”.

Phase 5 — Phase 4 dependency (corrected per S155 WP1.4 F2): Phase 5 eval and skill tasks (W1 Tasks 5.1.1-5.1.7, W2 Tasks 5.2.1-5.2.8, W3 Tasks 5.3.1-5.3.6, W4 whole workstream, W5 whole workstream, W6 Tasks 5.6.1-5.6.7) do NOT need to wait for Phase 4. They ship independently as long as their individual dependencies are met. Phase 5 adapter tasks (5.1.8, 5.2.9, 5.3.7, 5.6.8) DO depend on Phase 4 Tasks 4.1 + 4.2 landing first, because they create files under lib/refinement/adapters/ which Phase 4 is responsible for creating. Each adapter task below is marked with an explicit dependency line. See §3.6 for the adapter-dependency table.

2. Motivation

2.1 Concrete risks the phase addresses

R1: The highest-stakes touchpoint has zero measurement. Bid drafting is the platform’s revenue-critical AI interaction — customers pay for bid responses. The three-pass bid drafting pipeline (analysis, drafting, quality check) uses 800+ lines of skill guidance and the most expensive model (Opus). But the eval runner is stubbed (see docs/audits/ai-eval/ai-eval-gap-analysis-s149.md H5): the gold standard uses synthetic question IDs because no real bid workspace has been created. A model update or prompt change could silently degrade bid quality with no detection. The impact is direct revenue loss if a customer’s bid fails.

R2: Digest (Change Reports) is the most user-facing AI output and has no eval. Every digest is generated by lib/ai/digest.ts (615 lines) with two programmatic variants. Customers see these directly in their daily / weekly change reports. Quality is entirely unmeasured. There is no gold standard, no reference summaries, no narrative coherence score, no faithfulness check.

R3: Summarisation ROUGE thresholds are permissive. The summarisation eval exists with thresholds of 0.15 ROUGE-L (executive) / 0.1 ROUGE-L (detailed). The S149 gap analysis (M4) flags these as “too low to detect meaningful regressions. A 50% drop in summary quality could still pass.” Reference summaries themselves were noted as low-quality. The eval exists but its teeth are dull.

R4: Cost is invisible in aggregate. Individual AI call sites log response.usage (classify.ts does this in the Pass 1 / Pass 2 paths; others partially). But there is no aggregation: no per-touchpoint cost trend, no “how much did classification cost last week”, no pre-deploy cost estimate for a prompt change. S149 H4 and the roadmap §3.5.5 both call for systematic cost tracking.

R5: Output mode inconsistency is a maintenance burden. The platform uses 5 different output strategies: tool_use (forced) (8 touchpoints), Structured Outputs (2 touchpoints), Plain text JSON (3 touchpoints), Search Result Citations (1 touchpoint), Plain text (1 touchpoint). There is no documented policy on when to use which. New touchpoints inherit the choice of whoever shipped first. Each mode has different error surfaces and validation patterns.

R6: MCP tool descriptions are unevaluated against a scoring rubric. None of the 41 MCP tools have been audited for description quality. MCP eval Layer 4 (functional correctness) measures whether the tools work, not whether their descriptions help an LLM select them correctly. W6 ships a Claude-as-judge rubric to close this gap. Note (S155 WP1.4 F1): the original motivation referenced a tool-design skill at .claude/skills/context-engineering-collection/skills/tool-design/, which does not exist on disk — W6 has been rewritten to use a hand-rolled Claude-as-judge rubric rather than an external Python evaluator. See §3.5 and §5.6.

2.2 Why Phase 5 is last

Each workstream is individually valuable, but none blocks production use of the platform. Customers can use classification, entity extraction, SI, bid drafting, summarisation, and digests today — Phase 5 adds quality measurement and cost visibility, not new functionality. It is last because its effort-to-value ratio is worse than Phases 1-4, AND because several items (W1, W5) are blocked on prerequisites that are themselves roadmap items.

3. Dependencies

3.1 Workstream dependencies (per-item)

Workstream	Depends on	Blocker?
W1 Bid drafting rubric	§1.4 real bid data (roadmap)	YES — hard blocker
W2 Summarisation skill conversion	Phase 2 classification skill conversion pattern	Already shipped (S147) — non-blocker
W3 Digest evaluation	Stable digest output format	See §3.2 below
W4 Output mode standardisation	Architectural decision on default output mode	See §3.3 below
W5 Cost tracking	§9.7 structured logging (roadmap)	See §3.4 below — partial blocker
W6 MCP tool description audit	Claude-as-judge rubric (defined in §3.5) + MCP Eval Layer 4	Non-blocker (rubric defined in spec; Layer 4 already shipped)

3.2 W3 Digest dependency: stable output format

The digest generator uses two programmatic prompt variants (standard + daily) with different output shapes. A digest eval requires a stable reference format to compare against. Two options:

Option A: Freeze the current digest output format as the evaluable contract. Any future change becomes a “format version bump” with baseline migration.
Option B: Extract the digest output shape into a typed Zod schema in lib/validation/digest-schema.ts, then evaluate against that schema plus quality metrics (theme coherence, coverage, structural compliance).

Recommendation: Option B. The schema extraction is low-cost and gives the eval something concrete to validate against. W3 begins with schema extraction as Task 3.1.

3.3 W4 Output mode decision

Before implementing W4, the team must commit to an output mode policy. The spec proposes:

Proposed default policy (REVISED per S155 WP1.4 F8): the original spec proposed tool_use (forced) as default for structured tasks without acknowledging that the parent audit (docs/audits/ai-eval/ai-touchpoint-map-and-improvement-strategy.md §2.5 lines 215-218) recommends Structured Outputs as default. The F8 finding required either justifying the disagreement or adopting the parent audit’s recommendation. Resolution: adopt Structured Outputs as the default for new touchpoints; tool_use (forced) is the fallback for models or tasks where Structured Outputs is not yet supported.

New structured tasks (default): Structured Outputs. Per the parent audit, Structured Outputs is “more reliable than JSON-in-text parsing and more flexible than tool_use forcing.” Canonical future-state reference: new touchpoints introduced after W4 ships.

Classification-style tasks (existing): tool_use (forced) with a single-tool schema. Canonical reference: lib/ai/classify.ts. Existing tool_use (forced) call sites are NOT required to migrate — they work fine. Opportunistic migration is acceptable.

Generative tasks (bid drafting, digest narratives, summarisation prose): extended thinking + plain text for the prose body, with a Zod-schema-validated Structured Outputs response for metadata (word count, sections, citations). Two-output pattern: prose via plain text, metadata via Structured Outputs.

Hybrid tasks (question extraction with structure + reasoning): Structured Outputs is the default for new work; extended thinking is added if the task is reasoning-heavy.

Legacy Plain text JSON parsing is deprecated. Existing call sites (Python classification, relevance scoring, structured extraction) are migration candidates. They are migrated opportunistically, not on a forced schedule.

Exceptions (documented in the policy):

Python classification stays on plain-text JSON because the Python Anthropic SDK’s tool_use path is more painful. Exception granted for maintainability.
Search Result Citations mode for bid drafting Pass 2 is unique to that touchpoint (native Claude citation mechanism) — stays as is.
Vision/PDF analysis uses plain text because the user prompt is intentionally free-form. Stays as is.

The policy is a proposal, not yet committed. W4 begins with ADR (docs/adrs/ai-output-mode-policy.md) that documents the decision and its rationale.

3.4 W5 Structured logging dependency

§3.5.5 cost tracking depends on docs/specs/structured-logging-spec.md (roadmap §9.7, 14-19h effort, “Should” priority, not yet implemented). Structured logging provides the persistence layer for per-call cost records. Without it, cost tracking would need a bespoke persistence mechanism.

Two paths forward:

Path A: Wait for structured logging. W5 blocks until §9.7 is implemented. W5 then becomes “add cost fields to the structured log schema and ensure every AI call site populates them”.
Path B: Interim cost tracking. W5 ships a lightweight interim solution: a recordAiCall() helper in lib/ai/telemetry.ts that writes to a new ai_call_costs table (or to activity_history as a side channel), gated by a feature flag. When structured logging lands, W5 is refactored to use it.

Recommendation: Path B. The structured logging spec is a substantial piece of work (14-19h) and blocking cost tracking on it introduces unnecessary dependency chain. The interim solution is 2-3h of work, is additive, and has a clean migration path when §9.7 lands.

3.5 W6 MCP tool description evaluation approach

REVISED per S155 WP1.4 F1 (BLOCKER). The original spec assumed a tool-design skill at .claude/skills/context-engineering-collection/skills/tool-design/ with a ToolDescriptionEvaluator Python class. That path does not exist on disk. The only tool-design artefact found is a single Markdown reference doc at ~/.claude/plugins/marketplaces/claude-plugins-official/plugins/mcp-server-dev/skills/build-mcp-server/references/tool-design.md (not a Python evaluator).

Resolution: W6 is rewritten to use a hand-rolled Claude-as-judge rubric. A new scripts/eval-mcp-tool-descriptions.ts reads the 41 tool definitions, sends each to getModelForTier('analysis') with a fixed scoring prompt, and collects per-dimension scores. The five dimensions are defined inline in the spec (not inherited from the missing skill):

Clarity (1-5): Does the description clearly state what the tool does?
Parameter documentation (1-5): Are parameters named, typed, and described with examples?
Example quality (1-5): Are usage examples concrete, correct, and diverse enough to cover common cases?
Edge case coverage (1-5): Does the description note boundary behaviour (empty results, rate limits, permission failures)?
Error handling hints (1-5): Does the description tell the caller what errors to expect and how to recover?

Aggregate score per tool = average of the 5 dimensions. Flag tools with total ≤ 3.0 or any single dimension ≤ 2.0 for human review.

MCP Eval Layer 4 (functional correctness) exists at scripts/mcp-eval/functional-correctness.ts with 37 checks. It provides the live metric for whether tool descriptions help an LLM pick the right tool. W6 uses both: the Claude-as-judge rubric for static analysis and Layer 4 for live measurement.

3.6 Phase 4 integration

Phase 5 workstreams should emit signals and register adapters into Phase 4’s refinement infrastructure where possible. Per S155 WP1.4 F2, the adapter tasks below are the ONLY Phase 5 work that depends on Phase 4 — everything else ships independently.

Workstream	Phase 4 integration	Adapter task	Phase 4 dependency
W1	New `BidDraftingAdapter` + `BidDraftingGoldStandardEvaluator`	Task 5.1.8	Blocks on Phase 4 Tasks 4.1 + 4.2
W2	New `SummarisationSkillAdapter` (replaces the `InlinePromptAdapter`)	Task 5.2.9	Blocks on Phase 4 Tasks 4.1 + 4.2
W3	New `DigestAdapter` + `DigestGoldStandardEvaluator`	Task 5.3.7	Blocks on Phase 4 Tasks 4.1 + 4.2
W4	No adapter — feeds into A/B testing	—	No hard Phase 4 dependency
W5	Provides cost fields to every Phase 4 component	—	No hard Phase 4 dependency
W6	New `McpToolDescriptionAdapter` per category file	Task 5.6.8	Blocks on Phase 4 Tasks 4.1 + 4.2

The adapter tasks create files under lib/refinement/adapters/ — a directory that does not exist on main and is created only by Phase 4 Tasks 4.1 + 4.2. The evaluator and skill-conversion tasks do NOT depend on Phase 4 and may ship first. Each adapter task in §5 is marked with an explicit **Depends on:** Phase 4 Tasks 4.1 + 4.2 line per F2 resolution.

Phase 5 eval/skill tasks can ship without the adapter integrations (the integrations are additive); the adapter shapes must align with the Phase 4 interfaces defined in docs/specs/ai-eval-phase4-hitl-refinement-spec.md.

4. Architecture

4.1 Workstream scope map

1
                        Cost    Eval    Skill     A/B     Rollback
2
                        track   runner  convert   ready   ready
3
W1 Bid rubric              .       X       .        X        X
4
W2 Summ skill              .       X       X        X        X
5
W3 Digest eval             .       X       .        X        X
6
W4 Output mode             .       .       .        X        .
7
W5 Cost tracking           X       .       .        .        .
8
W6 MCP tool desc           .       X       .        .        X

(X = workstream delivers this; . = not in scope for that workstream.)

4.2 Shared interfaces

All workstreams reuse Phase 1’s shared eval infrastructure:

lib/eval/types.ts — EvalResult, EvalBaseline, RegressionResult
lib/eval/metrics.ts — ROUGE, BERTScore (wrapper for existing scripts/compute-bertscore.py per S155 WP1.4 F3), MRR, NDCG, precision/recall/F1
lib/eval/baseline.ts — save / load / compare baselines. Extended by Phase 3 to include severity: 'block' | 'warn' | 'info' and variance_band fields per Phase 3 §4.2.
lib/eval/reporter.ts — console + JSON output

New workstream-specific additions:

~~lib/eval/bertscore.ts (W2) — BERTScore implementation for summarisation~~ REMOVED per S155 WP1.4 F3: BERTScore already implemented at scripts/compute-bertscore.py and integrated into scripts/eval-summarisation.ts via --bertscore flag. W2 Task 5.2.5 verifies the existing integration; a TS wrapper is optional.
lib/eval/nli.ts (W2, W3) — NLI entailment for faithfulness checking (new)
lib/ai/telemetry.ts (W5) — recordAiCall() shared wrapper (new)
lib/refinement/adapters/bid-drafting.ts (W1) — Phase 4 adapter (DEPENDS ON PHASE 4)
lib/refinement/adapters/summarisation.ts (W2) — Phase 4 adapter (DEPENDS ON PHASE 4)
lib/refinement/adapters/digest.ts (W3) — Phase 4 adapter (DEPENDS ON PHASE 4)
lib/refinement/adapters/mcp-tool-description.ts (W6) — Phase 4 adapter (DEPENDS ON PHASE 4)

4.2.1 Per-metric severity and variance_band defaults (added per S155 WP1.4 F15 + F16)

Phase 3 §4.2 extends EvalBaseline with severity and variance_band fields per metric. Phase 5 workstreams must populate these when saving their first baselines. Default assignments:

Workstream	Metric	`severity`	`variance_band`	Phase 3 tier
W1 bid drafting	Compliance score (deterministic)	`warn`	±2%	manual on-demand
W1 bid drafting	Narrative quality (LLM-judge)	`warn`	±5%	manual on-demand
W1 bid drafting	Evidence strength (deterministic)	`warn`	±2%	manual on-demand
W1 bid drafting	Faithfulness (NLI)	`warn`	±3%	manual on-demand
W1 bid drafting	Completeness (deterministic)	`warn`	±2%	manual on-demand
W2 summarisation	ROUGE-L executive	`warn`	±2%	scheduled nightly
W2 summarisation	ROUGE-L detailed	`warn`	±2%	scheduled nightly
W2 summarisation	BERTScore (deterministic)	`warn`	±2%	scheduled nightly
W2 summarisation	NLI faithfulness	`warn`	±3%	scheduled nightly
W3 digest	Theme coherence (LLM-judge)	`warn`	±5%	scheduled nightly
W3 digest	Coverage completeness	`warn`	±3%	scheduled nightly
W3 digest	Structural compliance (Zod)	`block`	±0%	pre-merge (schema check)
W3 digest	Narrative quality (LLM-judge)	`warn`	±5%	scheduled nightly
W3 digest	Faithfulness (NLI)	`warn`	±3%	scheduled nightly
W4 output mode	Migration health (deterministic)	`block`	±0%	pre-merge AI-light
W5 cost tracking	Per-touchpoint cost (deterministic)	`info`	±10%	pre-merge unit test + nightly aggregation
W6 MCP tool descriptions	Per-dimension score (LLM-judge)	`warn`	±5%	manual on-demand

Defaults rationale: LLM-judge metrics default to warn with ±5% because LLM scoring has inherent variance. Deterministic metrics default to ±2% (deterministic but subject to fixture changes). block severity is reserved for schema-compliance checks (hard contracts) — everything else is warn or info so a single eval miss does not stop a merge. The F24 “retry once with 30-second backoff” policy applies to all LLM-judge metrics.

Revision trigger: after the first week of nightly runs, the variance bands should be re-measured empirically and the table updated. Tracked in §12 Q3.

5. Per-item Sections

5.1 W1 — §3.5.1 Bid Drafting Rubric

Goal: Build a multi-dimensional LLM-as-judge quality rubric for bid drafting responses. Replace the current 2-line “Score 0-100” quality check prompt with a structured scoring rubric.

Current state:

Eval runner: scripts/eval-bid-drafting.ts (415 lines) — structurally complete but cannot produce results because the gold standard uses synthetic question_id values and the eval queries the empty bid_responses table.
Gold standard: __tests__/fixtures/bid-drafting-eval-gold-standard.json — 24 synthetic items.
Skill files: lib/ai/skills/bid-writing.md (617 lines per S155 WP1.4 F18), lib/ai/skills/uk-procurement.md (335 lines per S155 WP1.4 F19).
Quality check (Pass 3): lib/ai/quality-check.ts — 2-line system prompt, no scoring rubric.

BLOCKER: §1.4 real bid data. This workstream cannot ship until a real bid workspace with real questions and real responses exists. The roadmap item §1.4 (1-2 sessions effort, Pending) is the hard prerequisite. See the roadmap’s §1.4 entry. W1 spec can be written now; implementation must wait.

Proposed design (implementation-ready, execute after §1.4):

The rubric has five scoring dimensions, each 0-10, weighted:

Dimension	Weight	What it measures	Reference
Completeness	25%	Does the response address all sub-questions?	Gold standard expected answer elements
Evidence strength	25%	Are claims backed by specific KB citations?	Citation count + source quality
Compliance language	20%	Does it use required UK procurement conventions (MEAT scoring language, word count adherence)?	`uk-procurement.md` skill
Structure	15%	Correct section headings, bullet structure, response format	Template schema
Faithfulness	15%	No hallucinated claims; groundedness against cited KB items	NLI entailment score

The rubric is implemented as a claude-sonnet-4-6 call with the rubric embedded in the system prompt and the candidate response in the user message. Structured output via tool_use (forced) with a schema per dimension.

Bid-specific fixtures (built during §1.4, consumed here):

At least 10 real bid questions
At least 10 human-written reference responses (ideally by the customer)
Human scores on the 5 rubric dimensions (for calibration)

Task breakdown:

Task 5.1.1: After §1.4 lands, populate real question IDs in the gold standard. Effort: 30 min.
Task 5.1.2: Refactor eval-bid-drafting.ts to read from the gold standard fixture directly (not via an empty DB table query — this is S149 H5 / roadmap §3.2.3). Effort: 1-2h (tracked separately in §3.2.3 but executed here if not already done).
Task 5.1.3: Define the 5-dimension rubric in a Zod schema. Effort: 30 min.
Task 5.1.4: Implement lib/ai/bid-quality-rubric.ts — the rubric evaluator. Effort: 3h.
Task 5.1.5: Rewrite lib/ai/quality-check.ts Pass 3 to use the rubric. Effort: 1h.
Task 5.1.6: Collect human scores on 10+ reference responses for calibration. Effort: 1-2h (requires customer input).
Task 5.1.7: Run the bid drafting eval with the rubric; save the first baseline. Effort: 30 min.
Task 5.1.8: Implement BidDraftingAdapter for Phase 4 refinement. Effort: 1h. Depends on: Phase 4 Tasks 4.1 + 4.2 (creates files under lib/refinement/adapters/ which Phase 4 owns — per S155 WP1.4 F2).
Task 5.1.9: Add NLI-based groundedness checking. Effort: 2h.

Acceptance:

bun run eval:bid-drafting runs end-to-end against real bid data and produces 5-dimension scores.
The first baseline is saved.
The rubric correlates with human scores ≥0.7 on the calibration set.
Phase 4 can consume bid drafting signals via the adapter.

Dependencies: §1.4 real bid data. §3.2.3 bid drafting eval fixture refactor. Phase 4 adapter infrastructure (optional but recommended).

Effort: 1-2 sessions (matches roadmap, executed after §1.4 ships).

5.2 W2 — §3.5.2 Summarisation Skill Conversion

Goal: Promote the hardcoded summarisation prompt to a skill file following the classification skill conversion template from Phase 2. Strengthen the summarisation eval with BERTScore and NLI faithfulness.

Current state:

lib/ai/summarise.ts (245 lines) — hardcoded ~15-line prompt inline in the summariseContent() function.
scripts/kb_pipeline/summarise.py — mirror Python implementation, also hardcoded.
lib/ai/skills/ — no summarisation.md skill file exists.
scripts/eval-summarisation.ts (721 lines per S155 WP1.4 F20) — two-tier scoring; ROUGE-L baseline at 36.1% executive / 25.4% detailed; thresholds at min 0.15 / min 0.1. Baseline exists.
No BERTScore implementation.
No NLI faithfulness check.

Proposed design:

Mirror the Phase 2 classification skill conversion, scoped to summarisation:

Phase 5.2a: Skill file creation

Extract the inline prompt into lib/ai/skills/summarisation.md.
Add content-type-specific guidance sections (Q&A pair summaries, article summaries, policy summaries, case study summaries) — mirrors the entity types reference pattern from Phase 2.
Add placeholders: {CONTENT_TYPE}, {TITLE}, {WORD_LIMIT}.
Update summariseContent() to load via loadSkill('summarisation').
Update the Python mirror to read from docs/reference/summarisation-prompt.md (new reference file, mirrors the classification reference prompt pattern).

Phase 5.2b: Eval strengthening

Add BERTScore implementation (lib/eval/bertscore.ts). Use a pre-trained model via the @xenova/transformers package or call an embedding API for a simpler approximation.
Add NLI faithfulness check (lib/eval/nli.ts). Uses Claude as an NLI judge: given source text + summary, return entailment / contradiction / neutral per claim.
Add a structural compliance check (executive / detailed / takeaways sections present).
Add length compliance check (200-char limit for ai_summary).
Re-calibrate thresholds after skill conversion. Raise ROUGE-L minimums from 0.15/0.1 to 0.30/0.25 (or whatever the new baseline supports).

Phase 5.2c: Content-type specialisation

Measure per-content-type ROUGE/BERTScore breakdown.
If Q&A pairs score significantly worse than articles (as suspected), add Q&A-specific guidance to the skill file.
Re-run eval; confirm improvement.

Task breakdown:

Task 5.2.1: Audit current summarisation prompt vs the Phase 2 classification pattern. Effort: 30 min.
Task 5.2.2: Create lib/ai/skills/summarisation.md. Effort: 2h.
Task 5.2.3: Refactor summariseContent() to use the skill. Effort: 1.5h.
Task 5.2.4: Mirror in Python via docs/reference/summarisation-prompt.md. Effort: 1.5h.
Task 5.2.5 (REVISED per S155 WP1.4 F3): ~~Implement lib/eval/bertscore.ts.~~ BERTScore is already implemented at scripts/compute-bertscore.py and integrated into scripts/eval-summarisation.ts via the --bertscore flag (see lines 23, 309-352 of that script). Verify the existing integration meets W2 needs; if Tier-1 BERTScore is needed by code outside the eval runner, extract a TypeScript wrapper lib/eval/bertscore.ts that shells out to the Python script. Effort: 30 min verification + optional 1h TS wrapper.
Task 5.2.6: Implement lib/eval/nli.ts faithfulness check. Effort: 2h.
Task 5.2.7: Extend eval-summarisation.ts to use BERTScore + NLI. Effort: 1h.
Task 5.2.8: Re-run eval, save new baseline with raised thresholds. Effort: 30 min.
Task 5.2.9: Implement SummarisationSkillAdapter for Phase 4. Effort: 1h. Depends on: Phase 4 Tasks 4.1 + 4.2 (creates files under lib/refinement/adapters/ which Phase 4 owns — per S155 WP1.4 F2).

Acceptance:

lib/ai/summarise.ts no longer contains inline prompt text.
bun run eval:summarisation reports ROUGE-L, BERTScore, NLI faithfulness, and structural compliance.
New baseline saved with thresholds at a level that would catch a 20% quality drop.
No regression in domain / subtopic accuracy of classification (the skill conversion must not accidentally modify classification output).
Pipeline parity test passes (TS and Python summarisation agree on behaviour).

Dependencies: Phase 2 classification skill conversion pattern (already shipped). Phase 4 adapter interface (optional).

Effort: 1 session (matches roadmap).

5.3 W3 — §3.5.3 Digest Evaluation

Goal: Build an eval suite for digest (Change Reports) generation. Close the gap that the most user-facing AI output has zero quality measurement.

Current state:

lib/ai/digest.ts (615 lines) — two programmatic prompt variants (standard + daily). Schema validated via Zod but no quality eval.
app/api/change-reports/* — change-report generation API endpoints.
lib/change-reports/change-reports-export.ts, digest-helpers.ts — post-generation formatting.
No gold standard; no eval runner; no baseline.

Proposed design:

Phase 5.3a: Stabilise output format

REVISED per S155 WP1.4 F4: ~~Extract the digest output shape into lib/validation/digest-schema.ts as a Zod schema.~~ The schema is already extracted to lib/validation/ai-schemas.ts:51-62 as DigestResponseSchema, imported by lib/ai/digest.ts:10. Verify this is the canonical evaluable contract.
Document the schema in the eval runner header and mark it as the evaluable contract.

Phase 5.3b: Gold standard creation

Generate digests for 10 reference time windows (e.g. 10 weeks of real KB activity).
Have a human (Liam or editor) rate each digest on the rubric below.
Save as __tests__/fixtures/digest-eval-gold-standard.json.

Phase 5.3c: Eval runner

Mirrors the pattern of eval-classification.ts and eval-summarisation.ts:

Load the gold standard.
For each reference time window, re-generate the digest via digest.ts.
Score against the rubric:

Metric	Method	Threshold
Theme coherence	LLM-as-judge rubric (0-10)	Drop > 1 point
Coverage completeness	% of input items mentioned / summarised	Drop > 15%
Structural compliance	Required sections present (themes, domains, opportunities)	Any regression
Narrative quality	Human-calibrated LLM-as-judge (0-10)	Drop > 1 point
Faithfulness	NLI entailment against source items	Drop > 5%

Compare against baseline via lib/eval/baseline.ts.
Save/regress gates via the shared infra.

Phase 5.3d: Skill conversion (optional, if time allows)

If the digest prompt proves hard to improve via inline editing, convert to lib/ai/skills/digest.md following the classification template. Otherwise, leave as a programmatic prompt and skip.

Task breakdown:

Task 5.3.1 (REVISED per S155 WP1.4 F4): ~~Extract digest schema to lib/validation/digest-schema.ts.~~ Verify DigestResponseSchema at lib/validation/ai-schemas.ts:51-62 is the canonical evaluable contract and document it in the eval runner header. Effort: 15 min.
Task 5.3.2: Generate + label 10 reference digests for the gold standard. Effort: 2h (requires human scoring time).
Task 5.3.3: Implement scripts/eval-digest.ts. Effort: 3h.
Task 5.3.4: Wire LLM-as-judge for theme coherence and narrative quality. Effort: 2h.
Task 5.3.5: Run eval; save baseline. Effort: 30 min.
Task 5.3.6: (Optional) Convert to skill file. Effort: 2h.
Task 5.3.7: Implement DigestAdapter for Phase 4 refinement. Effort: 1h. Depends on: Phase 4 Tasks 4.1 + 4.2 (creates files under lib/refinement/adapters/ which Phase 4 owns — per S155 WP1.4 F2).
Task 5.3.8: Add Vitest eval test wrapper gated behind EVAL_DIGEST=1. Effort: 30 min.

Acceptance:

bun run eval:digest runs end-to-end against the 10-item gold standard.
First baseline saved.
Theme coherence and narrative quality LLM-as-judge scores correlate ≥0.6 with human scores.
Eval integrated into Phase 3 regression detection (if Phase 3 has landed) or standalone if not.

Dependencies: Stable digest output format (see §3.2). Phase 4 adapter interface (optional).

Effort: 1 session (matches roadmap).

5.4 W4 — §3.5.4 Output Mode Standardisation

Goal: Commit to a default output mode policy and migrate non-conforming touchpoints opportunistically.

Current state:

5 different strategies across 18 touchpoints. Distribution documented in docs/audits/ai-eval/ai-touchpoint-map-and-improvement-strategy.md §2.5.
No written policy.

Proposed design:

Output mode standardisation is a cross-cutting decision, not per-item. The workstream delivers:

An ADR documenting the default policy and its exceptions (see §3.3 for the proposed policy text).
A migration guide for each deprecated pattern.
Tracked opportunistic migrations — each touchpoint that is modified for any reason gets migrated to the new policy as part of the change. No forced-march migration.

Default policy (proposed, to be confirmed in ADR):

Task type	Default output mode	Example touchpoint
Structured extraction with fixed schema	`tool_use (forced)`	Classification (TS) — canonical reference
Generative prose (bid drafts, digest narratives)	Extended thinking + plain text prose + metadata tool_use	Bid drafting Pass 2 (partial — uses Search Result Citations additionally)
Hybrid reasoning + structured output	`tool_use (forced)` with extended thinking	Question extraction (DOCX)
Simple text responses	Plain text	Vision/PDF analysis

Documented exceptions:

Python classification (plain text JSON) — SDK ergonomics exception
Bid drafting Pass 2 (Search Result Citations) — native feature exception
Vision analysis (plain text) — intentionally generic exception

Task breakdown:

Task 5.4.1: Draft the ADR at docs/adrs/ai-output-mode-policy.md. Effort: 2h.
Task 5.4.2: Review + approve with Liam. Effort: 30 min.
Task 5.4.3: Create migration guide at docs/reference/output-mode-migration-guide.md. Effort: 1h.
Task 5.4.4: Tag all non-conforming call sites with // TODO(output-mode): migrate per docs/adrs/ai-output-mode-policy.md. Effort: 30 min.
Task 5.4.5: Migrate lib/ai/extract-content.ts (plain text JSON → tool_use) as the first reference migration. Effort: 1h.
Task 5.4.6: Update docs/audits/ai-eval/ai-touchpoint-map-and-improvement-strategy.md §2.5 to reflect the policy and reduce the expected drift. Effort: 30 min.

Acceptance:

ADR committed and referenced from CLAUDE.md’s “Key Product Design Principles” section.
Migration guide exists with before/after examples for each deprecated pattern.
One reference migration completed and documented.
Every non-conforming touchpoint has a TODO comment linking to the ADR.

Dependencies: Architectural decision (this spec proposes it; needs Liam’s approval). The §3.5.4 roadmap item’s ~3h effort matches.

Effort: ~3h (matches roadmap).

5.5 W5 — §3.5.5 Cost Tracking Across Touchpoints

Goal: Provide per-touchpoint cost visibility and aggregation. Every AI call site records token usage + estimated cost. Aggregate dashboards show trends.

Current state:

lib/ai/classify.ts — logs Pass 1 + Pass 2 token usage (S149 Wave 3).
lib/ai/extract-content.ts — imports pricing.ts, logs usage.
lib/ai/summarise.ts, digest.ts, draft.ts, quality-check.ts, vision.ts, extract-questions.ts — log response.usage but do NOT estimate cost or persist it.
lib/ai/pricing.ts — pricing constants for Opus, Sonnet-4-5, Haiku-4-5, and claude-sonnet-4-6 (the most-used model, confirmed present at lib/ai/pricing.ts:32-37 per S155 WP1.4 F5 — the S149 L3 gap is closed).
No aggregation. No per-touchpoint cost trend dashboard.
No persistence of cost records beyond ad-hoc log lines.

Proposed design (Path B — interim solution, from §3.4):

Phase 5.5a: Shared telemetry wrapper

Create lib/ai/telemetry.ts with a recordAiCall() helper that uses tryQuery() from @/lib/supabase/safe (per S155 WP1.4 F11 — best-effort Result-returning pattern, never throws to caller). On persistence failure, calls logBestEffortWarn('ai_cost_tracking.record_ai_call.failed', msg, { err }).

1
import { tryQuery } from '@/lib/supabase/safe';
2
import { logBestEffortWarn } from '@/lib/supabase/telemetry';
3

4
export async function recordAiCall<T>(
5
  touchpointId: string,
6
  call: () => Promise<Anthropic.Message>,
7
  options?: {
8
    workspace_id?: string;
9
    content_item_id?: string;
10
    correlation_id?: string | null; // NEW per F12 — §9.7 forward-compat
11
  },
12
): Promise<Anthropic.Message> {
13
  const start = Date.now();
14
  const response = await call();
15
  const elapsed = Date.now() - start;
16
  const usage = response.usage;
17
  const cost = estimateCost(response.model, usage);
18
  const result = await tryQuery(
19
    supabase.from('ai_call_costs').insert({
20
      touchpoint_id: touchpointId,
21
      model: response.model,
22
      input_tokens: usage.input_tokens,
23
      output_tokens: usage.output_tokens,
24
      cache_creation_tokens: usage.cache_creation_input_tokens ?? 0,
25
      cache_read_tokens: usage.cache_read_input_tokens ?? 0,
26
      estimated_cost_usd: cost,
27
      latency_ms: elapsed,
28
      workspace_id: options?.workspace_id ?? null,
29
      content_item_id: options?.content_item_id ?? null,
30
      correlation_id: options?.correlation_id ?? null, // F12
31
    }),
32
  );
33
  if (!result.ok) {
34
    logBestEffortWarn(
35
      'ai_cost_tracking.record_ai_call.failed',
36
      `Failed to persist ai_call_costs row for ${touchpointId}`,
37
      { err: result.error },
38
    );
39
  }
40
  return response;
41
}

Wrap every AI call site in recordAiCall(). Start with the non-instrumented ones (summarise, digest, draft, quality-check, vision, extract-questions) and the three lib/intelligence/* call sites added per F17.

Sandbox gotcha (per S155 WP1.4 F10): any Bun script that calls recordAiCall() via the sandboxed Claude Code environment must run with dangerouslyDisableSandbox: true — otherwise the Supabase insert hangs indefinitely on HTTP/2 204 per the CLAUDE.md Bun-fetch-204 gotcha. Production (Vercel) is unaffected.

Phase 5.5b: Persistence

Create ai_call_costs table (schema updated per S155 WP1.4 F12 to include correlation_id for §9.7 structured logging forward-compat):

1
CREATE TABLE ai_call_costs (
2
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
3
  touchpoint_id text NOT NULL,
4
  model text NOT NULL,
5
  input_tokens integer NOT NULL,
6
  output_tokens integer NOT NULL,
7
  cache_creation_tokens integer DEFAULT 0,
8
  cache_read_tokens integer DEFAULT 0,
9
  estimated_cost_usd numeric(10, 6) NOT NULL,
10
  latency_ms integer NOT NULL,
11
  workspace_id uuid,
12
  content_item_id uuid,
13
  correlation_id text,  -- NEW per F12: forward-compat with §9.7 structured logging
14
  created_at timestamptz DEFAULT now()
15
);
16

17
CREATE INDEX idx_ai_call_costs_touchpoint_time ON ai_call_costs(touchpoint_id, created_at DESC);
18
CREATE INDEX idx_ai_call_costs_workspace ON ai_call_costs(workspace_id) WHERE workspace_id IS NOT NULL;
19
CREATE INDEX idx_ai_call_costs_correlation ON ai_call_costs(correlation_id) WHERE correlation_id IS NOT NULL;

Retention (per S155 WP1.4 F25): ai_call_costs rows older than 365 days are archived or deleted by a nightly admin-tooling job (to be specified in a future housekeeping session — tracked in §11 follow-ups and §12 Q5).

Writes are best-effort (logBestEffortWarn pattern from the silent-failure spec) — a cost persistence failure must NEVER block a production AI call.

Phase 5.5c: Pricing module fixes (REVISED per S155 WP1.4 F5)

~~Add claude-sonnet-4-6 to lib/ai/pricing.ts~~ (already present at lib/ai/pricing.ts:32-37 — the S149 L3 gap is closed).
Add a getPricing(model) helper with a sensible fallback for unrecognised models (return zero + log warning via logBestEffortWarn('ai_cost_tracking.unknown_model', ...); do not throw). Add unit test covering both the happy path and the unknown- model fallback.

Phase 5.5d: Aggregation dashboard (clarified per S155 WP1.4 F13)

Admin-only page at /admin/ai-costs showing:

Total cost this week / month / quarter, per touchpoint.
Cost trend (daily/weekly sparklines).
Cost per classification / per bid draft / per digest.
Top-N expensive workspaces.

Backed by SQL aggregate queries against ai_call_costs.

Implementation conventions (per F13 — these are non-negotiable):

Auth: const auth = await getAuthorisedClient(['admin']); if (!auth.success) return authFailureResponse(auth); — use the discriminated-union check (auth.success), NOT auth.authorised, per the CLAUDE.md getAuthorisedClient() gotcha.
Data fetching: TanStack Query exclusively. Keys in lib/query/query-keys.ts (new queryKeys.admin.aiCosts.* namespace). Fetchers in lib/query/fetchers.ts. NO SWR, NO raw fetch() in hooks.
Styling: Warm Meridian semantic tokens per docs/design/warm-meridian-implementation-spec.md. NO raw Tailwind colours (bg-red-500, etc.) — use semantic tokens (bg-destructive, bg-muted, etc.).
Component location: new dashboard components go under components/admin/ai-costs/ per the project’s component layout.
Accessibility: WCAG 2.1 AA — keyboard navigation, semantic headings, no colour-alone meaning.
UK English: “Cost”, “Behaviour”, “Organisation”, etc.

Phase 5.5e: Structured logging migration path

When §9.7 structured logging ships, refactor recordAiCall() to also emit a structured log line. The ai_call_costs table can be retired or kept as a denormalised query-friendly view.

Task breakdown:

Task 5.5.1 (REVISED per S155 WP1.4 F5): ~~Add claude-sonnet-4-6 to pricing.ts~~ (already present at lib/ai/pricing.ts:32-37). Add getPricing(model) helper with fallback for unrecognised models (return zero + log warning via logBestEffortWarn('ai_cost_tracking.unknown_model', ...)). Add unit test __tests__/lib/ai/pricing.test.ts covering both the happy path and the unknown-model fallback. Effort: 30 min.
Task 5.5.2: Create ai_call_costs table via migration. Effort: 30 min.
Task 5.5.3: Implement lib/ai/telemetry.ts with recordAiCall(). Effort: 2h.
Task 5.5.4: Wrap every AI call site. Effort: 2h.
Task 5.5.5: Pipeline parity for Python — Python wrapper in scripts/kb_pipeline/telemetry.py. Effort: 1h.
Task 5.5.6: Admin cost dashboard. Effort: 3h.
Task 5.5.7: Documentation: add a gotcha to CLAUDE.md about always wrapping AI calls in recordAiCall(). Effort: 15 min.
Task 5.5.8: Migration plan document for structured-logging handoff (what to refactor when §9.7 lands). Effort: 30 min.

Acceptance:

Every anthropic.messages.create(...) call site in lib/ai/** and app/api/** is wrapped in recordAiCall().
ai_call_costs table has rows for every touchpoint after one day of production traffic.
claude-sonnet-4-6 has a pricing entry.
The admin dashboard renders real cost data.
recordAiCall() failures are best-effort (do not throw, log via logBestEffortWarn).
ESLint rule (stretch goal) flags unwrapped anthropic.messages.create(...) calls.

Dependencies: §9.7 structured logging (partial — see §3.4). Best-effort logging pattern (already shipped).

Effort: 1 session (matches roadmap).

5.6 W6 — §3.5.6 MCP Tool Description Audit (Claude-as-judge rubric)

Goal: Score all 41 MCP tool descriptions against a 5-dimension rubric (clarity, parameter documentation, example quality, edge case coverage, error handling hints) and propose improvements for any tool with poor LLM selection reliability.

REVISED per S155 WP1.4 F1 (BLOCKER): original spec depended on a tool-design skill at .claude/skills/context-engineering-collection/skills/tool-design/ that does not exist. W6 now uses a hand-rolled Claude-as-judge rubric implemented in a new eval script — no external Python evaluator dependency. See §3.5 for the full dimension definitions.

Plugin prompt audit (was Phase 5.6e “bonus”): demoted to §11 References follow-up list per S155 WP1.4 F22, not part of W6 acceptance.

Current state:

41 MCP tools across 11 category files (lib/mcp/tools/*.ts). Inventory in docs/generated/mcp-inventory.md.
Tool descriptions are inline in each category file as part of the tool registration.
No audit has been performed.
MCP Eval Layer 4 (functional correctness) exists and measures whether tools work correctly when called — NOT whether their descriptions help selection.

Proposed design:

Phase 5.6a: Static audit via Claude-as-judge

Extract every tool description into a flat list at __tests__/fixtures/mcp-tool-descriptions.json, structured as [{ toolName, category, description, parameters: [...], examples: [...] }].
Implement scripts/eval-mcp-tool-descriptions.ts that reads the fixture and calls getModelForTier('analysis') with a fixed scoring prompt (committed alongside the script, not in an external skill file). For each tool, the model returns a JSON object with five integer scores (clarity, parameter_documentation, example_quality, edge_case_coverage, error_handling_hints) each in [1, 5], plus a per-dimension one-sentence justification.
The script aggregates results, computes the average score per tool, and writes a baseline to __tests__/fixtures/eval-baselines/mcp-tool-descriptions.baseline.json.
Rank tools by total score. Flag anything with average < 3.0 or any single dimension ≤ 2.
Write the audit report to docs/audits/mcp-tool-description-audit.md.

Phase 5.6b: Live measurement cross-reference

For each flagged tool, cross-reference with MCP Eval Layer 4 results. Are tools with poor descriptions also tools with poor functional correctness scores?
Identify the intersection: tools that are both badly described AND badly selected by LLMs. These are the highest-priority fixes.

Phase 5.6c: Improvement proposals

For each flagged tool:

Draft a candidate improved description manually OR by asking the same Claude-as-judge model to propose an improvement based on the per-dimension feedback it gave in Phase 5.6a (no separate description_generator.py dependency — the original spec’s reference to that script was tied to the non-existent tool-design skill per S155 WP1.4 F1).
Re-run the candidate through scripts/eval-mcp-tool-descriptions.ts to confirm the score improves.
Produce a PR-ready diff.

Phase 5.6d: Apply improvements

Batch-apply the proposed improvements as a single PR. Run the MCP fixture sync test (__tests__/mcp/mcp-fixture-sync.test.ts) and full MCP eval suite (L1, L3, L4) to confirm no regression.

Phase 5.6e: Plugin prompt audit (bonus)

Plugin skill prompt audit (demoted per S155 WP1.4 F22): the original spec bundled a Phase 5.6e “bonus” audit of Claude Code plugin command prompts (.claude/plugins/knowledge-hub/1.0.0/skills/*/SKILL.md). This has been demoted to a §11 References follow-up item, not part of W6 acceptance. Task 5.6.7 below is therefore OMITTED.

Task breakdown:

Task 5.6.1: Extract MCP tool descriptions to a flat JSON fixture at __tests__/fixtures/mcp-tool-descriptions.json. Effort: 30 min.
Task 5.6.2 (REVISED per S155 WP1.4 F1): Implement scripts/eval-mcp-tool-descriptions.ts — a new Claude-as-judge runner that reads the fixture and scores each tool’s description against the five dimensions defined in §3.5 (clarity, parameter documentation, example quality, edge case coverage, error handling hints). Uses getModelForTier('analysis'). Effort: 2h (new script) + ~$3 Claude API cost per run.
Task 5.6.3: Cross-reference with MCP Eval Layer 4 results. Effort: 30 min.
Task 5.6.4: Write audit report to docs/audits/mcp-tool-description-audit-s{N}.md. Effort: 1h.
Task 5.6.5: Generate improved descriptions for flagged tools. Effort: 1-2h.
Task 5.6.6: Apply improvements + run full MCP eval. Save new baseline to __tests__/fixtures/eval-baselines/mcp-tool-descriptions.baseline.json. Effort: 1h.
Task 5.6.7: OMITTED per F22 (plugin skill prompt audit moved to §11 References follow-up).
Task 5.6.8: Implement McpToolDescriptionAdapter for Phase 4. Effort: 1h. Depends on: Phase 4 Tasks 4.1 + 4.2.

Acceptance (revised per S155 WP1.4 F6):

Audit report committed with per-tool 5-dimension scores for all 41 tools.
Baseline file committed at __tests__/fixtures/eval-baselines/mcp-tool-descriptions.baseline.json.
All tools flagged below threshold (average ≤ 3.0 or any dimension ≤ 2) have candidate improvements scored higher than the originals.
Post-improvement MCP eval (L1, L3, L4) baseline score ≥ first-run baseline (no regression). Improvement is not a ship gate.

Dependencies: Claude-as-judge rubric (defined in §3.5). MCP Eval Layer 4 (already shipped). Phase 4 adapter interface (Task 5.6.8 only).

Effort: 1 session (~9.5h after the Claude-as-judge rewrite per F7). The original 7.5h estimate increases by ~2h for the new script implementation.

6. Cross-cutting concerns

6.1 Observability

W5 is the observability deliverable. Every other workstream emits through W5’s recordAiCall() wrapper once it exists.
For Phase 4 integration, see the Phase 4 spec’s §6.1.

6.2 Cost

W1 eval runs are expensive (Opus for bid drafting). Gate behind explicit triggers; do NOT run on every PR.
W3 digest eval is similarly expensive (multi-item synthesis per run).
W2 BERTScore: the existing scripts/compute-bertscore.py (already wired into scripts/eval-summarisation.ts via --bertscore per S155 WP1.4 F3) uses a local model via Python bert-score package — one-time download, no per-call API cost.
W6 audit is ~$3 per run (41 tools × Sonnet analysis). The output is static after the first run and only re-runs when descriptions change.

Bun fetch 204 sandbox bypass (per S155 WP1.4 F10): Any W5 script that calls recordAiCall() from a sandboxed Bun environment — eval scripts, ad-hoc data repair scripts, bun -e snippets — must run with dangerouslyDisableSandbox: true. Production (Vercel) is unaffected. Reference: CLAUDE.md “Bun fetch hangs on HTTP 204 through sandbox proxy” gotcha.

529 Overloaded handling (per S155 WP1.4 F24): The nightly workflow retries each eval once with 30-second backoff on Anthropic 529 / OpenAI 503. If the second attempt also fails, the eval is marked as completed_with_errors and the summary emits the failure. Phase 3’s retry/backoff policy applies once Phase 3 infrastructure lands.

6.3 Rollback

W2 skill conversion follows the Phase 2 rollback pattern: keep the old inline prompt as a comment in the first commit, then remove in a follow-up.
W4 output mode migrations are opportunistic — no big-bang revert.
W6 MCP tool description updates go through normal PR review; revert via git revert if issues arise.

6.4 UK English + accessibility

All new docs, ADRs, and admin UIs follow CLAUDE.md’s UK English and WCAG 2.1 AA rules. The cost dashboard is a new admin UI (W5 Task 5.5.6) — apply Warm Meridian tokens, semantic colours, keyboard navigation.

6.5 AI visibility policy

Per docs/reference/ai-visibility-policy.md, Phase 5 deliverables are invisible infrastructure. The cost dashboard is admin-only. The audit report is developer-facing. No user-visible AI product features.

7. Implementation phasing within the phase

Workstreams are independent and can ship in any order within their dependency constraints. Suggested sequence:

Wave A (W5 first — everything else depends on it for observability):

Session 5.A — W5 full implementation (6-7h, one session)

Wave B (W4 second — informs W2/W3 output choices):

Session 5.B — W4 ADR + migration guide + reference migration (3-4h)

Wave C (W2 + W6 in parallel — independent):

Session 5.C.1 — W2 summarisation skill conversion (3-4h)
Session 5.C.2 — W6 MCP tool description audit (3-4h)

Wave D (W3 digest eval — depends on stable output format decision):

Session 5.D — W3 digest eval (3-4h)

Wave E (W1 bid drafting rubric — BLOCKED on §1.4 real bid data):

Session 5.E.1 — W1 implementation once §1.4 ships (3-4h)
Session 5.E.2 — W1 calibration with real data (2-3h)

Total expected effort (REVISED per S155 WP1.4 F7): ~52 hours across 8-12 sessions (revised upward from the original 20-26h / 5-6 sessions claim, which did not sum against the per-task breakdown). Per workstream:

W1 post-§1.4: 9h (unchanged — still blocked on real bid data)
W2: 9.5h (was 11.5h; −2h from F3 BERTScore strike)
W3: 9.25h (was 10h; −0.75h from F4 digest schema strike, +2h from F14 NLI re-estimate)
W4: 5h (unchanged)
W5: 10.25h (unchanged — F10/F11/F12/F13 are clarifications, not additional work)
W6: 9.5h (was 7.5h; +2h for F1 Claude-as-judge script rewrite)

Blocked W1 effort (3-5h more after §1.4 ships) is in addition to the 52h.

8. Test strategy

8.1 Workstream-specific tests

W1: Rubric correlation with human scores (stretch target ≥0.7).
W2: Skill loading tests (mirror Phase 2 Task 5 pattern). Pipeline parity tests for summarisation. BERTScore unit tests. NLI unit tests.
W3: Eval runner unit tests. Gold standard fixture validation.
W4: ADR review (not a test). Reference migration verified by eval.
W5: recordAiCall() unit tests with mocked Anthropic responses. Best-effort failure handling test (persistence failure must not throw). Pipeline parity between TS and Python wrappers.
W6: Audit script unit tests. Improved description scoring regression tests.

8.2 Integration tests

W2 end-to-end: a summarisation call produces a valid summary with the new skill file and recordAiCall() records the cost.
W3 end-to-end: a digest generation call produces a valid digest scored against the gold standard.
W5 end-to-end: every AI touchpoint, when called, writes a row to ai_call_costs. A dashboard query returns the row.

8.3 Regression guards

After W2 skill conversion, classification and entity eval must NOT regress (cross-touchpoint safety).
After W6 improvements, MCP eval L1/L3/L4 must NOT regress.
After every W4 opportunistic migration, the affected touchpoint’s eval must NOT regress.

9. Out of scope

Relevance scoring enhancement — SI relevance scoring already has the best prompt management infrastructure (§2.2 in the pathway). Phase 4 handles its refinement loop; Phase 5 does not add a new eval for it.
Quality check Pass 3 rubric on its own — merged into W1 (bid drafting rubric) because Pass 3 is part of the bid drafting pipeline.
Forced-march migration of all output modes — W4 is opportunistic.
Structured logging implementation — that is §9.7, a separate spec. W5 is the interim cost-tracking solution that uses structured logging when it lands.
Cross-workspace cost aggregation for multi-tenant — the platform is single-tenant-per-workspace, one Supabase project per client. No cross-client aggregation needed.
Model comparison evaluation — that is Phase 3 §3.3.3 (model change protocol), not Phase 5.
CI integration of eval suites — that is Phase 3 §3.3.4.
Automated robot-PR for W6 description updates — same as Phase 4 §3.4.4; manual PR application for now.

10. Acceptance criteria (REVISED per S155 WP1.4 F6)

Every acceptance criterion is now stated with a concrete pass/fail signal. Vague terms (“raised thresholds”, “no regression, ideally an improvement”, “every AI call site wrapped”) from the original spec have been replaced with numeric targets or enumerated checks.

W1 (split into pre and post per F9):
- W1-pre (ships in Phase 5): lib/ai/bid-quality-rubric.ts evaluator module committed with a 5-dimension Zod schema, synthetic-fixture unit tests pass, dry-run against the 24 synthetic fixtures produces scored output without errors.
- W1-post (blocked on §1.4, not part of Phase 5 acceptance): once real bid data exists, bun run eval:bid-drafting runs end-to-end against real data and produces 5-dimension scores, baseline saved at __tests__/fixtures/eval-baselines/bid-drafting.baseline.json, LLM- judge correlation ≥0.7 with human reference scores on a 10-item calibration sample.
W2: lib/ai/summarise.ts loads its prompt from lib/ai/skills/summarisation.md (no inline prompt text); bun run eval:summarisation reports ROUGE-L, BERTScore, and NLI faithfulness in a single run; new baseline saved with ROUGE-L executive ≥0.30 and detailed ≥0.25 (current measured values 0.361/0.254 already exceed these; per F26, if W2 wants genuine improvement headroom it should target 0.40/0.35 — the baseline acceptance is the confirmation-of- current-state option).
W3: bun run eval:digest runs against a 10-item stratified gold standard (5 per item-count bucket); all 5 metrics (theme coherence, coverage, structural compliance, narrative quality, faithfulness) are within their first-run variance bands; LLM-as-judge calibration correlation ≥0.6 with human reference scores on the 10-item sample; baseline saved at __tests__/fixtures/eval-baselines/digest.baseline.json.
W4: ADR committed at docs/adrs/ai-output-mode-policy.md declaring Structured Outputs as the default for new touchpoints (per F8 and the parent audit docs/audits/ai-eval/ai-touchpoint-map-and-improvement-strategy.md §2.5 lines 215-218); migration guide exists listing all 17 current LLM-text touchpoint slots (per F21 corrected count, not 18) with current mode
- recommended target; at least one reference migration completed and eval non-regressing.
W5: Every anthropic.messages.create(...) call site enumerated in Appendix B (updated per F17 to include lib/intelligence/article- summariser.ts, lib/intelligence/guide-generator.ts, lib/intelligence/ feed-poller.ts) is wrapped in recordAiCall(). Enforcement test at __tests__/lib/ai/cost-tracking-coverage.test.ts asserts every grep match for anthropic.messages.create in lib/ai/**, lib/intelligence/**, app/api/** has a matching recordAiCall call site within 10 lines. ai_call_costs table (including the new correlation_id TEXT NULL column per F12) is populated with rows for every touchpoint after one day of production traffic. Admin dashboard at /admin/ai-costs renders real cost data using TanStack Query, Warm Meridian semantic tokens, and getAuthorisedClient(['admin']) with the auth.success discriminated-union check per F13. getPricing(model) helper has a unit test covering the unknown-model fallback per F5.
W6: Audit report committed at docs/audits/mcp-tool-description-audit-s{N}.md with per-tool 5-dimension scores for all 41 tools (clarity, parameter documentation, example quality, edge case coverage, error handling hints per §3.5). Baseline saved at __tests__/fixtures/eval-baselines/mcp-tool-descriptions.baseline.json. All tools with average ≤3.0 or any single dimension ≤2 have candidate improvements scored higher than the originals via re-running the Claude-as-judge eval. Post-improvement MCP eval L1/L3/L4 baseline score ≥ first-run baseline (no regression). Plugin skill prompt audit is NOT part of W6 acceptance per F22; tracked as a follow-up in §11.
Cross-workstream: All scripts that write to Supabase have the dangerouslyDisableSandbox: true bypass documented per F10. No touchpoint has lost observability — the F6 enforcement test (see W5 above) guarantees this on every CI run.
Phase 4 adapter integration: each new adapter (Tasks 5.1.8, 5.2.9, 5.3.7, 5.6.8) matches the Phase 4 PromptStorageAdapter interface. Adapter tasks are gated on Phase 4 Tasks 4.1 + 4.2 landing (see §3.6 dependency table).
UK English + WCAG 2.1 AA maintained in all new UI (cost dashboard).

11. References

Document	Location
Parent strategy	`docs/audits/ai-eval/ai-touchpoint-map-and-improvement-strategy.md`
Gap analysis	`docs/audits/ai-eval/ai-eval-gap-analysis-s149.md`
Post-MVP roadmap	`docs/reference/product-roadmap.md` §3.5 and §1.4
Phase 2 classification skill spec	`docs/specs/ai-eval-phase2-classification-skill-spec.md`
Phase 3 regression infrastructure spec	`docs/specs/ai-eval-phase3-regression-infrastructure-spec.md`
Phase 4 HITL refinement spec	`docs/specs/ai-eval-phase4-hitl-refinement-spec.md`
Structured logging spec	`docs/specs/structured-logging-spec.md`
Silent failure prevention spec	`docs/specs/silent-failure-prevention-spec.md`
MCP evaluation spec	`docs/specs/mcp-evaluation-spec.md`
AI integration layers	`docs/reference/ai-integration-layers.md`
AI integration strategy	`docs/reference/ai-integration-strategy.md`
AI visibility policy	`docs/reference/ai-visibility-policy.md`
~~tool-design skill~~	~~`.claude/skills/context-engineering-collection/skills/tool-design/`~~ (skill does not exist on disk per S155 WP1.4 F1; W6 rewritten to use hand-rolled Claude-as-judge rubric — see §3.5 and §5.6)
llm-evaluation skill	`.claude/skills/llm-evaluation/`

| MCP inventory | docs/generated/mcp-inventory.md | | Existing eval scripts | scripts/eval-classification.ts, scripts/eval-summarisation.ts, scripts/eval-bid-drafting.ts, scripts/eval-search.ts, scripts/eval-entity-classification.ts | | AI touchpoints (TS) | lib/ai/classify.ts, lib/ai/summarise.ts, lib/ai/digest.ts, lib/ai/draft.ts, lib/ai/quality-check.ts, lib/ai/extract-questions.ts, lib/ai/extract-content.ts, lib/ai/vision.ts | | Pricing constants | lib/ai/pricing.ts | | MCP tools | lib/mcp/tools/*.ts (11 category files, 41 tools total) |

Open follow-ups (not part of W6 scope):

Plugin skill prompt audit — audit .claude/plugins/knowledge-hub/1.0.0/skills/*/SKILL.md for clarity and error-handling hints using the same Claude-as-judge rubric as W6. Demoted from W6 bonus task 5.6e per S155 WP1.4 F22. Track as its own task in a future session.
ai_call_costs retention policy (W5 F25) — decide whether rows older than 365 days are archived or deleted. Assigned to next admin-tooling housekeeping session.

12. Open questions (added per S155 WP1.4 F27)

The original spec did not include a dedicated Open questions section. This section consolidates the embedded ambiguities plus the questions raised by the S155 adversarial review.

Q1 — W1 split acceptance (from F9). W1-pre ships in Phase 5 (rubric schema + synthetic-fixture evaluator). W1-post is gated on §1.4 real bid data. Is the W1-pre deliverable sufficient to close the Phase 5 workstream, or does Phase 5 acceptance require W1-post to exist as a tracked follow-up? Default: W1-pre is sufficient for Phase 5; W1-post is tracked separately under §1.4.

Q2 — W4 Structured Outputs scope (from F8). The revised §3.3 policy adopts Structured Outputs as default for new touchpoints. Is opportunistic migration of existing tool_use (forced) call sites in-scope for W4, or is W4 limited to a reference migration? Default: reference migration only (lib/ai/extract-content.ts per current §5.4); other migrations are opportunistic and not Phase 5 scope.

Q3 — Digest variance bands (from F16). Each new digest metric needs a variance_band default. LLM-judge metrics default to ±5%, deterministic metrics to ±2%. Should the W3 gold standard first-run explicitly measure the actual variance before committing the band, or do we ship the defaults and revise after the first nightly run? Default: ship defaults, revise after first week of nightly runs.

Q4 — W6 Claude-as-judge vs external evaluator (from F1). The revised W6 uses a hand-rolled Claude-as-judge rubric rather than the non-existent tool-design skill. If claude-plugins-official later ships a ToolDescriptionEvaluator with compatible dimensions, should W6 switch to it? Default: stay with the Claude-as-judge rubric; switching would require a separate migration spec.

Q5 — ai_call_costs retention (from F25). 365-day default retention is proposed in §11 follow-ups. Is that long enough for month-over-month billing reconciliation, or should it be 730 days? Default: 365; revisit after 6 months of production data.

Q6 — Admin dashboard auth scope (from F13). The /admin/ai-costs dashboard is admin-only per the spec. Per-workspace cost views could also be useful for editors auditing their own workspace’s spend. Should the dashboard expose per-workspace views to editors (gated by getAuthorisedClient(['admin', 'editor'])), or stay strictly admin-only? Default: strictly admin-only for Phase 5; a per-workspace view is a follow-up feature after admin dogfooding validates the data.

Q7 — W2 ROUGE-L headroom (from F26). Current summarisation.baseline.json metrics (0.361/0.254) already exceed the proposed floor (0.30/0.25). W2 is effectively a no-quality-change refactor under the proposed thresholds. If the user wants genuine improvement headroom, W2 should target 0.40/0.35 — but that requires actual quality work beyond the skill conversion. Default: ship W2 as refactor; revisit headroom after skill conversion proves stable.

Appendix A: Touchpoints without eval coverage at the start of Phase 5

Touchpoint	File	Current eval?	Phase 5 workstream
Bid drafting Pass 1 (analysis)	`lib/ai/draft.ts`	No	W1 (indirect)
Bid drafting Pass 2 (drafting)	`lib/ai/draft.ts`	Partial (structural only)	W1
Bid drafting Pass 3 (quality check)	`lib/ai/quality-check.ts`	No	W1 (merged)
Summarisation (TS)	`lib/ai/summarise.ts`	Yes (weak thresholds)	W2
Summarisation (Python)	`scripts/kb_pipeline/summarise.py`	Yes (shared with TS)	W2
Digest generation	`lib/ai/digest.ts`	No	W3
Question extraction (PDF)	`lib/ai/extract-questions.ts`	No	Not in scope (low-stakes)
Question extraction (DOCX)	`lib/ai/extract-questions.ts`	No	Not in scope (low-stakes)
Tender metadata extraction	`lib/ai/extract-questions.ts`	No	Not in scope (low-stakes)
Search query generation	`lib/ai/extract-questions.ts`	No	Not in scope (low-stakes)
Vision/PDF analysis	`lib/ai/vision.ts`	No	Not in scope (user-customisable)
Structured extraction	`lib/ai/extract-content.ts`	No	Not in scope (schema-driven)
MCP tool descriptions (41 tools)	`lib/mcp/tools/*.ts`	Partial (L4 functional correctness)	W6
MCP prompt templates (5 prompts)	`lib/mcp/resources.ts`	No	Not in scope
Relevance scoring	`lib/intelligence/relevance-scorer.ts`	Partial (pass rate only)	Phase 4 (refinement loop)

Appendix B: W5 cost tracking coverage matrix (REVISED per S155 WP1.4 F17)

The original matrix omitted three call sites in lib/intelligence/. All three are added below. Also: the “claude-sonnet-4-6 missing from pricing constants” claim in the summary was wrong — the model is present at lib/ai/pricing.ts:32-37 per F5. Summary corrected below.

Touchpoint	File	Current logging	Pricing module?	Post-W5 state
Classification (TS)	`lib/ai/classify.ts`	Usage logged	Imported	Wrapped + persisted
Classification (Python)	`scripts/kb_pipeline/classify.py`	Usage logged	Imported	Wrapped + persisted
Summarisation (TS)	`lib/ai/summarise.ts`	Usage logged	NOT imported	Wrapped + persisted
Summarisation (Python)	`scripts/kb_pipeline/summarise.py`	Usage logged	—	Wrapped + persisted
Digest	`lib/ai/digest.ts`	Usage logged	NOT imported	Wrapped + persisted
Bid drafting Pass 1	`lib/ai/draft.ts`	Usage logged	NOT imported	Wrapped + persisted
Bid drafting Pass 2	`lib/ai/draft.ts`	Usage logged	NOT imported	Wrapped + persisted
Quality check	`lib/ai/quality-check.ts`	Usage logged	NOT imported	Wrapped + persisted
Question extraction	`lib/ai/extract-questions.ts`	Partial	NOT imported	Wrapped + persisted
Extract content	`lib/ai/extract-content.ts`	Usage logged	Imported	Wrapped + persisted
Vision	`lib/ai/vision.ts`	Usage logged	NOT imported	Wrapped + persisted
Relevance scoring	`lib/intelligence/relevance-scorer.ts`	Partial	NOT imported	Wrapped + persisted
Article summariser (NEW per F17)	`lib/intelligence/article-summariser.ts`	Usage logged	NOT imported	Wrapped + persisted
Guide generator (NEW per F17)	`lib/intelligence/guide-generator.ts`	Usage logged	NOT imported	Wrapped + persisted
Feed poller (NEW per F17)	`lib/intelligence/feed-poller.ts`	Usage logged (where it calls Claude)	NOT imported	Wrapped + persisted
Flag analyser (NEW, S154 WP2 Phase 1b)	`lib/intelligence/flag-analyser.ts`	N/A (new in S155)	NOT yet	Wrapped + persisted
Flag analysis (Phase 4)	`lib/refinement/proposal-generator.ts`	N/A (new)	NOT yet	Wrapped + persisted

Gap to close: 12 touchpoints (updated from 9) do not currently import the pricing module. claude-sonnet-4-6 IS present at lib/ai/pricing.ts:32-37 — the original spec’s “missing” claim was stale per F5. W5’s work is to wrap each call site in recordAiCall(), import lib/ai/pricing.ts where needed, and persist the cost.

Enforcement test (F6 + F17 together): unit test at __tests__/lib/ai/cost-tracking-coverage.test.ts asserts every grep match for anthropic.messages.create in lib/ai/**, lib/intelligence/**, and app/api/** has a matching recordAiCall call site within 10 lines of the same function. This guarantees future call sites don’t silently drop out of coverage.

ai_call_costs schema note (per F12): the schema includes a correlation_id TEXT NULL column so §9.7 structured logging can retroactively join cost rows to request-level timelines without a backfill migration.