MCP Server Evaluation Infrastructure Spec

MCP Server Evaluation Infrastructure Spec

Date: 11 March 2026 (initial); Last refreshed: 28 April 2026 (kh-prod-readiness-S10 / WP-G4.4). Status: Layer 1 + Layer 3 + Layer 4 implemented; CI integration shipped under WP-G4.4. Layer 2 superseded by Layer 4. Layer 5 (separate .github/workflows/mcp-eval.yml for nightly + dispatch) deferred. Depends on: docs/reference/mcp-server-improvements.md (investigation findings, S82).

Implementation note (S90, ~S160, ~S180, S10): Three layers shipped under scripts/mcp-eval/:

• Layer 1 — Protocol Compliance (protocol-compliance.ts, ~786 lines, 42 checks). Run: bun run test:mcp-eval
• Layer 3 — Response Quality (response-quality.ts, ~1127 lines, 17 checks). Run: bun run test:mcp-eval:rq
• Layer 4 — Functional Correctness (functional-correctness.ts, ~3107 lines, 37 checks; live DB). Run: bun run test:mcp-eval:fc

Canonical tool list lives in scripts/mcp-eval/fixtures.ts (kept current). Counts shown in §3-§5 below are as-written from S82 — current state per docs/generated/mcp-inventory.md: 57 tools, 12 resources (3 templates + 5 static + 4 app), 7 prompts, 34 subtopics.

CI integration (WP-G4.4, S10 Wave 2): all three layers wired into .github/workflows/ci.yml as a Staging-scoped matrix job (mcp-eval (l1|l3|l4)), running in parallel with quality. Skip-flag policy per spec at docs/audits/kh-production-readiness-phase-1/specs/wp-g4.4-mcp-eval-ci-spec.md §3.2: default --skip-ai/--skip-search for cost-zero PR runs; full mode on push-to-main and on PRs touching MCP paths (lib/mcp/**, app/api/mcp/**, scripts/mcp-eval/**, lib/ai/**, lib/validation/schemas.ts, package.json).

---

1. Problem Statement

The Knowledge Hub MCP server exposes 23 tools, 9 resources, and 5 prompts to Claude Desktop, Claude.ai, and the Claude Code plugin. There are 191 unit tests across 7 test files covering formatters, app triggers, data contracts, and UI logic. However:

• No protocol-level testing — nothing verifies that a real MCP client can connect, list tools, call them, and receive valid responses.
• No functional correctness testing against the live database — all tests use createMockSupabaseClient(), so broken SQL, missing RPC functions, or RLS violations are invisible until manual testing.
• No response quality evaluation — there is no systematic way to measure whether tool responses are useful to Claude (concise, well-ranked, relevant).
• Stale search evaluation suite — scripts/search-evaluation.json references “7 domains, 30 subtopics” but the taxonomy now has 7 domains and 34 subtopics (4 added in S80: health-and-safety, environmental, modern-slavery, supply-chain).

The result is that regressions are caught late (e.g., S77 discovered search_for_bid_response had been broken since creation due to a missing extensions search_path). A layered evaluation infrastructure would catch these issues automatically.

---

2. Current Test Infrastructure

2.1 MCP Unit Tests (7 files, 191 tests)

File	Tests	What it covers
mcp-app-formatters.test.ts	37	Markdown formatter output for app data
mcp-app-trigger-tools.test.ts	28	App trigger tool registration and responses
mcp-app-contracts.test.ts	17	Data contracts between server tools and MCP Apps
mcp-app-ui-logic.test.ts	41	App-side rendering and interaction logic
mcp-new-tools.test.ts	28	Tools #17-21 (coverage gaps, audit, update, similar, batch)
mcp-tools-entity.test.ts	15	Entity relationship tool (#14)
mcp-entity-formatters.test.ts	25	Entity and citation formatter output

Strengths: Good coverage of formatter correctness and data contracts. All tests use a shared createMockSupabaseClient() from __tests__/helpers/mock-supabase.ts.

Gaps: Tools #1-13 and #15-16 have no dedicated test files (partially covered via formatter tests). No tests exercise real MCP protocol flow. No tests hit a real database.

2.2 Search Evaluation Suite

• File: scripts/search-evaluation.json (20 test cases)
• Guide: docs/reference/search-evaluation-guide.md
• Runner: Manual via Playwright MCP, API curl, or browser
• Categories: exact-topic (3), conceptual (3), cross-domain (4), technical (2), broad-theme (2), negative (2), keyword-overlap (2), content-type-specific (2)
• Problem: References 30 subtopics; taxonomy now has 34. No test cases for the 4 new compliance subtopics. No automated runner script.

2.3 MCP Server Endpoint

The MCP server is exposed at /api/mcp/[transport]/route.ts using the MCP SDK’s WebStandardStreamableHTTPServerTransport (not mcp-handler — see CLAUDE.md gotchas). Each request creates a fresh server + transport instance to avoid Vercel warm instance corruption.

2.4 Existing Supabase Connection

• Project: rovrymhhffssilaftdwd (eu-west-2 London)
• MCP connection: Already configured for Claude Desktop
• 186 content items (173 Q&A pairs), 999 entities, 913 relationships
• No separate test database — evaluation runs against the live instance

---

3. Layer 1: Protocol Compliance

Priority: Implement first. Catches structural regressions (missing tools, broken schemas, malformed responses) that unit tests cannot detect.

3.1 Approach

Use the MCP TypeScript SDK client (@modelcontextprotocol/sdk/client) to connect to the MCP server endpoint programmatically and exercise the protocol. This is automatable in CI, unlike mcp-inspector which is interactive.

The mcp-inspector CLI/UI remains useful for development and manual exploration, but the automated suite is the primary deliverable.

3.2 Test Harness

Create scripts/mcp-eval/protocol-compliance.ts:

1. Start the dev server (or use a running instance)
2. Create an MCP client via the SDK
3. Connect to http://localhost:3000/api/mcp/streamable-http
4. Authenticate with a test user token (service role for simplicity)
5. Run the protocol test suite
6. Report pass/fail per check

3.3 Protocol Checks

As-implemented count overrides: PC-01 expects 57 tools (was 23); PC-05 expects 9 resources (5 static + 4 app — was “at least 7 static”); PC-06 expects 3 templates as listed; PC-07 expects 7 prompts (was 5). Source of truth: scripts/mcp-eval/fixtures.ts constants TOOL_COUNT, STATIC_RESOURCE_URIS, RESOURCE_TEMPLATE_URIS, PROMPT_COUNT.

3.3.1 Discovery

Check	Method	Pass criteria
PC-01	tools/list	Returns exactly 23 tools
PC-02	tools/list	Every tool has name, description, inputSchema
PC-03	tools/list	Every tool name matches the canonical list
PC-04	tools/list	Every tool has annotation hints (readOnlyHint, idempotentHint, openWorldHint)
PC-05	resources/list	Returns resources (at least the 7 static URI resources)
PC-06	resources/templates/list	Returns resource templates (kb://items/{id}, kb://bids/{id}, kb://qa/{id})
PC-07	prompts/list	Returns exactly 5 prompts
PC-08	prompts/list	Every prompt has name, description

3.3.2 Tool Call Structure

For each of the 23 tools, call with minimal valid arguments and verify:

Check	Pass criteria
PC-10	Response has content array
PC-11	Each content block has type (text) and text (string)
PC-12	No tool throws an unhandled exception (all return structured error text)
PC-13	Read-only tools (18) return results without side effects
PC-14	Tools with required params reject calls with missing params (error, not crash)

Minimal valid arguments per tool:

#	Tool	Minimal args
1	search_knowledge_base	{query: "test"}
2	get_dashboard_summary	{} (no args)
3	list_active_bids	{}
4	get_content_item	{id: "<known_item_uuid>"}
5	get_reorientation	{}
6	get_bid_detail	{id: "<known_bid_uuid>"}
7	get_bid_question	{question_id: "<known_question_uuid>"}
8	get_quality_summary	{}
9	get_freshness_report	{}
10	classify_content	{item_id: "<known_item_uuid>"}
11	generate_summary	{item_id: "<known_item_uuid>"}
12	create_content_item	{title: "Eval test", content: "Test content", content_type: "note"}
13	search_qa_library	{query: "test"}
14	get_entity_relationships	{entity_type: "certification"}
15	cite_content	{content_item_id: "<uuid>", bid_response_id: "<uuid>"}
16	get_content_effectiveness	{content_item_id: "<known_item_uuid>"}
17	get_coverage_gaps	{}
18	audit_content	{}
19	update_content_item	{id: "<known_item_uuid>", fields: {notes: "eval test"}}
20	find_similar_items	{id: "<known_item_uuid>"}
21	get_content_items	{ids: ["<known_item_uuid>"]}
22	show_coverage_matrix	{}
23	show_bid_dashboard	{}

Write tool safety: Tools #10, #11, #12, #15, and #19 perform writes. The test harness must:

• Use a dedicated eval content item (created at suite start, deleted at end)
• For create_content_item: delete the created item after the test
• For classify_content and generate_summary: use the eval item
• For update_content_item: restore original values after the test
• For cite_content: use the eval item as both content and bid response IDs (will fail validation but should return a structured error, not crash)

3.3.3 Error Handling

Check	Method	Pass criteria
PC-20	tools/call with invalid UUID	Returns error text, not protocol error
PC-21	tools/call with missing required param	Returns validation error
PC-22	tools/call with wrong param type	Returns validation error
PC-23	resources/read with non-existent ID	Returns “not found” text

3.4 Output Format

MCP Protocol Compliance Report
==============================
Date: 2026-03-11T14:30:00Z
Server: http://localhost:3000/api/mcp/streamable-http

Discovery
  PC-01 tools/list count ............... PASS (23 tools)
  PC-02 tool schema completeness ....... PASS
  PC-03 tool name validation ........... PASS
  ...

Tool Calls (23 tools)
  search_knowledge_base ................ PASS (response: 1 content block, 847 chars)
  get_dashboard_summary ................ PASS (response: 1 content block, 2,341 chars)
  ...

Error Handling
  PC-20 invalid UUID ................... PASS
  PC-21 missing required param ......... PASS
  ...

Summary: 52/52 passed, 0 failed, 0 skipped

3.5 CI Integration

Add a GitHub Actions workflow step that:

1. Starts the dev server in background
2. Waits for the server to be ready (health check on /api/mcp/streamable-http)
3. Runs bun run scripts/mcp-eval/protocol-compliance.ts
4. Fails the build if any check fails

Requires: SUPABASE_SECRET_KEY, NEXT_PUBLIC_SUPABASE_URL, NEXT_PUBLIC_SUPABASE_ANON_KEY as GitHub Actions secrets (already needed for existing test infrastructure).

---

4. Layer 2: Functional Correctness

SUPERSEDED: The “Layer 2 functional correctness” of this design was implemented as Layer 4 (scripts/mcp-eval/functional-correctness.ts, 37 checks). Layer numbering drifted between spec (S82) and implementation (~S180); the scope is identical. See §1 Implementation note above for current run instructions. Section retained for historical context only.

Priority: Implement after Layer 1. Catches data-level regressions (broken RPCs, incorrect SQL, missing search_path settings, RLS violations).

4.1 Approach

Input/output pairs for each tool, executed against the live Supabase instance. Each test case specifies the tool name, arguments, and expected properties of the response. This is a superset of Layer 1 — it verifies not just that a response is structurally valid, but that the data is correct.

4.2 Test Data Strategy

Use the existing Supabase instance (rovrymhhffssilaftdwd) rather than provisioning a separate test database. Rationale:

• The KB has 186 real items with embeddings, classifications, and entities — this is the data that matters for correctness testing
• A separate test DB would require maintaining parallel seed data
• Write tool tests create and clean up their own data (eval item pattern from Layer 1)
• The Supabase project is single-tenant (one client) so there is no risk of affecting other users’ data

Eval fixtures: Create scripts/mcp-eval/fixtures.ts with known UUIDs and expected values, refreshed periodically by querying the live database.

4.3 Functional Test Cases

4.3.1 Search Tools (3 tools)

ID	Tool	Input	Expected
FC-01	search_knowledge_base	{query: "ISO 27001"}	>= 3 results, top result domain = security
FC-02	search_knowledge_base	{query: "ISO 27001", domain: "security"}	All results have domain security
FC-03	search_knowledge_base	{query: "quantum blockchain"}	0-3 results (negative test)
FC-04	search_knowledge_base	{query: "data protection GDPR"}	>= 3 results, subtopic includes data-protection
FC-05	search_qa_library	{query: "SLA response times"}	>= 2 results, all content_type = q_a_pair
FC-06	search_qa_library	{query: "SLA", limit: 3}	Exactly 3 results
FC-07	find_similar_items	{id: "<known_item>"}	Results sorted by similarity descending

4.3.2 Dashboard and Summary Tools (4 tools)

ID	Tool	Input	Expected
FC-10	get_dashboard_summary	{}	Response contains freshness counts, total items > 0
FC-11	get_quality_summary	{}	Response contains issue type breakdown
FC-12	get_freshness_report	{}	Response contains fresh/aging/stale/expired counts summing to total
FC-13	get_reorientation	{}	Response contains “attention items” section

4.3.3 Content Retrieval Tools (3 tools)

ID	Tool	Input	Expected
FC-20	get_content_item	{id: "<known_qa_item>"}	Returns title, domain, content_type = q_a_pair
FC-21	get_content_item	{id: "<nonexistent_uuid>"}	Returns “not found” error text
FC-22	get_content_items	{ids: ["<id1>", "<id2>"]}	Returns exactly 2 items

4.3.4 Bid Tools (3 tools)

ID	Tool	Input	Expected
FC-30	list_active_bids	{}	Returns array (may be empty), each with name and status
FC-31	get_bid_detail	{id: "<known_bid>"}	Returns bid with questions array
FC-32	get_bid_question	{question_id: "<known_q>"}	Returns question_text and status

4.3.5 Coverage and Quality Tools (3 tools)

ID	Tool	Input	Expected
FC-40	get_coverage_gaps	{}	Returns gaps array, each with domain and subtopic
FC-41	get_coverage_gaps	{min_items: 100}	Returns more gaps than default (lower bar = more gaps)
FC-42	audit_content	{issue_type: "no_domain"}	All returned items have primary_domain = null
FC-43	audit_content	{issue_type: "thin_content"}	All returned items have content length < 20

4.3.6 Entity Tools (2 tools)

ID	Tool	Input	Expected
FC-50	get_entity_relationships	{entity_type: "certification"}	Returns entities with type certification
FC-51	get_content_effectiveness	{content_item_id: "<known_item>"}	Returns citation_count (number) and win_rate

4.3.7 Write Tools (5 tools)

These tests create side effects and must clean up after themselves.

ID	Tool	Input	Expected	Cleanup
FC-60	create_content_item	{title: "Eval FC-60", content: "Test", content_type: "note"}	Returns created item with UUID	Delete item
FC-61	classify_content	{item_id: "<eval_item>", force: true}	Returns classification with domain	None (eval item)
FC-62	generate_summary	{item_id: "<eval_item>", force: true}	Returns summary text	None (eval item)
FC-63	update_content_item	{id: "<eval_item>", fields: {notes: "FC-63"}}	Returns updated item	Restore original notes
FC-64	cite_content	{content_item_id: "<eval_item>", bid_response_id: "<known_response>"}	Returns citation record	Delete citation

4.3.8 App Trigger Tools (2 tools)

ID	Tool	Input	Expected
FC-70	show_coverage_matrix	{}	Response contains HTML or structured app data
FC-71	show_bid_dashboard	{}	Response contains HTML or structured app data

4.4 Output Format

Same structured report as Layer 1 but with data assertions:

Functional Correctness Report
==============================

Search Tools
  FC-01 search ISO 27001 .............. PASS (7 results, top domain: security)
  FC-02 search ISO 27001 filtered ..... PASS (5 results, all security)
  FC-03 search quantum blockchain ..... PASS (0 results)
  ...

Summary: 28/28 passed, 0 failed

4.5 CI Integration

Runs as a separate step after Layer 1, using the same dev server instance. Layer 2 takes longer (real DB calls) so it should run on main branch merges rather than every PR, or as a nightly scheduled workflow.

Cost consideration: classify_content and generate_summary call the Claude API. In CI, these can be skipped (or run with a --skip-ai flag) to avoid API costs. The structural test (does the tool return a valid response?) is Layer 1’s job; Layer 2 verifies the data shape and database side effects.

---

5. Layer 3: Response Quality

Priority: Implement after Layer 2. Evaluates whether tool responses are optimised for LLM consumption — concise, well-structured, and useful.

5.1 Approach

Response quality evaluation measures three dimensions:

1. Token efficiency — Are responses concise enough for model context windows? (Character count and estimated token count per response.)
2. Search relevance — Does search_knowledge_base rank relevant results highly? (Extends the existing search evaluation framework.)
3. Structural clarity — Are Markdown responses well-structured with headers, lists, and clear hierarchy?

5.2 Token Efficiency Checks

For each tool, establish baseline response sizes and flag outliers:

Tool	Expected range	Flag if
search_knowledge_base (10 results)	2,000-5,000 chars	> 8,000 chars
get_dashboard_summary	1,500-4,000 chars	> 6,000 chars
get_content_item	500-3,000 chars	> 5,000 chars
get_bid_detail	1,000-8,000 chars	> 12,000 chars
get_reorientation	2,000-6,000 chars	> 10,000 chars
get_coverage_gaps	1,000-4,000 chars	> 8,000 chars
audit_content	1,000-5,000 chars	> 8,000 chars

The existing CHARACTER_LIMIT and truncateResponse() in formatters.ts already enforce a ceiling, but these checks verify that typical responses are well within it.

5.3 Search Relevance Evaluation

Build on the existing search evaluation framework (scripts/search-evaluation.json) to evaluate MCP tool search quality specifically:

ID	Query	Via tool	Expected
RQ-01	”data protection GDPR policies”	search_knowledge_base	Top 3 results contain data-protection subtopic
RQ-02	”ISO 27001 certification ISMS”	search_knowledge_base	Results span security and compliance domains
RQ-03	”SLA response times”	search_qa_library	Top result is an SLA Q&A pair
RQ-04	”staff qualifications CVs”	search_knowledge_base	Results in corporate/staffing
RQ-05	”quantum computing blockchain”	search_knowledge_base	0-3 results, all low relevance

These mirror the search evaluation test cases but execute via MCP tool calls rather than the REST API, verifying the MCP layer does not degrade search quality.

5.4 Structural Quality Checks

Automated checks on response Markdown:

Check	Pass criteria
RQ-10	Dashboard summary has section headers (## or bold)
RQ-11	Search results use consistent item formatting
RQ-12	Bid detail includes progress indicators (fractions or percentages)
RQ-13	Coverage gaps are grouped by domain
RQ-14	Entity relationships are structured as a list, not prose

5.5 Output Format

Response Quality Report
=======================

Token Efficiency
  search_knowledge_base ............... PASS (3,412 chars, est. 850 tokens)
  get_dashboard_summary ............... WARN (5,891 chars — above typical range)
  ...

Search Relevance (5 test cases)
  RQ-01 GDPR data protection ......... PASS (3/3 top results correct)
  RQ-02 ISO 27001 .................... PASS (both domains represented)
  ...

Structural Quality
  RQ-10 dashboard headers ............. PASS
  ...

Summary: 18/20 passed, 1 warning, 1 failed

---

6. Search Evaluation Refresh Plan

The search evaluation suite (scripts/search-evaluation.json and docs/reference/search-evaluation-guide.md) needs updating to reflect the current taxonomy.

6.1 Taxonomy Update

The evaluation metadata references 30 subtopics. The current taxonomy has 34 subtopics. The 4 additions (all in the compliance domain, added in S80):

Subtopic	Description	Source
health-and-safety	H&S policy, risk assessments, RIDDOR, CDM	S80 taxonomy expansion
environmental	Carbon reduction, net zero, ISO 14001, PPN 06/20	S80 taxonomy expansion
modern-slavery	Modern slavery statement, supply chain due diligence, PPN 02/23	S80 taxonomy expansion
supply-chain	Supply chain management, prompt payment, subcontractor oversight	S80 (added to corporate)

Note: supply-chain was added under the corporate domain, not compliance. The search evaluation guide’s taxonomy table needs updating for both domains.

6.2 Changes to search-evaluation.json

1. Update metadata description: Change “30 subtopics” to “34 subtopics”
2. Update taxonomy.domains: No change (still 7 domains)
3. Add 4 new test cases (SE-21 through SE-24):

ID	Category	Query	Expected domains	Expected subtopics
SE-21	exact-topic	”health and safety risk assessment CDM regulations”	compliance	health-and-safety
SE-22	exact-topic	”carbon reduction plan net zero environmental policy ISO 14001”	compliance	environmental
SE-23	exact-topic	”modern slavery statement supply chain due diligence”	compliance	modern-slavery
SE-24	cross-domain	”supply chain management subcontractor oversight prompt payment”	corporate	supply-chain

6.3 Changes to search-evaluation-guide.md

1. Update taxonomy table: Add the 4 new subtopics to the correct domains
   • compliance: add health-and-safety, environmental, modern-slavery
   • corporate: add supply-chain
2. Update domain coverage table: Add SE-21, SE-22, SE-23 to compliance column; SE-24 to corporate column
3. Update test case count: “20 test cases” becomes “24 test cases”
4. Update category table: exact-topic goes from 3 to 6; cross-domain from 4 to 5

6.4 Review of Existing 20 Test Cases

Review each existing test case for accuracy given the expanded taxonomy:

• SE-16 (Cyber Essentials): Currently expects compliance/certification and security/cyber-security. Still valid.
• SE-09 (staff qualifications): Currently expects corporate/staffing. The new supply-chain subtopic should not affect this. Still valid.
• SE-17 (professional indemnity insurance): Currently expects corporate/insurance. Still valid.
• All other test cases: Unaffected by the new subtopics.

No changes needed to existing test cases. The new subtopics are additive and do not alter existing classification boundaries.

6.5 Evaluation Runner Script

Currently there is no automated runner. The guide describes three manual options (Playwright MCP, API curl, browser). As part of this work, create an automated runner:

File: scripts/mcp-eval/search-evaluation-runner.ts

1. Read scripts/search-evaluation.json
2. For each test case:
   a. Call POST /api/search with the query
   b. Compare results against expectations
   c. Record pass/fail per criterion
3. Output structured report (same format as Layer 1/2/3)
4. Exit with non-zero code if any critical/major failures

This runner can be invoked standalone or as part of the Layer 3 response quality evaluation.

---

7. CI Integration Plan

SUPERSEDED by WP-G4.4 (28 April 2026): The CI integration shipped under docs/audits/kh-production-readiness-phase-1/specs/wp-g4.4-mcp-eval-ci-spec.md uses an in-ci.yml matrix job (mcp-eval (l1|l3|l4)) scoped to the Staging GitHub environment, not a separate .github/workflows/mcp-eval.yml as originally proposed below. Skip-flag policy: cost-zero --skip-ai/ --skip-search on default PR runs; full mode on push-to-main and on PRs touching MCP paths (paths-filter). Section retained for historical context only.

7.1 Workflow Structure

.github/workflows/mcp-eval.yml

name: MCP Evaluation
on:
  push:
    branches: [main]
    paths:
      - 'lib/mcp/**'
      - 'app/api/mcp/**'
      - 'scripts/mcp-eval/**'
  schedule:
    - cron: '0 6 * * 1' # Weekly Monday 06:00 UTC

7.2 Jobs

Job	Trigger	Duration	Secrets needed
Layer 1: Protocol compliance	Every push to lib/mcp/**	~2 min	Supabase URL + keys
Layer 2: Functional correctness	Weekly + main merges	~5 min	Supabase URL + keys + service role
Layer 3: Response quality	Weekly scheduled	~3 min	Supabase URL + keys
Search evaluation	Weekly scheduled	~2 min	Supabase URL + keys

7.3 Prerequisites

• GitHub Actions secrets: SUPABASE_SECRET_KEY, NEXT_PUBLIC_SUPABASE_URL, NEXT_PUBLIC_SUPABASE_ANON_KEY, OPENAI_API_KEY (for search embedding)
• Dev server startup in CI (Next.js build + start, or dev mode)
• The --skip-ai flag for Layer 2 to avoid Claude API costs on every run

7.4 Failure Handling

• Layer 1 failure: Blocks the merge. Protocol compliance is a hard gate.
• Layer 2 failure: Posts a comment on the PR but does not block. Functional correctness failures may be caused by database state changes.
• Layer 3 failure: Informational only. Response quality regressions are tracked over time but do not block merges.
• Search evaluation failure: Posts a summary to a Slack webhook or GitHub issue (future, not in scope for initial implementation).

---

8. Implementation Phases and Effort

Phase 1: Layer 1 — Protocol Compliance (4-6 hours)

Task	Effort
Create scripts/mcp-eval/ directory structure	15 min
Implement MCP client connection and auth	1 hour
Implement discovery checks (PC-01 through PC-08)	1 hour
Implement tool call checks for all 23 tools	1.5 hours
Implement error handling checks (PC-20 through PC-23)	30 min
Report formatting and exit codes	30 min
Test and debug against live server	30 min

Deliverables:

• scripts/mcp-eval/protocol-compliance.ts
• scripts/mcp-eval/constants.ts (tool names, resource URIs, prompt names)
• Run command: bun run scripts/mcp-eval/protocol-compliance.ts

Phase 2: Search Evaluation Refresh (2-3 hours)

Task	Effort
Update scripts/search-evaluation.json metadata and add SE-21 to SE-24	30 min
Update docs/reference/search-evaluation-guide.md	30 min
Create scripts/mcp-eval/search-evaluation-runner.ts	1 hour
Verify all 24 test cases against live database	30 min

Deliverables:

• Updated scripts/search-evaluation.json (24 test cases)
• Updated docs/reference/search-evaluation-guide.md
• scripts/mcp-eval/search-evaluation-runner.ts

Phase 3: Layer 2 — Functional Correctness (6-8 hours)

Task	Effort
Create fixtures file with known UUIDs and expected values	1 hour
Implement search tool test cases (FC-01 to FC-07)	1.5 hours
Implement dashboard/summary tool test cases (FC-10 to FC-13)	1 hour
Implement content retrieval test cases (FC-20 to FC-22)	30 min
Implement bid tool test cases (FC-30 to FC-32)	30 min
Implement coverage/quality test cases (FC-40 to FC-43)	30 min
Implement entity tool test cases (FC-50 to FC-51)	30 min
Implement write tool test cases with cleanup (FC-60 to FC-64)	1.5 hours
Implement app trigger test cases (FC-70 to FC-71)	15 min
Add --skip-ai flag for CI	15 min

Deliverables:

• scripts/mcp-eval/functional-correctness.ts
• scripts/mcp-eval/fixtures.ts

Phase 4: Layer 3 — Response Quality (3-4 hours)

Task	Effort
Implement token efficiency checks	1 hour
Implement search relevance checks (RQ-01 to RQ-05)	1 hour
Implement structural quality checks (RQ-10 to RQ-14)	45 min
Integrate with search evaluation runner	30 min

Deliverables:

• scripts/mcp-eval/response-quality.ts

Phase 5: CI Integration (2-3 hours)

Task	Effort
Create .github/workflows/mcp-eval.yml	45 min
Configure GitHub Actions secrets	15 min
Dev server startup/shutdown in CI	45 min
Test workflow on a feature branch	30 min
Add bun run eval:mcp convenience script to package.json	15 min

Deliverables:

• .github/workflows/mcp-eval.yml
• package.json scripts: eval:mcp, eval:mcp:protocol, eval:mcp:functional, eval:mcp:quality, eval:search

Total Effort

Phase	Effort	Cumulative
Phase 1: Protocol compliance	4-6 hours	4-6 hours
Phase 2: Search eval refresh	2-3 hours	6-9 hours
Phase 3: Functional correctness	6-8 hours	12-17 hours
Phase 4: Response quality	3-4 hours	15-21 hours
Phase 5: CI integration	2-3 hours	17-24 hours

Phases 1 and 2 are independent and can be done in the same session. Phases 3 and 4 depend on Phase 1 (reuse the client connection infrastructure). Phase 5 depends on all others.

---

9. Acceptance Criteria

Layer 1: Protocol Compliance

• MCP client connects to the server endpoint successfully
• tools/list returns exactly 23 tools, all with valid schemas
• resources/list returns resources including the 3 template URIs
• prompts/list returns exactly 5 prompts
• Every tool can be called with minimal valid args and returns a content array
• Error cases (invalid UUID, missing params, wrong types) return structured error text, not protocol-level errors or crashes
• Write tool tests clean up after themselves (no residual eval data)
• Script exits with code 0 on all-pass, non-zero on any failure
• Runs in under 60 seconds

Layer 2: Functional Correctness

• All 28+ test cases have defined expected values
• Search tools return relevant results for known queries
• Dashboard tools return non-empty data with correct structure
• Content retrieval returns correct items by UUID
• Coverage gaps correctly reflects the 34-subtopic taxonomy
• Audit tool filters work correctly (each issue_type returns matching items)
• Write tools create, modify, and clean up data without errors
• --skip-ai flag skips classify_content and generate_summary tests
• Script runs in under 120 seconds (with --skip-ai)

Layer 3: Response Quality

• Token efficiency baselines established for all 23 tools
• Search relevance checks pass for at least 4/5 test cases
• Structural quality checks pass for all 5 checks
• Report identifies any tools producing unusually large responses

Search Evaluation Refresh

• search-evaluation.json updated to reference 34 subtopics
• 4 new test cases added (SE-21 to SE-24) for the new subtopics
• search-evaluation-guide.md taxonomy table updated with all 34 subtopics
• Domain coverage table updated to reflect new test cases
• Automated runner script produces a pass/fail report
• All 24 test cases verified against live database

CI Integration

• GitHub Actions workflow triggers on MCP-related file changes
• Layer 1 runs on every push to MCP paths
• Layers 2-3 and search eval run on weekly schedule
• Layer 1 failure blocks the merge
• All layers produce structured output parseable by CI

---

File Structure

scripts/mcp-eval/
  protocol-compliance.ts       # Layer 1: protocol checks
  functional-correctness.ts    # Layer 2: input/output pairs
  response-quality.ts          # Layer 3: token efficiency + relevance
  search-evaluation-runner.ts  # Automated search eval runner
  fixtures.ts                  # Known UUIDs, expected values
  constants.ts                 # Tool names, resource URIs, prompt names
  utils.ts                     # Shared: MCP client creation, report formatting

---

References

• docs/reference/mcp-server-improvements.md — Investigation findings (S82)
• lib/mcp/tools.ts — 23 tool registrations
• lib/mcp/resources.ts — 9 resources, 5 prompts
• lib/mcp/formatters.ts — Response formatters
• app/api/mcp/[transport]/route.ts — MCP server endpoint
• scripts/search-evaluation.json — 20 search test cases (to be updated)
• docs/reference/search-evaluation-guide.md — Search evaluation guide
• __tests__/mcp-*.test.ts — 7 existing MCP unit test files
• docs/reference/classification-prompt.md — v4.1 taxonomy (34 subtopics)