Search Evaluation Guide

Search Evaluation Guide

This document describes how to run, interpret, and maintain the Knowledge Hub search evaluation test suite.

Overview

The search evaluation suite (scripts/search-evaluation.json) contains 24 test cases that exercise Knowledge Hub semantic search across different query types, domains, and content types. The suite is designed to detect regressions and validate improvements when changes are made to the search pipeline.

What it tests

• Semantic search quality — Do semantically related items appear in results?
• Domain routing — Do queries about data protection return security items, not unrelated domains?
• Cross-domain retrieval — Do broad queries correctly pull from multiple domains (e.g. implementation + security for “secure migration”)?
• Content type matching — Do queries surface the expected content types (q_a_pair, policy, case_study, etc.)?
• Negative cases — Do off-topic queries correctly return few or no results?
• Similarity threshold — Is the default threshold (0.35) appropriate?

Search architecture

The search flow is:

1. User enters query in the search bar at /search?q=...
2. Frontend calls POST /api/search with { query, threshold: 0.35, limit: 20 }
3. API generates an embedding via OpenAI text-embedding-3-large (1024 dims)
4. Supabase hybrid_search() RPC combines embedding similarity (70%), title match (15%), keyword match (10%), summary/author match (5%), and recency boost (5%) into a final score
5. Results returned sorted by combined score, filtered by threshold
6. Content snippets (200 chars around keyword match) returned alongside results

Taxonomy reference

The knowledge base uses 7 bid-management domains:

Domain	Subtopics
security	data-protection, cyber-security, encryption, access-control, iso-27001
compliance	standards, regulatory, audit, certification, health-and-safety, environmental, modern-slavery
implementation	deployment, migration, onboarding, integration
support	sla, helpdesk, maintenance, incident
corporate	company-info, financial, insurance, references, staffing, supply-chain
product-feature	functionality, technical, reporting, usability
methodology	approach, project-management, quality, delivery

Valid content types: article, blog, pdf, note, research, other, q_a_pair, case_study, policy, certification, compliance, methodology, capability, product_description.

Content is primarily Q&A pairs extracted from bid library documents, plus policies, case studies, certifications, and capability statements.

How to run the evaluation

Option A: Via Playwright MCP (recommended)

Use the Playwright MCP browser tools to run each test case through the live UI:

1. Start the dev server: bun dev --port 6100
2. For each test case in scripts/search-evaluation.json: a. Navigate to http://localhost:6100/search?q={encoded_query} b. Wait for results to load (skeleton disappears, result count appears) c. Take a snapshot of the results page d. Record: result count, visible titles, domain badges, content type icons e. Compare against expectations in the test case

Playwright MCP sequence per test case:

browser_navigate -> http://localhost:6100/search?q={query}
browser_wait_for -> text "results for" (or "No matches found")
browser_snapshot -> capture result state

Option B: Via API directly

Call the search API endpoint for each test case:

curl -X POST http://localhost:6100/api/search \
  -H "Content-Type: application/json" \
  -d '{"query": "data protection GDPR policies", "threshold": 0.25, "limit": 20}'

The response includes structured data:

{
  "results": [
    {
      "id": "uuid",
      "suggested_title": "...",
      "primary_domain": "security",
      "primary_subtopic": "data-protection",
      "content_type": "q_a_pair",
      "similarity": 0.432
    }
  ],
  "count": 12
}

Option C: Manual browser testing

1. Open http://localhost:6100 in a browser
2. Enter each query in the search bar
3. Visually inspect results against the test case expectations
4. Note which expected titles appear and at what position

How to interpret results

Pass criteria per test case

A test case passes if all of the following are met:

Criterion	Description
Min results	Result count >= min_results
Max results	Result count <= max_results (if specified)
Must-include titles	All titles in must_include_titles appear in results
Domain coverage	At least one result from each expected_domains
Content type spread	At least one result of each expected_content_types

Severity levels

• Critical failure: Must-include title missing from results (precision issue)
• Major failure: min_results not met (recall issue)
• Minor failure: Expected domain or content type not represented
• Warning: Negative test returning more than max_results

Scoring

For an overall health score:

Score = (passed_test_cases / total_test_cases) * 100

• 90-100%: Search is healthy
• 70-89%: Some regressions — investigate failed cases
• Below 70%: Significant search quality issue — likely a pipeline or threshold problem

Similarity score analysis

When using the API (Option B), also check similarity scores:

• Top result similarity > 0.4: Strong match
• Top result similarity 0.3-0.4: Good match
• Top result similarity 0.25-0.3: Borderline — may indicate the query is at the edge of the collection’s coverage
• All results < 0.3: Weak matches — the collection may lack content on this topic

Test case categories

Category	Count	Purpose
exact-topic	5	Specific bid topics with clear keyword matches (GDPR, ISO 27001, SLA, H&S, environmental)
conceptual	3	Semantic meaning beyond keywords (secure migration, BCP, agile delivery)
cross-domain	5	Queries spanning multiple taxonomy domains
technical	2	Precise technical terms (API integration, encryption)
broad-theme	2	Wide-ranging themes within a domain
negative	2	Off-topic queries that should return few results
keyword-overlap	3	Where keyword and semantic signals overlap
content-type-specific	2	Queries that should surface specific content types (case studies, product descriptions)

Domain coverage

All 7 bid domains are covered by at least 2 test cases. The “focus” column lists cases where the domain is the primary query intent; “also covers” lists cases where the domain appears in expected_domains as a secondary match. Array ordering in the JSON is not meaningful — focus is determined by query intent.

Domain	Focus test cases	Also covers
security	SE-01, SE-02, SE-08, SE-11	SE-04, SE-16
compliance	SE-16, SE-21, SE-22, SE-23	SE-02, SE-08
implementation	SE-04, SE-07, SE-13, SE-20	SE-10
support	SE-03, SE-05	—
corporate	SE-09, SE-12, SE-17, SE-18, SE-24	SE-23
product-feature	SE-10, SE-19	—
methodology	SE-06	SE-07, SE-20

When to re-run

Re-run the evaluation after any of the following changes:

• Embedding model change — Switching from text-embedding-3-large or changing dimensions
• Similarity threshold change — Adjusting the default 0.35 threshold
• Classification prompt update — New classification version may change domain/subtopic assignments
• Bulk re-embedding — After regenerating embeddings for existing items
• Significant content additions — After ingesting 100+ new items (shifts the distribution)
• Search RPC modification — Changes to hybrid_search() function logic
• Major content type additions — Adding a new content type
• Taxonomy changes — Adding, renaming, or restructuring domains or subtopics

How to add new test cases

1. Choose a category

Pick from the categories above, or create a new one if needed. Aim for balance — each category should have 2-3 test cases.

2. Query the database

Run SQL against Supabase to find real content that should match:

-- Find items about a topic
SELECT suggested_title, primary_domain, primary_subtopic, content_type
FROM content_items
WHERE suggested_title ILIKE '%your topic%'
ORDER BY captured_date DESC
LIMIT 20;

-- Check which domains/subtopics cover a topic
SELECT primary_domain, primary_subtopic, count(*)
FROM content_items
WHERE suggested_title ILIKE '%your topic%'
GROUP BY primary_domain, primary_subtopic
ORDER BY count DESC;

3. Define expectations

Use the query results to set realistic expectations:

• min_results: Set conservatively (usually 2-5) — better to catch regressions than generate false failures
• must_include_titles: Pick 1-3 titles that are the strongest matches. Use exact titles from the database. Leave empty if titles have not been verified against the live database.
• expected_domains: Include all domains that genuinely contain relevant content. Use lowercase domain names: security, compliance, implementation, support, corporate, product-feature, methodology
• expected_content_types: Use valid types: article, blog, pdf, note, research, other, q_a_pair, case_study, policy, certification, compliance, methodology, capability, product_description
• max_results: Only set for negative tests (typically 0-3)

4. Add to the JSON

Follow the existing format:

{
  "id": "SE-21",
  "category": "your-category",
  "query": "your search query",
  "expectations": {
    "min_results": 3,
    "expected_domains": ["security", "compliance"],
    "expected_subtopics": ["data-protection", "certification"],
    "expected_content_types": ["q_a_pair"],
    "must_include_titles": ["Exact Title From Database"],
    "notes": "Why this test matters and what it validates"
  }
}

5. Validate manually

Run the query once via the UI or API to confirm your expectations are reasonable before committing.

Populating must_include_titles

The current test cases have empty must_include_titles arrays because titles have not been verified against the live database. To populate them:

1. Run each test case query via the API or bun run scripts/kb-search.ts
2. Identify the top 1-3 results that should always appear
3. Copy exact titles from the database into must_include_titles
4. Re-run to confirm the titles appear consistently

This step is recommended after the knowledge base has been populated with bid-domain content and classifications have been applied.

Note: The search uses hybrid_search() which combines embedding similarity with keyword matching, so results may differ from pure semantic search. Test cases should be evaluated with this in mind.