Two-Pass Entity Validation Architecture

Two-Pass Entity Validation Architecture

Version: 1.0 Date: 05/04/2026 Status: Design document for Phase 2d Task 13 Implements: Task 14 (two-pass validation in lib/ai/classify.ts)

---

1. Overview

Entity precision in the classification pipeline currently sits at 43.6%. The dominant failure mode is false positives: the model extracts internal documents, generic concepts, job titles, protocols, and algorithms as though they were named entities. The existing deterministic post-extraction filters (shouldExcludeEntity and its constituent checks in lib/ai/classify.ts) catch many of these, but a significant residual false-positive rate remains because the filters only match known patterns — novel false positives slip through.

This document specifies a second validation pass that reviews Pass 1 entity extractions using the diagnostic questions from the entity type taxonomy. Pass 2 is an LLM call focused exclusively on entity quality — it does not re-do domain/subtopic classification, keyword extraction, or summarisation.

---

2. Pass 1 / Pass 2 Responsibilities

Pass 1 (existing, unchanged)

Function: classifyContent() in lib/ai/classify.ts Model: claude-sonnet-4-6 (configurable via AI_SUMMARY_MODEL) Responsibilities:

• Domain and subtopic classification (primary + secondary)
• Keyword extraction and normalisation
• AI summary and suggested title generation
• Classification confidence and reasoning
• Initial entity extraction (names, types, canonical names)
• Relationship extraction
• Temporal reference detection

Output: ClassificationResult containing all classification fields plus an initial entities array.

Pass 1 continues to operate exactly as it does today. No changes to its prompt, tool schema, or processing logic.

Pass 2 (new)

Function: validateEntities() (new function in lib/ai/classify.ts) Model: claude-haiku-4-5 (see Section 4 for rationale) Responsibilities:

• Review each Pass 1 entity against the Named Entity Test and External Reference Test
• Apply the five universal exclusion rules (named entity, external reference, policy/procedure/plan, role title, generic concept)
• Validate entity type assignments against per-type diagnostic questions
• Return a verdict for each entity: confirmed, removed, or retyped

Input:

Field	Source	Description
entities	Pass 1 output	Array of ExtractedEntity objects
content_excerpt	Same truncated text sent to Pass 1	First 2,000 characters of the plain text (reduced from Pass 1’s 5,000 chars — see Section 4)
content_title	Content item title	Title of the content item
content_type	Content item type	e.g. q_a_pair, policy, article

Output:

interface EntityValidationResult {
  validated_entities: ValidatedEntity[];
  removed_count: number;
  retyped_count: number;
  confirmed_count: number;
}

interface ValidatedEntity {
  name: string;
  type: ExtractedEntity['type'];
  canonical_name: string;
  verdict: 'confirmed' | 'removed' | 'retyped';
  original_type?: string; // populated when verdict is 'retyped'
  reason: string; // brief justification for the verdict
}

Only entities with verdict confirmed or retyped proceed to storage. Entities with verdict removed are discarded.

---

3. Pass 2 Prompt Design

The prompt must be concise and focused. It does NOT load the full 959-line classification skill. Instead, it embeds a compressed version of the five universal rules and the 12 diagnostic test questions.

Draft Prompt Template

You are an entity validation assistant for a UK knowledge base. You will review
a list of entities extracted from content by a classification pipeline and
determine whether each entity is valid.

For EACH entity, apply these tests in order:

1. NAMED ENTITY TEST: Is this a specific, named thing that exists independently
   of the document discussing it? Generic concepts (information security, data
   protection, encryption, business continuity, risk management, etc.) FAIL.

2. EXTERNAL REFERENCE TEST: Could someone outside this organisation look this
   up and find an independent definition or registration? Internal documents
   (policies, procedures, plans, registers, agreements, statements) FAIL.

3. ROLE TITLE TEST: Is this a job title or role description rather than a
   person's name? (Managing Director, DPO, Project Manager, IT Director) FAIL.

4. PROTOCOL/FORMAT TEST: Is this a protocol (HTTPS, SSH, TLS), file format
   (PDF, CSV), programming language (Python, JavaScript), or cryptographic
   algorithm (AES-256, RSA)? These are NOT entities. FAIL.

5. TYPE ACCURACY TEST: Does the assigned type match the entity? Apply these
   diagnostic questions:
   - organisation: Does it have a legal registration or government charter?
   - certification: Is it obtained by assessment with an issuing body and
     renewal cycle?
   - regulation: Does non-compliance carry legal penalties?
   - framework: Is it published guidance for voluntary adoption, without legal
     force?
   - capability: Would the organisation list this on its website as a service
     it offers?
   - person: Is this the actual name of a specific individual?
   - technology: Is this a specific named platform with a vendor and version?
   - project: Is this a named piece of work with a start, scope, and end?
   - sector: Is this a recognised industry classification?
   - product: Is this a named branded offering the organisation sells?
   - standard: Is this a numbered document published by a standards body?
   - methodology: Is this a named approach to work with its own body of
     knowledge?

Common false positives to catch:
- ISMS/QMS/EMS are management systems, not certifications or frameworks
- Insurance products (professional indemnity, public liability) are NOT entities
- GDPR artefacts (DPIA, ROPA, lawful basis, consent) are NOT standalone entities
- Contract types (NDA, SLA, DPA) are NOT entities
- Security principles (defence in depth, zero trust, least privilege) are NOT
  entities or methodologies
- Geographic regions (England, Wales, Scotland) are NOT sectors
- Internal departments (IT Department, HR Team) are NOT organisations

For each entity, return a verdict:
- "confirmed" if it passes all tests with the correct type
- "retyped" if the entity is valid but has the wrong type (provide corrected
  type)
- "removed" if it fails any test (provide reason)

Content title: {TITLE}
Content type: {CONTENT_TYPE}

Content excerpt:
{CONTENT_EXCERPT}

Entities to validate:
{ENTITY_LIST}

Tool Schema for Pass 2

{
  name: 'return_entity_validation',
  description: 'Return validated entity list with verdicts',
  input_schema: {
    type: 'object',
    properties: {
      validated_entities: {
        type: 'array',
        items: {
          type: 'object',
          properties: {
            name: { type: 'string' },
            type: { type: 'string', enum: [/* 12 entity types */] },
            canonical_name: { type: 'string' },
            verdict: { type: 'string', enum: ['confirmed', 'removed', 'retyped'] },
            original_type: { type: ['string', 'null'] },
            reason: { type: 'string' }
          },
          required: ['name', 'type', 'canonical_name', 'verdict', 'reason']
        }
      }
    },
    required: ['validated_entities']
  }
}

Prompt Size Estimate

The validation prompt template above is approximately 450 words (~600 tokens). With content excerpt (2,000 chars ~500 tokens) and entity list formatting (~15 tokens per entity), a typical validation call with 10 entities will have an input of approximately 1,250 tokens.

---

4. Cost Model

Pricing Reference (from lib/ai/pricing.ts)

Model	Input ($/M tokens)	Output ($/M tokens)	Cache Read ($/M)	Cache Write ($/M)
claude-opus-4-6	15.00	75.00	1.50	18.75
claude-sonnet-4-5	3.00	15.00	0.30	3.75
claude-haiku-4-5	0.80	4.00	0.08	1.00

Token Estimates Per Classification

Pass 1 (existing, Sonnet):

Component	Tokens
System/skill prompt	~4,500
Content (5,000 chars)	~1,250
Tool schema	~500
Total input	~6,250
Output (tool response)	~800

Pass 1 cost: (~6,250 / 1M x $3.00) + (~800 / 1M x $15.00) = $0.019 + $0.012 = $0.031 per item

Pass 2 (new, Haiku):

Component	Tokens
Validation prompt	~600
Content excerpt (2,000 chars)	~500
Entity list (10 entities avg)	~150
Tool schema	~200
Total input	~1,450
Output (10 verdicts)	~400

Pass 2 cost: (~1,450 / 1M x $0.80) + (~400 / 1M x $4.00) = $0.0012 + $0.0016 = $0.003 per item

Cost Comparison

Mode	Cost per item	Relative
Pass 1 only (current)	$0.031	Baseline
Pass 1 + Pass 2 (Haiku)	$0.034	+10%
Pass 1 + Pass 2 (Sonnet)	$0.062	+100%

Recommendation: Use Haiku for Pass 2. The validation task is well-structured (apply N tests to each entity, return a verdict) and does not require the reasoning depth of Sonnet. The diagnostic questions are explicit enough that Haiku can apply them reliably. At $0.003 per item, the validation pass adds only 10% to the classification cost. Using Sonnet would double the cost for a task that is essentially pattern matching against clear rules.

Batch Cost Projections

For the current knowledge base (~400 items):

Mode	Total cost
Full reclassification (Pass 1 only)	$12.40
Full reclassification (Pass 1 + Pass 2, Haiku)	$13.60
Pass 2 only (entity revalidation, Haiku)	$1.20

The “Pass 2 only” mode is particularly cost-effective for retroactive entity quality improvement without re-running the full classification.

---

5. Triggering Conditions

Options Considered

Strategy	Description	Pros	Cons
Always	Run Pass 2 after every classification	Maximum quality	Cost on items with 0-2 entities (wasted)
Entity count threshold	Only when >N entities extracted	Targets entity-heavy content	Misses low-count false positives
Low confidence only	Only when classification_confidence < 0.7	Targets uncertain classifications	Confidence measures domain accuracy, not entity quality
Batch mode only	Only during batch-reclassify.ts runs	Zero impact on interactive latency	Misses interactive classifications
Parameter-controlled	Caller decides via validate boolean	Full flexibility	Pushes decision to every call site

Recommendation: Parameter-controlled, default OFF for now

Interface: Add validate?: boolean to ClassifyParams. Default: false.

Rationale:

1. Low confidence is a poor proxy. Classification confidence measures domain/subtopic accuracy, not entity extraction quality. A 0.95 confidence item can still have 5 false-positive entities.

2. Entity count threshold is crude. Items with 3 entities can have 2 false positives. The threshold would need to be so low (>0 entities) that it becomes equivalent to “always”.

3. Default OFF preserves backward compatibility. Interactive classification via the UI should remain fast and unchanged. The validation pass adds latency (~1-2 seconds for the Haiku call) that is acceptable in batch mode but noticeable in interactive mode.

4. Batch reclassification is the primary use case. The batch-reclassify.ts script is where entity quality matters most and where the cost is budgeted. Adding --validate as a CLI flag is straightforward.

5. Task 15 will measure the trade-off. After implementation, the eval suite will determine whether Pass 2 improves precision enough to justify default-on. The parameter makes A/B comparison trivial.

Planned call sites:

Call site	Default	Rationale
/api/classify route	false	Interactive, latency-sensitive
MCP classify_content tool	false	Interactive, but could be overridden
batch-reclassify.ts	Controlled by --validate flag	Primary quality improvement path
Classification eval suite	true	Always validate during evaluation

Post Task 15 decision: If the eval shows a significant precision improvement (target: >60% precision, up from 43.6%), reconsider making validation default-on for all classifications. The 10% cost increase and ~1-2 second latency increase are acceptable if precision improves by 15+ percentage points.

---

6. Implementation Interface

Function Signature Change

export interface ClassifyParams {
  supabase: SupabaseClient<Database>;
  itemId: string;
  force: boolean;
  userId: string;
  validate?: boolean; // NEW — default false
}

New Function: validateEntities()

export async function validateEntities(
  entities: ExtractedEntity[],
  contentExcerpt: string,
  contentTitle: string,
  contentType: string,
): Promise<EntityValidationResult> {
  // 1. Skip if no entities to validate
  if (!entities.length) {
    return {
      validated_entities: [],
      removed_count: 0,
      retyped_count: 0,
      confirmed_count: 0,
    };
  }

  // 2. Build validation prompt from template
  // 3. Call Claude Haiku via tool_choice
  // 4. Parse and return EntityValidationResult
}

Where Pass 2 Fits in the Flow

The current classifyContent() flow (from classification-architecture.md):

1.  Fetch content item
2.  Cache check
3.  Empty content guard
4.  Load classification skill
5.  Build taxonomy from DB
6.  Prepare content (truncate to 5,000 chars)
7.  Call Claude API (Pass 1)
8.  Extract tool result
9.  Validate domains
10. Normalise keywords
11. Update content item
12. Regenerate embedding
13. Store temporal references
14. Load entity aliases
15. Store entity mentions        <-- Pass 2 goes BEFORE this
16. Store entity relationships
17. Bridge temporal references

Pass 2 inserts between steps 14 and 15:

14.  Load entity aliases
14a. IF validate && entities.length > 0:
       Run validateEntities() with Pass 1 entities + content excerpt
       Replace result.entities with validated entities (confirmed + retyped only)
       Log validation summary (removed/retyped/confirmed counts + token usage)
15.  Store entity mentions (now uses validated entities)
16.  Store entity relationships (now uses validated entities)
17.  Bridge temporal references

Token Usage Tracking

Both passes should track token usage via the response.usage object returned by the Anthropic SDK:

interface ClassificationTokenUsage {
  pass1: {
    input_tokens: number;
    output_tokens: number;
    model: string;
  };
  pass2?: {
    input_tokens: number;
    output_tokens: number;
    model: string;
  };
  total_input_tokens: number;
  total_output_tokens: number;
  estimated_cost_usd: number;
}

Token usage should be logged (not stored in DB) for cost monitoring. The estimateCost() function in lib/anthropic.ts already supports multi-model cost calculation.

---

7. Integration with Existing Filters

Current Post-Extraction Filter Chain

The existing deterministic filters in lib/ai/classify.ts (lines 26-365):

1. isExcludedEntity() — identifier patterns (SIC codes, VAT numbers, DUNS)
2. isInternalDocument() — suffix patterns (Policy, Procedure, Plan, etc.)
3. isGenericConcept() — ~80 known abstract terms
4. isRoleTitle() — regex patterns for job titles
5. isProtocolOrFormat() — ~35 protocols, formats, algorithms
6. isInsuranceOrContract() — ~8 insurance products and contract types
7. isManagementSystemAcronym() — ISMS, QMS, EMS, IMS
8. isGdprArtefact() — ~20 GDPR sub-concepts
9. isFrameworkLot() — G-Cloud lot numbers
10. isCompoundEntity() — slash-separated compounds

These filters are applied via shouldExcludeEntity() during entity storage (step 15).

Recommended Ordering: Deterministic Filters FIRST, Then Pass 2

Pass 1 entities
    |
    v
shouldExcludeEntity() -- deterministic, free, instant
    |
    v
Filtered entities (known false positives removed)
    |
    v
validateEntities() -- LLM, costs $0.003, ~1-2 seconds
    |
    v
Validated entities (novel false positives removed)
    |
    v
Storage (entity_mentions + entity_relationships)

Rationale for filters-first ordering:

1. Cost efficiency. Every entity removed by the deterministic filters is one fewer entity sent to the LLM. If Pass 1 extracts 15 entities and the filters remove 5, Pass 2 only needs to validate 10. This reduces input tokens and output tokens proportionally.

2. The filters are free and instant. There is no reason to pay an LLM to identify entities that a regex can catch. The deterministic filters handle the “known unknowns” (patterns we have already identified). Pass 2 handles the “unknown unknowns” (novel false positives the filters do not cover).

3. Better LLM focus. With obvious false positives already removed, the LLM can focus its attention on the harder cases: borderline entities where type accuracy matters, or novel false positives that do not match existing patterns.

4. Validation prompt is smaller. Fewer entities in the prompt means fewer tokens, faster response, and less chance of the model losing focus on individual entities in a long list.

5. No quality downside. The deterministic filters have very high precision (they almost never remove real entities). The statutory allowlist (STATUTORY_ALLOWLIST) handles the known exceptions. Running the LLM first would not save any true positives that the filters would otherwise catch.

Implementation in classifyContent()

// Step 14a: Apply deterministic filters FIRST
const deterministicallyFiltered = (result.entities ?? []).filter(
  (e) => !shouldExcludeEntity(e),
);

// Step 14b: IF validate, run LLM validation on the survivors
let finalEntities = deterministicallyFiltered;
if (params.validate && deterministicallyFiltered.length > 0) {
  const validation = await validateEntities(
    deterministicallyFiltered,
    contentForClassification.slice(0, 2000),
    item.title,
    item.content_type,
  );
  finalEntities = validation.validated_entities.filter(
    (v) => v.verdict !== 'removed',
  );
}

// Step 15: Store entity mentions using finalEntities

---

8. Content Excerpt Length for Pass 2

Pass 1 sends 5,000 characters of content to Claude for full classification. Pass 2 does not need as much content because it is not classifying — it only needs enough context to verify whether each entity genuinely appears in the text and whether the type assignment is correct.

Recommendation: 2,000 characters for Pass 2.

• Sufficient for entity mention verification (most entities appear in the first third of the content)
• Reduces input tokens by ~750 compared to sending the full 5,000 characters
• For edge cases where an entity appears only in the later portion of the text, the validation prompt instructs the model to confirm the entity if it is a clearly valid named entity even without a visible mention in the excerpt

---

9. Error Handling

Pass 2 is non-blocking, following the same pattern as entity storage, relationship storage, and temporal bridging in the existing pipeline.

try {
  const validation = await validateEntities(...);
  finalEntities = validation.validated_entities.filter(v => v.verdict !== 'removed');
} catch (validationErr) {
  console.error('Entity validation (Pass 2) failed:', validationErr);
  // Fall back to deterministically-filtered entities (graceful degradation)
  finalEntities = deterministicallyFiltered;
}

If Pass 2 fails (network error, model error, timeout), classification still succeeds with the unvalidated entities. This matches the existing error handling philosophy: entity quality is important but must never break classification.

---

10. Batch Reclassification Integration

The batch-reclassify.ts script gains a --validate flag:

bun run scripts/batch-reclassify.ts --validate --execute --limit 50

When --validate is set, the script passes validate: true to classifyContent(). The validation summary (removed/retyped/confirmed counts) is included in the per-item log output and the batch summary.

For entity-only mode (--entities-only), a standalone validation-only mode should also be considered: re-validate existing entities without re-running Pass 1. This would use the stored entities from entity_mentions and the stored content from content_items, calling only validateEntities(). This is the cheapest option for retroactive quality improvement ($1.20 for 400 items).

---

11. Evaluation Integration

The classification eval suite should test both modes:

1. Pass 1 only — baseline entity precision/recall (current: 43.6% precision)
2. Pass 1 + Pass 2 — validated entity precision/recall

The eval fixtures in __tests__/fixtures/entity-eval-gold-standard.json already define expected_entities and excluded_entities per content item. The eval runner compares extracted entities against these fixtures. With Pass 2 enabled, the expectation is:

• Precision improvement: Target >60% (up from 43.6%). Pass 2 should remove false positives that deterministic filters miss.
• Recall preservation: Target >= current recall. Pass 2 should not remove true positives. The confirmation step ensures legitimate entities are retained.
• Type accuracy improvement: Target >95% for retyping (already 99.2% in S140 eval, so this is a secondary benefit).

---

12. Future Considerations

Prompt Caching for Pass 2

If validation becomes default-on, the static portion of the validation prompt (~600 tokens of rules and diagnostic questions) could benefit from Anthropic’s prompt caching. At Haiku’s cache read rate ($0.08/M vs $0.80/M input), this would reduce the per-item cost from $0.003 to approximately $0.002. Worth implementing if Pass 2 becomes default-on after Task 15 evaluation.

Feedback Loop to Deterministic Filters

When Pass 2 consistently removes the same entity pattern (e.g., a specific phrase appears in 5+ items and is always removed), that pattern should be added to the deterministic filters. This progressively makes Pass 2 cheaper (fewer entities to validate) and faster. A quarterly review of Pass 2 removal patterns would inform filter updates.

Standalone Validation API

A /api/validate-entities endpoint could allow manual entity validation without re-running classification. This would be useful for the review queue workflow where a reviewer wants to clean up entities on a specific item.

---

13. File Reference

File	Role
lib/ai/classify.ts	Primary implementation file — add validateEntities(), modify classifyContent()
lib/ai/pricing.ts	Model pricing constants (Haiku rates for cost tracking)
lib/anthropic.ts	getAnthropicClient(), estimateCost()
lib/ai/skills/classification-entity-types.md	Entity type reference (informing validation prompt)
docs/reference/entity-type-taxonomy-spec.md	Authoritative entity type spec (source of truth for diagnostic questions)
__tests__/fixtures/entity-eval-gold-standard.json	Eval fixtures for measuring precision/recall
scripts/batch-reclassify.ts	Batch script — add --validate flag
docs/reference/classification-architecture.md	Architecture doc — update after implementation

---

14. Summary of Decisions

Decision	Choice	Rationale
Pass 2 model	Haiku (claude-haiku-4-5)	10% cost increase vs 100% for Sonnet; task is structured pattern matching
Triggering	Parameter-controlled, default OFF	Preserve interactive latency; batch mode is primary use case
Filter ordering	Deterministic filters FIRST, then Pass 2	Filters are free and instant; reduces LLM token cost
Content excerpt	2,000 characters (vs 5,000 for Pass 1)	Sufficient for entity verification; reduces token cost
Error handling	Non-blocking (graceful degradation)	Matches existing pipeline philosophy
Token tracking	Logged, not stored	Cost monitoring without schema changes
Default future state	Revisit after Task 15 eval results	Data-driven decision on default-on vs default-off