Two-Pass Validation Results — Entity Classification Eval

Re-filed S504 (ruling R9), from reference/: eval results; the durable companion is reference/two-pass-validation-architecture.md.

Two-Pass Validation Results — Entity Classification Eval

Status: Measured 06/04/2026 (Session 151 WP3) Baseline source: __tests__/fixtures/eval-baselines/entity-classification.baseline.json (corrected 08/04/2026 per S155 WP1.1 L-2; original header referenced the non-existent lib/eval/baseline/ directory — the actual location is hardcoded in lib/eval/baseline.ts:16 BASELINE_DIR) Eval script: bun run scripts/eval-entity-classification.ts --live --validate --confirm --save-baseline Source spec: .planning/.archive/.specs/ai-eval-phase2-classification-skill-spec.md Task 15 (archived S168 — all tasks complete) Superseded by: docs/audits/two-pass-cost-quality-measurement.md (S168 WP2 — full single-pass vs two-pass contrast on current code) Roadmap: §3.1 (entity precision measurement) and §3.2.2 (two-pass tuning)

This document captures the first end-to-end measurement of two-pass entity validation against the 95-item gold standard fixture, after the scripts/eval-entity-classification.ts --validate flag wiring was repaired in Session 150 (AI-H2).

---

TL;DR

Two-pass validation did not improve precision over the cached baseline.

Metric	S141 cached baseline	S151 live —validate	Δ	Launch target
Precision	43.6%	41.7%	-1.9 pp	≥70%
Recall	(not recorded)	60.2%	—	≥70%
F1	(not recorded)	49.3%	—	—
Type accuracy	(not recorded)	98.3%	—	≥80%
Exclusion compliance	(not recorded)	44.8%	—	≥50%
Cross-item consistency	(not recorded)	93.2%	—	≥80%

Run cost: $5.50 estimated → ~$5.30 actual at Sonnet 4.6 (Pass 1 ~$0.06/item

• Pass 2 ~$0.003/item × 95 items + retries). Latency: ~25 minutes wall clock with the script’s 1 req/sec rate limit. Result: FAIL — exclusion_compliance 44.8% below the 50% threshold.

---

Recommendation

Two-pass validation alone is insufficient for launch. It is not the wrong intervention, but it is being asked to fix a problem the prompt should not be producing in the first place.

The headline numbers confirm three things at once:

1. Type accuracy (98.3%) — when an entity is genuinely an entity, the classifier picks the right type almost every time. Pass 2 is doing its job there.

2. Cross-item consistency (93.2%) — the same entity gets the same type across documents. Pass 2 is also doing its job here.

3. Precision (41.7%) and exclusion compliance (44.8%) are floor-level. The classifier is extracting the wrong things in the first place. Pass 2 then “validates” them as confirmed because the LLM judge agrees they look like entities — but they should never have been extracted.

The Pass 2 LLM judge is too forgiving on borderline cases. It can also only remove things; it can never go back and find what the extractor missed.

Launch decision: AI-C2 cannot close as “two-pass validation = launch ready”. Either:

• (a) Tighten Pass 1 extraction (the recommended path), or
• (b) Tighten Pass 2 to be more aggressive about removing capabilities, methodology concepts, and policy nouns, or
• (c) Both, with re-evaluation

See “Next steps” below.

---

Headline numbers and what they mean

Aggregate metrics (n=95, 290 entities extracted, 201 expected)

• TP: 121 (correct extractions)
• FP: 169 (wrong extractions — 58% of everything we extracted)
• FN: 80 (missed real entities)
• Type errors: 2 (only 2 wrong types out of 121 correct extractions — nearly perfect when the entity itself is right)
• Exclusion failures: 116 (we extracted 116 entities the gold standard explicitly says not to extract)

Per-domain breakdown (worst → best)

Domain	Items	Precision	Recall	TP	FP	FN	Notes
methodology	9	12%	13%	2	15	14	Worst — methodology items extract everything as entities
support	5	14%	100%	2	12	0	Recall perfect; precision floor-level
compliance	18	32%	39%	15	32	23	Largest absolute error contribution
implementation	11	39%	46%	11	17	13
security	29	41%	70%	45	66	19	Largest item set; capability bleed dominates
corporate	14	60%	64%	18	12	10	Best non-product domain
product-feature	9	65%	97%	28	15	1	Best — product/capability boundary works here

Type errors (only 2)

Entity	Expected	Actual
ISO 27001	certification	standard
ISO 9001	certification	standard

Both of these are the same disambiguation question: is “ISO 27001” the standard (the published document) or the certification (the verifiable claim “we hold ISO 27001”)? In the gold standard it is treated as a certification. The classifier picks standard. This is a prompt diagnostic issue, not a failure of the extractor — and Pass 2 actually flips two of the four ISO mentions to standard, contributing to the cross-item inconsistency below.

Cross-item type inconsistencies (13)

The same entity name gets two different types across documents:

• iso 27001 → [certification, standard] ← same disambiguation as type errors
• iso 9001 → [certification, standard]
• iso 14001 → [standard, certification]
• bitdefender → [product, technology]
• worldpay → [organisation, technology]
• access paysuite, adalante smartpay, opayo, pay360 → [product, technology]
• phew knowledge academy, phew lunchtime learning, phew virtual clinic → [product, capability]
• multi-academy trust → [organisation, sector]

These point at concrete prompt-level disambiguation gaps that the existing “diagnostic questions” are not resolving:

• Software product vs underlying technology (Bitdefender, Opayo, etc.)
• Branded service vs capability (Phew Lunchtime Learning, etc.)
• Sector vs organisation type (Multi-Academy Trust)

---

False-positive deep dive (the 116 exclusion failures)

The 116 exclusion failures are dominated by three buckets. These are exactly the categories the entity taxonomy spec (docs/reference/entity-type-taxonomy-spec.md) and the post-extraction filters (lib/ai/classify.ts + skill diagnostic questions) were supposed to catch.

Bucket 1 — Capabilities and security techniques extracted as entities

The skill says capabilities like “encryption”, “access control”, or “penetration testing” are not entities. The exclusion list in lib/ai/classify.ts contains many of these. Pass 2 still confirms them.

Examples from the run:

• ICO Guide: encryption, penetration testing, pseudonymisation, vulnerability scanning, BYOD, CIA Triad, information risk analysis
• NCSC 10 Steps: Asset Management, Identity and Access Management, Incident Management, Logging and Monitoring, Supply Chain Security, Vulnerability Management, Risk Management
• Information Security Controls: Multi-Factor Authentication, principle of least privilege, clear desk policy, clear screen policy
• Are employee laptops encrypted: email encryption, endpoint encryption, server encryption, password hashing

Why Pass 2 misses these: the validator is asked “is this an entity?” and answers “yes” because the named technique is real. It is not asked “is this a named, branded entity, or is it a capability the organisation demonstrates?”. The diagnostic questions added in Phase 2c are not operationalised in Pass 2 — only in Pass 1 prompting.

Bucket 2 — Methodology / framework concepts extracted as entities

The skill says abstract methodology concepts (information risk analysis, privacy by design, phased approach, early life support) are not entities. They are activities or concepts.

Examples:

• information risk analysis, Risk Management (NCSC)
• Data Protection Impact Assessment, privacy by design (GDPR overview)
• principle of least privilege (Information Security Controls)
• Escalation Procedures, Phased Approach, UAT (Risk Mitigation)
• User Acceptance Testing (Resource Commitments)

These mostly come from documents whose entire purpose is to describe a methodology. The classifier is interpreting “this document is about X” as “X is an entity in this document”. It is not.

Bucket 3 — Policies, procedures, and internal documents

Internal documents (a “Data Processing Agreement”, a “Records of Processing Activities”, a “Privacy Management Framework”) are not entities — they are artefacts the organisation produces.

Examples:

• Data Processing Agreement, Records of Processing Activities, Data Protection Impact Assessment (GDPR overview)
• Phew Information Management System (Information Security Controls)
• equal opportunity policy, remote and flexible working policy, social value (Social Value Statement)
• Privacy Management Framework (ICO Accountability)

The skill addresses these via the INTERNAL_DOCUMENT_SUFFIXES filter (policy$, procedure$, plan$, register$, etc.) and the GENERIC_CONCEPTS set in lib/ai/classify.ts. Both are present in code, but they fail on this run because:

1. The names extracted by Claude are slightly different forms (e.g. “Privacy Management Framework” instead of “…framework” — note the noun is at the end, but it does match the regex).
2. Pass 2 is not aware of the deterministic filter rules at all and can “confirm” things that Pass 1 should have rejected.

Why the deterministic filters are not biting

The deterministic filters in shouldExcludeEntity() run between Pass 1 and Pass 2 (see lib/ai/classify.ts:1071-1074). They are working — the deterministicallyFiltered array IS smaller than result.entities. But the filters only catch a specific subset (suffix patterns, generic concepts, identifier patterns). The 116 exclusion failures are entities that:

1. Pass the deterministic filter (correct domain, correct shape)
2. But are still semantically wrong (capability, methodology, internal document)
3. And Pass 2 confirms them anyway

---

Worst-performing items (top 10)

Item	P	R	ExCmpl	Headline issue
What is your organisation doing regarding Social Value	0%	100%	0%	Every extraction was a capability/concept
Risk Mitigation	0%	100%	0%	Method-document hallucinated entire process as “entities”
Performance Measures	0%	100%	20%	SLA descriptions extracted as capabilities
Do you undertake any additional training for system administrators	0%	100%	0%	Job titles + concepts
Are employee laptops encrypted?	17%	100%	100%	Encryption variants extracted as capabilities
ISO 9001 Quality Management Systems	20%	50%	100%	Sister-standard pollution (ISO 13485, ISO 18091, ISO 22163, ISO 29001 cross-referenced and extracted)
NCSC 10 Steps	30%	100%	13%	Each “step” extracted as a capability
Resource Commitments	13%	100%	22%	Job titles extracted as capabilities
Phew Design Ltd – Social Value Statement	14%	100%	17%	Benefits extracted as capabilities/frameworks
ICO Guide to Data Security	43%	100%	29%	Same NCSC pattern + acronym noise

Pattern: the extractor scores P=0% on documents whose subject is controls / capabilities / methodologies. It reads the body, sees a list of named techniques, and dutifully extracts them. The Pass-2 validator confirms because, considered in isolation, “MFA” or “encryption” really is a real thing the document discusses. The problem is that those things are not entities in the taxonomy sense — they are properties of the organisation, not nameable nouns we want to graph against.

---

Two-pass cost vs quality trade-off

Two-pass validation in this run:

• Cost: added ~$0.40 across 95 items (Pass 2 average ~$0.004/item with retries) on top of the ~$5.50 Pass 1. ~7% incremental cost.
• Latency: added ~0.5–1.5s per item. With the 1s rate limit between items, the wall-clock impact was modest (~25 min total vs ~14 min for Pass 1 only).
• Quality: negligible improvement on the metric the run was designed to measure. Two-pass validation is currently a tax with no observable return.

This does not mean Pass 2 is broken. It means the two-pass validator is asked the wrong question. It is asked to confirm extractions that Pass 1 should have refused to make. Removing things is its only lever, and it exercises that lever conservatively.

---

Recommendation: NOT “always” / “conditional” / “batch-only”

The Phase 2d Task 15 deliverable was supposed to recommend one of:

• Always — run Pass 2 on every classification
• Conditional — run Pass 2 only when Pass 1 confidence is low
• Batch-only — run Pass 2 only during nightly batch reclassification

Based on the data, none of these are the right launch posture, because all three assume Pass 2 is the bottleneck. It is not. Pass 1 is the bottleneck.

The right launch posture is therefore:

1. Tighten Pass 1 prompt to refuse capability / methodology / internal document extraction at source — see “Next steps” below.
2. Re-measure with the same fixture against the same baseline.
3. Then decide whether Pass 2 should be Always / Conditional / Batch-only. At that point Pass 2’s job becomes “catch the residual ~5–10% Pass 1 still leaks”, which is exactly the problem two-pass validation is good at.

This document records that Step 1 (Pass 1 tightening) is the prerequisite, not Step 3 (Pass 2 deployment posture).

---

Next steps

These map to the existing roadmap and product-backlog work. None require new specs.

1. Roadmap §1.5 (Entity classification prompt tightening) — a new focused spec is being written at docs/specs/entity-classification-prompt-tightening-spec.md (S154 WP2b). The S154 WP1 audit (docs/audits/s154-si-prompt-refinement-spec-review.md) established that the SI Prompt Refinement Skill (§2.2) targets a different AI touchpoint — SI feed relevance scoring, not entity extraction — and cannot close the Pass 1 bottleneck described above. The classification tightening spec instead targets direct edits to lib/ai/skills/classification.md, filter expansions in lib/ai/classify.ts, and the cross-item consistency / ISO-cert diagnostic rules below. The spec’s framing is diagnostic-first — understand why previous changes have not moved the numbers before iterating — and its acceptance criteria target production-readiness of the touchpoint rather than a rigid precision number, on the basis that past “hit the target” framings drove changes that did not move the needle. HIGH priority. Multi-session.

2. Roadmap §3.2.2 (two-pass tuning) — once Pass 1 is tightened, re-run bun run scripts/eval-entity-classification.ts --live --validate --confirm and decide Pass 2 deployment posture against the new precision floor.

3. Pre-launch decision (roadmap §1) — until precision crosses ~65–70% on this fixture, the eval is not “PASS”. The pre-launch quality bar needs to be re-stated explicitly: are we launching with 41.7% precision on entity extraction, or do we hold launch on §2.2?

4. Cross-item consistency fixes — the 13 cross-item inconsistencies are smaller, more tractable fixes than the precision problem. They need:

   • Software product vs technology disambiguation (Bitdefender, Opayo)
   • Sector vs organisation disambiguation (Multi-Academy Trust)
   • Branded service vs capability disambiguation (Phew Lunchtime Learning)

   Each is a single line in the diagnostic-question section of lib/ai/skills/classification.md. ~30 minutes total.

5. Type-error fixes (ISO standards as certifications) — single disambiguation rule needed: “An ISO/IEC standard mentioned in the context of a holder — ‘we are ISO 27001 certified’, ‘our suppliers must hold ISO 27001’ — is a certification. A standard mentioned as a published document — ‘ISO 27001 specifies…’, ‘as defined in ISO 27001 §A.5’ — is a standard.” Add this single sentence to the skill. ~10 minutes.

6. Defensive ai_keywords handling in lib/ai/classify.ts — one item in the run failed with result.ai_keywords.map is not a function because Claude omitted that field from a tool response despite the schema marking it required. ~5 minutes. Tracked as a S151 main-session task.

---

Methodology notes

• Fixture: 95 items in __tests__/fixtures/entity-eval-gold-standard.json. This is the version expanded in S149 from 85 → 95 items, with hand-labelled expected entities and excluded entities per item.
• Validator (Pass 2): Sonnet 4.6 with temperature: 0, prompt at lib/ai/classify.ts:validateEntities. Verdict choices: confirmed, retyped, removed.
• Deterministic filters: shouldExcludeEntity() runs between Pass 1 and Pass 2 — see lib/ai/classify.ts:1071-1074.
• Sandbox workaround: the eval was run with dangerouslyDisableSandbox: true to bypass the Bun-fetch + Claude Code sandbox proxy interaction documented in CLAUDE.md gotchas. Without it, every post-classification DB write hangs on a 204 response. This is not a production issue.
• Baseline saved: the new precision=0.417, exclusion_compliance=0.448 baseline is now stored via lib/eval/baseline.ts, the on-disk baseline module used by all eval scripts (via loadBaseline(SUITE_NAME) and saveBaseline(SUITE_NAME, metrics)). The earlier reference in this doc to a lib/eval/baseline/entity-classification-baseline.json file was incorrect — no such directory exists. The old S141 cached baseline (precision=0.436) was overwritten by this run; if we need the historical comparison again, restore from the git history of lib/eval/baseline.ts and whichever file the module persists its state to.