RESEARCH — {449.1} Extraction review spike

RESEARCH — ID-449 Extraction review spike

Task: id-449 — review entity extraction end-to-end and measure whether opening the entity_type vocabulary improves or degrades extraction quality on real tier, so the owner can rule finally on D11 (S554 board). Spec-chain artefact: {449.1} RESEARCH (of {449.1} → {449.2} → {449.3}). Date: 11/08/2026. Baseline: main at e24ee6199 — post-id-433 strip (no alias layer, no ISO branch, no canonicalise_for_relationship, no holder_rule.py), post-id-437 (the entity-resolution-llm extra declared at the 1.0.18 pin), and post-{434.3} TECH. Every file:line below is at that SHA. The id-434 reshape has not been implemented, so the partition and key structure described in §6.6 is the post-reshape target shape ({434.3} §2.3/§2.4) unless stated otherwise. Posture: findings, evidence, and a design-only run plan. No extraction was executed, no LLM was called, no database was written, nothing was committed. AC2 is specified here, not run.

---

0. Headline findings

1. The S545 mock-bypass claim is verified and is stronger than it was stated. The mock does not merely bypass the prompt’s content — it uses the prompt text only as a dictionary key, and emits exactly two of the twelve entity_type values, from a regex and a first-line slice (§3). It cannot referee extraction quality, and it cannot express an out-of-vocabulary type even if the CHECK opened.
2. Two prompt sections are orphaned — their stated consumers were deleted by id-433. The RELATIONSHIP_PROMPT’s 2,603-char HOLDER DISAMBIGUATION block names a downstream that no longer exists, and contains a live defect: the literal token {CLIENT_ORGANISATION_NAME} is sent to the model unsubstituted in two worked examples (§2.2). The ENTITY_MENTION_PROMPT’s canonical_name guidance asks for a normalisation form DR-140 retired (§2.1 row 4).
3. The closed vocabulary has four enforcement points, not the three the board named — the producer ontology linter is a fourth, and entity_type_override has no CHECK at all (§4). The verdict must cover both columns or the override is a silent bypass.
4. The Trigger-1/2/3 tabular-recall rules are unmeasured and untested. Their only provenance is a commit message; the S203 WP-C2 measurement they were ported from targeted the pre-cocoindex TypeScript pipeline and survives in neither repository. They have zero test coverage of any kind, not even a drift guard (§2.1 row 10, §2.5).
5. AC2 has no reference set and cannot be executed until one is authored. The only document-keyed golden set this repo ever had was deleted on 2026-08-04, was keyed to the dropped content_items table, and described a client bid corpus rather than the Platform corpus (§6.5). This is the AC2 blocker, and it is judgement work, not spend.
6. Cost is not a constraint. Order of magnitude for the whole comparison is single-digit dollars (§6.4). The real constraints are the memo-carry-over hazard, the CHECK constraint blocking the open variants from writing, and (5).
7. Prompt caching is declared but almost certainly never fires at the current model and prompt sizes — a free observation the spike gets from an existing log line (§5).

---

1. The prompt surface — what is actually sent

Four static instruction constants in scripts/cocoindex_pipeline/prompts.py, each sent as a cache_control: ephemeral system block with only the per-document content_text in the uncached user message (extraction.py:874-902, :1021, :1049, :1076, :1108).

Constant	Lines	Chars	≈ tokens	Extractor	max_tokens
CLASSIFICATION_PROMPT	prompts.py:61-95	2,714	~680	extract_classification (extraction.py:1005)	4,096 (:745)
Q_A_FORM_PROMPT	prompts.py:98-147	3,450	~860	extract_qa_form (:1033)	32,768 (:742)
ENTITY_MENTION_PROMPT	prompts.py:150-204	6,223	~1,555	extract_entity_mentions (:1059)	16,384 (:743)
RELATIONSHIP_PROMPT	prompts.py:207-293	5,704	~1,425	extract_relationships (:1088)	16,384 (:744)

Model is a hardcoded constant — ANTHROPIC_MODEL = "claude-opus-4-6" (extraction.py:117) — not env-overridable. Token figures are chars ÷ 4; they are estimates and are used only for the order-of-magnitude cost model in §6.4.

Note the id-449 task file and the S554 board both describe the tabular-recall rules as “~1,400-char”. The measured figures are 2,726 chars for the whole TABULAR AND INDEX EXTRACTION RECALL section and 2,190 chars for its four rule bullets alone. The section is roughly twice the size the carried figure implies, and is 44% of the entity prompt.

---

2. AC1 — the prompt reviewed end to end

For each section: (a) the requirement it serves, (b) that requirement’s current source, (c) whether it has ever been measured. “Measured” means a measurement of extraction behaviour, not a test asserting that a substring is present in a Python string (DR-139 discipline). The full test inventory is §2.5.

2.1 ENTITY_MENTION_PROMPT (prompts.py:150-204)

#	Section	Requirement it serves	Current source	Measured?
1	Preamble + OUTPUT FORMAT (:150-163)	The emitted object matches EntityMentionExtraction’s seven fields so TypeAdapter.validate_json round-trips directly	extraction.py:333-351 + id-36 TECH §2.1/§3.1	No. Contract tests assert substrings only
2	entity_type enum line (:168)	The closed 12-value vocabulary	Four enforcement points (§4). No DR ratifies the content of the twelve values; DR-141 withdrew this exact shape one level up	No — and this is D11’s subject
3	entity_name verbatim (:169)	The surface form must occur in content_text so DR-135 anchoring can succeed	DR-135 (accepted, S543)	Partially — the code-side refusal is measured; the prompt’s contribution to it is not
4	canonical_name guidance (:170) — “ISO 27001:2022 -> iso_27001”	Was: a normalised dedup key	None found — candidate for disposition. DR-140 clause 3 + id-433 reduced the key function to strip → NFKD → lower; nothing now produces an underscored form, and id-434 D2/DR-147 key identity on the resolved canonical, not on anything the model emits	No
5	Span offsets (:171)	DR-135: an unanchored mention is refused, not written empty	DR-135	Partially (as row 3)
6	mention_confidence banding (:172)	Telemetry only	S554 transfer-type floor: “entity_mentions.confidence survives as telemetry only and must never surface as a bundle trust signal”	No
7	12 type meanings (:176-188)	Disambiguating the twelve values, one line + examples each	None found — candidate for disposition. Inherited from the pre-cocoindex TypeScript classification skill (lib/ai/skills/classification.md, deleted)	No
8	Internal-departments exclusion (:189-191)	An internal function is not an organisation	None current. Origin is ID-109 PC-2 (d551c283e); id-433 (7db759811) trimmed its source_scope parenthetical but kept the rule. The requirement is defensible on its own terms; it has no live source	No
9	UK-English caveat (:192)	House convention; explicitly exempts the verbatim entity_name	prompts.py module docstring :33	No
10	TABULAR AND INDEX EXTRACTION RECALL (:194-201) — 2,726 chars	Recall on markdown tables, source-index lists, and em-dash attributions	Commit message only (§2.5)	No — and zero test coverage of any kind

Row 7 is where the S551 divergence actually lives. The prompt’s own meanings say certification: "ISO 27001:2022", "Cyber Essentials Plus" and standard: "OAuth 2.0", "TLS 1.3" (prompts.py:178, :187). Under those definitions a real extractor should type ISO 9001:2015 as certification. The DB says standard — but that is a mock artefact, not extractor behaviour (§3). Whatever id-451 and D9 assume about the divergence’s origin, the prompt is not the cause.

2.2 RELATIONSHIP_PROMPT (prompts.py:207-293)

#	Section	Requirement	Current source	Measured?
1	Preamble + OUTPUT FORMAT (:207-216)	Raw triples matching RelationshipExtraction	extraction.py:354-381 + {101.6} PC-1	No
2	10-value relationship table (:222-235)	The closed relationship vocabulary	extraction.py:369-380; producer/validator.py ALLOWED_RELATIONSHIP_TYPES; the public ontology fixture	No
3	HOLDER DISAMBIGUATION (:239-285) — 2,603 chars	Was: attributing a certification to a third-party holder rather than the author org	None — ORPHANED. Its own text names the consumer at :269: “The downstream system will infer holder attribution from source_entity vs the configured client organisation name.” holder_rule.py (232 lines) and derive_holder_metadata were deleted whole by id-433 item 3 (c17c5a522), and PIPELINE_CLIENT_ORG — the configured client organisation name — was removed from every deploy manifest at 87a1e250d as reader-less	No
4	Closing GUIDANCE (:287-291)	Empty list rather than invented triples	id-36 TECH	No

Live defect inside row 3. {CLIENT_ORGANISATION_NAME} appears as a literal at prompts.py:271, :275, and :285 and is never substituted. The prompts are plain module constants handed verbatim to _cached_system_block; there is no .format(), no template render, and the only replaceAll('{CLIENT_ORGANISATION_NAME}', …) in the tree is scripts/quality-gate.ts:1026, which operates on an expectations JSON and not on any prompt. The model is therefore shown an unresolved template token inside two worked examples, in a section whose consumer no longer exists. This is a defect, not a design question — see OQ-2.

Historical note worth carrying. id-93’s C1 table (specs/id-93-pre-reingest-intent-gap/RESEARCH.md:50) dispositioned bl-92, the Provider-Disambiguation holder rule, as “N/A: the cocoindex path has no entity-relationship extraction”, and id-64’s close-out records “both backlog items retired”. Relationship extraction and the holder rules were then added anyway by ID-101 and ID-109. The retirement was recorded and then reversed without the record being updated.

2.3 CLASSIFICATION_PROMPT (prompts.py:61-95)

Section	Requirement	Current source	Measured?
content_type 7-value enum (:80-81)	The one surviving runtime taxonomy gate	Explicitly UNRATIFIED. extraction.py:120-128: the “S451 owner-ratified freeze” this once claimed “was asserted in a commit message, not in any ruling”; ontology/04-content-type.md still reads “flagged for the owner”. id-417 OQ5 owns the successor	No — but this is the one section with a genuine bidirectional parity test (§2.5 #8)
primary_domain / primary_subtopic (:82-83)	Kebab-case domain labels	None. DR-130 deleted both gates; values are written as-emitted	No
suggested_title (:84)	Title for the content_items write path	ID-64.10; consumed at flow.py ci_target.declare_row	No
classification_confidence banding (:85)	Confidence calibration	None found	No
secondary_classifications / rationale (:86-87)	Optional enrichment	None found	No

2.4 Q_A_FORM_PROMPT (prompts.py:98-147)

Section	Requirement	Current source	Measured?
form_metadata.form_type 7-value list (:130)	Was: a validated form-type key	None — advisory only. DR-130 deleted the runtime gate, and FormMetadata’s own docstring (extraction.py:236-243) records that the walk has DISCARDED this value since ID-136. Real enforcement is a DB FK on the app upload path, which this prompt never reaches	No
qa_pairs shape (:113-137)	Maps to q_a_extractions	id-36 TECH §2.1 / PRODUCT inv 2	No
expected_response_kind (:134)	2-value CV; info_only unratified	extraction.py:281	No — and test #9 is misleading (§2.5)
question_phrasings 3-5 (:137)	Rephrase-invariant search_qa_library matching, captured at the cheapest moment	ID-94.1, ratified ID-93 register §6.7	No
Non-form → qa_pairs: [] (:143)	Don’t synthesise pairs from prose	id-36 PRODUCT inv 2; extraction.py:1036-1038	No

On the Platform corpus this whole lane is expected to return [] for all nine documents by design — docs/reference/testing/corpus-manifest.json records the ruling that “a synthetic corpus yielding zero q_a_pairs is the EXPECTED post-retirement state, not a gap to backfill”. It still costs ~860 system tokens plus the document on every walk. Flagged, not dispositioned (OQ-6).

2.5 What the tests actually measure

scripts/tests/test_cocoindex_prompts.py — 16 test functions, 28 parametrised cases. Of those, 27 are substring or smoke assertions over a static Python string, one is a genuine bidirectional parity check, and zero measure extraction quality. No test in the file calls an LLM, loads a document, or compares output to anything.

• #8 test_classification_enumerates_content_types (:120) is the only real gate: it regex-parses the FIELD CONSTRAINTS block and asserts exact set parity with extraction.py::_VALID_CONTENT_TYPES. It would catch a value added on either side.
• #10 (entity types, :163) and #11 (relationship types, :184) look like parity checks and are not. Each compares the prompt against a set literal duplicated inside the test, so they can detect a value removed from the prompt but not a value added to the canonical set. If D11 opens the vocabulary, #10 will stay green while the prompt drifts.
• #9 (:152) is actively misleading. Its docstring says the prompt should forbid info_only; its assertion is "info_only" in Q_A_FORM_PROMPT.lower(), which passes if the string appears for any reason at all.
• #12/#13/#14 (:213, :235, :253) pin the orphaned holder block in place by asserting its eight trigger phrases, four disclaimer exemplars, and worked example verbatim. Deleting §2.2 row 3 means deleting these three tests with it.
• The tabular-recall section (prompts.py:194-201) has no test of any kind — not a quality measurement, not a drift guard, nothing.

Provenance of the tabular rules. One commit, 3d6fce39c (2026-06-08): “port bl-93 table-heavy entity-recall guidance into ENTITY_MENTION_PROMPT … (S203 WP-C2 Test 6)”. The word port is load-bearing: the rules were copied from the TypeScript-era lib/ai/skills/classification.md, which no longer exists. The gate they satisfied was id-64’s §6.4 G1, and its close-out records the gate as “G1 SATISFIED (bl-93 recall guidance ported …)” — the gate was port it, never measure it. The underlying S203 WP-C2 measurement targeted the old pipeline’s surfaces and is in neither repository (searched: bl-93, WP-C2, S203, table-heavy, tabular across the repo and the docs-site; the only working-tree hit is the prompt text itself). S545’s “no measurement in either direction” is exact.

---

3. The mock tier — the S545 claim verified

Claim (S545, carried into id-449’s Scope): “the live tier is the mock, which bypasses the prompt entirely.” Verdict: VERIFIED, and understated.

How the mock uses the prompt. Discrimination is an exact string match of the request’s system-block text against the four prompts.py constants (mock_llm.py:454-468, _ROUTES). The prompt is a dictionary key and nothing else — its content is never read, parsed, or honoured. A prompt edit changes the key, so the mock 400s loudly rather than silently serving the wrong shape (mock_llm.py:29-34), which is a good property and is also the whole of the prompt’s role in the mock tier.

What the mock emits for entity mentions (_entity_mentions_payload, mock_llm.py:250-279):

1. One organisation mention whose entity_name is the document’s first markdown heading, else its first line with words in it, truncated at 90 chars (_document_anchor, :206-247). It is typed organisation unconditionally (:262) — so on this corpus the mock types document titles as organisations, a category error no real extractor would make.
2. Up to five standard mentions (_ECHO_MENTION_CAP = 5, :151) matching the regex \b[A-Z]{2,6} ?\d{3,6}\b (_CERT_TOKEN_RE, :147), typed standard unconditionally (:272).

So the mock exercises — the wire shape and SSE grammar; strict Pydantic round-trip (its payloads are constructed from the real classes and revalidated through extraction.py’s own TypeAdapters at import, :35-37); memo serde; DR-135 span anchoring; Stage-5 near-match semantics; and the natural-key uniqueness constraints. All of that is real CI value.

It exercises none of — entity_type selection (2 of 12 values ever emitted); the twelve type meanings; the internal-departments exclusion; any of Triggers 1/2/3; canonical_name; mention_confidence calibration; recall; precision; or any judgement whatsoever.

Two consequences that bear directly on D11.

• The mock cannot express an out-of-vocabulary type even if the CHECK opened. Its payloads are constructed from EntityMentionExtraction and revalidated at import. Opening the Literal changes what is legal; it does not change what the mock emits. A mock-tier run after the constraint moves would look identical to one before it. The S554 real-tier posture is not a preference here — it is the only way to observe the variable.
• Every entity row in Platform staging and prod types ISO tokens as standard because mock_llm.py:272 hardcodes it, while the prompt’s own guidance (prompts.py:178) says certification. The S551 standard-vs-certification divergence, as currently visible in the database, is a mock artefact. Reading the live type distribution as evidence about the real extractor is the DR-104 rows-in-a-dev-DB trap, and id-451’s framing should carry this correction.

---

4. The enforcement surface — four points, not three

The S554 board names three (DB CHECK + Python Literal + TS union). There is a fourth, and a fifth column with no bound at all.

#	Site	Shape
1	supabase/migrations/20260617130000_squash_baseline.sql:5933	CONSTRAINT entity_mentions_entity_type_check CHECK (entity_type = ANY (ARRAY[...12 values...]))
2	scripts/cocoindex_pipeline/extraction.py:333-346	entity_type: Literal[...12 values...]
3	lib/validation/schemas.ts:1239-1252	export const VALID_ENTITY_TYPES = [...12...] as const (the extraction.py:16 docstring cites 1506-1519 — stale line reference)
4	scripts/cocoindex_pipeline/producer/validator.py:187-203	ALLOWED_ENTITY_TYPES frozenset — the BI-13 OKF concept-frontmatter semantic lint, deliberately not imported from extraction.py (collection-safety), with test_producer_validator.py:473 asserting len(...) == 12
5	entity_mentions.entity_type_override	No CHECK constraint. Read through COALESCE(entity_type_override, entity_type) at squash :2574, :2760, :2766

Parity is guarded asymmetrically. scripts/tests/test_cocoindex_extraction.py:908-931 parses VALID_ENTITY_TYPES out of the TS source and asserts it matches the Python Literal — a real bidirectional gate between (2) and (3). Point (4) has only a count assertion, and the prompt (§2.5 #10) has only a one-directional substring check. So a widening at (2)+(3) would pass the parity guard, fail test_ontology_has_exactly_12_entity_types_and_10_relationship_types, and leave the prompt silently un-widened.

The natural unique key. entity_mentions_canonical_name_entity_type_source_document_id_key — created in the squash as …_content_item_id_key (:8203) and renamed at 20260628200000_id131_extract_reparent.sql:43-44. entity_type is its second component. Post-reshape the surrogate PK agrees with it by construction: uuid5(ns, "em:{sd_id}:{resolved}:{type}") (DR-147 clause 1; {434.3} §2.4 step 1).

---

5. Prompt caching is declared but almost certainly never fires

_cached_system_block (extraction.py:874-902) stamps cache_control: {"type": "ephemeral"} on every extractor’s system block, closing GAP-Q-EX2-002 under ID-61.1. The documented minimum cacheable prefix on claude-opus-4-6 is 4,096 tokens. The largest of the four prompts is ~1,555 tokens (§1). A prefix under the minimum silently does not cache — no error, cache_creation_input_tokens: 0.

If that holds, ID-61.1’s stated saving is not being realised at the current model and prompt sizes, and every extractor call bills its full system prompt at the uncached rate.

This is cheap to settle and the spike gets it for free. _log_prompt_cache_usage (extraction.py:905-925) already emits one INFO line per call carrying cache_creation_input_tokens and cache_read_input_tokens. A single real-tier run answers it definitively. Stated as a finding to confirm, not as a conclusion: the token count is a chars÷4 estimate and the minimum is a documented per-model figure, so the log line is the arbiter. Recorded here because it changes the cost model in §6.4 (no cache discount is assumed) and because it is an owner-facing cost fact independent of D11.

---

6. AC2 — the run plan (DESIGN ONLY, nothing executed)

6.1 The variants

Three. A is the control; C is the shipping candidate; B isolates the pure question.

	Emission contract	Schema	Prompt delta vs A
A — bounded (control)	One of exactly twelve	Literal[...12...] unchanged	none (but see OQ-2)
B — open	”a short lower-case type label”	entity_type: str	Enum line at :168 replaced; the twelve meanings at :176-188 reframed as examples, not a menu
C — guided-open	The twelve as preferred labels in order; “if none fits, emit a new short lower-case label and use it consistently within the document”	entity_type: str	Enum line reframed as preference; meanings retained verbatim; one residual clause added

C is DR-141’s own shape applied one level down — preferred routing plus a residual that guarantees nothing is silently dropped — which is why it is the likely answer rather than a compromise. If budget or time forces two variants, run A and C; A-vs-B alone answers a question nobody proposes to ship.

6.2 How a real-tier supervised extraction is invoked today

There is exactly one real-tier path in the repository, and it is a full /walk.

1. Stage the corpus. deploy/onprem/sync-platform-corpus.sh — Hop A rsyncs the vendored tree (scripts/cocoindex_pipeline/fixtures/platform-corpus, :79) to --dest with a diff -r self-check (:139-148); Hop B (--on-prem) docker cps it into ${COCOINDEX_CONTAINER:-ca-cocoindex-platform}:${COCOINDEX_CORPUS_DIR:-/cocoindex-state/corpus} (:155-163). It never triggers the walk (:112-114).
2. Point the flow at it. COCOINDEX_SOURCE_PATH=/cocoindex-state/corpus (flow.py:4105; unset or missing ⇒ idle, and /walk 400s at server.py:1607-1618).
3. Trigger. POST /walk (server.py:2124 → _walk_handler:1529) with Authorization: Bearer $PIPELINE_TRIGGER_SECRET (:1580-1596; unset ⇒ 503, wrong ⇒ 401). Optional body field full_reprocess only (:1011-1034). Returns 202 {status, requestId, fullReprocess}; poll GET /walk-status/{request_id} (server.py:2126 → :1663) to completed / fence_busy / failed.
4. Select the real tier. There is no LLM_TIER variable in any Python code. Tier is purely the three Anthropic SDK env vars, scrubbed of empty strings at extraction.py:868-871 and then read natively by a zero-arg AsyncAnthropic(): real tier = ANTHROPIC_BASE_URL unset/empty, ANTHROPIC_AUTH_TOKEN unset/empty, ANTHROPIC_API_KEY a live key.

The existing wrapper. .github/workflows/cocoindex-nightly.yml has a workflow_dispatch input llm_tier ∈ mock|openrouter|anthropic, default mock (:307-326, :388). Its anthropic branch passes neither ANTHROPIC_BASE_URL nor ANTHROPIC_AUTH_TOKEN into the sidecar (:728-749) — exactly the real-tier shape above. It also runs a live credential preflight at minute one (:396-438) and the id-414 terminal gate afterwards (:1004-1029). It already stages the vendored corpus via the sync script (:649). This is the cheapest legitimate harness for a confirming full walk.

Env the flow reads at startup: COCOINDEX_DB_DSN (required — flow.py:1777-1789 raises at lifespan entry), COCOINDEX_DB (the engine’s LMDB path), COCOINDEX_SOURCE_PATH, PIPELINE_TRIGGER_SECRET, SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY, OPENAI_API_KEY, IMAGE_SHA, PORT, optionally PIPELINE_RUN_WEBHOOK_URL.

What does not exist, checked: no local CLI, pytest marker, or script issues real Anthropic extraction calls. Every Python test stubs the SDK behind a dummy key; the gated integration suites (KH_RUN_PG_INTEGRATION, KH_RUN_STAGE5_INTEGRATION) are Postgres-gated, not LLM-gated. POST /extract (server.py:2127 → :1973) is not an extraction seam — it is the pure Trafilatura HTML cleaner behind EXTRACT_API_TOKEN, with no LLM call in it at all.

6.3 Three execution hazards — read before spending anything

(a) Memo carry-over will silently void the comparison. extract_entity_mentions is @coco.fn(memo=True) with memo key (content_text,), and the logic fingerprint folds the function’s canonicalised AST — module, qualname, body (extraction.py:1139-1151). ENTITY_MENTION_PROMPT appears in that body as a name reference, not as its value. Therefore editing prompts.py does not change the fingerprint, and variants B and C would replay variant A’s cached payloads with no error and no warning. Widening the Literal does not help either: a widened schema still deserialises the narrower cached payloads.

Mitigations, in order of confidence: (i) a separate COCOINDEX_DB LMDB path per variant — guaranteed, zero code change, recommended. (ii) A version= bump on the extractor (the S460 mechanism, extraction.py:961-1004) — effective, but that block’s own warning says do not bump reflexively, and it burns the whole corpus for that extractor. (iii) full_reprocess=true is UNVERIFIED as a memo-buster and must not be assumed (OQ-3).

(b) The CHECK constraint blocks B and C from writing. entity_mentions_entity_type_check (squash :5933) rejects any out-of-vocabulary value — which B and C produce by design. Writes would fail per item and the id-414 terminal gate would red the run. Two ways out: drop the CHECK on a scratch database, or capture at the extractor boundary before the write.

Recommendation: capture at the boundary. A small, uncommitted harness that calls extract_entity_mentions directly over the nine converted corpus documents and writes JSON to disk needs no database, no writer fence, no Stage-5, and no walk, and it isolates the one variable under test. It is still a real-tier supervised extraction over the vendored Platform corpus, which is what the S554 verification posture requires. Two caveats to state plainly: it is a new harness and therefore wants Coordinator sign-off rather than being assumed; and it does not exercise the write path, so it cannot measure the resolution-partition effect. Hence: variants compared by direct call; one confirming /walk at the winning variant to measure partition effects end to end (§6.5 metric 4).

(c) Conversion must be reproduced, not skipped. Three of the nine corpus files are binaries (.pdf, .xlsx, .docx) that the walk converts through docling. A direct-call harness that reads raw bytes would compare variants against different inputs than the walk sees. It must run the same conversion. This matters most for Trigger 1: the manifest asserts synthetic-sector-spend.xlsx converts to 4,238 chars of table markdown, and that is the only markdown-table surface anywhere in the corpus.

6.4 Call volume and cost

Corpus (counted, docs/reference/testing/corpus-manifest.json + direct measurement): 9 files, all content documents, 47,850 bytes on disk, ~24,600 chars of extracted text (docling-realistic, taking the manifest’s 4,238-char figure for the xlsx) ≈ ~6,150 tokens total, ~683 tokens per document. Extraction calls the LLM once per document, not per chunk (mock_llm.py:109-112 records the correction).

Pricing for claude-opus-4-6: $5.00 / 1M input, $25.00 / 1M output. No cache discount is assumed (§5).

Run	Calls	Input tok	Output tok	Cost
Entity lane, one variant, one repetition	9	~20k	~14k	~$0.45
3 variants × 3 repetitions	81	~181k	~122k	≈ $4
One confirming full /walk (4 extractors × 9 docs)	36	~65k	~23k	≈ $0.90

Order of magnitude: single-digit dollars; under $20 including pilots, mistakes, and re-runs. Output is the dominant term, and it is bounded — the corpus contains roughly 20–24 distinct entities and 120–157 raw mentions in total, concentrated in five of the nine documents (three are deliberately anonymous and will return near-empty).

Cost is not a constraint on this spike. The constraints are §6.3(a), §6.3(b), and the absent reference set below.

6.5 The comparison protocol — what is measured, against what

1. Type distribution. Per variant, the multiset of emitted entity_type values across the nine documents; for B and C, the set of values falling outside the twelve and their frequencies. The decision-relevant shape of this result: a long tail of near-synonyms (org / organisation / company) is the cost of opening; a small stable set of genuinely-missing types (legislation for Procurement Act 2023, a geography type for the regions the corpus names) is the benefit. Report the shape, not just the count.

2. Mention recall and precision — and the reference problem. There is no reference set, and this is AC2’s blocker. The only document-keyed golden set this repository ever had — __tests__/fixtures/eval-gold/entity-eval-gold-standard.json, 93 items with 301 expected_entities and 302 excluded_entities — was deleted on 2026-08-04 at 14a4d4e36, together with its scorer (scripts/eval-entity-classification.ts, 918 lines) and its test. It was keyed to content_item_id (a table id-131 dropped) and described the client’s .docx bid library, not the Platform corpus. The retirement commit’s own stated reason: “0 of 140 gold IDs resolve against source_documents (measured SQL) — with no CI lane and no eval_touchpoints registration.”

scripts/quality-gate.ts is not a substitute: it measures corpus-wide presence of 12 hand-listed entities with no per-document keying and no false-positive term; its filename_matchers (FAQs, Security, Funtionality [sic]) match none of the nine corpus files; and every config file it loads was deleted at 5a3a77547, so it exits 2 before reaching the database.

Therefore the spike’s first deliverable, before any LLM call, is an authored reference set for the nine documents. Recommended shape: the deleted file’s schema — expected_entities plus excluded_entities, the latter supplying the false-positive term — rekeyed off the surviving docs/reference/testing/corpus-manifest.json fixture ids and sha256 rather than a row id. The scorer already exists and is currently dead code: lib/eval/metrics.ts::precision / recall / f1Score have zero production callers.

Three honest limits on what that reference can support, all of which belong in the verdict:

• A reference set authored by the same session that runs the comparison is a judgement artefact, not ground truth (OQ-4). Report per-document, not only aggregate, and treat the aggregate delta as directional.
• The corpus contains no GDPR, Cyber Essentials, Bribery Act, or ICO strings — precisely the vocabulary the DR-140 17-pair resolver measurement was built on. This corpus cannot reproduce that measurement, and exercises regulation only via Procurement Act 2023 (2 occurrences).
• The corpus has no source-index lists and no em-dash bullet attributions, so Triggers 2 and 3 have no fixture at all and cannot be measured by this spike on this corpus (OQ-7).

3. Type-assignment agreement. For each entity that all variants find, does the assigned type differ? This is measurable without a reference set — inter-variant agreement, with owner adjudication needed only on the disagreements, a small bounded set. It is also where the S551 standard-vs-certification question gets its first real-tier answer, since ISO 9001:2015 appears 6 times across 4 documents.

4. Resolution-partition effects — measured on the confirming /walk, not the direct-call harness. Post-reshape, resolve_entities is mounted once per entity_type present ({434.3} §2.3), so each distinct type value is a partition. Record: (a) partition count; (b) partitions of size 1 — a singleton means the resolver never runs and the mention keeps its per-document form; (c) whether any two mentions of the same real-world entity land in different partitions and therefore can never merge. Item (c) is the structural cost made concrete and is the single most decision-relevant number the run can produce.

5. Prompt-cache counters. Free, from the existing INFO line — settles §5.

6.6 The verdict must address the two structural costs the board named

Cost 1 — entity_type is a component of the natural unique key. entity_mentions_canonical_name_entity_type_source_document_id_key (20260628200000_id131_extract_reparent.sql:43-44), and post-reshape the surrogate primary key agrees with it by construction — uuid5(ns, "em:{sd_id}:{resolved}:{type}") (DR-147 clause 1; {434.3} §2.4 step 1).

The consequence, stated precisely, and it is not on the board. DR-147 clause 2 — a re-run may grow a group but never rename it — constrains only the resolved canonical. It says nothing about the type. So an unstable type label reaches the same failure through the other axis: if a re-run types the same entity standard on one walk and certification on the next, the row’s primary key changes, and under cocoindex’s contract a changed declared key is a clean delete-and-recreate. That is exactly the whole-corpus churn DR-147 exists to forbid — “correct by the engine’s contract, wrong as product behaviour” — arriving by the type axis. Opening the vocabulary widens the space in which that can happen.

The verdict must pick one and say so: (a) extend the DR-147 stability rule to the type (types may be added, never re-labelled, with a probe as an AC); (b) remove entity_type from the key; or (c) keep the vocabulary closed as a dated deviation. Flag that (b) touches id-434’s shipped identity design and is therefore not this spike’s to take — routing it back is the correct move, not deciding it.

Cost 2 — entity_type partitions resolution. {434.3} §2.3: one mounted resolve_entities child per type present, each seeded from SELECT DISTINCT canonical_name FROM entity_mentions WHERE entity_type = $T plus effective-type pins. Two spellings of one entity typed differently sit in different partitions and cannot be resolved to one canonical by construction — no amount of resolver quality closes it. Opening the vocabulary increases partition count and therefore the probability of the split. Metric 4(c) quantifies it.

The interaction the owner should see: DR-141’s answer to the same problem one level up was a residual grain, and id-451 is building the entity-level equivalent. A residual fixes coverage. It does not fix partition. DR-141’s own rider states that coverage and resolution are orthogonal; this cost is on the resolution side, and id-451 does not absorb it.

Cost 3, not on the board, surfaced here — the effective-type predicate. All pin reads align to COALESCE(entity_type_override, entity_type) (D7; {434.3} §2.5; squash :2574, :2760, :2766), and entity_type_override carries no CHECK constraint of its own. Whatever the verdict does to the base column must be decided for the override in the same breath, or the override becomes a silent bypass of whatever bound survives — and it is the column a human curator writes.

---

7. Open questions and UNDECIDABLEs

• OQ-1 — Does this spike’s disposition scope include the orphaned holder block? UNDECIDABLE RULED (owner, S555): the block is strip-class complexity — retired in full at canonical 3303f992a. One attribution sentence survives (source is whoever the content names); the three port-fidelity tests pinning the orphan were replaced by a retirement guard. Nothing routes to id-435.
• OQ-2 — Fix {CLIENT_ORGANISATION_NAME} before the baseline run? MOOT (S555): the token died with the block. Variant A’s control is RELATIONSHIP_PROMPT at 3303f992a — no HEAD+1 caveat needed.
• OQ-3 — Does full_reprocess=true bust the @coco.fn memo? UNVERIFIED. Cheap to settle with a mock-tier probe before spending anything real-tier. Until settled, §6.3(a) mitigation (i) — a per-variant LMDB path — is the only assumption-free option.
• OQ-4 — Who authors the reference set? open CLOSED (owner, S555): the Coordinator-authored set (canonical 35e615465) was owner-reviewed on the s555-entity-reference-review lavish board — all nine document cards confirmed and the APPROVE verdict recorded. The real-tier comparison run is authorised against it.
• OQ-5 — Does the verdict bind entity_type_override? See §6.6 Cost 3. Owner-facing.
• OQ-6 — The Q&A lane returns [] on this corpus by design yet costs ~860 system tokens per document per walk. In scope for disposition here, or id-417/id-45 ground? Flagged, not resolved.
• OQ-7 — Corpus adequacy. Triggers 2 and 3 have no fixture in the Platform corpus and cannot be measured by this spike. Does the owner want per-test-content fixtures authored to exercise them, or are they dispositioned unmeasured on the strength of §2.5’s provenance finding? This is the one AC1 item that cannot be closed by measurement without new corpus content.
• OQ-8 — Sections with no source found at all (§2.1 rows 4, 7, 8; §2.3 rows 2 and 4; §2.4 row 1): each is a candidate for disposition under AC1’s own terms. [S555 partial ruling: §2.1 row 4 ADDRESSED — the canonical_name prompt guidance is deleted at canonical 7b54b421d (verified: the emitted field had zero readers; schema field kept tolerated-and-discarded under extra="forbid", retiring with {434.3} §2.2). Rows 7 (type meanings) and the tabular section: OWNER CONFIRMED (S555) — both stay through the spike; the meanings are the semantics the D11 variants reshape (menu → examples), and the tabular section gets a ±-axis measurement on the leading variant rather than a blind delete. OQ-4’s reference set is drafted at canonical 35e615465 (docs/reference/testing/entity-extraction-reference.json), awaiting owner review.] Retaining any of them requires naming a current source; if none can be named, the verdict is UNDECIDABLE and it stays only until the owner rules (the id-434 AC2 pattern).

---

8. What is NOT evidence here

• The live type distribution in Platform staging or prod. Every entity row is mock-tier, and the mock hardcodes two type values (§3). Reading the DB’s standard-vs-certification split as extractor behaviour is the DR-104 rows-in-a-dev-DB trap, one step removed.
• A green test_cocoindex_prompts.py. 27 of its 28 cases assert substrings in a Python string. A prompt section can be wrong, orphaned, or actively harmful and stay green (§2.5) — and three of those tests exist only to pin an orphan in place.
• “The pipeline runs fine today.” It runs on the mock tier, which cannot fail the judgement being questioned.
• Grep absence for the tabular rules’ provenance. Absence prompted the look; the finding is that the gate they satisfied was port it, recorded in id-64’s own close-out, and that the underlying measurement targeted a pipeline that no longer exists (§2.5).
• A prior task’s disposition. id-93 recorded bl-92 as “N/A” and “retired”; ID-101 and ID-109 then built it anyway (§2.2). A recorded retirement is evidence of intent at a point in time (DR-123/DR-139), not of current state.
• The number of call sites touching the twelve values. Consumer-counting. Four enforcement points is a measure of how far the shape spread, not of whether it is right.