Cocoindex Extraction Contract — Verifier Report (S242 WP1.1)

Reviewer: worktree-agent-a3039e3ae1c7043b5 (verifier) Drafter commit under review: c62c1c1c (worktree branch worktree-agent-ab3eb54a8d155f21f) Base HEAD: 70336a3c (S242 continuation prompt head; branch content-items-investigation) Spec files under review:

docs/specs/id-36-cocoindex-extraction-contract/PRODUCT.md (96 lines, 26 numbered Behavior invariants)
docs/specs/id-36-cocoindex-extraction-contract/TECH.md (487 lines)

Verdict: FAIL — three blocking substantive shape errors must be fixed before merge. Citation-discipline floor is otherwise high; the layered fn-shape framing, gap-flag structure, and ratification provenance are well-grounded.

Axis-by-axis findings

1. Brief compliance — PASS-WITH-NOTES

Every acceptance criterion from docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md §4.1 T1.1+T1.2 is touched:

Discriminated-union 3 variants (PRODUCT inv 1; TECH §2.1) — present.
ExtractByLlm typed output_type citation — present (PRODUCT inv 1 cites phase-b-prerequisite-2a-cocoindex-examples.md §“Capability: cocoindex.functions.ExtractByLlm” lines 311-329; verified accurate at source).
Layered fn-shape inner-tier content_text: str — present (PRODUCT inv 12, TECH §3.1 with citation to S9 spike §7.1).
Validation rules — PRODUCT inv 11/13/14/18/19 cover pre-UPSERT validation, failure ledger row, failure non-terminal supersession, no-retry-on-mismatch, invalid_discriminator severity.
Test plan — TECH §3.1-§3.6 maps each Behavior invariant to a test type with table format.
Gap-flags §5.3 format — three flags (GAP-Q-EX2-001/002/003) all carry Category + Sources checked + Prerequisite work fields (verified against construction-guide §5.3 pattern).
UK English — clean (no color/organization/favorite/criticize/analyze/optimize/MM-DD-YYYY found).
Ratification header — present on both files (line 1: RATIFIED-S241 — Q-EX2 discriminated-union direction...).

Note: Drafter’s claim of “26 invariants” matches reality (1-26 numbered). Drafter’s claim of “3 gap-flags” matches reality.

2. No-fabrication — FAIL

Spot-checked 9 citations (6 PRODUCT + 3 TECH). Findings:

lib/validation/schemas.ts:1495-1508 (PRODUCT inv 3, TECH §2.1 line 147) — VERIFIED. 12 values exactly as listed (organisation/certification/regulation/framework/capability/person/technology/project/sector/product/standard/methodology).
lib/validation/schemas.ts:43-52 (PRODUCT inv 4, TECH §2.1 line 168) — VERIFIED. VALID_CONTENT_TYPES = CONTENT_TYPE_VALUES re-export at line 52.
lib/validation/schemas.ts:899 (PRODUCT inv 7) — VERIFIED. Line 899 contains ['q_a_pair', 'case_study', 'policy', 'methodology', 'capability']; the 4 non-form types cited are correct.
phase-b-prerequisite-2a-cocoindex-examples.md lines 207-221 (PRODUCT inv 2 source citation) — PARTIALLY ACCURATE. Lines 207-221 do contain the illustrative Pydantic shape, but TECH §2.1’s FormMetadata / QAPair shapes drift from the source — see Axis 6 below.
phase-b-prerequisite-2a-cocoindex-examples.md lines 311-329 (PRODUCT inv 1, inv 11, inv 21) — VERIFIED. Capability section title + the cache-hash claim at line 329 (“Memoised: output_type + instruction + LLM model + input markdown all participate in the code-hash. Changing the schema or instruction → all rows re-extract.”) match exactly.
02-data-flow.md §3.1 (PRODUCT inv 1, inv 6, inv 8, inv 10) — VERIFIED. 6-stage topology table with cocoindex primitives + KH purpose columns matches.
02-data-flow.md §5.1 N7 hybrid op_id (PRODUCT inv 5, inv 15) — VERIFIED. §5.1 carries the N7 RESOLVED-S236 ratification language exactly.
02-data-flow.md §6 (TECH Context line 14, lib/extraction/ retire claim) — CITATION ERROR. TECH Context line 14 reads RATIFIED-RETIRE-POST-PHEW-MIGRATION per 02-data-flow.md §6 + phase-b-prerequisite-2-cocoindex-deep-dive.md §4 Recommendation 1. §6 of 02-data-flow.md is “RLS pattern — forward-reference to RLS-PATTERN spec pair”, not Pattern A/B retire. The Pattern A/B retire lives at §10.1 (“Pattern A/B Q&A parser retire (B1 RESOLVED)”). The deep-dive §4 Recommendation 1 reference is correct; the 02-data-flow.md §6 reference is wrong. (Non-blocking — accurate ratification exists at §10.1; citation just needs updating.)
docs/specs/core-docs-pathway-assessment/S239-still-open-consolidation.md lines 216-219 (PRODUCT line 5 ratification provenance) — VERIFIED. Row 6 (Q-EX2) at line 219 carries the CATEGORISED-S241 tech-spec-needed verdict with destination docs/specs/id-36-cocoindex-extraction-contract/{PRODUCT,TECH}.md.

FAIL trigger: TECH §2.1 QAPair.question_kind: Literal["mandatory", "optional", "info_only"] includes info_only which appears NOWHERE in source citations. The illustrative deep-dive at line 341 says Literal["mandatory", "optional", "info-only"] (note: hyphen info-only, not snake-case info_only), and line 212 says Literal["mandatory", "optional"] (two values, no third). A repo-wide grep -rn "info_only" returns ZERO hits across docs/, lib/, and scripts/. The value is fabricated (or, at best, snake-cased from a deep-dive pseudo-code shape that itself lacks ratification). See blocking issue [B-1] below.

3. Gap-flag format (§5.3) — PASS

All three gap-flags [GAP-Q-EX2-001/002/003] carry the required triplet (Category + Sources checked + Prerequisite work). Concrete check:

GAP-Q-EX2-001 (routing content_types post-T2): Category=schema alignment; Sources checked = lib/validation/schemas.ts:899, docs/ontology/26-form-type.md line 85, PLAN.md §4.2 T2 subtask 1; Prerequisite work named (T2 lands form_types CV instance table). All verified at source.
GAP-Q-EX2-002 (Anthropic prompt-cache passthrough): Category=cost-projection verification; Sources checked = deep-dive §5 + S239 closure row 7; Prerequisite work = T13 spike per PLAN.md §4.13 subtask 1. Citations verified.
GAP-Q-EX2-003 (nested-schema depth limits): Category=capability verification; Sources checked = deep-dive lines 191-204; Prerequisite work = cocoindex 1.0.3 docs read against LlmApiType.ANTHROPIC. Citation 191-204 anchors the patient_intake nested-dataclass example (verified).

No bare TBD/XXX/TODO/FIXME placeholders found in either file (grep clean).

4. Tag vocabulary discipline — PASS

[RATIFIED-S241] appears 23 times across PRODUCT.md (correctly applied to invariants 1-10, 11-19, 21-26 — all behavior + anti-pattern invariants). [DEFERRED-v1.1] correctly applied to inv 24 (Ollama/OpenAI). [GAP-Q-EX2-NNN] used per §5.3 format. No [LOCKSTEP-...] is needed in this spec (it’s a contract spec, not a migration). No mis-applied tags found.

5. Feature-spec pattern enforcement — PASS

PRODUCT.md uses Summary + Behavior (numbered invariants) shape per write-product-spec skill. TECH.md uses Context + Proposed changes + Testing + Risks + Follow-ups shape per write-tech-spec skill. Neither file uses the construction-guide §2 P-N pattern. Critical-rule 5 (S241) compliance: confirmed.

6. Pydantic v2 + ExtractByLlm correctness — FAIL

Two substantive shape errors:

(a) QAPair.question_kind vocabulary collides with the canonical question_kind discriminator on question_matches. PRODUCT inv 2 reads “question_kind from form_types” — but form_types CV holds form-type values (bid/rfp/pqq/etc. per docs/ontology/26-form-type.md line 52). TECH §2.1 line 125 then defines question_kind: Literal["mandatory", "optional", "info_only"] — a different vocabulary (and info_only is fabricated, per axis 2 above). The canonical architecture per 05-qa-flow.md §7.2 explicitly aligns the name question_kind to the form_types vocabulary on the question_matches table; using the SAME column name question_kind on the new QAPair extraction shape with a different vocabulary creates a downstream naming collision when an extracted QAPair row is later joined to or projected through question_matches. See blocking issue [B-1].

(b) FormMetadata.form_type Literal is incomplete vs the canonical CV register. TECH §2.1 lines 102-105: Literal["bid", "rfp", "pqq", "itt", "tender", "framework", "dps", "gcloud"] — 8 values. The canonical CV at docs/ontology/26-form-type.md lines 65-79 has 11 baseline values (adds checklist, questionnaire, sales_proposal_template). Line 52 of that ontology doc names only the 8 procurement values (the Q-OQR1-02 ratification subset), but 26-form-type.md line 50 explicitly says non-procurement forms (checklist, questionnaire, sales_proposal_template) are CORE for their respective applications. The §3.4 parity test in TECH (line 454) claims form_type (8 values) ... docs/ontology/26-form-type.md line 52 (CV markdown register) — but 26-form-type.md is NOT the CV markdown register, and the register at lines 65-79 has 11 values. The parity test would FAIL on first run. See blocking issue [B-2].

(c) ExtractByLlm.transform(content_text) is API-incorrect. TECH §3.1 lines 235-242 wraps ExtractByLlm(...).transform(content_text) inside an @coco.fn(memo=True) async def extract_classification(...) body, passing a raw str to .transform(). The canonical cocoindex pattern (verified at phase-b-prerequisite-2a-cocoindex-examples.md lines 196-203, 362-368) places ExtractByLlm at flow level: doc["markdown_sidecar"].transform(cocoindex.functions.ExtractByLlm(...)) — .transform() is invoked on a flow column/data-source, not on a raw Python value inside an @coco.fn async wrapper. The S9 spike inner-fn shape (line 68-95) uses pure Python work in @coco.fns, NOT wrapped ExtractByLlm calls. As written, the spec’s example code would not execute correctly against cocoindex 1.0.3. See blocking issue [B-3].

Pydantic v2 surface checks (PASS):

ConfigDict(strict=True, extra="forbid") correctly applied per Pydantic v2 idiom.
Annotated[Union[...], Field(discriminator="extraction_kind")] matches the Pydantic v2 discriminated-union API.
model_copy(update={...}) (§3.2 line 319) matches Pydantic v2 model-copy semantics.
model_validate_json(...) test references (§3.1 row 2) match Pydantic v2 API.

7. CLAUDE.md gotcha application — PASS-WITH-NOTES

cocoindex 1.0.3 requires dangerouslyDisableSandbox: true — applied (TECH §7 bullet 1).
localfs.walk_dir defaults recursive=False — applied (TECH §7 bullet 2, marked as orthogonal but flagged for T8).
Python background output: PYTHONUNBUFFERED=1 — applied (TECH §7 bullet 3).
content_items.content_text_hash is GENERATED ALWAYS — NOT mentioned. PRODUCT inv 4 says classification writes back content_items.content_type / primary_domain / classification_confidence; the contract should explicitly call out that content_text_hash is omitted from any UPSERT payload per the gotcha. The spec writes content_items columns but doesn’t disclaim the GENERATED ALWAYS column — non-blocking but worth a one-line gotcha note in §5 or §7.
summary not ai_summary — NOT applicable (spec doesn’t touch summary column).
Embedding-vector JSON.stringify for RPC — NOT applicable (Python pipeline doesn’t go through RPC at extraction time per the architecture).
Entity classification false-positives ≠ type errors — applied (PRODUCT inv 9 cites the gotcha and the entity-type-taxonomy-spec; mention_confidence threshold framing aligns with false-positive guard).
classifyContent userId must be a UUID — flagged as legacy path (TECH §7 bullet 4) but the production gotcha was the literal-string bug; the spec correctly notes it doesn’t apply to cocoindex extraction.

8. Test plan quality — PASS-WITH-NOTES

Every numbered Behavior invariant from PRODUCT.md maps to at least one row in the TECH §3 tables. AST-scan enforcement is concrete enough to verify:

§3.5 row 2 (“no extract-everything variant”): “any @coco.fn whose return annotation is ExtractionOutput (the root union) instead of a single variant fails the test” — this is verifiable with a mypy + ast.walk check on function annotations.
§3.5 row 1 (“no raw_llm_response columns”): migration-review checklist + knip scan — pragmatic.
§3.5 row 3 (“no LlmApiType.OLLAMA/OPENAI”): explicit grep -n "LlmApiType\.\(OLLAMA\|OPENAI\)" scripts/cocoindex_pipeline/ regex provided — directly runnable.

Note 1: TECH §3.1 row 5 (_ExtractionBase fields test) cites strict-mode UUID parsing, but uuid.UUID accepts any RFC-4122 string by default in Pydantic v2 — the test as written would only catch genuinely malformed UUIDs, not the v1/v4 distinction the CLAUDE.md “Zod UUID validation” gotcha implies. Non-blocking but worth nuancing.

Note 2: Inv 9 (confidence threshold) is tested at “Mock entity-resolution stage” (§3.3 row 4) — but the contract spec doesn’t OWN the entity-resolution stage (per PRODUCT inv 8, that stage emits raw mentions; the threshold is applied at write-back). The test as written would have to mock a stage outside the contract’s scope — worth a note that the threshold test belongs to the T8 implementation, not this contract’s acceptance.

9. UK English — PASS

10. Forward-ref discipline — PASS-WITH-NOTES

PRODUCT and TECH reference the canonical pipeline P-1/P-2 framing without restating (PRODUCT line 5 ratification provenance, TECH §1 Context line 11 architecture-flow citation). The 6-stage topology is cited via pointer to 02-data-flow.md §3.1 (e.g. PRODUCT inv 1, inv 6, inv 8) without restating the table — verified clean. Note: TECH §3.4 parity-test table inlines form_type (8 values) — but the form_type Literal value list is itself the canonical-vocabulary spec for the LLM contract, so inlining is appropriate; the issue is the count drift (Axis 6b above), not forward-ref discipline.

Blocking issues (must-fix before merge)

[B-1] TECH §2.1 line 125 — QAPair.question_kind: Literal["mandatory", "optional", "info_only"] is fabricated and collides with the canonical question_kind form-type discriminator on question_matches. Fix paths:
- (a) Rename the field to a non-colliding name (e.g. question_obligation with values mandatory/optional/info_only) AND ratify the new vocabulary with citation, OR
- (b) Drop the field entirely from v1 and surface as [GAP-Q-EX2-004] pending ratification, OR
- (c) If mandatory/optional/info-only is the genuine intent (lifted from the deep-dive pseudo-code), explicitly flag the source (phase-b-prerequisite-2a-cocoindex-examples.md lines 207-221 + lines 335-355 — pseudo-code, NOT ratified) and add a separate ratification line. Note also the snake/kebab inconsistency (info_only vs source’s info-only).
- PRODUCT inv 2 sentence “question_kind from form_types” is wrong regardless — form_types CV does not contain mandatory/optional values.
[B-2] TECH §2.1 lines 102-105 + §3.4 row 3 — FormMetadata.form_type Literal omits the 3 non-procurement values (checklist, questionnaire, sales_proposal_template) from the canonical CV at docs/ontology/26-form-type.md lines 65-79. The §3.4 parity test would fail on first run. Fix paths:
- (a) Add the 3 missing values to the Literal AND update PRODUCT inv 2 (“procurement form… ”) to clarify the full v1 routing scope, OR
- (b) Scope the v1 contract explicitly to procurement-only forms (cite Q-OQR1-02 at line 52 for the 8-value subset) and surface the non-procurement form_type routing as [GAP-Q-EX2-NNN] until the sales-proposal application_type spec lands.
- Either way, also fix PRODUCT inv 2’s list of 7 values (omits tender from the procurement subset — tender IS in line 52, the spec just missed transcribing it).
[B-3] TECH §3.1 lines 235-242, 256-263, 278-285 — ExtractByLlm(...).transform(content_text) invoked inside an @coco.fn(memo=True) async def ... is API-incorrect. The canonical pattern (verified at phase-b-prerequisite-2a-cocoindex-examples.md lines 196-203 + 362-368) places ExtractByLlm at flow-scope on a data-source column, not inside an async wrapper. Fix: either
- (a) restructure the example to show @coco.fn-decorated PURE Python functions (e.g. preprocessing) and a separate flow-scope doc["..."].transform(ExtractByLlm(...)) for the LLM extraction (with the prompt-template parameterisation moved to flow assembly), OR
- (b) cite the cocoindex 1.0.3 API surface that justifies the .transform(content_text) pattern on a raw str argument — if such a surface exists (e.g. cocoindex 1.0.4 added a Python-callable wrapper), the spec must cite the source.
- Without the fix, every consumer of this spec (notably T8 implementation) would copy non-functional reference code.

Non-blocking notes (should-fix, won’t block merge)

[N-1] TECH §1 Context line 14 cites 02-data-flow.md §6 for lib/extraction/ Pattern A/B retire. §6 of that file is “RLS pattern — forward-reference”. Correct citation is §10.1 (Pattern A/B Q&A parser retire). Verified at source.
[N-2] TECH §3.1 line 213 hard-codes ANTHROPIC_MODEL = "claude-opus-4-7" with the comment “Per CLAUDE.md project memory (Opus 4.7 1M context)”. CLAUDE.md mentions Opus 4.7 1M context as the Claude Code session model, NOT as the pipeline production model. Production usage (verified in repo: scripts/kb_pipeline/config.py:29, lib/anthropic.ts:29, scripts/run-1m-context.ts:66, scripts/tests/test_config_env_vars.py:28) is claude-opus-4-6. The cocoindex-examples.md line 365 uses claude-opus-4-7 only in a forward-looking pseudo-code snippet, not as a ratified production reference. Fix: either cite the actual production-model constant CLASSIFICATION_MODEL from scripts/kb_pipeline/config.py, or surface a [GAP-Q-EX2-NNN] flag asking which model to target at v1.
[N-3] TECH §7 should add a one-line note that content_items.content_text_hash is GENERATED ALWAYS per the CLAUDE.md gotcha — the classification variant writes back content_items columns and downstream T8 implementers will need the reminder.
[N-4] PRODUCT inv 2 says “question_kind from form_types” — clarify whether the intent is the form_type discriminator (per the canonical question_matches.question_kind naming) or a per-Q&A obligation field (the TECH §2.1 invented vocabulary). They are different concepts and the spec text should pick one.
[N-5] TECH §3.1 row 5 _ExtractionBase UUID test plan: Pydantic v2 strict-mode UUID validation accepts any RFC-4122 form; the CLAUDE.md “Zod UUID validation is strict” gotcha is TypeScript-side and does NOT apply to Python. Re-frame the test as “UUID parse error” rather than “v4-compliance”.
[N-6] TECH §3.3 row “Inv 9 — confidence threshold” tests an entity-resolution-stage filter that lives OUTSIDE this contract’s scope (per PRODUCT inv 8, inv 9). The test would have to mock a downstream stage, which is brittle. Push this test row to the T8 acceptance criteria and remove from the contract’s test plan.

Suggestions (optional improvements)

[S-1] PRODUCT inv 5 names three outer-tier fields (op_id, content_items_id, extracted_at). Consider also naming extractor_kind as the link to q_a_extractions.extractor_kind enum — currently the enum lineage is only mentioned in TECH §3.1 row 4 + PRODUCT inv 2 (extractor_kind='llm_extraction'). Surfacing it in the invariants makes the downstream UPSERT-target binding explicit.
[S-2] TECH §4.1 “Pydantic strict-mode handling” table is excellent — consider adding the inverse mapping (a Pydantic ValidationError to error_class string) as a typed enum/dict in code, not just a doc table. Otherwise the _classify_pydantic_error(exc) helper at §4.2 line 363 is implicit.
[S-3] TECH §3.4 enum-parity guard table is good. Consider promoting it to a §3.4-pre-launch checklist with a specific runbook entry — these parity tests are the load-bearing guard against silent drift, and a one-paragraph runbook (“run python3 -m pytest scripts/tests/cocoindex_pipeline/test_*_parity.py after every change to lib/validation/schemas.ts or docs/ontology/26-form-type.md”) would help future agents.
[S-4] Both files could benefit from a forward-pointer to docs/specs/id-36-cocoindex-ledger-api/TECH.md (sibling T1.3) being more prominent — currently the cross-reference is at TECH Follow-ups line 484 only. Drafter brief said this spec’s failure-path depends on the sibling spec’s record_extraction_failure signature; surfacing the dependency in §1 Context or §4.2 would help reviewers.

Overall verdict rationale

The drafter has produced a substantively grounded spec pair with strong citation discipline at the macro level: ratification provenance is correct, gap-flags follow §5.3 format, layered fn-shape is captured per S9 spike §7.1, UK English is clean, no inline TBD/placeholders, and the discriminated-union direction matches Q-EX2 closure row 6. The Behavior invariants are testable, the test plan maps to invariants, and the AST-scan enforcement strategy is concrete.

However, three blocking shape errors prevent merge: (1) the QAPair.question_kind vocabulary is fabricated AND collides with the canonical question_kind discriminator name; (2) FormMetadata.form_type omits 3 of the 11 canonical CV values, which would fail the spec’s own parity test; (3) the ExtractByLlm(...).transform(content_text) pattern as written is API-incorrect per cocoindex’s documented surface and would mislead the T8 implementation. These are not style issues — they are substantive contract claims that would propagate into the production pipeline if not caught.

The N-tier non-blocking notes (citation typos at §6 vs §10.1, model-name conflation claude-opus-4-6 vs claude-opus-4-7, missing content_text_hash gotcha note) are easy to fix and do not block independently — they would have made it PASS-WITH-NOTES were the three B-tier issues absent.

Recommended next step: drafter agent fixes the three blocking issues — most importantly clarifying the question_kind vocabulary collision and the ExtractByLlm integration pattern — then resubmits for a second verifier pass. The framing, ratification header, and test-plan structure can stay as-is.