Skip to content
Canonical

Cocoindex Extraction Contract — PRODUCT

Cocoindex Extraction Contract — PRODUCT

Section titled “Cocoindex Extraction Contract — PRODUCT”

Status: RATIFIED-S241 (Q-EX2 — discriminated-union Pydantic with ExtractByLlm typed output_type). Ratification source: docs/specs/core-docs-pathway-assessment/S239-still-open-consolidation.md lines 216-219 (S241 closure addendum row 6). Originating recommendation: docs/plans/phase-0-investigation/phase-b-prerequisite-2-cocoindex-deep-dive.md §4 Recommendation 1 (“single biggest missing primitive in KH plan”).

Summary

Section titled “Summary”

The cocoindex extraction contract defines the typed Python schemas that govern every LLM extraction in the Knowledge Hub pipeline. A single discriminated-union root type — ExtractionOutput — is supplied to cocoindex’s ExtractByLlm primitive as output_type, so each LLM response is validated against the declared shape before any Postgres UPSERT. Three variants are mandatory at v1: a typed Q&A form extraction (one form metadata block plus zero-or-more Q&A pairs), a typed entity-mention extraction (named-entity dedup substrate per the canonical 12-value entity taxonomy), and a typed classification result (content_type + primary_domain + confidence). Every extraction carries a discriminator field, a stable correlation handle (op_id), and source-attribution back to the content_items row that triggered it. Validation failures are loud (the extraction is recorded via pipeline_runs with a typed error class and no partial write reaches the target tables); silent fallback to untyped output is forbidden.

Behavior

Section titled “Behavior”

Each numbered invariant below is [RATIFIED-S241] unless explicitly tagged otherwise. Per CLAUDE.md no-fabrication discipline, source citations are inline (file:line where possible). Gap-flag format per construction-guide §5.3 used for genuine open items.

Schema shape — the discriminated union

Section titled “Schema shape — the discriminated union”

  1. ExtractionOutput is a discriminated-union Pydantic root type. [RATIFIED-S241] Every LLM extraction call in the cocoindex pipeline (per the 6-stage topology in docs/plans/phase-0-investigation/architecture/02-data-flow.md §3.1 — “LLM extraction” stage) supplies output_type=ExtractionOutput (or a sub-variant) to its ExtractByLlm invocation. The discriminator field is extraction_kind: Literal["q_a_form", "entity_mention", "classification"]. No untyped or dict[str, Any] extraction shape is permitted in v1. Source: phase-b-prerequisite-2a-cocoindex-examples.md §“Capability: cocoindex.functions.ExtractByLlm” lines 311-329 (the ExtractByLlm capability section confirms output_type as a typed Python class — dataclass, Pydantic, NamedTuple — with nested-schema support).

  2. The q_a_form variant carries form metadata plus zero-or-more Q&A pairs. [RATIFIED-S241] When the extractor fires against a procurement form (one of the eight procurement form_type values — bid / rfp / pqq / itt / tender / framework / dps / gcloud per docs/ontology/26-form-type.md lines 65-79) or a non-procurement form (checklist / questionnaire / sales_proposal_template per the same canonical CV) or against a curated Q&A markdown sidecar, the response shape is one FormMetadata block (form_type, form_format, optional form_title, optional issuing_organisation, optional deadline) plus a list of QAPair blocks (question_text, optional answer_text, expected_response_kind, optional evaluation_criteria, optional evidence_requirements). The q_a_form variant binds to a single downstream target: q_a_extractions (per Q&A pair, with extractor_kind='llm_extraction' per 05-qa-flow.md §3.2). Its FormMetadata block is not persisted by this LLM path — form_templates / form_template_fields are written by the deterministic Path-B form extractor (ExtractedForm, ID-52), which Inv-19 keeps disjoint from the Path-A Q&A write. [Factual reconciliation S287 / bl-184(a): the original “per FormMetadata → form_templates (where Theme A wiring applies)” binding never materialised through the LLM variant; form-table writes flow only from Path-B. See docs/research/s286-bl184-formmetadata-persistence-gap.md §1.6.] Source for the full 11-value form_type enum: docs/ontology/26-form-type.md lines 65-79 baseline values table.

    expected_response_kind rename rationale. The drafter spec named this field question_kind, which collides with question_matches.question_kind per docs/plans/phase-0-investigation/architecture/05-qa-flow.md §7.2. That column is the form-type discriminator on the question_matches table — it holds form-type values (bid / rfp / pqq / etc.). To avoid name collision when extracted Q&A rows project into or join against question_matches, this contract renames the per-Q&A obligation field to expected_response_kind. Canonical values are Literal["mandatory", "optional"] per the ratified shape at phase-b-prerequisite-2a-cocoindex-examples.md line 212 (the deep-dive’s Pydantic illustration). A third “info-only” variant appears in pseudo-code at line 341 of the same file but is NOT ratified and is omitted from the v1 closed enumeration; if downstream procurement workflows surface a third state, it lands via spec-update with citation rather than via prompt drift.

  3. The entity_mention variant carries one named entity plus its source span. [RATIFIED-S241] Each entity_mention extraction names exactly one entity from the canonical 12-value taxonomy in lib/validation/schemas.ts:1495-1508 (organisation, certification, regulation, framework, capability, person, technology, project, sector, product, standard, methodology). The variant carries entity_type, entity_name (raw mention text), optional canonical_name (pre-resolution), source-span character offsets within content_text (start + end), and a mention_confidence float. This variant binds downstream to entity_mentions (per 02-data-flow.md §3.1 “Entity resolution” stage). Source: 12-value entity taxonomy per lib/validation/schemas.ts:1495-1508 + docs/reference/entity-type-taxonomy-spec.md (cited in CLAUDE.md Gotcha “Entity classification: false positives, not type errors”).

  4. The classification variant carries content_type, primary_domain, and confidence. [RATIFIED-S241] Each classification extraction declares a content_type from the closed enumeration in lib/validation/schemas.ts:43-52 (VALID_CONTENT_TYPES re-exported from lib/ontology/content-type-registry.ts), a primary_domain string (free-form but constrained by the per-workspace domain taxonomy at extraction time), and a classification_confidence float in [0.0, 1.0]. Optional fields: secondary_classifications (list of content_type values for multi-topic content) and rationale (model-supplied free text). This variant binds downstream to content_items.content_type + content_items.primary_domain + content_items.classification_confidence. Source: lib/validation/schemas.ts:43-52 (existing content_type enum); supersedes the bespoke classify.ts chain per 02-data-flow.md §3.3 RATIFIED-RETIRE classification telemetry.

  5. Every variant carries op_id, content_items_id, and extracted_at. [RATIFIED-S241] Each ExtractionOutput instance carries an op_id: UUID (the cocoindex per-flow op_id per 02-data-flow.md §5.1 hybrid op_id pattern + N7 RESOLVED-S236), a content_items_id: UUID (FK to the content_items row whose content_text was the extraction input — this is the source-attribution marker per CLAUDE.md no-fabrication discipline), and extracted_at: datetime (UTC timestamp set by the extractor at LLM-call time). These three fields are populated by the outer-tier cocoindex flow wrapper, not by the LLM — the model does not generate them.

When each variant fires

Section titled “When each variant fires”

  1. The classification variant fires for every newly-ingested content_items row. [RATIFIED-S241] The classification stage runs on every content_text that enters the cocoindex flow (per the 6-stage topology in 02-data-flow.md §3.1). It supersedes the pre-cocoindex classifyContent path per 02-data-flow.md §3.3 RATIFIED-RETIRE roadmap §3 classification telemetry. The output rewrites content_items.content_type / primary_domain / classification_confidence for the row. Re-runs on subsequent ingest only when content_text changes (per the inner-tier memo behaviour established in invariant 12).

  2. The q_a_form variant fires only when the content_type indicates a form or Q&A sidecar. [RATIFIED-S241] Cocoindex routes a content row to the q_a_form extractor when the classification stage (or the manual ingest seed) declares content_type in {q_a_pair, plus form variants per form_types CV in docs/ontology/26-form-type.md}. Non-form content (case_study, policy, methodology, capability per lib/validation/schemas.ts:899) does NOT route through this extractor. Open question: the exact set of routing content_types depends on the form_types CV instance table landing in T2 of the canonical-pipeline PLAN (per 04-workspace-types.md §8 item 7) — pre-launch this list is constrained to the four legacy content_types until the CV table lands. [GAP-Q-EX2-001] Category: schema alignment. Sources checked: lib/validation/schemas.ts:899, docs/ontology/26-form-type.md lines 65-79, docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md §4.2 T2 subtask 1. Prerequisite work: T2 (Q-OQR1-16 combined PR) lands the form_types CV instance table; routing finalises post-T2.

  3. The entity_mention variant fires when the entity-resolution stage runs. [RATIFIED-S241] Per 02-data-flow.md §3.1 “Entity resolution” stage, the cocoindex pipeline emits zero-or-more entity-mention extractions per content_text. The extractor runs on every content row regardless of content_type (entities can appear in any text), but invariants 9-10 bound which mentions are persisted. Selective adoption per phase-b-prerequisite-2-cocoindex-deep-dive.md §4 Recommendation 3: named-entity dedup is in scope (organisation / person / product / technology per Phew domain); Q&A pair dedup and content_items dedup remain KH-side.

  4. Entity mentions below mention_confidence threshold are not persisted. [RATIFIED-S241] Each entity_mention whose mention_confidence < 0.5 (threshold ratified at the entity-resolution stage per Phew’s named-entity domain — see CLAUDE.md Gotcha “Entity classification: false positives, not type errors”) is dropped before write-back to entity_mentions. The threshold is configurable per entity_type but defaults to 0.5 globally. The filter is applied at the entity-resolution stage (downstream of this contract per invariant 8 + §3.1 row 5) — not at the LLM extraction stage. This invariant therefore documents the persistence rule; enforcement is owned by T8 entity-resolution wiring, not by the Pydantic shape.

  5. A single content_text can yield mixed variants in one flow run. [RATIFIED-S241] Per 02-data-flow.md §3.1, the LLM extraction and entity-resolution stages run in the same cocoindex flow run for a given content_items row. The output of one run can therefore be: one classification + many entity_mention (always); plus, when routed per invariant 7, one q_a_form containing nested Q&A pairs. Each output carries the same op_id per invariant 5, providing the correlation handle across variants.

Validation rules (what counts as well-formed)

Section titled “Validation rules (what counts as well-formed)”

  1. Pydantic validation runs before Postgres UPSERT. [RATIFIED-S241] The cocoindex ExtractByLlm primitive validates the LLM response against the declared output_type schema before passing the extraction downstream. Per phase-b-prerequisite-2a-cocoindex-examples.md line 329 (“Memoised: output_type + instruction + LLM model + input markdown all participate in the code-hash”), validation is part of the cocoindex contract. A response that does not parse as the declared discriminated union is treated as a validation failure per invariant 13.

  2. Inner-tier post-processing functions consume content_text: str and the typed extracted column, never FileLike. [RATIFIED-S241] Per phase-b-prerequisite-2-cocoindex-deep-dive.md §3 + 0.9-spike-S9-cocoindex-idempotency.md §7.1 + §7.2, the layered fn-shape requirement is load-bearing for memoisation correctness. Any @coco.fn(memo=True) post-processing helper that runs against the LLM output (for example, span-offset reconciliation, normalisation, or confidence-threshold filtering) takes content_text: str (and where needed the typed ExtractionOutput value) as its primary input parameter, NOT FileLike. If a file-handle-tier wrapper is added (e.g. process_content_item(file: FileLike, ...)), it is the OUTER tier — it parses the file once via the binary-conversion stage, then delegates to the flow-scope ExtractByLlm step with the parsed content_text. Violations (passing FileLike to an inner tier) re-run the expensive LLM call on every metadata-only edit, defeating idempotency.

  3. A validation failure writes a structured failure record via pipeline_runs, not a half-typed payload. [RATIFIED-S241] When the LLM response fails Pydantic parsing (missing required fields, invalid enum value, unknown discriminator value, type-coercion error), the cocoindex flow run records a structured failure via the pipeline_runs rollup (per 02-data-flow.md §5.1 trigger-driven audit_log + recordPipelineRun() from @/lib/pipeline/record-run per CLAUDE.md Gotcha). The failure record carries: op_id, content_items_id, extraction_kind (attempted variant), error_class (Pydantic error category — missing_required, invalid_enum, invalid_discriminator, type_coercion, unexpected_field), and the raw LLM response (redacted of PII per the standard logging surface). No partial extraction row is written to q_a_extractions / entity_mentions / content_items. The pipeline continues processing subsequent rows; the failed row is queued for HITL review or retry per cocoindex’s native DLQ (per 02-data-flow.md §3.2 retry/back-off/DLQ).

  4. A subsequent successful extraction supersedes a prior failure for the same content_items_id. [RATIFIED-S241] Failures are not terminal. When the same content_items_id re-runs (e.g. after a prompt-template version bump per 0.9-spike-S9-cocoindex-idempotency.md §7.4 version-bump cascade, or after operator-initiated retry), a successful extraction overwrites the prior failure record on the target tables. The failure record in pipeline_runs is retained for audit per its retention policy but no longer blocks downstream UPSERTs.

Source-attribution invariants

Section titled “Source-attribution invariants”

  1. Every extraction row in q_a_extractions / entity_mentions / content_items carries the originating op_id. [RATIFIED-S241] Per 02-data-flow.md §5.1 N7 hybrid op_id pattern, the cocoindex per-flow op_id propagates into the target Postgres tables. This gives the pipeline-correlation surface (“which rows did extraction run X produce?”) via direct SQL on the target table — no separate ledger join required for that question.

  2. Every extraction row references content_items.id, never source_documents.id directly. [RATIFIED-S241] The extraction contract operates on parsed content_text, which is the content_items body. The source-document linkage is recovered via content_items.source_document_id if present, NOT via a direct extraction → source_documents edge. This preserves the per-02-data-flow.md §2.1 invariant that the source folder is canonical and content_items is the materialised index — extractions belong to the index, not to the canonical source.

Failure modes + error envelope

Section titled “Failure modes + error envelope”

  1. An LLM API failure (network, rate limit, provider 5xx) retries per cocoindex native back-off. [RATIFIED-S241] Per 02-data-flow.md §3.2 “Retry / back-off / DLQ — cocoindex provides these natively. KH does not build a pipeline_failures table (COCO.7 DO-NOT-BUILD per 00-synthesis-v2.md §3.10)”. Retry behaviour is configured at the cocoindex flow level; this contract does not override it. After exhausting retries, the failure routes to invariant 13’s pipeline_runs failure record.

  2. An Anthropic API response that mismatches the declared output_type is treated as a validation failure, not a transient retry. [RATIFIED-S241] Per phase-b-prerequisite-2a-cocoindex-examples.md line 329 + the typing contract in invariant 11, a malformed response (e.g. the LLM returned entity_type: "unknown" when the enum requires one of the 12 canonical values) is a schema mismatch, not a transient API issue. The flow does NOT retry on validation mismatch — retrying yields the same malformed shape. The failure is recorded per invariant 13 and surfaced for prompt-template or schema review.

  3. A discriminator-field absence is the highest-severity validation failure. [RATIFIED-S241] If the LLM response omits the extraction_kind field entirely (or returns a discriminator value not in the union), the failure is logged with error_class='invalid_discriminator'. This signal is the load-bearing alert that the prompt is no longer producing typed output — distinct from individual-field validation failures which can indicate prompt drift on a single variant.

Provenance + visibility

Section titled “Provenance + visibility”

  1. Extraction failures are visible to operators via pipeline_runs MCP surface. [RATIFIED-S241] Per the AI-invisible-infrastructure invariant (CLAUDE.md “AI is invisible infrastructure”), the user-facing surface does not display “LLM extraction failed for this content” — that would surface AI plumbing. Instead, failed extractions surface to operators via pipeline_runs and the cocoindex ledger API (the sibling spec being drafted in T1.3 — docs/specs/id-36-cocoindex-ledger-api/TECH.md). End users see only that the extraction has not yet completed (e.g. q_a_extractions row absent, content_items.content_type still NULL); they do not see error envelopes.

  2. Prompt-template version is tracked in the cocoindex code-hash. [RATIFIED-S241] Per phase-b-prerequisite-2a-cocoindex-examples.md line 329 (“Memoised: output_type + instruction + LLM model + input markdown all participate in the code-hash. Changing the schema or instruction → all rows re-extract.”), the prompt template’s text is part of the memoisation hash. Updating the prompt invalidates all cached extractions and triggers full re-extraction per 0.9-spike-S9-cocoindex-idempotency.md §7.4. The contract does not declare a separate prompt_version column; the cocoindex code-hash IS the version identifier.

Anti-patterns (RATIFIED-DO-NOT-BUILD)

Section titled “Anti-patterns (RATIFIED-DO-NOT-BUILD)”

  1. The contract does not include a raw_llm_response field as a routine output column. [RATIFIED-S241] Persisting the raw LLM response per extraction row would violate the AI-invisible-infrastructure invariant (it surfaces the LLM as a visible artefact rather than infrastructure). Raw responses are retained only in pipeline_runs for failed extractions per invariant 13, where they serve operator debugging.

  2. The contract does not include a per-call “extract-everything” variant. [RATIFIED-S241] Each ExtractByLlm invocation declares ONE variant (classification, q_a_form, or entity_mention). Per 02-data-flow.md §3.1, separate cocoindex stages run separate extractors — bundling all three into one mega-prompt would conflate memoisation tiers and violate the layered fn-shape requirement in invariant 12. The discriminated union is for response-validation at the cocoindex contract layer, not for prompt design.

  3. The contract does not bind to LlmApiType.OLLAMA at v1. [RATIFIED-S241] Per phase-b-prerequisite-2a-cocoindex-examples.md line 320 (“api_type can be OLLAMA for on-premise, OPENAI, ANTHROPIC, GEMINI, etc.”), cocoindex supports multiple LLM backends. KH v1 binds to LlmApiType.ANTHROPIC exclusively per the Phew procurement domain’s accuracy requirements. Local/on-premise variants are [DEFERRED-v1.1].

Open questions surfaced for verification

Section titled “Open questions surfaced for verification”

  1. Anthropic prompt-cache passthrough for LlmApiType.ANTHROPIC is unverified. [GAP-Q-EX2-002] Category: cost-projection verification. Sources checked: docs/plans/phase-0-investigation/phase-b-prerequisite-2-cocoindex-deep-dive.md §5 still-open table row "Anthropic prompt-cache passthrough", docs/specs/core-docs-pathway-assessment/S239-still-open-consolidation.md S241 closure addendum row 7 (CATEGORISED-S241 — investigation spike needed, 1-2h). Prerequisite work: T13 prompt-cache verification spike (per canonical-pipeline PLAN §4.13 subtask 1) confirms whether cache_controlblocks pass through to the Anthropic API or are stripped by the cocoindexLlmApiType.ANTHROPIC adapter. Outcome affects cost projections for high-volume extraction (Phew's bid library at ~395 q_a_pairs + ongoing ingest). Per phase-b-prerequisite-2-cocoindex-deep-dive.md §5, this is a MEDIUM-priority verification, NOT a v1 blocker — extraction works without cache passthrough, just at higher cost.

  2. Per-extraction-kind nested-schema depth limits are unverified for Anthropic. [GAP-Q-EX2-003] Category: capability verification. Sources checked: docs/plans/phase-0-investigation/phase-b-prerequisite-2a-cocoindex-examples.md §"patient_intake_extraction" lines 191-204 (confirms nested dataclass support for OpenAI gpt-4o). Prerequisite work: cocoindex 1.0.3 docs read against the LlmApiType.ANTHROPIC adapter to confirm whether nested Pydantic models (e.g. Formcontaininglist[QAPair] per the q_a_form variant in invariant 2) survive the Anthropic JSON-mode contract end-to-end. If a depth-flattening transform is needed, it lands in the outer-tier wrapper before the flow-scope ExtractByLlm step. This is a capability-confirmation question, not a design-block — the discriminated-union direction holds regardless of how the nesting is materialised.

Cross-references

Section titled “Cross-references”
  • Implementation references: see companion TECH.md in this directory.
  • 6-stage cocoindex topology: docs/plans/phase-0-investigation/architecture/02-data-flow.md §3.1.
  • Layered fn-shape rationale: docs/plans/phase-0-investigation/0.9-spike-S9-cocoindex-idempotency.md §7 (especially §7.1 + §7.2).
  • ExtractByLlm API capability + canonical flow-scope pattern: docs/plans/phase-0-investigation/phase-b-prerequisite-2a-cocoindex-examples.md §“Capability: cocoindex.functions.ExtractByLlm” lines 311-329 + lines 362-368 (flow-scope doc["..."].transform(ExtractByLlm(...))).
  • Q-EX2 ratification provenance: docs/specs/core-docs-pathway-assessment/S239-still-open-consolidation.md lines 216-219 (S241 closure addendum row 6).
  • Downstream consumer (q_a_extractions extractor_kind enum): docs/plans/phase-0-investigation/architecture/05-qa-flow.md §3.2.
  • Question-matches question_kind collision rationale: docs/plans/phase-0-investigation/architecture/05-qa-flow.md §7.2.
  • Canonical entity taxonomy: lib/validation/schemas.ts:1495-1508 (VALID_ENTITY_TYPES).
  • Canonical content_type enumeration: lib/validation/schemas.ts:43-52 (VALID_CONTENT_TYPES).
  • Form-type CV (11 baseline values): docs/ontology/26-form-type.md lines 65-79.
  • Pattern A/B parser retire context: docs/plans/phase-0-investigation/architecture/02-data-flow.md §10.1.
  • T1.1 / T1.2 task scope: docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md §4.1 lines 122-139.
  • Sibling spec (failure-path consumer): docs/specs/id-36-cocoindex-ledger-api/TECH.md (T1.3 draft) — owns record_extraction_failure signature.