Skip to content

ID-54 {54.3} TECH — qaextractions lossy-write fix (S273 OQ-52-LOSSY)

ID-54 {54.3} TECH — q_a_extractions lossy-write fix (S273 OQ-52-LOSSY)

Section titled “ID-54 {54.3} TECH — q_a_extractions lossy-write fix (S273 OQ-52-LOSSY)”

Artefact: TECH ({N.3}), retro-documentation of an ADOPTED as-built design. PRODUCT ({54.2}) was deliberately skipped — this spec maps 1:1 from RESEARCH.md ({54.1}, committed b0e34388, ratified-as-PASS). UK English throughout; relative paths; file:line citations.

Status — ADOPTED / BLESSED (this session). The implementation that was begun prematurely (ahead of TECH ratification) has now been adopted as canonical by the product owner. This document therefore no longer proposes a design — it documents and blesses the as-built shape and definitively resolves the scope_tags vocabulary question that RESEARCH §3 left open. The as-built artefacts are:

  • Migration supabase/migrations/20260530195956_id54_q_a_extractions_form_question_fields.sql is in main and ALREADY APPLIED to staging (turayklvaunphgbgscat). It is the canonical schema. This document does not propose creating it, and does not propose altering it.
  • The complete implementation (migration + flow.py + pytest + database.types.ts regen) is committed at 644f5455 on branch worktree-agent-afe287804df50e414. The parent orchestrator will checker-gate + integrate that commit — not this lane. This TECH references the commit; it does not re-plan or re-implement it.

This lane authored/committed THIS document only. It did not create a migration, edit flow.py, run the migration, or touch any ledger.


§0 — Code-intelligence orientation (N/A rationale, for the Checker)

Section titled “§0 — Code-intelligence orientation (N/A rationale, for the Checker)”

This Task touches .py, .sql, and generated-.ts files only: scripts/cocoindex_pipeline/flow.py, the migration supabase/migrations/20260530195956_id54_q_a_extractions_form_question_fields.sql, scripts/tests/test_cocoindex_flow_write_path.py, and the regenerated supabase/types/database.types.ts.

  • GitNexus (gitnexus_query / gitnexus_context) indexes the TypeScript corpus only (per the “GitNexus — Code Intelligence” block in CLAUDE.md). It is structurally blind to the .py / .sql files under change here — i.e. INVISIBLE, not merely unused.
  • ast-dataflow is ts-morph-only and explicitly “does not cover Python or SQL files” (per .ast-dataflow/CLAUDE.md).
  • The single TypeScript touch-point — supabase/types/database.types.ts — is a generated artefact (never hand-edited; CI-guarded by supabase-types-parity), regenerated post- migration, not authored. It is not an ast-dataflow target.

The prescribed substrate for Python + SQL is therefore grep/rg, used both by RESEARCH and re-used here for re-verification of every as-built citation. Code-intelligence orientation is correctly judged N/A for this Subtask.

Re-verification log (this revision, against the as-built commit + live worktree)

Section titled “Re-verification log (this revision, against the as-built commit + live worktree)”
Claim relied onCited atRe-verified
Migration DDL — 4 columns, named CHECK on expected_response_kind, two text[] NOT NULL DEFAULT '{}'git show 644f5455:supabase/migrations/20260530195956_id54_q_a_extractions_form_question_fields.sql✓ verbatim
flow.py Q_A_EXTRACTIONS_SCHEMA gains 4 ColumnDefsgit show 644f5455 -- …/flow.py (@@ …835,6)✓ verbatim
flow.py qa_target.declare_row gains 4 _field keysgit show 644f5455 -- …/flow.py (@@ …1234,6)✓ verbatim
extractor_kind write line NOT touched by 644f5455 (context anchor, not a + line)git show 644f5455 -- …/flow.py"extractor_kind": content_type or "q_a_form", shown unchanged✓ (OQ-54-E latent)
Test extends fixture to present-value + default-fallback pairs; len(qa.rows) == 2git show 644f5455 -- …/test_cocoindex_flow_write_path.py✓ verbatim
database.types.ts gains 4 fields in Row/Insert/Updategit show 644f5455 -- …/database.types.ts✓ verbatim
QAPair fields + strict=True, extra="forbid"extraction.py:250-257 (RESEARCH §1)✓ (grounding)
q_a_extractions.extractor_kind CHECK = 4 values, none q_a_form/content_type…225456…full_schema.sql:109-115✓ verbatim
q_a_pairs.scope_tag / anti_scope_tag text[] NOT NULL DEFAULT ARRAY[]::text[] (SINGULAR)…120828…b_form_type_split.sql:358-359✓ verbatim
caller_scope_tags array-overlap contract scope_tag && caller_scope_tags, CALLER-SIDE…231524…q_a_search_rpcs.sql:31-34, 68✓ verbatim
workspaces.scope_tag absent (only q_a_pairs-family hits across all migrations)rg -ln "scope_tag" supabase/migrations/ → 3 files, all q_a_pairs-family
No anti_scope in extractorrg "anti_scope" scripts/cocoindex_pipeline/ → empty (RESEARCH §3)
form_template_requirements.requirement_type CHECK = 7 coarse values, single 'evidence' bucket…102457_pre_squash_reconciliation.sql:4167✓ verbatim

§1 — Context / problem (what the adopted design fixes)

Section titled “§1 — Context / problem (what the adopted design fixes)”

The cocoindex Path-A Q&A-form extractor (extract_qa_formQAFormExtractionqa_pairs: list[QAPair]) emits four form-question fields per pair — expected_response_kind, evaluation_criteria, evidence_requirements, scope_tags (extraction.py:252-257) — that were silently dropped at write time. The loss was structural across three layers (all confirmed in RESEARCH §preamble):

  1. DB table public.q_a_extractions carried no column for any of the four (…225456…full_schema.sql:103-127).
  2. cocoindex schema Q_A_EXTRACTIONS_SCHEMA declared only 7 columns (flow.py:831-842).
  3. Write site qa_target.declare_row(...) populated only 7 keys (flow.py:1230-1244); TableTarget.declare_row(row=...) writes only the keys present in the row= dict, so the four fields never reached Postgres — not even stashed in extraction_metadata jsonb.

The downstream consumer chain is DORMANT (RESEARCH §2). No production reader of any of the four fields exists in scripts/ lib/ app/ supabase/ components/ hooks/ types/ (extractor + prompt + their own tests only); the q_a_extractions → q_a_pairs promotion path (UC5, promoted_to_pair_id) has zero code; the scope-search RPCs (q_a_search / q_a_get_verbatim) read q_a_pairs only, never q_a_extractions, and have no production caller; and workspaces.scope_tag does NOT exist — the canonical scope vocabulary the &&-overlap retrieval contract depends on is unmaterialised (RESEARCH §3 CRITICAL FINDING; only comment references at …full_schema.sql:279-280).

ID-54’s adopted goal is narrow: stop the data loss at the extraction cache so the four fields are persisted and available when the dormant promotion + search consumers are built later — NOT to wire the full extraction → pair → search contract now. This is a forward-substrate fix; the cost being avoided is having to re-extract to recover dropped LLM output once the consumers land. “No current reader” is a sequencing fact, not a reason to defer (RESEARCH §2 bottom line). There is no user-visible behaviour today (no UI/API surface consumes these fields); the observable contract is purely at the write path and is covered by the §7 as-built tests.


§2 — The blessed column design (as-built)

Section titled “§2 — The blessed column design (as-built)”

The as-built migration adds exactly four columns. Each is confirmed sound against the extractor’s strict=True, extra="forbid" Pydantic guarantees (RESEARCH §1). The driving principle the as-built honours: the DB column round-trips the Pydantic guarantee on the LLM write path, while remaining tolerant of historical / edge / non-extractor-origin rows (the table is shared by four extractor_kind values, only one of which — the LLM path — guarantees the four fields).

From supabase/migrations/20260530195956_id54_q_a_extractions_form_question_fields.sql:

ALTER TABLE public.q_a_extractions
ADD COLUMN expected_response_kind text NULL
CONSTRAINT q_a_extractions_expected_response_kind_check
CHECK (expected_response_kind IN ('mandatory', 'optional')),
ADD COLUMN evaluation_criteria text NULL,
ADD COLUMN evidence_requirements text[] NOT NULL DEFAULT '{}',
ADD COLUMN scope_tags text[] NOT NULL DEFAULT '{}';
ColumnAs-built DDLWhy this is correct
expected_response_kindtext NULL CONSTRAINT q_a_extractions_expected_response_kind_check CHECK (expected_response_kind IN ('mandatory','optional'))Pydantic-REQUIRED on the LLM path (extraction.py:254) yet the DB col is NULL — blessed; see §2.1. The bare CHECK (col IN (...)) passes on NULL by SQL semantics (a CHECK is satisfied unless it evaluates to FALSE; NULL IN (...) yields NULL, not FALSE), so nullable + CHECK coexist without an explicit IS NULL OR clause — see §2.2. The named constraint mirrors the extractor_kind text + CHECK controlled-vocab idiom at …full_schema.sql:109-115.
evaluation_criteriatext NULLMatches str | None exactly (extraction.py:255); precedent extracted_answer_text text NULL (…full_schema.sql:117).
evidence_requirementstext[] NOT NULL DEFAULT '{}'Matches the never-null-empty-list guarantee (list[str] = Field(default_factory=list), extraction.py:256); mirrors the q_a_pairs.scope_tag text[] NOT NULL DEFAULT ARRAY[]::text[] precedent (…120828…b_form_type_split.sql:358). Free-form, no CHECK — correct: no fine-grained evidence taxonomy exists (RESEARCH §3; §3.2 below).
scope_tagstext[] NOT NULL DEFAULT '{}'PLURAL name mirrors the extractor field (extraction.py:257) — see §4. Never-null guarantee per extraction.py:257. Correctly NOT wired to scope-matching (free-text v1, §3) — semantically inert for retrieval until the alignment prerequisites land, which is the right place to be (§3).

Equivalence to the RESEARCH-recommended shape. The as-built shapes match the RESEARCH §4 precedent-recommended options exactly except for one deliberate, sound simplification on expected_response_kind: RESEARCH §4 sketched the CHECK as IS NULL OR expected_response_kind IN (...), whereas the as-built uses the bare CHECK (... IN (...)). These are semantically identical on the nullable column (§2.2), so the as-built is blessed without amendment.

§2.1 — Why expected_response_kind is NULL (not NOT NULL) — blessed

Section titled “§2.1 — Why expected_response_kind is NULL (not NOT NULL) — blessed”

expected_response_kind is REQUIRED with no default in Pydantic (extraction.py:254), so every row the LLM extractor writes carries a value. The argument for NOT NULL is fidelity to that guarantee. The adopted text NULL is nonetheless correct and blessed:

  1. ADD COLUMN on a populated table. Staging already has q_a_extractions rows. A ADD COLUMN ... NOT NULL with no DEFAULT fails on a non-empty table; a ADD COLUMN ... NOT NULL DEFAULT '<sentinel>' would inject a fake third value, which the 2-value CV explicitly forbids (info_only unratified — extraction.py:247, prompts.py:111). NULL avoids both hazards and keeps the migration clean-additive.
  2. The table is multi-origin. extractor_kind permits four values ('prior_bid_response' | 'llm_extraction' | 'yaml_frontmatter_v1' | 'markdown_heading_v1', …full_schema.sql:109-115). Non-LLM extractor paths legitimately leave expected_response_kind NULL — a NOT NULL column would mis-model those rows.
  3. The strict Pydantic model still supplies a value on every LLM-path write. The strict Literal["mandatory","optional"] hard-rejects out-of-CV values upstream, so the value reaching declare_row on the LLM path is always one of the two literals. Practical fidelity is therefore identical to NOT NULL; only the schema-level floor differs — and the DB CHECK still catches any future non-LLM writer that tries a third value, without breaking historical/edge/NULL rows.

§2.2 — Why a bare CHECK coexists with nullability (document the SQL semantics)

Section titled “§2.2 — Why a bare CHECK coexists with nullability (document the SQL semantics)”

A SQL CHECK constraint is satisfied unless it evaluates to FALSE. For a NULL value, NULL IN ('mandatory','optional') evaluates to NULL (three-valued logic), not FALSE — so the row passes. The as-built CHECK (expected_response_kind IN ('mandatory','optional')) therefore permits NULL implicitly, with no need for an explicit expected_response_kind IS NULL OR ... disjunction. This is why a nullable column and a value-list CHECK coexist cleanly in the adopted migration. (Documented here so the Checker — and any future reader puzzled by the absence of an IS NULL OR arm — sees the rationale.)


§3 — ★ DEFINITIVE RESOLUTION: the scope_tags vocabulary question

Section titled “§3 — ★ DEFINITIVE RESOLUTION: the scope_tags vocabulary question”

This is the load-bearing new content of the revision. RESEARCH §3 surfaced the question and laid out the trade-offs but deferred the call; the product owner’s adoption decision lets TECH now resolve it cleanly.

Must q_a_extractions.scope_tags align with the EXISTING caller_scope_tags array-overlap vocabulary used by the search RPCs (scope_tag && caller_scope_tags, …231524…q_a_search_rpcs.sql:33)?

No — not by this migration, and the adopted column is correct in NOT attempting it.

The adopted free-text text[] column-add STOPS THE LOSS — it persists what the LLM emits so it is no longer dropped. But the column-add does not, by itself, achieve alignment with the scope_tag && caller_scope_tags overlap contract, and it should not try to. Alignment to caller_scope_tags semantics requires three things the column alone cannot provide:

  1. A canonical scope-tag vocabulary source — which does not exist today. RESEARCH §3 established (and this revision re-verified: rg -ln "scope_tag" supabase/migrations/ returns only the three q_a_pairs-family files) that workspaces.scope_tag does NOT exist as a column. The scope_tag && workspaces.scope_tag filter described at …full_schema.sql:279-280 and in 05-qa-flow.md is an intended/future contract. There is nothing to overlap against today — so there is no meaningful vocabulary to “align” the adopted column to.
  2. The plural→singular promotion mapping. The retrieval contract operates on the promoted corpus q_a_pairs.scope_tag (singular, …b_form_type_split.sql:358), not on q_a_extractions. The dormant UC5 promotion path (promoted_to_pair_id, …full_schema.sql:120-121 — zero code, RESEARCH §2) would have to map q_a_extractions.scope_tags (plural) → q_a_pairs.scope_tag (singular). That mapping does not exist yet, so the adopted column cannot reach the overlap contract by any live path.
  3. An LLM/prompt constraint to a controlled vocabulary. The LLM is currently instructed only to emit “zero or more scope identifiers” (prompts.py:113) with no enumerated value listscope_tags is free-text. Free-text tags will never &&-overlap a controlled caller_scope_tags / workspaces.scope_tag set with any reliability, which would make the column semantically inert for scope-matching even if (1) and (2) existed.

Therefore: the adopted scope_tags text[] NOT NULL DEFAULT '{}' column is correct AS A LOSS-STOP and is correctly NOT wired to scope-matching. Persisting free-text now is strictly cheaper than blocking: the data shape is text[] regardless of which vocabulary eventually backs it; normalisation can happen at promotion time; and an unconsumed column cannot misfire. Alignment is NOT a change to the adopted ID-54 migration — per product-owner instruction, do NOT propose altering migration 20260530195956. Alignment is captured below as a follow-up recommendation for a future Task/backlog item.

★ §3.1 — Follow-up recommendation (FUTURE Task/backlog — NOT an ID-54 change)

Section titled “★ §3.1 — Follow-up recommendation (FUTURE Task/backlog — NOT an ID-54 change)”

To eventually make q_a_extractions.scope_tags participate in the scope_tag && caller_scope_tags overlap contract, a future Task should (in order):

  1. Materialise the canonical scope vocabulary. Either add the workspaces.scope_tag column (the source the migration comment + 05-qa-flow.md assume) or confirm an equivalent canonical source (e.g. derive from an existing taxonomy table). Until this exists there is nothing to overlap against. (RESEARCH §3 “Unverified”: this is on no committed Task today.)
  2. Define the plural→singular promotion mapping. Build the UC5 promotion path and map q_a_extractions.scope_tags (plural) → q_a_pairs.scope_tag (singular) — a column-alias in the promotion INSERT (cheap; paid in the dormant promotion path, not the shipped write path — see §4).
  3. Decide prompt-constraint vs normalise-at-promotion. Either constrain the LLM prompt to the now-materialised vocabulary (mirroring the form_type snapshot-validator pattern at extraction.py:228-239), or persist free-text and normalise to the controlled set at promotion time. Prompt-constraint presupposes step 1.
  4. ONLY THEN wire q_a search scope-matching — i.e. let the search caller’s scope_tag && caller_scope_tags filter consume promoted scope tags. No search-RPC change is in ID-54 scope (RESEARCH §2); this is the future Task’s terminal step.

§3.2 — evidence_requirements (lighter parallel analysis — same disposition)

Section titled “§3.2 — evidence_requirements (lighter parallel analysis — same disposition)”

The adopted evidence_requirements text[] NOT NULL DEFAULT '{}' is likewise a free-form v1 loss-stop, correctly not wired to any taxonomy. RESEARCH §3 (re-verified here) found no fine-grained evidence-artefact controlled vocabulary anywhere in the migrations or scripts/ corpus. The only adjacent enum is form_template_requirements.requirement_type — a CHECK of seven coarse values ('policy' | 'statement' | 'evidence' | 'data' | 'narrative' | 'declaration' | 'reference', …pre_squash_reconciliation.sql:4167) with a single 'evidence' bucket, not a list of evidence kinds like iso27001_certificate. So it is a poor direct vocabulary. Unlike scope_tags, evidence_requirements carries no inert-column retrieval risk — nothing filters on it. Follow-up (future, NOT ID-54): if an evidence taxonomy is ever needed, form_template_requirements.requirement_type is the future normalisation anchor to start from — but ID-54 must not block on inventing one.


§4 — anti_scope + naming — resolved (blessed, not open)

Section titled “§4 — anti_scope + naming — resolved (blessed, not open)”

These were RESEARCH §3 sub-problems / OQs; the adoption decision resolves them as adopted / blessed, not open questions:

  • v1 extractions are inclusion-only. The extractor emits no anti-scope field (QAPair has only scope_tags; rg "anti_scope" scripts/cocoindex_pipeline/ is empty). The workspace-overlap exclusion arm (AND NOT (anti_scope_tag && workspaces.scope_tag), …full_schema.sql:279-280) lives on the corpus q_a_pairs.anti_scope_tag (singular, …b_form_type_split.sql:359), which is dormant and defaults to {}. Exclusion is correctly a curation/promotion-time concern; the adopted ID-54 design does not add an anti_scope_tag column to q_a_extractions nor an anti-scope extractor field. Growing the extractor an anti-scope field is a wider change (extractor + prompt + model) and out of scope.
  • The column is named PLURAL scope_tags, mirroring the extractor (extraction.py:257), not the corpus-singular scope_tag. This is blessed: it gives a clean, mechanical _field(pair, "scope_tags", []) write with a 1:1 key↔column↔Pydantic-field name (lowest defect surface; clearest forensic lineage from LLM output → cache column). The promotion path (when built) maps plural→singular — a rename paid in the dormant promotion path rather than the shipped write path, which is the cheaper trade.

§5 — flow.py — as-built deltas (blessed)

Section titled “§5 — flow.py — as-built deltas (blessed)”

Both edits are localised and additive; no existing line was moved. Re-verified verbatim against git show 644f5455 -- scripts/cocoindex_pipeline/flow.py.

§5.1 — Q_A_EXTRACTIONS_SCHEMA — four ColumnDefs added

Section titled “§5.1 — Q_A_EXTRACTIONS_SCHEMA — four ColumnDefs added”
# in Q_A_EXTRACTIONS_SCHEMA (after extracted_answer_text):
"expected_response_kind": ColumnDef(type="text", nullable=True),
"evaluation_criteria": ColumnDef(type="text", nullable=True),
"evidence_requirements": ColumnDef(type="text[]", nullable=False),
"scope_tags": ColumnDef(type="text[]", nullable=False),

nullable mirrors the §2 DDL exactly: expected_response_kind / evaluation_criteria are nullable=True; the two arrays are nullable=False (never-null, DB-defaulted '{}'). Arrays use type="text[]" per the existing in-schema array precedent — blessed. asyncpg maps Python list[str] to the text[] column.

§5.2 — qa_target.declare_row row dict — four keys added

Section titled “§5.2 — qa_target.declare_row row dict — four keys added”
# in qa_target.declare_row row dict:
"expected_response_kind": _field(pair, "expected_response_kind"),
"evaluation_criteria": _field(pair, "evaluation_criteria"),
"evidence_requirements": _field(pair, "evidence_requirements", []),
"scope_tags": _field(pair, "scope_tags", []),

The keys read off each pair via the existing _field(obj, name, default=None) helper (flow.py:1025-1036); 1:1 key↔column↔Pydantic-field naming. Default-fallback semantics (blessed): production extractors return Pydantic models (getattr returns the guaranteed value), but the write-path test stubs return plain dicts (_field reads via dict.get). The defaults make the write robust to a dict stub omitting a key:

  • expected_response_kindNone (matches the §2 NULL column; a stub without the key writes NULL rather than crashing — the None is defensive, since the Pydantic model itself guarantees the value on the LLM path).
  • evaluation_criteriaNone.
  • evidence_requirements / scope_tags[] (matches the never-null '{}' DB default; a stub without the key writes an empty array, not None, satisfying the NOT NULL column).

This mirrors the pre-existing _field(pair, "question_text", "") and _field(pair, "answer_text") defaulting at the same site.


§6 — database.types.ts — as-built regeneration (blessed)

Section titled “§6 — database.types.ts — as-built regeneration (blessed)”

database.types.ts is generated — never hand-edited (CLAUDE.md gotcha); CI guard supabase-types-parity enforces this. The as-built commit regenerated it from staging (turayklvaunphgbgscat — the project the migration was pushed to), and the diff adds four fields to each of Row / Insert / Update of the q_a_extractions block (re-verified at git show 644f5455 -- supabase/types/database.types.ts):

  • expected_response_kind: string | null (Row); optional | null in Insert/Update.
  • evaluation_criteria: string | null.
  • evidence_requirements: string[] (Row; NOT NULL ⇒ no | null); string[] optional in Insert/Update via the DB DEFAULT.
  • scope_tags: string[] (Row); Insert/Update as for evidence_requirements.

No database-overrides.ts change was needed — none of the four columns is a JSONB domain type.


§7 — As-built test + verification evidence

Section titled “§7 — As-built test + verification evidence”

Rather than a forward test plan to dispatch, this section documents what the adopted tests already cover and the staging-applied + types-regen facts.

File: scripts/tests/test_cocoindex_flow_write_path.py. The as-built commit extended the existing write-path test test_ingest_file_declares_rows_with_op_id_field (which stubs extract_qa_form with a plain dict, exercising the _field dict-path defaulting). Re-verified verbatim at git show 644f5455 -- scripts/tests/test_cocoindex_flow_write_path.py.

As-built coverage (two paths):

  1. Present-value pair — the stubbed qa_pairs first entry now carries all four fields with non-default values (expected_response_kind: "mandatory", evaluation_criteria: "scored on completeness", evidence_requirements: ["certificate"], scope_tags: ["lot-1"]). The test asserts all four reach qa.rows[0] verbatim.
  2. Default-fallback pair — a second qa_pairs entry ("What is Z?") omits the four fields. The test asserts len(qa.rows) == 2 and that qa.rows[1] carries the _field defaults: expected_response_kind is None, evaluation_criteria is None, evidence_requirements == [], scope_tags == [] — proving the §5.2 defaults match the §2 column nullability/defaults exactly.

Staging + types facts (as-built, integrated by parent):

  • Migration 20260530195956_id54_q_a_extractions_form_question_fields.sql is in main and already applied to staging (turayklvaunphgbgscat) per the product-owner adoption decision.
  • database.types.ts was regenerated from staging (the four columns appear in the q_a_extractions Row/Insert/Update — §6).

Acceptance contract (PRODUCT was skipped) — decomposed from the §1 forward-substrate goal:

Acceptance criterionVerified by
All four extractor fields reach the q_a_extractions write (no silent drop)§7 present-value assertions (as-built)
Missing optional keys default to NULL/[], not crash§7 default-fallback assertions (as-built)
Column shapes round-trip the §1 extractor guarantees§2 DDL + §5.1 ColumnDef nullability + the two test paths
No regression to the existing 7-key write (op_id, FK, question text)pre-existing assertions remain green

Execution note (for the parent’s integration checker): cocoindex uses an LMDB mmap the sandbox blocks (S252 precedent) — run pytest UNSANDBOXED (python3 -m pytest scripts/tests/test_cocoindex_flow_write_path.py -q). The guard tests pipeline-parity.test.ts / doc-freshness.test.ts run on every Vitest invocation; if pipeline-parity asserts the cocoindex schema column set against the DB, its fixture must include the four new columns (CLAUDE.md “Guard tests break on structural changes”).


§8 — Out-of-scope / adjacent: OQ-54-E (follow-up for the PARENT’s integration checker)

Section titled “§8 — Out-of-scope / adjacent: OQ-54-E (follow-up for the PARENT’s integration checker)”

OQ-54-E — pre-existing extractor_kind CHECK-violation, STILL LATENT in the adopted commit.

Verified this revision: 644f5455 did NOT touch the extractor_kind write line. In the as-built flow.py diff, "extractor_kind": content_type or "q_a_form", appears as an unchanged context anchor (the line immediately above the four added keys), not a + line. On this worktree’s HEAD it is flow.py:1234; on the as-built branch it shifts to ~:1241 after the schema insert, but the line content is identical and unmodified.

The q_a_extractions.extractor_kind column carries a CHECK permitting only 'prior_bid_response' | 'llm_extraction' | 'yaml_frontmatter_v1' | 'markdown_heading_v1' (…225456…full_schema.sql:109-115, re-verified verbatim). Neither "q_a_form" nor any content_type taxonomy value (e.g. "case_study", the write-path test fixture value) is in that CHECK set. Therefore the first real q_a_form ingest against a CHECK-enforcing DB would fail the upsert. It has gone unnoticed because the write-path tests use a _FakeTarget stub that does not enforce the CHECK — so the test suite is green while the live-INSERT hazard remains.

This is a latent constraint-violation bug still present in the adopted implementation. It is NOT part of the four-field loss and is NOT decided here — it is surfaced for the parent’s integration checker to route to the product owner. Two options (presented, not chosen — the parent/owner owns the call):

  • Option A — map the write to a CHECK-valid value. Change content_type or "q_a_form" to write 'llm_extraction' (the correct enum value for the LLM Path-A extractor) on the q_a_extractions write. One line, co-located with the ID-54 edits; removes the live-INSERT hazard on the path ID-54 otherwise “fixes”.
  • Option B — extend the CHECK to include the q_a_form extractor value(s). Keeps the written string but widens the constraint vocabulary (a new migration).

Not in this lane’s scope to fix. This lane authors/commits TECH only. The recommendation is to surface it as a blocking note on the parent’s integration checker so the live-INSERT hazard is not merged silently.

(Adjacent, also deferred per RESEARCH §2: the UC5 promotion path and the q_a_search consumer chain — both DORMANT — are explicitly not built or modified by ID-54.)


  • scope_tags alignment to the overlap contract — §3.1: a future Task must (1) materialise the canonical scope vocabulary (workspaces.scope_tag or equivalent), (2) define the plural→singular UC5 promotion mapping, (3) decide prompt-constraint vs normalise-at-promotion, and (4) only then wire q_a search scope-matching. NOT a change to migration 20260530195956.
  • UC5 promotion path (q_a_extractions → q_a_pairs, promoted_to_pair_id) — DORMANT; when built, maps scope_tags (plural) → q_a_pairs.scope_tag (singular) and normalises free-text scope tags to the canonical vocabulary.
  • evidence_requirements taxonomy — §3.2: free-form for v1; form_template_requirements .requirement_type is a coarse future normalisation anchor only. No inert-column risk.
  • Extractor anti-scope field (§4) — inclusion-only is blessed for v1; an anti-scope field is a wider extractor+prompt change, only if exclusion-arm scoping is later needed at extraction time.
  • OQ-54-E extractor_kind CHECK violation — §8; LATENT in 644f5455. Surfaced to the parent’s integration checker: Option A (map to 'llm_extraction') or Option B (extend the CHECK). Pending parent/owner decision.