ID-54 {54.3} TECH — qaextractions lossy-write fix (S273 OQ-52-LOSSY)
ID-54 {54.3} TECH — q_a_extractions lossy-write fix (S273 OQ-52-LOSSY)
Section titled “ID-54 {54.3} TECH — q_a_extractions lossy-write fix (S273 OQ-52-LOSSY)”Artefact: TECH ({N.3}), retro-documentation of an ADOPTED as-built design. PRODUCT
({54.2}) was deliberately skipped — this spec maps 1:1 from RESEARCH.md ({54.1},
committed b0e34388, ratified-as-PASS). UK English throughout; relative paths; file:line
citations.
Status — ADOPTED / BLESSED (this session). The implementation that was begun prematurely (ahead of TECH ratification) has now been adopted as canonical by the product owner. This document therefore no longer proposes a design — it documents and blesses the as-built shape and definitively resolves the
scope_tagsvocabulary question that RESEARCH §3 left open. The as-built artefacts are:
- Migration
supabase/migrations/20260530195956_id54_q_a_extractions_form_question_fields.sqlis inmainand ALREADY APPLIED to staging (turayklvaunphgbgscat). It is the canonical schema. This document does not propose creating it, and does not propose altering it.- The complete implementation (migration +
flow.py+ pytest +database.types.tsregen) is committed at644f5455on branchworktree-agent-afe287804df50e414. The parent orchestrator will checker-gate + integrate that commit — not this lane. This TECH references the commit; it does not re-plan or re-implement it.This lane authored/committed THIS document only. It did not create a migration, edit
flow.py, run the migration, or touch any ledger.
§0 — Code-intelligence orientation (N/A rationale, for the Checker)
Section titled “§0 — Code-intelligence orientation (N/A rationale, for the Checker)”This Task touches .py, .sql, and generated-.ts files only:
scripts/cocoindex_pipeline/flow.py, the migration
supabase/migrations/20260530195956_id54_q_a_extractions_form_question_fields.sql,
scripts/tests/test_cocoindex_flow_write_path.py, and the regenerated
supabase/types/database.types.ts.
- GitNexus (
gitnexus_query/gitnexus_context) indexes the TypeScript corpus only (per the “GitNexus — Code Intelligence” block inCLAUDE.md). It is structurally blind to the.py/.sqlfiles under change here — i.e. INVISIBLE, not merely unused. - ast-dataflow is
ts-morph-only and explicitly “does not cover Python or SQL files” (per.ast-dataflow/CLAUDE.md). - The single TypeScript touch-point —
supabase/types/database.types.ts— is a generated artefact (never hand-edited; CI-guarded bysupabase-types-parity), regenerated post- migration, not authored. It is not an ast-dataflow target.
The prescribed substrate for Python + SQL is therefore grep/rg, used both by RESEARCH and
re-used here for re-verification of every as-built citation. Code-intelligence orientation is
correctly judged N/A for this Subtask.
Re-verification log (this revision, against the as-built commit + live worktree)
Section titled “Re-verification log (this revision, against the as-built commit + live worktree)”| Claim relied on | Cited at | Re-verified |
|---|---|---|
Migration DDL — 4 columns, named CHECK on expected_response_kind, two text[] NOT NULL DEFAULT '{}' | git show 644f5455:supabase/migrations/20260530195956_id54_q_a_extractions_form_question_fields.sql | ✓ verbatim |
flow.py Q_A_EXTRACTIONS_SCHEMA gains 4 ColumnDefs | git show 644f5455 -- …/flow.py (@@ …835,6) | ✓ verbatim |
flow.py qa_target.declare_row gains 4 _field keys | git show 644f5455 -- …/flow.py (@@ …1234,6) | ✓ verbatim |
extractor_kind write line NOT touched by 644f5455 (context anchor, not a + line) | git show 644f5455 -- …/flow.py — "extractor_kind": content_type or "q_a_form", shown unchanged | ✓ (OQ-54-E latent) |
Test extends fixture to present-value + default-fallback pairs; len(qa.rows) == 2 | git show 644f5455 -- …/test_cocoindex_flow_write_path.py | ✓ verbatim |
database.types.ts gains 4 fields in Row/Insert/Update | git show 644f5455 -- …/database.types.ts | ✓ verbatim |
QAPair fields + strict=True, extra="forbid" | extraction.py:250-257 (RESEARCH §1) | ✓ (grounding) |
q_a_extractions.extractor_kind CHECK = 4 values, none q_a_form/content_type | …225456…full_schema.sql:109-115 | ✓ verbatim |
q_a_pairs.scope_tag / anti_scope_tag text[] NOT NULL DEFAULT ARRAY[]::text[] (SINGULAR) | …120828…b_form_type_split.sql:358-359 | ✓ verbatim |
caller_scope_tags array-overlap contract scope_tag && caller_scope_tags, CALLER-SIDE | …231524…q_a_search_rpcs.sql:31-34, 68 | ✓ verbatim |
workspaces.scope_tag absent (only q_a_pairs-family hits across all migrations) | rg -ln "scope_tag" supabase/migrations/ → 3 files, all q_a_pairs-family | ✓ |
No anti_scope in extractor | rg "anti_scope" scripts/cocoindex_pipeline/ → empty (RESEARCH §3) | ✓ |
form_template_requirements.requirement_type CHECK = 7 coarse values, single 'evidence' bucket | …102457_pre_squash_reconciliation.sql:4167 | ✓ verbatim |
§1 — Context / problem (what the adopted design fixes)
Section titled “§1 — Context / problem (what the adopted design fixes)”The cocoindex Path-A Q&A-form extractor (extract_qa_form → QAFormExtraction →
qa_pairs: list[QAPair]) emits four form-question fields per pair —
expected_response_kind, evaluation_criteria, evidence_requirements, scope_tags
(extraction.py:252-257) — that were silently dropped at write time. The loss was
structural across three layers (all confirmed in RESEARCH §preamble):
- DB table
public.q_a_extractionscarried no column for any of the four (…225456…full_schema.sql:103-127). - cocoindex schema
Q_A_EXTRACTIONS_SCHEMAdeclared only 7 columns (flow.py:831-842). - Write site
qa_target.declare_row(...)populated only 7 keys (flow.py:1230-1244);TableTarget.declare_row(row=...)writes only the keys present in therow=dict, so the four fields never reached Postgres — not even stashed inextraction_metadatajsonb.
The downstream consumer chain is DORMANT (RESEARCH §2). No production reader of any of the
four fields exists in scripts/ lib/ app/ supabase/ components/ hooks/ types/ (extractor +
prompt + their own tests only); the q_a_extractions → q_a_pairs promotion path (UC5,
promoted_to_pair_id) has zero code; the scope-search RPCs (q_a_search / q_a_get_verbatim)
read q_a_pairs only, never q_a_extractions, and have no production caller; and
workspaces.scope_tag does NOT exist — the canonical scope vocabulary the &&-overlap
retrieval contract depends on is unmaterialised (RESEARCH §3 CRITICAL FINDING; only comment
references at …full_schema.sql:279-280).
ID-54’s adopted goal is narrow: stop the data loss at the extraction cache so the four fields are persisted and available when the dormant promotion + search consumers are built later — NOT to wire the full extraction → pair → search contract now. This is a forward-substrate fix; the cost being avoided is having to re-extract to recover dropped LLM output once the consumers land. “No current reader” is a sequencing fact, not a reason to defer (RESEARCH §2 bottom line). There is no user-visible behaviour today (no UI/API surface consumes these fields); the observable contract is purely at the write path and is covered by the §7 as-built tests.
§2 — The blessed column design (as-built)
Section titled “§2 — The blessed column design (as-built)”The as-built migration adds exactly four columns. Each is confirmed sound against the
extractor’s strict=True, extra="forbid" Pydantic guarantees (RESEARCH §1). The driving
principle the as-built honours: the DB column round-trips the Pydantic guarantee on the LLM
write path, while remaining tolerant of historical / edge / non-extractor-origin rows (the
table is shared by four extractor_kind values, only one of which — the LLM path — guarantees
the four fields).
As-built DDL (VERBATIM — do NOT alter)
Section titled “As-built DDL (VERBATIM — do NOT alter)”From supabase/migrations/20260530195956_id54_q_a_extractions_form_question_fields.sql:
ALTER TABLE public.q_a_extractions ADD COLUMN expected_response_kind text NULL CONSTRAINT q_a_extractions_expected_response_kind_check CHECK (expected_response_kind IN ('mandatory', 'optional')), ADD COLUMN evaluation_criteria text NULL, ADD COLUMN evidence_requirements text[] NOT NULL DEFAULT '{}', ADD COLUMN scope_tags text[] NOT NULL DEFAULT '{}';Per-column blessing
Section titled “Per-column blessing”| Column | As-built DDL | Why this is correct |
|---|---|---|
expected_response_kind | text NULL CONSTRAINT q_a_extractions_expected_response_kind_check CHECK (expected_response_kind IN ('mandatory','optional')) | Pydantic-REQUIRED on the LLM path (extraction.py:254) yet the DB col is NULL — blessed; see §2.1. The bare CHECK (col IN (...)) passes on NULL by SQL semantics (a CHECK is satisfied unless it evaluates to FALSE; NULL IN (...) yields NULL, not FALSE), so nullable + CHECK coexist without an explicit IS NULL OR clause — see §2.2. The named constraint mirrors the extractor_kind text + CHECK controlled-vocab idiom at …full_schema.sql:109-115. |
evaluation_criteria | text NULL | Matches str | None exactly (extraction.py:255); precedent extracted_answer_text text NULL (…full_schema.sql:117). |
evidence_requirements | text[] NOT NULL DEFAULT '{}' | Matches the never-null-empty-list guarantee (list[str] = Field(default_factory=list), extraction.py:256); mirrors the q_a_pairs.scope_tag text[] NOT NULL DEFAULT ARRAY[]::text[] precedent (…120828…b_form_type_split.sql:358). Free-form, no CHECK — correct: no fine-grained evidence taxonomy exists (RESEARCH §3; §3.2 below). |
scope_tags | text[] NOT NULL DEFAULT '{}' | PLURAL name mirrors the extractor field (extraction.py:257) — see §4. Never-null guarantee per extraction.py:257. Correctly NOT wired to scope-matching (free-text v1, §3) — semantically inert for retrieval until the alignment prerequisites land, which is the right place to be (§3). |
Equivalence to the RESEARCH-recommended shape. The as-built shapes match the RESEARCH §4 precedent-recommended options exactly except for one deliberate, sound simplification on
expected_response_kind: RESEARCH §4 sketched the CHECK asIS NULL OR expected_response_kind IN (...), whereas the as-built uses the bareCHECK (... IN (...)). These are semantically identical on the nullable column (§2.2), so the as-built is blessed without amendment.
§2.1 — Why expected_response_kind is NULL (not NOT NULL) — blessed
Section titled “§2.1 — Why expected_response_kind is NULL (not NOT NULL) — blessed”expected_response_kind is REQUIRED with no default in Pydantic (extraction.py:254), so
every row the LLM extractor writes carries a value. The argument for NOT NULL is fidelity to
that guarantee. The adopted text NULL is nonetheless correct and blessed:
ADD COLUMNon a populated table. Staging already hasq_a_extractionsrows. AADD COLUMN ... NOT NULLwith no DEFAULT fails on a non-empty table; aADD COLUMN ... NOT NULL DEFAULT '<sentinel>'would inject a fake third value, which the 2-value CV explicitly forbids (info_onlyunratified —extraction.py:247,prompts.py:111).NULLavoids both hazards and keeps the migration clean-additive.- The table is multi-origin.
extractor_kindpermits four values ('prior_bid_response' | 'llm_extraction' | 'yaml_frontmatter_v1' | 'markdown_heading_v1',…full_schema.sql:109-115). Non-LLM extractor paths legitimately leaveexpected_response_kindNULL — aNOT NULLcolumn would mis-model those rows. - The strict Pydantic model still supplies a value on every LLM-path write. The
strictLiteral["mandatory","optional"]hard-rejects out-of-CV values upstream, so the value reachingdeclare_rowon the LLM path is always one of the two literals. Practical fidelity is therefore identical toNOT NULL; only the schema-level floor differs — and the DB CHECK still catches any future non-LLM writer that tries a third value, without breaking historical/edge/NULL rows.
§2.2 — Why a bare CHECK coexists with nullability (document the SQL semantics)
Section titled “§2.2 — Why a bare CHECK coexists with nullability (document the SQL semantics)”A SQL CHECK constraint is satisfied unless it evaluates to FALSE. For a NULL value,
NULL IN ('mandatory','optional') evaluates to NULL (three-valued logic), not FALSE — so
the row passes. The as-built CHECK (expected_response_kind IN ('mandatory','optional'))
therefore permits NULL implicitly, with no need for an explicit expected_response_kind IS NULL OR ... disjunction. This is why a nullable column and a value-list CHECK coexist cleanly in the
adopted migration. (Documented here so the Checker — and any future reader puzzled by the
absence of an IS NULL OR arm — sees the rationale.)
§3 — ★ DEFINITIVE RESOLUTION: the scope_tags vocabulary question
Section titled “§3 — ★ DEFINITIVE RESOLUTION: the scope_tags vocabulary question”This is the load-bearing new content of the revision. RESEARCH §3 surfaced the question and laid out the trade-offs but deferred the call; the product owner’s adoption decision lets TECH now resolve it cleanly.
The question (product owner)
Section titled “The question (product owner)”Must
q_a_extractions.scope_tagsalign with the EXISTINGcaller_scope_tagsarray-overlap vocabulary used by the search RPCs (scope_tag && caller_scope_tags,…231524…q_a_search_rpcs.sql:33)?
The answer
Section titled “The answer”No — not by this migration, and the adopted column is correct in NOT attempting it.
The adopted free-text text[] column-add STOPS THE LOSS — it persists what the LLM emits so
it is no longer dropped. But the column-add does not, by itself, achieve alignment with the
scope_tag && caller_scope_tags overlap contract, and it should not try to. Alignment to
caller_scope_tags semantics requires three things the column alone cannot provide:
- A canonical scope-tag vocabulary source — which does not exist today. RESEARCH §3
established (and this revision re-verified:
rg -ln "scope_tag" supabase/migrations/returns only the threeq_a_pairs-family files) thatworkspaces.scope_tagdoes NOT exist as a column. Thescope_tag && workspaces.scope_tagfilter described at…full_schema.sql:279-280and in05-qa-flow.mdis an intended/future contract. There is nothing to overlap against today — so there is no meaningful vocabulary to “align” the adopted column to. - The plural→singular promotion mapping. The retrieval contract operates on the promoted
corpus
q_a_pairs.scope_tag(singular,…b_form_type_split.sql:358), not onq_a_extractions. The dormant UC5 promotion path (promoted_to_pair_id,…full_schema.sql:120-121— zero code, RESEARCH §2) would have to mapq_a_extractions.scope_tags (plural) → q_a_pairs.scope_tag (singular). That mapping does not exist yet, so the adopted column cannot reach the overlap contract by any live path. - An LLM/prompt constraint to a controlled vocabulary. The LLM is currently instructed only
to emit “zero or more scope identifiers” (
prompts.py:113) with no enumerated value list —scope_tagsis free-text. Free-text tags will never&&-overlap a controlledcaller_scope_tags/workspaces.scope_tagset with any reliability, which would make the column semantically inert for scope-matching even if (1) and (2) existed.
Therefore: the adopted scope_tags text[] NOT NULL DEFAULT '{}' column is correct AS A
LOSS-STOP and is correctly NOT wired to scope-matching. Persisting free-text now is
strictly cheaper than blocking: the data shape is text[] regardless of which vocabulary
eventually backs it; normalisation can happen at promotion time; and an unconsumed column cannot
misfire. Alignment is NOT a change to the adopted ID-54 migration — per product-owner
instruction, do NOT propose altering migration 20260530195956. Alignment is captured below
as a follow-up recommendation for a future Task/backlog item.
★ §3.1 — Follow-up recommendation (FUTURE Task/backlog — NOT an ID-54 change)
Section titled “★ §3.1 — Follow-up recommendation (FUTURE Task/backlog — NOT an ID-54 change)”To eventually make q_a_extractions.scope_tags participate in the scope_tag && caller_scope_tags overlap contract, a future Task should (in order):
- Materialise the canonical scope vocabulary. Either add the
workspaces.scope_tagcolumn (the source the migration comment +05-qa-flow.mdassume) or confirm an equivalent canonical source (e.g. derive from an existing taxonomy table). Until this exists there is nothing to overlap against. (RESEARCH §3 “Unverified”: this is on no committed Task today.) - Define the plural→singular promotion mapping. Build the UC5 promotion path and map
q_a_extractions.scope_tags (plural) → q_a_pairs.scope_tag (singular)— a column-alias in the promotion INSERT (cheap; paid in the dormant promotion path, not the shipped write path — see §4). - Decide prompt-constraint vs normalise-at-promotion. Either constrain the LLM prompt to the
now-materialised vocabulary (mirroring the
form_typesnapshot-validator pattern atextraction.py:228-239), or persist free-text and normalise to the controlled set at promotion time. Prompt-constraint presupposes step 1. - ONLY THEN wire q_a search scope-matching — i.e. let the search caller’s
scope_tag && caller_scope_tagsfilter consume promoted scope tags. No search-RPC change is in ID-54 scope (RESEARCH §2); this is the future Task’s terminal step.
§3.2 — evidence_requirements (lighter parallel analysis — same disposition)
Section titled “§3.2 — evidence_requirements (lighter parallel analysis — same disposition)”The adopted evidence_requirements text[] NOT NULL DEFAULT '{}' is likewise a free-form v1
loss-stop, correctly not wired to any taxonomy. RESEARCH §3 (re-verified here) found no
fine-grained evidence-artefact controlled vocabulary anywhere in the migrations or scripts/
corpus. The only adjacent enum is form_template_requirements.requirement_type — a CHECK of
seven coarse values ('policy' | 'statement' | 'evidence' | 'data' | 'narrative' | 'declaration' | 'reference', …pre_squash_reconciliation.sql:4167) with a single 'evidence'
bucket, not a list of evidence kinds like iso27001_certificate. So it is a poor direct
vocabulary. Unlike scope_tags, evidence_requirements carries no inert-column retrieval
risk — nothing filters on it. Follow-up (future, NOT ID-54): if an evidence taxonomy is
ever needed, form_template_requirements.requirement_type is the future normalisation anchor
to start from — but ID-54 must not block on inventing one.
§4 — anti_scope + naming — resolved (blessed, not open)
Section titled “§4 — anti_scope + naming — resolved (blessed, not open)”These were RESEARCH §3 sub-problems / OQs; the adoption decision resolves them as adopted / blessed, not open questions:
- v1 extractions are inclusion-only. The extractor emits no anti-scope field (
QAPairhas onlyscope_tags;rg "anti_scope" scripts/cocoindex_pipeline/is empty). The workspace-overlap exclusion arm (AND NOT (anti_scope_tag && workspaces.scope_tag),…full_schema.sql:279-280) lives on the corpusq_a_pairs.anti_scope_tag(singular,…b_form_type_split.sql:359), which is dormant and defaults to{}. Exclusion is correctly a curation/promotion-time concern; the adopted ID-54 design does not add ananti_scope_tagcolumn toq_a_extractionsnor an anti-scope extractor field. Growing the extractor an anti-scope field is a wider change (extractor + prompt + model) and out of scope. - The column is named PLURAL
scope_tags, mirroring the extractor (extraction.py:257), not the corpus-singularscope_tag. This is blessed: it gives a clean, mechanical_field(pair, "scope_tags", [])write with a 1:1 key↔column↔Pydantic-field name (lowest defect surface; clearest forensic lineage from LLM output → cache column). The promotion path (when built) maps plural→singular — a rename paid in the dormant promotion path rather than the shipped write path, which is the cheaper trade.
§5 — flow.py — as-built deltas (blessed)
Section titled “§5 — flow.py — as-built deltas (blessed)”Both edits are localised and additive; no existing line was moved. Re-verified verbatim
against git show 644f5455 -- scripts/cocoindex_pipeline/flow.py.
§5.1 — Q_A_EXTRACTIONS_SCHEMA — four ColumnDefs added
Section titled “§5.1 — Q_A_EXTRACTIONS_SCHEMA — four ColumnDefs added”# in Q_A_EXTRACTIONS_SCHEMA (after extracted_answer_text):"expected_response_kind": ColumnDef(type="text", nullable=True),"evaluation_criteria": ColumnDef(type="text", nullable=True),"evidence_requirements": ColumnDef(type="text[]", nullable=False),"scope_tags": ColumnDef(type="text[]", nullable=False),nullable mirrors the §2 DDL exactly: expected_response_kind / evaluation_criteria are
nullable=True; the two arrays are nullable=False (never-null, DB-defaulted '{}'). Arrays
use type="text[]" per the existing in-schema array precedent — blessed. asyncpg maps Python
list[str] to the text[] column.
§5.2 — qa_target.declare_row row dict — four keys added
Section titled “§5.2 — qa_target.declare_row row dict — four keys added”# in qa_target.declare_row row dict:"expected_response_kind": _field(pair, "expected_response_kind"),"evaluation_criteria": _field(pair, "evaluation_criteria"),"evidence_requirements": _field(pair, "evidence_requirements", []),"scope_tags": _field(pair, "scope_tags", []),The keys read off each pair via the existing _field(obj, name, default=None) helper
(flow.py:1025-1036); 1:1 key↔column↔Pydantic-field naming. Default-fallback semantics
(blessed): production extractors return Pydantic models (getattr returns the guaranteed
value), but the write-path test stubs return plain dicts (_field reads via dict.get). The
defaults make the write robust to a dict stub omitting a key:
expected_response_kind→None(matches the §2NULLcolumn; a stub without the key writes NULL rather than crashing — theNoneis defensive, since the Pydantic model itself guarantees the value on the LLM path).evaluation_criteria→None.evidence_requirements/scope_tags→[](matches the never-null'{}'DB default; a stub without the key writes an empty array, notNone, satisfying theNOT NULLcolumn).
This mirrors the pre-existing _field(pair, "question_text", "") and
_field(pair, "answer_text") defaulting at the same site.
§6 — database.types.ts — as-built regeneration (blessed)
Section titled “§6 — database.types.ts — as-built regeneration (blessed)”database.types.ts is generated — never hand-edited (CLAUDE.md gotcha); CI guard
supabase-types-parity enforces this. The as-built commit regenerated it from staging
(turayklvaunphgbgscat — the project the migration was pushed to), and the diff adds four
fields to each of Row / Insert / Update of the q_a_extractions block (re-verified at
git show 644f5455 -- supabase/types/database.types.ts):
expected_response_kind: string | null(Row); optional| nullin Insert/Update.evaluation_criteria: string | null.evidence_requirements: string[](Row;NOT NULL⇒ no| null);string[]optional in Insert/Update via the DB DEFAULT.scope_tags: string[](Row); Insert/Update as forevidence_requirements.
No database-overrides.ts change was needed — none of the four columns is a JSONB domain type.
§7 — As-built test + verification evidence
Section titled “§7 — As-built test + verification evidence”Rather than a forward test plan to dispatch, this section documents what the adopted tests already cover and the staging-applied + types-regen facts.
File: scripts/tests/test_cocoindex_flow_write_path.py. The as-built commit extended the
existing write-path test test_ingest_file_declares_rows_with_op_id_field (which stubs
extract_qa_form with a plain dict, exercising the _field dict-path defaulting). Re-verified
verbatim at git show 644f5455 -- scripts/tests/test_cocoindex_flow_write_path.py.
As-built coverage (two paths):
- Present-value pair — the stubbed
qa_pairsfirst entry now carries all four fields with non-default values (expected_response_kind: "mandatory",evaluation_criteria: "scored on completeness",evidence_requirements: ["certificate"],scope_tags: ["lot-1"]). The test asserts all four reachqa.rows[0]verbatim. - Default-fallback pair — a second
qa_pairsentry ("What is Z?") omits the four fields. The test assertslen(qa.rows) == 2and thatqa.rows[1]carries the_fielddefaults:expected_response_kind is None,evaluation_criteria is None,evidence_requirements == [],scope_tags == []— proving the §5.2 defaults match the §2 column nullability/defaults exactly.
Staging + types facts (as-built, integrated by parent):
- Migration
20260530195956_id54_q_a_extractions_form_question_fields.sqlis inmainand already applied to staging (turayklvaunphgbgscat) per the product-owner adoption decision. database.types.tswas regenerated from staging (the four columns appear in theq_a_extractionsRow/Insert/Update — §6).
Acceptance contract (PRODUCT was skipped) — decomposed from the §1 forward-substrate goal:
| Acceptance criterion | Verified by |
|---|---|
All four extractor fields reach the q_a_extractions write (no silent drop) | §7 present-value assertions (as-built) |
Missing optional keys default to NULL/[], not crash | §7 default-fallback assertions (as-built) |
| Column shapes round-trip the §1 extractor guarantees | §2 DDL + §5.1 ColumnDef nullability + the two test paths |
| No regression to the existing 7-key write (op_id, FK, question text) | pre-existing assertions remain green |
Execution note (for the parent’s integration checker): cocoindex uses an LMDB mmap the
sandbox blocks (S252 precedent) — run pytest UNSANDBOXED
(python3 -m pytest scripts/tests/test_cocoindex_flow_write_path.py -q). The guard tests
pipeline-parity.test.ts / doc-freshness.test.ts run on every Vitest invocation; if
pipeline-parity asserts the cocoindex schema column set against the DB, its fixture must
include the four new columns (CLAUDE.md “Guard tests break on structural changes”).
§8 — Out-of-scope / adjacent: OQ-54-E (follow-up for the PARENT’s integration checker)
Section titled “§8 — Out-of-scope / adjacent: OQ-54-E (follow-up for the PARENT’s integration checker)”OQ-54-E — pre-existing extractor_kind CHECK-violation, STILL LATENT in the adopted commit.
Verified this revision: 644f5455 did NOT touch the extractor_kind write line. In the
as-built flow.py diff, "extractor_kind": content_type or "q_a_form", appears as an
unchanged context anchor (the line immediately above the four added keys), not a + line.
On this worktree’s HEAD it is flow.py:1234; on the as-built branch it shifts to ~:1241 after
the schema insert, but the line content is identical and unmodified.
The q_a_extractions.extractor_kind column carries a CHECK permitting only
'prior_bid_response' | 'llm_extraction' | 'yaml_frontmatter_v1' | 'markdown_heading_v1'
(…225456…full_schema.sql:109-115, re-verified verbatim). Neither "q_a_form" nor any
content_type taxonomy value (e.g. "case_study", the write-path test fixture value) is in
that CHECK set. Therefore the first real q_a_form ingest against a CHECK-enforcing DB would
fail the upsert. It has gone unnoticed because the write-path tests use a _FakeTarget stub
that does not enforce the CHECK — so the test suite is green while the live-INSERT hazard
remains.
This is a latent constraint-violation bug still present in the adopted implementation. It is NOT part of the four-field loss and is NOT decided here — it is surfaced for the parent’s integration checker to route to the product owner. Two options (presented, not chosen — the parent/owner owns the call):
- Option A — map the write to a CHECK-valid value. Change
content_type or "q_a_form"to write'llm_extraction'(the correct enum value for the LLM Path-A extractor) on theq_a_extractionswrite. One line, co-located with the ID-54 edits; removes the live-INSERT hazard on the path ID-54 otherwise “fixes”. - Option B — extend the CHECK to include the q_a_form extractor value(s). Keeps the written string but widens the constraint vocabulary (a new migration).
Not in this lane’s scope to fix. This lane authors/commits TECH only. The recommendation is to surface it as a blocking note on the parent’s integration checker so the live-INSERT hazard is not merged silently.
(Adjacent, also deferred per RESEARCH §2: the UC5 promotion path and the q_a_search consumer
chain — both DORMANT — are explicitly not built or modified by ID-54.)
§9 — Follow-ups (deferred, named)
Section titled “§9 — Follow-ups (deferred, named)”scope_tagsalignment to the overlap contract — §3.1: a future Task must (1) materialise the canonical scope vocabulary (workspaces.scope_tagor equivalent), (2) define the plural→singular UC5 promotion mapping, (3) decide prompt-constraint vs normalise-at-promotion, and (4) only then wire q_a search scope-matching. NOT a change to migration20260530195956.- UC5 promotion path (
q_a_extractions → q_a_pairs,promoted_to_pair_id) — DORMANT; when built, mapsscope_tags (plural) → q_a_pairs.scope_tag (singular)and normalises free-text scope tags to the canonical vocabulary. evidence_requirementstaxonomy — §3.2: free-form for v1;form_template_requirements .requirement_typeis a coarse future normalisation anchor only. No inert-column risk.- Extractor anti-scope field (§4) — inclusion-only is blessed for v1; an anti-scope field is a wider extractor+prompt change, only if exclusion-arm scoping is later needed at extraction time.
- OQ-54-E
extractor_kindCHECK violation — §8; LATENT in644f5455. Surfaced to the parent’s integration checker: Option A (map to'llm_extraction') or Option B (extend the CHECK). Pending parent/owner decision.