ID-59 {59.19} PRODUCT — Corpus q_a_extractions → q_a_pairs promotion (route i, extracted_from_corpus)

ID-59 {59.19} PRODUCT — Corpus Q&A promotion (route i, `origin_kind=extracted_from_corpus`)

Status: PRODUCT draft — authored 13/06/2026 by a FRESH planner instance (Q-PLANNER-2). This is a SEPARATE feature from the sibling PRODUCT.md in this dir (per-UC write-back + edit_intent capture); it does NOT overwrite it. Inputs read in full: the {59.19} subtask design grounding (S350 Q&A-coverage trace; Spike-S16 §6 two-tier model), the route-iii promotion endpoint shipped by {59.14} (app/api/q-a-pairs/promote/route.ts), the q_a_pairs / q_a_extractions schema, the q_a_search RPC, and the ID-45 INV-23 definition.

GAP-1 (S350): the corpus-wide q_a_extractions → q_a_pairs promotion (route i) was DESIGNED in S16 (two-tier model: extractions are the derived cache, pairs are the corpus surface) but never BUILT — only route iii (form-responses → pairs, UC5 {59.14} ✅) ships. This PRODUCT specifies the behaviour of route i. The companion {59.20} TECH (write-tech-spec) is the next chain step.

Summary

After the source-driven re-ingest (ID-45) re-walks the full corpus, extracted Q&A lands in the q_a_extractions derived cache — but Q&A search (q_a_search) only reads q_a_pairs, so extracted Q&A is INVISIBLE to consumers until promoted. Route i promotes each live (non-invalidated) corpus q_a_extraction into a q_a_pairs row tagged origin_kind='extracted_from_corpus', with promoted_to_pair_id linkage written back to the source extraction, idempotently (re-walking the corpus never duplicates pairs), and drives the promoted pair to a state in which it is visible to q_a_search — satisfying ID-45 INV-23.

Problem

The platform stores Q&A in a two-tier model (S16 §6): q_a_extractions is a derived, pipeline-UPSERTed cache (one row per extracted question, FK to the source content_item), and q_a_pairs is the curated corpus surface that consumers read. The promotion taxonomy has multiple routes: route i = corpus extractions (extractor_kind ∈ {llm_extraction, markdown_heading_v1, yaml_frontmatter_v1}) → pairs (origin_kind='extracted_from_corpus'); route iii = form-responses → pairs (origin_kind='derived_from_form_response', UC5 {59.14} ✅). Route i was designed in S16 §6 (promoted_to_pair_id lineage, origin_kind enum value) and 05-qa-flow §3.1/§9, but §9’s promotion flow only ever specced route iii. The decision-graph (S231 Q3.4/Q3.5) framed LLM-extraction as a one-shot migration helper rather than a recurring step, so historical Q&A was assumed converted once via origin_kind='imported_legacy'. ID-45 changes that assumption: the re-ingest re-walks the FULL corpus fresh into q_a_extractions on every run, so extraction → pair promotion must run for the re-ingested corpus, repeatably.

This GAP is a pre-cutover dependency for ID-45 INV-23: until extracted Q&A is promoted to q_a_pairs in a search-visible state, corpus Q&A is invisible to q_a_search after re-ingest.

Code-intelligence orientation

gitnexus_query({query: 'q_a_extractions q_a_pairs promotion'}) (repo knowledge-hub, 13/06/2026) returned the route-iii promotion flow plus the Python extraction producers; cited verbatim below. gitnexus_context({name: 'POST', file_path: 'app/api/q-a-pairs/promote/route.ts'}) confirmed the route-iii call graph. ast-dataflow is TS-only and does not cover the Python pipeline or SQL migrations, so the q_a_extractions/q_a_search claims are grounded by a grep sweep over scripts/, supabase/migrations/ plus direct migration reads.

promoted_to_pair_id has ZERO writers — confirming the GAP. grep -rniE 'promoted_to_pair_id' over lib/ app/ components/ types/ scripts/ supabase/ (13/06/2026) returns exactly ONE hit — the schema column definition (supabase/migrations/20260520225456_t6_q_a_pairs_full_schema.sql:120, promoted_to_pair_id uuid NULL REFERENCES public.q_a_pairs(id) ON DELETE SET NULL). No TS, Python, or SQL code WRITES it. Route i is greenfield against a ratified design.
q_a_extractions is read only by Python pipeline tests — no TS reader. grep -rniE 'q_a_extractions' over TS dirs returns only scripts/tests/test_cocoindex_*.py matches (the cocoindex UPSERT-target tests). Confirms the {59.19} grounding: “zero TS reads q_a_extractions”.
origin_kind='extracted_from_corpus' is a valid value with ZERO live writers. The CHECK constraint q_a_pairs_origin_kind_check (…t6_q_a_pairs_full_schema.sql:68) admits ('extracted_from_corpus', 'curated_explicit', 'derived_from_form_response', 'imported_legacy'). The ONLY live origin_kind writer is app/api/q-a-pairs/promote/route.ts:159 writing 'derived_from_form_response' (route iii). 'extracted_from_corpus' is admissible but never written — the GAP.
Route-iii promotion shape (the pattern route i mirrors) — gitnexus_context verbatim. POST at app/api/q-a-pairs/promote/route.ts:70 (process proc_97_post) calls getAuthorisedClient, authFailureResponse, parseBody, tryQuery, coerceIntent, safeErrorMessage (outgoing calls). It does NOT call arbitrate/arbitrateMany (single-actor). It INSERTs a q_a_pairs row with origin_kind='derived_from_form_response', publication_status='draft', lineage (source_form_response_id, source_question_id), via the RLS-scoped authorised client (no service-role escalation). Route i diverges from this shape in three load-bearing ways: (a) it is corpus-batch not per-response; (b) it writes promoted_to_pair_id lineage on the source EXTRACTION, not response/question FKs on the pair; (c) it must reach search-visibility, not stop at draft — see INV-9/INV-10.
q_a_search visibility predicate (the INV-23 gate) — verbatim. public.q_a_search (supabase/migrations/20260520231524_t6_q_a_search_rpcs.sql:71) ranks pairs WHERE qap.question_embedding IS NOT NULL AND qap.publication_status = 'published' (lines 117–118). So a promoted pair is search-visible ONLY when BOTH publication_status='published' AND question_embedding IS NOT NULL. Route iii stops at draft with NO embedding — insufficient for INV-23. grep -rniE 'question_embedding' over lib/ app/ scripts/ returns NO application or pipeline writer of q_a_pairs.question_embedding (only the migration/RPC definitions) — route i is the first writer of this column on the pair surface (generateEmbedding lives at lib/ai/embed).
q_a_extractions shape — the idempotency + invalidation surface. …t6_q_a_pairs_full_schema.sql:103: id, source_content_item_id, extractor_kind, extracted_question_text, extracted_answer_text, extraction_metadata jsonb, promoted_to_pair_id uuid NULL, invalidated_at timestamptz NULL, created_at, updated_at. The schema comment (lines 122–124): “Invalidation audit trail — set by cocoindex when source content changes (§3.1). Invalidated rows are NOT deleted; new extraction rows are emitted for new content.” So invalidated_at IS NULL is the live-row predicate, and promoted_to_pair_id IS NOT NULL is the already-promoted predicate — both load-bearing for idempotency (INV-5).
ID-45 INV-23 verbatim (specs/id-45-full-corpus-reingest/PRODUCT.md:257): “Extraction count is plausible vs source QA count … and the UC5 promotion q_a_extractions → q_a_pairs succeeds post-run as a separate assertion. Validation: … post-promotion q_a_pairs row count > 0 and consistent with promoted extractions.” (INV-23 says “UC5” but means the extraction→pair promotion, i.e. route i — UC5 is route iii of form-responses and does not read q_a_extractions.)

Goals / Non-goals

Goals. Promote live corpus q_a_extractions into q_a_pairs tagged origin_kind='extracted_from_corpus'; write promoted_to_pair_id lineage back to the source extraction; make the promotion idempotent so re-walking the corpus never duplicates pairs; drive each promoted pair to a state visible to q_a_search (publication_status='published' AND question_embedding IS NOT NULL); satisfy ID-45 INV-23.

Non-goals (v1). Route iii (form-responses → pairs — UC5 {59.14} ✅, separate route). Route ii / any other taxonomy route. Cross-workspace dedup of promoted pairs (UC8 — separate {59} slice). The edit_intent arbitration substrate (sibling PRODUCT.md — route-i promotion is single-actor / batch, does NOT invoke arbitrate()). One-shot legacy migration (origin_kind='imported_legacy' — historical, already-run). Re-extraction itself (the cocoindex flow OWNS writing q_a_extractions; route i only READS them). The DDL/migration mechanics (a {59.20} TECH concern); this PRODUCT specifies the behaviour the promotion enables.

Figma: none provided. The only user-facing surface is the optional curated review queue (INV-13/INV-14); when built it follows Warm Meridian semantic tokens and the existing review-tab pattern.

Behavior

“User” here is twofold: the operator running the post-re-ingest promotion (and the ID-45 INV-23 Checker asserting it succeeded), and the downstream consumer of q_a_search (MCP / UI) who must find the re-ingested corpus Q&A. Behaviour is specified so both can rely on it.

A. Promotion scope and trigger

Promotion source = live corpus extractions only. Route i promotes q_a_extractions rows that are (a) NOT invalidated (invalidated_at IS NULL) and (b) NOT already promoted (promoted_to_pair_id IS NULL). An invalidated extraction (its source content changed; cocoindex has superseded it with a fresh row) is NEVER promoted. An already-promoted extraction is NEVER re-promoted (INV-5). Rows with no usable answer text are skipped, not failed (INV-7).
Promotion reads, never writes, the extraction cache’s identity. Route i is downstream of the cocoindex flow: cocoindex OWNS creating/invalidating q_a_extractions rows; route i only READS them and WRITES promoted_to_pair_id back. Route i MUST NOT create, delete, or re-key q_a_extractions rows, and MUST NOT write any source file or content_item — promotion is a KH-DB-only operation over the two Q&A tables.
Trigger — post-re-ingest batch (v1 default). Route i runs as a batch step AFTER an ID-45 re-ingest completes and BEFORE the cutover gate is asserted, walking all live, unpromoted extractions in the corpus in one operation. The batch is operator-invokable (and ID-45-pipeline-invokable) and reports a structured summary: counts of extractions considered, promoted, skipped (with reason), and already-promoted.
- Open question (OQ-59-19-1): auto (post-re-ingest batch, v1 default above) vs curated (human review queue, draft → in_review → published per INV-13/INV-14). Recommendation: ship the auto batch for v1 because ID-45 INV-23 requires post-run search-visibility without a human in the loop; offer the curated queue as an opt-in/v1.1 affordance layered ON the same promotion primitive. Needs Liam ratification — it determines whether INV-9 drives pairs to published directly or parks them at in_review for human publish.

B. Tagging, lineage, and field mapping

origin_kind tagging. Every pair created by route i is tagged origin_kind='extracted_from_corpus' (a value the q_a_pairs_origin_kind_check already admits). Route i MUST NOT write any other origin_kind, and MUST NOT mutate the origin_kind of a pair it did not create. A pair’s origin_kind is how a consumer (and the Checker) distinguishes corpus-promoted Q&A from form-response-promoted (route iii, derived_from_form_response) or curated (curated_explicit) Q&A.
promoted_to_pair_id lineage — exactly-once linkage. On promoting an extraction, route i sets that extraction’s promoted_to_pair_id to the id of the newly created pair. The linkage is the idempotency key (INV-5/INV-6): an extraction with promoted_to_pair_id already set is treated as already-promoted and skipped. The linkage MUST be written in the same logical operation as the pair insert so a created pair is never left without its back-link (no orphaned pair, no double-promotion window — INV-8). The relationship is one extraction → one pair (1:1); route i does NOT merge multiple extractions into one pair in v1 (dedup is UC8, a non-goal).
Field mapping. The promoted pair carries: question_text from the extraction’s extracted_question_text; answer_standard from extracted_answer_text; alternate_question_phrasings from the extraction’s phrasings (carried through the cache per the ID-94.1/G4 path) when present, else the empty default. The pair’s source_content_item_id-derived provenance (the extraction’s source_content_item_id) is preserved so a consumer can trace a promoted pair back to its corpus content item. The pair is NOT given form-response FKs (source_form_response_id / source_question_id are route-iii lineage and remain NULL for route-i pairs).

C. Idempotency on re-ingest

Skip, don’t fail, on unpromotable rows. An extraction with no usable answer text (NULL/empty extracted_answer_text) is SKIPPED with a recorded reason and does not create a pair (mirrors the route-iii 422 “no answer text to promote” guard, applied as a batch skip rather than an HTTP error). Skips are counted and surfaced in the batch summary so the operator/Checker can reconcile “considered vs promoted” without a silent drop.
Idempotent re-run — no duplicate pairs. Running route i twice over the same corpus state produces the SAME set of q_a_pairs — the second run finds every still-live extraction already carries promoted_to_pair_id and promotes nothing new. Re-walking the corpus (a fresh ID-45 re-ingest) MUST NOT duplicate pairs for extractions that were already promoted and remain live. The invariant a consumer relies on: the count of extracted_from_corpus pairs equals the count of live extractions that have ever been promoted — never more.
Re-ingest that supersedes an extraction does not orphan its pair silently. When cocoindex invalidates an extraction (invalidated_at set) and emits a fresh extraction for the changed content, the NEW extraction (invalidated_at IS NULL, promoted_to_pair_id IS NULL) is eligible for promotion and produces a new pair on the next route-i run; the OLD invalidated extraction is never re-promoted (INV-1). The disposition of the pair created from the now-invalidated extraction (supersede vs leave-published) is the lifecycle concern of INV-12.
- Open question (OQ-59-19-2): when an extraction is invalidated after its pair was published, does route i mark the stale pair archived/superseded_by the new pair, or leave it published until a separate reconciliation step acts? Recommendation: v1 leaves the stale pair published (no silent corpus loss) and records the supersession lineage where the schema allows (superseded_by); active stale-pair retirement is v1.1 / a reconciliation concern. Needs Liam ratification — it bounds whether route i is purely additive or also retires.

D. Search visibility (satisfying ID-45 INV-23)

A promoted pair becomes visible to q_a_search. q_a_search returns a pair ONLY when BOTH publication_status='published' AND question_embedding IS NOT NULL (verified verbatim from the RPC). Therefore a route-i promotion is not “done” — and ID-45 INV-23 is not satisfied — until each promoted pair (under the auto-batch trigger, INV-3) reaches publication_status='published' AND has a non-NULL question_embedding. A pair sitting at draft with no embedding (the route-iii resting state) is NOT search-visible and does NOT satisfy INV-23.
question_embedding is populated as part of promotion. Because no existing writer populates q_a_pairs.question_embedding, route i is responsible for setting it (from the pair’s question_text, using the platform’s standard 1024-dim embedding, consistent with the value q_a_search cosine-compares against). A pair that cannot be embedded (embedding step fails) is NOT silently published as invisible: it is recorded as a promotion failure for that extraction (its promoted_to_pair_id is still set so it is not re-attempted blindly, but the failure is surfaced in the batch summary so the operator/Checker sees the shortfall) — the platform MUST NOT report INV-23 success while pairs are silently un-embedded.
- Open question (OQ-59-19-3): on embedding failure for a row, retry-in-batch vs defer to a re-embed pass. Recommendation: surface-and-continue in v1 (do not abort the whole batch for one embedding failure), with the failure count gating the INV-23 “consistent with promoted extractions” assertion. Needs Liam ratification.
Lifecycle transitions are honoured. Under the curated trigger (INV-13), promoted pairs move draft → in_review → published through the existing publication-status lifecycle; under the auto trigger (INV-3, v1 default) route i drives them to published directly (with the embedding set, INV-11). In BOTH cases the pair MUST NOT skip the question_embedding requirement: a pair reaching published without an embedding is a defect, because it would be silently absent from q_a_search despite appearing published. (testStrategy of {59.19}: “lifecycle draft → in_review → published honoured” — satisfied by INV-12; the auto path is the published-direct specialisation.)

E. Roles, scoping, and AI-invisibility

Curated review queue (if built — INV-3 OQ disposition). If the curated trigger is chosen, route-i promotions land as q_a_pairs drafts in a review surface where an admin/editor reviews and publishes each (or in bulk), mirroring the existing review-tab pattern. The surface uses Warm Meridian semantic tokens, distinguishes states by more than colour (WCAG 2.1 AA), shows an explicit empty state when the queue is empty, and uses UK English (DD/MM/YYYY). This surface is READ-then-publish; it does not let a reviewer edit the extraction cache.
Authorisation. The promotion operation (batch or per-row publish) is restricted to admin/editor roles, following the route-iii guard pattern (getAuthorisedClient(['admin', 'editor']), failures via authFailureResponse). The batch runs through an authorised/operator context (not anon, not a public route); any HTTP surface is authenticated and NOT added to proxy.ts publicRoutes. Viewers/reviewers cannot promote or publish.
Workspace scoping. Route i is corpus-level (the corpus is the re-ingest scope); it MUST NOT leak Q&A across a workspace boundary the acting operator/consumer cannot otherwise reach. Promoted pairs inherit q_a_pairs RLS exactly as curated pairs do; q_a_search continues to apply its caller-side scope-tag overlap filter, so a promoted corpus pair is visible to a consumer only within that consumer’s scope, identically to any other published pair.
AI-invisible infrastructure. Where the extraction (LLM extractor) or the embedding involves AI, that involvement is NOT surfaced as a user-facing “AI feature”: a promoted corpus pair behaves in q_a_search and the Q&A library exactly as if the Q&A had always been curated. A consumer cannot tell from the search/get contract that the pair was machine-extracted other than via the neutral origin_kind='extracted_from_corpus' provenance value.

Open questions (collected)

OQ-59-19-1 (INV-3): Auto post-re-ingest batch (v1 default, drives to published) vs curated review queue (draft → in_review → published). Recommend auto for v1 (INV-23 needs post-run visibility without a human gate); curated as v1.1 opt-in on the same primitive. Liam ratify.
OQ-59-19-2 (INV-9): When an extraction is invalidated after its pair published — retire the stale pair (archived/superseded_by) or leave it published until a reconciliation step? Recommend leave-published + record lineage for v1; active retirement v1.1. Liam ratify.
OQ-59-19-3 (INV-11): Embedding-failure policy — surface-and-continue (recommended) vs abort-batch vs deferred re-embed pass. Liam ratify.

End of PRODUCT. Behaviour invariants INV-1..INV-16. Acceptance a Checker can verify against: (INV-4) every route-i pair has origin_kind='extracted_from_corpus'; (INV-5) each promoted extraction has promoted_to_pair_id set to its pair, 1:1; (INV-8) re-running promotion over the same corpus state creates zero new pairs and the extracted_from_corpus count equals the live-ever-promoted-extraction count (a fresh ID-45 re-walk emits new extraction rows and may legitimately add pairs — the guarantee is scoped to identical corpus state, per INV-8 body); (INV-10/INV-11) promoted pairs are publication_status='published' AND question_embedding IS NOT NULL, and appear in q_a_search results post-run with count > 0 consistent with promoted extractions (ID-45 INV-23); (INV-12) lifecycle draft → in_review → published honoured (auto path = published-direct). Companion {59.20} TECH (write-tech-spec) is the next chain step.