Phase 0.9 — Spike S16 (NEW) — KH canonical Q&A schema design

Phase 0.9 — Spike S16 (NEW) — KH canonical Q&A schema design

Audit date: 2026-05-11 (KH session S230) Branch: content-items-investigation (worktree-agent-a753854cc4f5d20e4 → spike-s16-qa-schema-design) Spike trigger: Liam clarification at S230 start — “We are in a position to be able to define what Q&A pairs should look like before they’re added to the main documentation file… Each client will be different, and it’s likely that we need to make sure that the format is suited to however we need it to be.” Pre-v1, pre-launch, pre-re-ingest schema design opportunity. Test corpus: docs/client-documentation-base/ (8 binary docx/pdf + 25 markdown + 2 forms — first-client Phew material; not modified) Spike scope: DESIGN ONLY — no schema files, validation files, migrations, or client docs modified. Outputs are recommended shapes + open questions for ratification at review.

Spike status: DELIVERED — DESIGN-RECOMMENDED. v1 canonical Q&A shape proposed with column list + ingest envelope + MCP retrieval contract + migration sketch. Client-input policy: predetermined markdown shape (YAML-frontmatter per pair) — KH defines the canonical wire format pre-ingestion. Existing Phew content (5 historical shapes) migrates to canonical shape via one-shot tooling, not as a recurring v1 ingestion path. Aligned with all S2/S3/S10/S12/edit-flow §6 ratifications.

Verdict: READY-FOR-REVIEW. No implementation work in this spike — schema + envelope + MCP shapes are recommendations for Liam’s review at the next decision-graph rewrite pass.

---

1. Executive summary

Question	Answer
What is the v1 canonical KH Q&A pair shape?	A new q_a_pairs table (curated, golden record) + a new q_a_extractions table (derived cache). Both temporal-aware (Shape A — valid_from / valid_to / provenance / confidence). 14-column q_a_pairs core + 10-column q_a_extractions cache, both with question-embedding for retrieval. Inherits the existing per-row temporal + ingest_source + publication_status + supersession patterns from content_items so the rest of the platform’s invariants apply.
Does it round-trip through cocoindex localfs (S2)?	Yes — but only for the q_a_extractions path (cocoindex flow populates extractions; Pattern A/B parser kept as the docx adapter, repurposed). The q_a_pairs curated table is KH-DB-only (per UC5 + UC6 RATIFIED). cocoindex Δ on a source file invalidates the corresponding q_a_extractions rows but never deletes q_a_pairs rows (these are user-curated artefacts; soft-archive via publication_status='archived').
Does it support dedup-with-temporal (UC8 + DRAFT-vs-final)?	Yes. valid_to + confidence + superseded_by columns inherited from S12 (provenance enum) and the existing content_items.dedup_status state machine. q_a_pairs.superseded_by enables UC8 user-driven merge; the supersession trigger pattern matches content_items.superseded_by precedent (S183/186 WP-B.1). Cross-source dedup substrate is S10’s responsibility — schema is ready for either S10 winner.
Does it map to mempalace KG triples (Shape A)?	Indirectly. Q&A pairs are not edges per se — they are subject artefacts that generate edges. The triple shape (q_a_pair_id, asserts_fact_about, entity_id) plus (q_a_pair_id, cites, content_chunk_id) rides on entity_relationships (Shape A) and the new citations table from intended-arch §4.3. No new triple table for Q&A. Q&A → triple emission is Pass 2’s job at write time, not the schema’s.
Does it surface in search_knowledge_base_strict MCP?	Yes — via a NEW MCP tool q_a_search plus q_a_get. Two-step pattern per S3 §4.1 (list/preview → get/verbatim). q_a_search returns {q_a_pair_id, question_preview, scope_tags, confidence, valid_from, current} ranked by question_embedding cosine OR question fulltext OR both (hybrid). q_a_get(q_a_pair_id) returns verbatim {question_text, answer_standard, answer_advanced, citations[], provenance}. Mirrors mempalace’s mempalace_list_drawers + mempalace_get_drawer exactly.
Does it accommodate Pass 2 / graphify confidence (S12 + Q4.14)?	Yes — single provenance text column (EXTRACTED / INFERRED / AMBIGUOUS) + numeric confidence numeric(3,2) at both q_a_pairs and q_a_extractions row level (same shape S12 ratified for entity_relationships). Pass 2 emission rules from graphify SKILL.md §confidence_score apply identically. No backfill (pre-launch re-ingest per CX.22 RESOLVED).
Client-input policy?	Predetermined markdown shape (YAML-frontmatter per pair) — KH defines the canonical wire format pre-ingestion. v1 ingestion path = predetermined markdown. Existing Phew content (5 historical shapes catalogued in §4) migrates to canonical shape via one-shot tooling; the existing Pattern A/B docx extractor + a one-shot markdown-heading + YAML-frontmatter + (optional) LLM-extraction tooling get reused as migration helpers, not as recurring v1 ingestion paths. Spec in §7.
Migration sketch needed?	Yes — 2 new tables (q_a_pairs + q_a_extractions) + 1 history table (q_a_pair_history) + q_a_pair removed from content_items.content_type CHECK constraint post-cutover. Polymorphic citations table inherited from intended-arch §4.3 — migration deferred per Q-S16-5. content_items.answer_standard / answer_advanced drop in Migration 5 (post-cutover, NOT in this spike). Sketch in §10 — staged across 4 migrations. Per CLAUDE.md anon-EXECUTE revoke rule, every new SECDEF function gets explicit revoke.

G16 decision-gate verdict: DESIGN-READY-FOR-REVIEW — recommended schema + ingest envelope + MCP tool shapes are internally consistent and tracked against every S2/S3/S10/S12/edit-flow §6 ratified decision. No implementation in this spike. Final binding awaits Liam’s review pass at the next decision-graph rewrite + Q3.5 ([RATIFY-AT-REVIEW]) Pattern A/B parser fate resolution.

---

2. Method (design-only, no live extraction)

2.1 Materials read

The spike is design-only — no code, no schema apply, no live DB query. Inputs:

• docs/reference/SCHEMA-QUICK-REFERENCE.md — current content_items Q&A-related columns (answer_standard, answer_advanced, content_type='q_a_pair'); bid_questions + bid_responses + content_citations adjacent surfaces; CHECK constraints; metadata JSONB keys for import.
• lib/validation/schemas.ts — VALID_CONTENT_TYPES, EDITABLE_FIELDS, body validation for editor PATCH on Q&A fields, KBIntegrationBodySchema for bid → KB promotion.
• scripts/import_bid_library.py (909 LOC) — current pipeline orchestrator: extract → dedup → classify → embed → store; chunk-on-insert; auto-supersede; entity extraction follow-up.
• lib/bid-library-ingest/extract-qa-pairs.ts (497 LOC) — Pattern A/B/C docx table parser; output shape {questionText, answerStandard, answerAdvanced, sectionName, sourceFile, tableIndex, rowIndex}.
• docs/client-documentation-base/ — 35-file client corpus (8 binary + 25 markdown + 2 forms); inventoried in §3.
• docs/plans/phase-0-investigation/0.9-intended-architecture.md §4.3 + §6 — proposed q_a_pairs + q_a_extractions schemas, Pattern A/B repurpose.
• docs/plans/phase-0-investigation/0.9-edit-flow-investigation.md §6.5 + §6.6 + §6.8 — UC5 (bid → Q&A promotion) + UC6 (Q&A revision) + UC8 (dedup merge) RATIFIED decisions.
• docs/plans/phase-0-investigation/0.9-spike-S2-cocoindex-folder-binding.md — localfs-only v1 connector list; cocoindex Δ semantics for q_a_extractions invalidation.
• docs/plans/phase-0-investigation/0.9-spike-S3-mempalace-observe.md §4 + §5.3 + §5.4 — Shape B two-step retrieval pattern; mempalace miner does NOT extract Q&A from non-transcript markdown (KH must own); triple emission is KH’s job not the miner’s.
• docs/plans/phase-0-investigation/0.9-spike-S12-graphify-confidence.md — provenance + confidence column pattern ratified for entity_relationships; applied identically here.
• docs/plans/phase-0-investigation/0.9-context.md — North Star; OQ ratifications; canonical baseline corpus.

2.2 Investigation tasks executed (from prompt)

1. Audit current state — §3.
2. Audit client input shapes — §4. 11+ client files inspected (4 final audit docx + 3 DRAFT docx + Phew-Bid-Library + LMS_Bid_Library + Website_Bid_Library + Advanced_Audits_Bid_Library + 2 forms); 5 distinct structural shapes catalogued (Phew historical baseline; not v1 ingestion paths).
3. Audit KH-extracted shape — §5. Pass 2 today produces zero structured Q&A; graphify extract.py is also not a Q&A extractor (per §5.3 in S3 spike). Conclusion: KH must own the per-source-shape adapter set.
4. Design canonical shape — §6.
5. Client recommendations — §7.
6. Ingest format spec — §8.
7. MCP retrieval shape — §9.
8. Migration plan — §10.

2.3 Out of scope

• Live DB query. The target tables don’t exist on staging or prod (verified — no q_a_extractions or q_a_pairs migrations in supabase/migrations/); design-only spike doesn’t need observation.
• Code prototypes. Adapter implementations, Zod schemas, MCP tool registration code — all deferred to the implementation phase that follows this spike’s review.
• S10 substrate decision. UC8 cross-record dedup substrate (cocoindex @coco.fn vs mempalace KG entity-resolution vs hybrid) is ratified by S10 — this spike defines q_a_pairs.superseded_by + supersession trigger pattern but does NOT decide where the detection signal originates.
• Q3.5 Pattern A/B parser final fate. Working recommendation per intended-arch §6.3 is “repurpose as corpus-mining tool”; this spike treats that as the working assumption but flags the [RATIFY-AT-REVIEW] status. If S8 spike returns “retire entirely”, §10 migration sketch is unchanged — only the adapter library inventory in §7 changes.

---

3. Audit — current state

3.1 Q&A in the live database (today)

Surface	Status today	Notes
q_a_extractions table	Does not exist.	Mentioned in intended-architecture.md §4.3 as a NEW table to add. No migration.
q_a_pairs table	Does not exist.	Same as above — NEW per intended-arch.
content_items with content_type='q_a_pair'	Exists today.	Per intended-arch.md Q3.1 RESOLVED: “Separate domain — own tables.” Existing rows (the 395 prod rows per intended-arch §6.1) become imported_legacy origin_kind at migration time.
content_items.answer_standard (text, nullable)	Exists.	Migrates to q_a_pairs.answer_standard per intended-arch §4.2 REMOVE list.
content_items.answer_advanced (text, nullable)	Exists.	Same — migrates to q_a_pairs.answer_advanced.
bid_questions table	Exists.	Stays — represents tender questions inside bid workspaces (project_id is FK to workspaces; not Q&A library content).
bid_responses table	Exists.	Stays — represents bid-question answers (in-flight; can be promoted to q_a_pairs per UC5 RATIFIED).
content_citations table	Exists (bid_response_id ↔ content_item_id).	Per intended-arch §4.3 superseded by new polymorphic citations table (citing_entity ∈ {bid_response, sales_proposal, q_a_pair, mcp_search_response}); legacy content_citations migrates and gets archived.
content_items.content_type CHECK	Includes 'q_a_pair'.	Per intended-arch §4.2 REMOVE: drop q_a_pair from CHECK once q_a_pairs table is the canonical home.
content_items.metadata keys for Q&A	section_name, table_index, row_index, has_standard, has_advanced (per §36 metadata key reference)	These migrate to q_a_extractions typed columns (per §6 design).

3.2 extractQaPairs output today (lib/bid-library-ingest/extract-qa-pairs.ts)

The TS Pattern A/B/C parser produces this output per Q&A pair:

interface QaPair {
  questionText: string;       // plain text, not markdown
  answerStandard: string;     // GFM markdown (post-Turndown)
  answerAdvanced: string;     // GFM markdown; empty string if no advanced column
  sectionName: string;        // heading text above the table (deduplicated)
  sourceFile: string;         // bare filename for provenance
  tableIndex: number;         // 0-indexed table position in doc
  rowIndex: number;           // 0-indexed row position in table
}

Three table formats recognised + 2 positional fallbacks:

Format	Cols	Schema
audit_6col (Pattern A)	6	No, Section, Question, Standard, Advanced, Notes
draft_5col (Pattern B)	5	No, Section, Question, Standard, Notes
numbered_6col (Pattern C)	6	No, Question, Standard, Advanced, Section, Notes
positional_5col (fallback)	5	Question, Standard, [3 unused]
positional_6col (fallback)	6	Question, Standard, Advanced, [3 unused]

HEADER_MAP (TS: 98 entries; Python parallel adds standard-selection-questionnaire entries — combined ~128) normalises column-header variants per canonical name (question, standard, advanced, section, number, notes). Pattern A/B/C selection is deterministic from header detection.

extractQaPairs calls mammoth.convertToHtml() → custom extractHtmlTables() regex split → per-cell HTML → htmlToMarkdown() (Turndown + GFM plugin per CLAUDE.md gotcha “mammoth convertToMarkdown() drops tables”).

3.3 Python pipeline orchestrator (scripts/import_bid_library.py)

Current ingest sequence (per §3 step list):

1. Find .docx files (non-recursive *.docx glob).
2. Track Changes detection per scripts/docx_utils.py has_tracked_changes().
3. Extract Q&A pairs via extract_qa_from_docx() (Python parallel of TS extractQaPairs).
4. Cross-file title dedup (dedup_across_files_by_title()) — keeps first-seen normalised title.
5. Exact dedup (MD5 of normalised question text).
6. Near-duplicate detection at default 0.85 threshold; informational only (flagged, not removed).
7. Classify via kb_pipeline.keyword_classifier.classify_pairs().
8. Quality validation — flag empty content + fragments (<20 chars).
9. Idempotency check — check_question_exists(question_text) REST query against content_items (matches title ILIKE first 80 chars; content_type='q_a_pair').
10. Embed — kb_pipeline.embed.build_embedding_text() + generate_embedding(); text = title (≤120 chars) + summary (≤500 chars) + content; OpenAI text-embedding-3-large at 1024 dims (Matryoshka shortened).
11. Store — kb_pipeline.store.insert_content_item() writing to content_items (rows tagged ingest_source='qa_import'); chunks via kb_pipeline.chunk.store_chunks(); auto-supersede if --auto-supersede flag set (content-hash dedup + DRAFT→final filename heuristic).
12. Entity extraction (--entities opt-in) via kb_pipeline.classify.classify() then run_post_insert() with store_entities_flag=True.

build_content_record(pair, batch_name) at line 215 returns this dict (the current write payload):

{
  "title": <truncated question, ≤120>,
  "content": "Q: <question>\n\n<answer_standard>\n\n<answer_advanced>",
  "answer_standard": <answer_standard or None>,
  "answer_advanced": <answer_advanced or None>,
  "content_type": "q_a_pair",
  "platform": "extraction",
  "source_url": "",
  "source_domain": "",
  "primary_domain": <or None>,
  "primary_subtopic": <or None>,
  "secondary_domain": <or None>,
  "secondary_subtopic": <or None>,
  "classification_confidence": <0.0+>,
  "classified_at": <ISO ts>,
  "summary": <truncated answer, ≤200>,
  "ai_keywords": [<3-5 deterministic keywords>],
  "source_file": <basename>,
  "metadata": {
    "section_name": <heading>,
    "table_index": <int>,
    "row_index": <int>,
    "has_standard": <bool>,
    "has_advanced": <bool>,
    "import_batch": <batch_name>,
    "has_tracked_changes": <bool>,
  },
  "ingest_source": "qa_import",
  "layer": <inferred>,
  "user_tags": [<batch_tag>] if set,
}

Observations on current state:

• The Q&A pair is stored as a content_items row today. Its question goes into title; the answer goes into answer_standard + answer_advanced (typed columns) and into content (concatenated for FTS / embedding). This conflates two distinct concerns in one table.
• No first-class question_embedding — embedding is built over title + summary + content (question + answer text together), which is fine for “find related Q&A” but suboptimal for “find Q&A matching THIS question shape” (which is what bid-question matching wants).
• No temporal columns on Q&A. created_at + updated_at exist but no valid_from / valid_to for Shape A temporal queries. Versioning is via content_history row.
• No origin classification. Today every Q&A row is functionally the same — no distinction between “curated by editor”, “extracted from docx”, “promoted from bid response”, “imported legacy”.
• No extraction-confidence vs curation-confidence distinction. classification_confidence is per-row classifier output; nothing distinguishes “Pattern A high-confidence table cell” vs “LLM-derived from prose”.
• metadata JSONB carries provenance (section_name, table_index, row_index) — violates CLAUDE.md “Prefer proper schema — tables and columns over JSONB for key data”. These columns ought to be typed in q_a_extractions.
• No FK to source content_items (corpus). A Q&A extracted from doc X has no row-level pointer to doc X; today’s lineage rides on source_file (basename) which is brittle (different paths, renames break it).

3.4 Pattern A vs Pattern B observation (current parser)

Pattern A (audit_6col) and Pattern C (numbered_6col) both produce separate standard + advanced answer columns — two answer variants per question. Pattern B (draft_5col) produces standard only (no advanced). The positional fallbacks default to question + standard (5-col) or question + standard + advanced (6-col).

The “standard / advanced” pattern is specifically Phew’s idiom — different audit pricing tiers warranting different answer detail. It is not universal: future clients are vanishingly unlikely to use the same two-tier shape. The KH-canonical schema must accommodate “single answer” as the dominant case + “tiered answers” as an optional Phew-style extension.

Working design choice (§6): q_a_pairs.answer_standard TEXT NOT NULL + q_a_pairs.answer_advanced TEXT NULL mirrors today’s column layout (zero data loss for Phew migration) and degrades gracefully for non-Phew clients (advanced is just NULL).

---

4. Audit — client input shapes

Inspected (no extraction; structural inventory only):

4.1 Shape A — Final docx with audit-6col tables

File: docs/client-documentation-base/binary/2026 Audit - Tender and Bid Library Template - FAQs .docx + .../Funtionality.docx + .../Implementation & Support.docx + .../Security & Compliance.docx.

Markdown equivalent: docs/client-documentation-base/markdown/2026 Audit - Tender and Bid Library Template - FAQs .md (58 lines).

Structure:

• H1 / H2 headings sectioning the document (e.g. ”# Product Overview”, ”# Security & Data Management”).
• Each section contains a single GFM markdown table with 6 columns: Question | Answer for Standard Audit System | Answer for Advanced Audits | Source of Information | Date Added | LT Verified for accuracy, date and initial.
• 12-25 rows per table; ~40 Q&A pairs total per document.

Adapter today: Pattern A (audit_6col) in extractQaPairs. Works. Mapping: question → question; “Answer for Standard Audit System” → standard; “Answer for Advanced Audits” → advanced; “Source of Information” → drop (could promote to citation entry); “Date Added” → drop (could promote to valid_from); “LT Verified…” → drop (could promote to verified_at + verified_by).

4.2 Shape B — DRAFT docx with draft-5col tables (no advanced column)

File: docs/client-documentation-base/binary/DRAFT 2026 Tender and Bid Library Template for Phew - FAQs - Copy (1).docx.

Markdown equivalent: docs/client-documentation-base/markdown/DRAFT 2026 Tender and Bid Library Template for Phew - FAQs - Copy (1).md (157 lines).

Structure:

• H1 / H2 headings sectioning the document (”## General FAQs”, ”### General Information”, ”### Business Activities”, …).
• Each section contains a 5-column markdown table: Question | Answer | Source of Information | Content Added Date | Verified for accuracy, date and initials.
• Significantly more rows per section than the final (~60-80 Q&A pairs per document) — DRAFT versions tend to include WIP entries that get pruned to the final.
• DRAFT files exist as siblings of the corresponding final files (different filenames, partially-overlapping content).

Adapter today: Pattern B (draft_5col) in extractQaPairs. Works. Per import_bid_library.py --auto-supersede, ingesting a final after the corresponding DRAFT auto-flips the DRAFT row’s superseded_by to the final row when content-hash matches AND filename heuristic recognises DRAFT→final transition.

Implication for q_a_pairs design: every Q&A pair carries publication_status (per the existing content_items pattern) + superseded_by. DRAFT-extracted Q&A pairs default to publication_status='draft'; final-extracted default to 'published'; supersession is applied at ingest time if the DRAFT/final pattern fires (per UC8 §6 §6.0.6 + ratified §6.8 v1 user-driven dedup).

4.3 Shape C — Phew-internal bid-library markdown with H4-per-pair headings + topics index

File: docs/client-documentation-base/markdown/Phew-Bid-Library-2026-v4_4.md (86,100 bytes).

Structure:

• H1 doc title.
• H2 “BID RESPONSE TOPIC INDEX” — a topic-tag index linking topic-name → bullet list of entries.
• H2 sections per topic-cluster (”## Company Details”, ”## Persons of Significant Control”, ”## Insurance Coverage”, …) — 116 such sections (grep -c "^##" returned 116).
• Each entry is a plain-prose H2 section with metadata lines:
  • **From**: Q23, Q24, Q25 (origin questions in legacy spreadsheet)
  • **Merged From**: Q23, Q24, Q25 (curation history)
  • **Topics**: Insurance & Commercial (manual scope tags)
  • > ⚠️ **SENSITIVE DATA**: Contains personal information. (sensitivity flag)
  • Bold question variants (**What parts of the service are sub-contracted?**) followed by answer prose paragraphs.
• ~65 logical Q&A pairs distributed across the 116 H2 sections (some sections aggregate multiple variants of the same topic).
• Markdown tables appear inline for tabular content (insurance limits, statutory yes/no answers).

Adapter today: None. This shape is NOT consumed by extractQaPairs (which is table-only) and is NOT consumed by ingest_markdown.py as Q&A (it would ingest as content_type='other' or similar). The pipeline today loses the Q&A structure when ingesting this kind of file.

Implication for q_a_pairs design: Shape C is one of the 5 historical Phew shapes that must be migrated once to the v1 canonical shape (§7). A one-shot markdown_heading_v1 migration helper handles the structural pattern (H2/H3 + **Metadata**: value + bold-question convention); any entries the heuristic misses can be triaged by editor review or routed through an optional LLM-extraction migration helper (Q-S16-4 — scope only if coverage gap surfaces). After migration, this format is not re-ingested — Phew’s bid-library Q&A pairs live in canonical shape going forward.

4.4 Shape D — LMS/Audit/Website bid library markdown with FAQ-ID-coded H4 entries + context variants

Files: docs/client-documentation-base/markdown/LMS_Bid_Library_v2.2.md (80,870 bytes), .../Website_Bid_Library_v4_2.md (48,949 bytes), .../Advanced_Audits_Bid_Library_v5.md (55,588 bytes).

Structure:

• H1 doc title with metadata lines (last-updated, entries-count, dedup-reductions).
• H2 / H3 section structure (## 1. FAQs → ### Product Overview).
• H4 entries per Q&A pair with FAQ-IDs (#### FAQ-001: What are the LMS best practices and attributes?, #### FUNC-025: What technology platform...?, #### SEC-001: ...?).
• Entry body: bold metadata (**ID**: FAQ-003, **Source**: FAQs, **Topics**: Delegate Experience & Onboarding) followed by free-prose answer and zero-or-more **Context Variant** blockquotes (> **When asked**: ... **Source**: ... **Detail**: ...).
• Cross-references inline (See FAQ-002, > **Cross-reference:** For accessibility detail → §2 Technical Specs).
• Dedup-flagged entries (#### ⚠️ DEDUP FLAG — GCLOUD-001 duplicates FAQ-002 context variant).
• ~40-50 entries per document.

Adapter today: None. Same as Shape C — this is heading-with-metadata-prose, not a table. extractQaPairs ignores it; markdown ingest swallows the whole doc as one content_items row.

Implication for q_a_pairs design: first-class support for “context variants” is needed — a single Q&A pair can have multiple legitimate phrasings of the question (and slight variations of the answer) that should be searchable but not deduplicated. Design choice (§6): q_a_pairs.alternate_question_phrasings TEXT[] (array of question variants) so all phrasings embed into one row’s question_embedding via concatenation OR via separate q_a_question_variants rows (deferred to v1.1; v1 ships TEXT[] as the simpler shape). Each variant’s “source” is captured in provenance notes.

4.5 Shape E — Advanced Audits library with YAML-frontmatter-like blocks

File: docs/client-documentation-base/markdown/Advanced_Audits_Bid_Library_v5.md (55,588 bytes; partially also Shape D).

Structure (specific to the “Schema-Mapped” section):

• --- separator
• Inline YAML-like metadata block (id: AUD-014, question: What is the Phew Advanced Audits system?, products: [audit], modules: [standard, advanced], topics: [overview, functionality])
• --- separator
• Module-tagged answer body: [ALL] The Phew Audit System is... / [ADVANCED] This includes audits involving sensitive data...

Adapter today: None. This is pseudo-YAML inside --- separators, not real frontmatter (no YAML parser would accept topics: [overview, functionality] followed by plain text body without a closing --- for the doc). It’s an authoring convention that requires bespoke parsing.

Implication for q_a_pairs design: the canonical client-authored template (per §7 recommendation) should use real YAML frontmatter per pair, which is unambiguous and trivially parseable. Working template shape in §7.2.

4.6 Shape F — Forms (PDF + XLSX) — questions only, no answers

Files: docs/client-documentation-base/forms/Cloud Security Principles Checklist V5_3 - PHEW.xlsx + standard-selection-questionnaire-ppn-03-24.pdf.

Structure:

• XLSX with question rows + empty answer columns (the platform’s job is to AUTO-COMPLETE these).
• PDF with formatted questions + checkbox / fill-in fields.

Implication for q_a_pairs design: forms are NOT Q&A input — they are Q&A consumers. The shape they want is the inverse: given a list of questions, find the best Q&A pair match + cite. This is handled by the MCP q_a_search tool (§9) + bid-question-matching flow per intended-arch §6.2 — not the schema.

4.7 Summary — Phew’s 5 historical shapes across 11+ inspected files

The shapes below describe Phew’s existing corpus (the first-client baseline). They are NOT v1 ingestion paths — v1 ingestion uses a predetermined markdown shape KH defines (§7). The shapes below are relevant only as one-shot migration sources to land Phew’s existing content into canonical shape.

Shape	File(s)	Structure	Existing extractor	One-shot migration target
A — Audit-6col docx tables	4 final .docx	Heading + 6-col table (Q / Standard / Advanced / Source / Date / Verified)	extractQaPairs Pattern A	q_a_extractions rows; extractor_kind='docx_pattern_a'; promoted to canonical q_a_pairs post-extraction
B — DRAFT-5col docx tables	3 DRAFT .docx	Heading + 5-col table (Q / Standard / Source / Date / Verified)	extractQaPairs Pattern B	q_a_extractions rows; extractor_kind='docx_pattern_b'; superseded by Shape A counterpart
C — Phew-internal bid library markdown	1 .md (86KB)	H2-sectioned plain prose with **Metadata**: value + bold-question convention	None (Q&A structure lost in current ingest)	One-shot markdown_heading_v1 migration helper OR llm_extraction for unrecognised entries
D — Product bid library markdown	3 .md	H4-coded entries (FAQ-001, FUNC-025) + context variants in blockquotes	None	One-shot markdown_heading_v1 migration helper; multi-variant question phrasings preserved
E — YAML-frontmatter-like blocks	1 .md (mixed Shape D)	--- blocks + pseudo-YAML + module-tagged answers	None	Convert to the v1 canonical YAML-frontmatter shape directly (Shape E is closest to the canonical, just authored loosely)
F — Forms (XLSX + PDF)	1 .xlsx + 1 .pdf	Questions-only; answers blank	Q-extraction via extract_docx_tables.py cousins (TBD per Q-S16-3)	Consumer of q_a_pairs, not a Q&A producer

Verdict — v1 ingestion = predetermined markdown shape (§7). The 5 historical Phew shapes are migrated once, then retired as ingestion paths. The existing Pattern A/B docx extractor is repurposed as a migration helper; markdown_heading_v1 + yaml_frontmatter_v1 + LLM-extraction stay scoped to that one-shot migration. Going forward, all new Q&A content (Phew + future clients) authors against the canonical predetermined shape from day 1.

Open question on forms (Shape F) — see Q-S16-3 below: how were the existing Standard Selection Questionnaire and Charnwood ITT Services question sets extracted historically? What’s the intended form-question-extraction path going forward?

---

5. Audit — KH-extracted shape today

5.1 Pass 2 (lib/ai/classify.ts + scripts/kb_pipeline/classify.py)

Pass 2 today is content classification, not Q&A extraction. It classifies a content_items row’s content_type (q_a_pair, case_study, article, etc.) + primary_domain + primary_subtopic + secondary_* + extracts entities for entity_mentions + emits relationships for entity_relationships.

What it does NOT emit: structured question + answer pairs from arbitrary prose. If you feed Pass 2 a paragraph of company-overview prose, you get an entity list + classification — not a {question, answer} pair.

Implication: the “extract Q&A from arbitrary corpus prose” path requires an LLM-driven extractor. Per §7 client-input policy, this is NOT a v1 recurring ingestion path — v1 takes Q&A authored in the canonical shape. An LLM-extraction migration helper is in scope only if Phew’s one-shot cutover surfaces content that the deterministic helpers (Pattern A/B docx, markdown_heading_v1) can’t cover — see Q-S16-4.

5.2 Graphify extract.py

Per ~/.claude/skills/graphify/SKILL.md inspection: graphify’s extract.py is an entity + relationship extractor, not a Q&A extractor. It produces a graphify-out/graph.json with {nodes: [...], edges: [...]} typed by community + confidence_label per S12. No {question, answer} emission.

Implication: even if KH adopts graphify’s semantic-extractor as the Pass 1/2 substrate (intended-arch §6.3 + graphify-evaluation §5.2.E), it does NOT close the “extract Q&A from corpus” gap. The optional LLM-extraction migration helper (Q-S16-4) is genuinely new work — scoped to one-shot migration only.

5.3 Mempalace miner

Per S3 spike §5.3: the mempalace miner only chunks Q&A-shaped content for files with ≥3 >-quoted-turn markers (Claude Code JSONL transcripts after normalize.py rewrites). For client docs with explicit Q: / A: structure, the miner treats them as plain text and chunks at 800 chars — straddling Q/A boundaries.

Implication: mempalace is the wrong tool for KH’s Q&A extraction. Borrow the schema (Shape A — already adopted at the entity_relationships layer per intended-arch §4.4), not the extraction logic.

5.4 Summary — what produces Q&A today vs what should

v1 ingestion path (recurring): predetermined markdown shape only (§7). All future Q&A authoring (Phew + new clients) lands in the canonical shape directly. The mappings below are one-shot migration sources for Phew’s existing content — they get processed once, results land in q_a_pairs (curated golden record), and the migration tooling retires.

Source	One-shot migration path	Outcome
Final audit-6col docx (Shape A)	extractQaPairs Pattern A → q_a_extractions rows (extractor_kind='docx_pattern_a') → editor-curate-promote to q_a_pairs	Phew’s final-audit Q&A pairs land as curated golden records
DRAFT-5col docx (Shape B)	extractQaPairs Pattern B → q_a_extractions rows with publication_status='draft'; superseded when Shape A counterpart promoted	DRAFT content preserved for audit; not promoted to golden record
Phew-internal markdown bid library (Shape C)	One-shot markdown_heading_v1 helper → q_a_extractions rows → editor review	Phew bid-library entries land in curated shape
Product bid library markdown (Shape D)	One-shot markdown_heading_v1 helper; context-variant phrasings preserved via alternate_question_phrasings	Product library Q&A lands in canonical shape
Pseudo-YAML markdown (Shape E)	Convert directly to canonical YAML-frontmatter (Shape E is closest to canonical)	Authored into v1 shape
Arbitrary corpus prose	NOT a v1 ingestion path. Pass 2 emits entities only (no Q&A surfacing). If specific corpus content needs to seed Q&A pairs, run one-shot LLM-extraction migration helper as a manual tool, not a recurring pipeline.	Out of v1 scope; manual tool only

Net new work for v1 recurring path: yaml_frontmatter_v1 adapter for the canonical shape + envelope + writer (§8) + MCP tools (§9). The migration helpers (docx_pattern_a/b/c/positional, markdown_heading_v1, llm_extraction) are one-shot tooling — built for Phew’s content cutover and retired post-migration.

---

6. Canonical shape — proposed v1 design

6.1 Design principles

Inherited from CLAUDE.md + the Phase 0.9 architecture:

1. One golden record, many extractions. q_a_pairs is the curated truth; q_a_extractions is a cache of LLM/parser output that may or may not get promoted. Maps to intended-arch §6.1 Q3.2 RESOLVED.
2. Temporal-aware (Shape A). Every row has valid_from / valid_to / confidence / provenance so “what was our position on PCI-DSS as of Q1 2026?” is answerable. Inherits S12 Q4.7+Q4.14 + S3 §3 ratifications.
3. AI-consumer-first. Question embedding is separate from corpus chunk embedding so bid-question matching can target Q&A shape directly. Two-step retrieval (list/preview → get/verbatim) per S3 §4 + intended-arch §6.2.
4. Provenance ≠ confidence. provenance (categorical: EXTRACTED / INFERRED / AMBIGUOUS) + confidence (numeric 0-1) follow graphify rubric per S12 §6.
5. Proper schema over JSONB. All extraction provenance (section_name, table_index, row_index, source_chunk_id) lives in typed columns, not metadata. JSONB only for genuinely-unconstrained adapter-specific context (e.g. context variants from Shape D pending v1.1 typed migration).
6. Hybrid retrieval. Question embedding (vector) + question fulltext (tsvector) + scope_tag filters + provenance filters all queryable in one RPC; matches the existing hybrid_search precedent.
7. No barrel re-exports for the new lib dir (lib/qa/ per intended-arch §6.3); direct file imports.
8. Anon EXECUTE revoke on every new function (CLAUDE.md gotcha).

6.2 q_a_pairs — golden record table

Inherits intended-arch §4.3 sketch with these refinements:

CREATE TABLE q_a_pairs (
  id                          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  workspace_id                UUID NOT NULL REFERENCES workspaces(id) ON DELETE RESTRICT,

  -- ── Core content ──────────────────────────────────────────────────
  question_text               TEXT NOT NULL,
  -- Variants of the question phrasing (Shape D context variants).
  -- v1 keeps as TEXT[] for ergonomics; v1.1 may split to q_a_question_variants table.
  alternate_question_phrasings TEXT[] NOT NULL DEFAULT '{}',
  answer_standard             TEXT NOT NULL,
  answer_advanced             TEXT,                    -- nullable; Phew-style tiered answers
  -- Answer body in canonical GFM markdown (the format used for both source-to-canonical
  -- and canonical-to-target rendering).

  -- ── Provenance ────────────────────────────────────────────────────
  source_content_item_id      UUID REFERENCES content_items(id) ON DELETE SET NULL,
  source_chunk_id             UUID REFERENCES content_chunks(id) ON DELETE SET NULL,
  promoted_from_extraction_id UUID REFERENCES q_a_extractions(id) ON DELETE SET NULL,
  promoted_from_bid_response_id UUID REFERENCES bid_responses(id) ON DELETE SET NULL,
  origin_kind                 TEXT NOT NULL CHECK (origin_kind IN (
    'extracted_from_corpus',     -- promoted from q_a_extractions
    'curated_explicit',          -- editor created via UI / MCP
    'derived_from_bid_response', -- bid response promoted to KB (UC5)
    'imported_legacy'            -- existing 395 content_items q_a_pair rows
  )),

  -- ── Retrieval substrate ───────────────────────────────────────────
  question_embedding          vector(1024),
  -- Embedded over: question_text + alternate_question_phrasings concatenated.
  -- Re-embed trigger fires on UPDATE of either column.

  -- ── Classification (KH taxonomy parallel to content_items) ────────
  primary_domain              TEXT,
  primary_subtopic            TEXT,
  secondary_domain            TEXT,
  secondary_subtopic          TEXT,
  classification_confidence   NUMERIC(3,2) CHECK (classification_confidence BETWEEN 0 AND 1),

  -- ── Scope tags (UC9 — per workspace vocabulary) ──────────────────
  scope_tag                   TEXT[] NOT NULL DEFAULT '{}',
  anti_scope_tag              TEXT[] NOT NULL DEFAULT '{}',

  -- ── Confidence + provenance (S12 + Q4.14) ─────────────────────────
  -- 'provenance' is the graphify confidence-label taxonomy.
  -- 'confidence' is the graphify confidence_score (rubric per S12 §2.2).
  provenance                  TEXT NOT NULL DEFAULT 'EXTRACTED' CHECK (provenance IN (
    'EXTRACTED', 'INFERRED', 'AMBIGUOUS'
  )),
  confidence                  NUMERIC(3,2) NOT NULL DEFAULT 1.0,

  -- ── Temporal validity (Shape A) ──────────────────────────────────
  valid_from                  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  valid_to                    TIMESTAMPTZ,    -- NULL = currently valid

  -- ── Publication lifecycle (matches content_items pattern) ─────────
  publication_status          TEXT NOT NULL DEFAULT 'draft' CHECK (publication_status IN (
    'draft', 'in_review', 'published', 'archived'
  )),

  -- ── Supersession (UC8 merge support) ─────────────────────────────
  superseded_by               UUID REFERENCES q_a_pairs(id) ON DELETE SET NULL,
  CHECK (superseded_by IS NULL OR superseded_by <> id),

  -- ── Governance ───────────────────────────────────────────────────
  verified_at                 TIMESTAMPTZ,
  verified_by                 UUID REFERENCES auth.users(id) ON DELETE SET NULL,
  governance_review_status    TEXT CHECK (governance_review_status IS NULL OR governance_review_status IN (
    'pending', 'approved', 'reverted', 'changes_requested', 'review_overdue'
  )),
  governance_review_due       TIMESTAMPTZ,
  governance_reviewer_id      UUID REFERENCES auth.users(id) ON DELETE SET NULL,
  next_review_date            DATE,
  review_cadence_days         INTEGER CHECK (review_cadence_days IS NULL OR (review_cadence_days BETWEEN 1 AND 1095)),

  -- ── Lineage / audit ──────────────────────────────────────────────
  created_at                  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at                  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  created_by                  UUID REFERENCES auth.users(id) ON DELETE SET NULL,
  updated_by                  UUID REFERENCES auth.users(id) ON DELETE SET NULL,
  archived_at                 TIMESTAMPTZ,
  archived_by                 UUID REFERENCES auth.users(id) ON DELETE SET NULL,
  archive_reason              TEXT,

  -- ── Ingest attribution (mirrors content_items.ingest_source) ─────
  ingest_source               TEXT,
    -- Canonical values: 'qa_promote_from_extraction', 'qa_curate_manual',
    -- 'qa_promote_from_bid', 'qa_import_legacy', 'qa_mcp_create'.

  -- ── Layer (progressive depth — same pattern as content_items.layer) ─
  layer                       VARCHAR REFERENCES layer_vocabulary(key) ON DELETE SET NULL
);

-- Indexes
CREATE INDEX idx_q_a_pairs_workspace ON q_a_pairs(workspace_id);
CREATE INDEX idx_q_a_pairs_source_content ON q_a_pairs(source_content_item_id);
CREATE INDEX idx_q_a_pairs_question_embedding ON q_a_pairs
  USING hnsw (question_embedding vector_cosine_ops) WITH (m=16, ef_construction=64);
CREATE INDEX idx_q_a_pairs_question_text_fts ON q_a_pairs
  USING gin (to_tsvector('english', question_text || ' ' || array_to_string(alternate_question_phrasings, ' ')));
CREATE INDEX idx_q_a_pairs_active_published ON q_a_pairs(workspace_id, valid_from DESC)
  WHERE valid_to IS NULL AND publication_status = 'published' AND archived_at IS NULL;
CREATE INDEX idx_q_a_pairs_governance_due ON q_a_pairs(governance_review_due)
  WHERE governance_review_due IS NOT NULL AND archived_at IS NULL;
CREATE INDEX idx_q_a_pairs_scope_tag ON q_a_pairs USING gin (scope_tag);
CREATE INDEX idx_q_a_pairs_anti_scope_tag ON q_a_pairs USING gin (anti_scope_tag);
CREATE INDEX idx_q_a_pairs_superseded_by ON q_a_pairs(superseded_by) WHERE superseded_by IS NOT NULL;

Triggers (NEW):

• enforce_q_a_pair_archive_state_consistency — mirror of enforce_archive_state_consistency (publication_status=‘archived’ ↔ archived_at IS NOT NULL).
• auto_v1_history_on_q_a_pair_insert — mirror of auto_v1_on_insert (writes initial q_a_pair_history row).
• update_q_a_pair_updated_at — standard BEFORE UPDATE trigger.
• re_embed_q_a_pair_on_question_change — sets question_embedding=NULL when question_text or alternate_question_phrasings change (deferred embedding pickup).

6.3 q_a_extractions — derived cache table

CREATE TABLE q_a_extractions (
  id                          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  workspace_id                UUID NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE,
  source_content_item_id      UUID NOT NULL REFERENCES content_items(id) ON DELETE CASCADE,
  source_chunk_id             UUID REFERENCES content_chunks(id) ON DELETE SET NULL,

  -- ── Extracted content ────────────────────────────────────────────
  question_text               TEXT NOT NULL,
  answer_text                 TEXT NOT NULL,
  -- For Pattern A/C (tiered answers): split into answer_standard + answer_advanced;
  -- otherwise the same content goes to both for downstream sameness.
  answer_standard             TEXT,
  answer_advanced             TEXT,

  question_embedding          vector(1024),

  -- ── Extractor lineage ────────────────────────────────────────────
  extractor_kind              TEXT NOT NULL CHECK (extractor_kind IN (
    'docx_pattern_a',          -- audit_6col (Q / Standard / Advanced / + 3 meta)
    'docx_pattern_b',          -- draft_5col (Q / Standard / + 3 meta)
    'docx_pattern_c',          -- numbered_6col
    'docx_pattern_positional', -- fallback all-empty-header
    'markdown_heading_v1',     -- NEW — H2/H3/H4-with-bold-question (Shape C/D)
    'yaml_frontmatter_v1',     -- NEW — canonical client-authored template
    'llm_extraction'           -- NEW — one-shot migration helper (Q-S16-4); not a recurring v1 ingestion path
  )),
  extractor_version           TEXT NOT NULL DEFAULT '1.0',
    -- Adapter-version tag for change-tracking + replay; deprecate-old + re-extract pattern.

  -- ── Source positional context (proper schema, not metadata.section_name) ─
  source_file_basename        TEXT,             -- 'FAQs.docx'
  source_section_path         TEXT[],           -- ['## Security & Data Management', '### Access Control']
  source_table_index          INTEGER,          -- 0-indexed; NULL for prose extractions
  source_row_index            INTEGER,          -- 0-indexed; NULL for prose extractions
  source_id_token             TEXT,             -- 'FAQ-003', 'AUD-014', 'FUNC-025' (when present)

  -- ── Provenance + confidence (S12 + Q4.14, same shape as q_a_pairs) ─
  provenance                  TEXT NOT NULL DEFAULT 'EXTRACTED' CHECK (provenance IN (
    'EXTRACTED', 'INFERRED', 'AMBIGUOUS'
  )),
  extraction_confidence       NUMERIC(3,2) NOT NULL DEFAULT 1.0,

  -- ── Promotion / lifecycle ────────────────────────────────────────
  promoted_to_pair_id         UUID REFERENCES q_a_pairs(id) ON DELETE SET NULL,
  promoted_at                 TIMESTAMPTZ,
  promoted_by                 UUID REFERENCES auth.users(id) ON DELETE SET NULL,

  -- ── Cocoindex Δ invalidation ─────────────────────────────────────
  -- Set when source file's content_hash changes (cocoindex post-flow trigger).
  -- Once invalidated, NOT auto-removed — kept for audit + comparison; new
  -- extraction rows emitted for the new content.
  invalidated_at              TIMESTAMPTZ,
  invalidation_reason         TEXT,

  -- ── Audit ────────────────────────────────────────────────────────
  created_at                  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  cocoindex_run_id            TEXT,             -- cocoindex App.update() run identifier
  metadata                    JSONB DEFAULT '{}'::jsonb
    -- Only adapter-specific overflow (e.g. raw "topics" string from Shape C/D
    -- before scope_tag canonicalisation). Avoid using this for first-class data.
);

-- Indexes
CREATE INDEX idx_q_a_extractions_workspace ON q_a_extractions(workspace_id);
CREATE INDEX idx_q_a_extractions_source ON q_a_extractions(source_content_item_id);
CREATE INDEX idx_q_a_extractions_source_chunk ON q_a_extractions(source_chunk_id);
CREATE INDEX idx_q_a_extractions_question_embedding ON q_a_extractions
  USING hnsw (question_embedding vector_cosine_ops) WITH (m=16, ef_construction=64);
CREATE INDEX idx_q_a_extractions_active ON q_a_extractions(source_content_item_id)
  WHERE invalidated_at IS NULL AND promoted_to_pair_id IS NULL;
CREATE INDEX idx_q_a_extractions_promoted ON q_a_extractions(promoted_to_pair_id)
  WHERE promoted_to_pair_id IS NOT NULL;
CREATE INDEX idx_q_a_extractions_extractor ON q_a_extractions(extractor_kind, extractor_version);

6.4 q_a_pair_history — version history (mirrors content_history)

CREATE TABLE q_a_pair_history (
  id                  UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  q_a_pair_id         UUID NOT NULL REFERENCES q_a_pairs(id) ON DELETE CASCADE,
  version_number      INTEGER NOT NULL,         -- monotonic; v1, v2, ...
  -- Snapshot fields (every editable column)
  question_text       TEXT NOT NULL,
  alternate_question_phrasings TEXT[],
  answer_standard     TEXT NOT NULL,
  answer_advanced     TEXT,
  scope_tag           TEXT[],
  anti_scope_tag      TEXT[],
  primary_domain      TEXT,
  primary_subtopic    TEXT,
  publication_status  TEXT,
  -- Change attribution
  change_type         TEXT NOT NULL CHECK (change_type IN (
    'initial_ingest', 'edit', 'promote_from_extraction', 'promote_from_bid',
    'merge', 'split', 'supersede', 'archive', 'reclassify', 'governance_review_action'
  )),
  change_reason       TEXT,
  changed_by          UUID REFERENCES auth.users(id) ON DELETE SET NULL,
  changed_at          TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  -- Cross-record operation grouping (UC8 merge / UC3 sweep)
  operation_id        UUID,
  UNIQUE (q_a_pair_id, version_number)
);

CREATE INDEX idx_q_a_pair_history_pair ON q_a_pair_history(q_a_pair_id, version_number DESC);
CREATE INDEX idx_q_a_pair_history_operation ON q_a_pair_history(operation_id) WHERE operation_id IS NOT NULL;

6.5 citations — inherited from intended-arch §4.3; migration deferred

Per intended-arch §4.3, the new polymorphic citations table covers q_a_pair-as-citation-source AND q_a_pair-as-citation-target. This spike inherits the design verbatim but does NOT migrate content_citations → polymorphic citations here — see Q-S16-5. Migration sketch is bundled with the citation re-anchor work (UC6 + UC8 §6 ratifications), not in §10.

6.6 What’s REMOVED from content_items (post-Q&A extraction)

Per intended-arch §4.2:

Column	Action	Rationale
answer_standard	DROP	Moves to q_a_pairs.answer_standard
answer_advanced	DROP	Moves to q_a_pairs.answer_advanced
q_a_pair value in content_type CHECK	REMOVE from enum	Q&A pairs no longer live in content_items
parent_id (if Q&A-only usage)	VERIFY then DROP	Stream 1 item 6 verifies if any non-Q&A use survives

source_bid, source_file, notes removals are independent of this spike (intended-arch §4.2 lists them but they’re broader cleanup).

6.7 Delta from intended-arch §4.3 sketch

Refinements introduced in this spike vs intended-arch §4.3:

#	Refinement	Source
1	Added alternate_question_phrasings TEXT[] for Shape D context variants	§4.4 audit
2	Added anti_scope_tag TEXT[] (parallels scope_tag) per Q1.7 RESOLVED + UC9 RATIFIED §6.0.6	edit-flow §6.9.1 4.8.Q6
3	Added superseded_by + identity CHECK constraint (parallel to content_items.superseded_by)	UC8 §6.8 + S186 WP-B.1
4	Added publication_status + enforce_archive_state_consistency-style trigger	content_items pattern + UC5 §6.5 4.6.Q2 RATIFIED
5	Added governance_review_* cluster + next_review_date + review_cadence_days (mirrors content_items §5.5)	S200 WP5 parallel
6	Added provenance text CHECK (graphify Q4.14) + repurposed numeric confidence per rubric	S12 §6
7	Added q_a_pair_history table separate from content_history (analogous but distinct)	edit-flow §6.6 4.3.Q6 (multi-Q&A op support)
8	Promoted section_name / table_index / row_index from metadata to typed columns; added source_section_path TEXT[] + source_id_token TEXT	§3.3 observation (JSONB → schema migration per CLAUDE.md)
9	Added extractor_kind enum expansion (markdown_heading_v1, yaml_frontmatter_v1)	§4 audit findings
10	Added extractor_version for adapter-version replay	operational concern from S2 spike
11	Added cocoindex_run_id for cross-run invalidation chains	S2 spike Δ semantics
12	Added promoted_from_bid_response_id FK on q_a_pairs	UC5 §6.5 lineage
13	Added ingest_source text on q_a_pairs (mirrors content_items pattern; canonical values listed)	content_items pattern
14	Added layer FK on q_a_pairs (mirrors content_items pattern for progressive depth)	Layer vocabulary feature parity

These refinements are additive — none breaks intended-arch §4.3 or §6.1 decisions.

---

7. Client-input policy — predetermined markdown shape

7.1 Decision framing

Per Liam’s S230 clarification + post-spike feedback: KH defines the v1 canonical Q&A wire format pre-ingestion. Clients (Phew + future) author Q&A content in that shape from day 1. Existing Phew content (5 historical shapes catalogued in §4) gets migrated once via one-shot tooling — see §7.3 — and is not a recurring ingestion path.

Rationale:

• Single recurring ingestion path simplifies the pipeline. One adapter (yaml_frontmatter_v1), one envelope writer, one dispatcher. No format detection at recurring ingest time.
• Authored shape matches the schema. YAML frontmatter keys map 1:1 to typed columns on q_a_pairs (workspace_id, scope_tag, anti_scope_tag, provenance, valid_from, source pointers). No “guess the metadata from prose” step.
• One-shot migration is bounded work. Phew’s 5 historical shapes are catalogued (§4); migration tooling runs once to land that content into canonical shape; then those code paths retire.
• AI consumer first. The canonical shape is structured for LLM consumption (citations + scope filters + temporal slices) — clients shouldn’t author content in a free-form way that LLMs then have to disentangle.

Implication: the existing Pattern A/B docx extractor + markdown_heading_v1 + LLM-extraction tooling are repurposed as migration helpers (one-shot for Phew’s existing corpus), not as ongoing v1 ingestion adapters.

7.2 The canonical shape — YAML frontmatter per pair

KH provides docs/templates/client-qa-bundle-template.md (a NEW reference document; not in this spike’s deliverable but specified here for the implementation phase). Markdown with YAML frontmatter per pair, separated by ---:

# {{ workspace_name }} — Q&A library

Last updated: 2026-05-11 by Liam
Workspace ID: {{ workspace_uuid }} (or leave blank to bind at import)

---

id: phew-faq-001
question: What is the Phew Audit System?
alternate_phrasings:
  - Can you describe the Phew Audit platform?
  - What does the Phew Audit product do?
scope_tags: [overview, functionality]
anti_scope_tags: [pricing]
primary_domain: overview
primary_subtopic: product-description
confidence: 1.0
provenance: EXTRACTED
publication_status: published
valid_from: 2024-01-01
sources:
  - file: 2026 Audit FAQs.docx
    table_index: 0
    row_index: 1
verified_at: 2024-05-22
verified_by: matt.burgess@phew.org.uk

---

**Standard answer:**

An Audit System Platform to build, design, send, collate and report on the data
recorded.

**Advanced answer:**

An Audit System Platform to build, design, send, collate and report on the data
recorded. Including the capture of sensitive data.

---

id: phew-faq-002
question: How does the pricing work?
...

Adapter: yaml_frontmatter_v1 (NEW). Parses real YAML frontmatter (using a real parser, not regex — js-yaml or pyyaml). Maps each ----separated block to one q_a_extractions row (or, if --curate-as-pair flag set on the import, direct to q_a_pairs with origin_kind='curated_explicit'). Reserved YAML keys map 1:1 to typed columns; unrecognised keys land in metadata JSONB.

Authoring workflow we recommend to clients:

1. Start with the template (we ship it as a markdown file they download).
2. Fill the first 5-10 Q&A pairs by hand to learn the format.
3. Bulk-port from existing content using a one-off conversion script we provide per source shape (docx-table → template; FAQ-md → template; etc.). Optional.
4. Commit the template file into the canonical folder; cocoindex picks it up at next scan.
5. Edit in place (in editor of choice, or via KH UI per UC1+UC2+UC4 RATIFIED).

The template IS the canonical shape — REQUIRED for v1 recurring ingestion. Clients with existing libraries run one-shot migration (§7.3) to land that content in canonical shape; from cutover onward, all authoring uses the template.

7.3 One-shot migration from Phew’s 5 historical shapes

For Phew’s existing corpus, run a bounded migration program:

Source shape	Migration tool	Outcome
Shape A — Audit-6col docx	extractQaPairs Pattern A → q_a_extractions (extractor_kind='docx_pattern_a') → editor curates + promotes to q_a_pairs	Final-audit Q&A pairs land as curated golden records
Shape B — DRAFT-5col docx	extractQaPairs Pattern B → q_a_extractions (publication_status='draft'); superseded when Shape A counterpart promoted	DRAFT content preserved for audit; not promoted
Shape C — Phew-internal bid library markdown	One-shot markdown_heading_v1 helper → q_a_extractions → editor review	Phew bid-library entries land in canonical shape
Shape D — Product bid library markdown	Same — markdown_heading_v1 helper; context-variant phrasings preserved via alternate_question_phrasings	Product library Q&A migrated
Shape E — pseudo-YAML markdown	Convert directly to canonical YAML-frontmatter (Shape E is closest to canonical; small authoring fix-up)	Authored into v1 shape
Shape F — Forms (XLSX + PDF)	NOT a Q&A producer — see Q-S16-3 below for form-question-extraction strategy	Out of Q&A migration scope

Migration tooling lives under a clearly-scoped folder (suggested scripts/qa-migration/) so its lifecycle is obvious: built for Phew cutover, retired post-migration.

7.4 Documentation deliverable (post-spike)

docs/templates/client-qa-bundle-template.md — the template file with inline instructions. Onboarding docs reference this as “this is the canonical Q&A shape; new content authored here from day 1; existing libraries get a one-time migration.”

---

8. Ingest envelope spec (extract adapter ↔ q_a_extractions writer)

8.1 Envelope shape (JSON / Zod)

Every adapter (Pattern A/B/C, markdown_heading_v1, yaml_frontmatter_v1, llm_extraction) emits the same envelope. Writer consumes envelope → inserts q_a_extractions row.

// lib/qa/envelope.ts — NEW
import { z } from 'zod';

export const QaExtractionEnvelopeSchema = z.object({
  // ── Required ──────────────────────────────────────────────────
  workspace_id: z.string().uuid(),
  source_content_item_id: z.string().uuid(),
  question_text: z.string().trim().min(1).max(2000),
  answer_text: z.string().trim().min(1).max(50_000),
  extractor_kind: z.enum([
    'docx_pattern_a', 'docx_pattern_b', 'docx_pattern_c', 'docx_pattern_positional',
    'markdown_heading_v1', 'yaml_frontmatter_v1', 'llm_extraction',
  ]),
  extractor_version: z.string().min(1).default('1.0'),

  // ── Recommended ───────────────────────────────────────────────
  source_chunk_id: z.string().uuid().nullable().default(null),
  answer_standard: z.string().trim().max(50_000).nullable().default(null),
  answer_advanced: z.string().trim().max(50_000).nullable().default(null),

  // ── Source positional context ─────────────────────────────────
  source_file_basename: z.string().max(500).nullable().default(null),
  source_section_path: z.array(z.string().max(500)).default([]),
  source_table_index: z.number().int().min(0).nullable().default(null),
  source_row_index: z.number().int().min(0).nullable().default(null),
  source_id_token: z.string().max(100).nullable().default(null),

  // ── Provenance (S12 Q4.14) ────────────────────────────────────
  provenance: z.enum(['EXTRACTED', 'INFERRED', 'AMBIGUOUS']).default('EXTRACTED'),
  extraction_confidence: z.number().min(0).max(1).default(1.0),

  // ── Cocoindex Δ context ───────────────────────────────────────
  cocoindex_run_id: z.string().max(100).nullable().default(null),

  // ── Adapter-specific overflow ─────────────────────────────────
  metadata: z.record(z.unknown()).default({}),
});

export type QaExtractionEnvelope = z.infer<typeof QaExtractionEnvelopeSchema>;

Writer responsibilities (lib/qa/writer.ts — NEW):

• Validate envelope (QaExtractionEnvelopeSchema.parse).
• Embed question_text via OpenAI text-embedding-3-large at 1024 dims (Matryoshka). Cache by hash of question text + workspace_id.
• INSERT q_a_extractions row.
• Optionally trigger promotion-to-pair workflow if envelope carries auto_promote=true flag (curated_explicit origin_kind).

8.2 Per-adapter envelope construction

Adapter	Envelope construction notes
docx_pattern_a	From existing extractQaPairs QaPair: maps questionText → question_text; answerStandard → answer_standard + answer_text (former is typed split, latter is canonical body); answerAdvanced → answer_advanced; sectionName → source_section_path[]; sourceFile → source_file_basename; tableIndex / rowIndex → source_table_index / source_row_index; provenance='EXTRACTED'; confidence=1.0 (table-structured = high-confidence).
docx_pattern_b	Same as Pattern A minus answer_advanced.
docx_pattern_c	Same as Pattern A; source_section_path reads from cell column (not heading).
markdown_heading_v1	(NEW) — parses H2/H3/H4 + bold-question convention; question_text from first bold paragraph after heading; answer_text from prose between bold-question and next heading; source_section_path from heading chain; source_id_token from **ID**: FAQ-003 line; provenance='EXTRACTED'; confidence=0.9 (lower because heading detection is heuristic).
yaml_frontmatter_v1	(NEW) — js-yaml parses --- blocks; required keys: question; recommended: id → source_id_token, scope_tags → routed to q_a_pairs if --curate-as-pair; confidence from YAML key or default 1.0; provenance from YAML key or ‘EXTRACTED’.
llm_extraction	(NEW) — Pass Q&A LLM extracts {question, answer, confidence, reasoning} from a chunk; chunk-id captured; provenance='INFERRED' always (LLM output is by definition inferred); confidence from graphify rubric per S12 (0.55-0.95 for INFERRED with strong reasoning; 0.3 below = AMBIGUOUS).

8.3 Why Zod over JSON Schema

• Codegen TS types from Zod, no extra step.
• Validation at adapter boundary uses same Zod instance — single source of truth.
• Inline transforms / refinements (default values, trimming) live in the schema.
• Matches existing lib/validation/schemas.ts precedent.

8.4 Cocoindex flow integration

cocoindex App.update() flow runs adapter dispatcher per file → emits N envelopes per file → writer inserts N rows. On subsequent runs with changed content_hash:

• Engine invokes flow → new envelopes emitted → new rows inserted (NOT UPDATEs — rows are immutable extraction snapshots).
• Old rows: trigger fires UPDATE q_a_extractions SET invalidated_at=NOW(), invalidation_reason='source_content_changed' WHERE source_content_item_id = ? AND invalidated_at IS NULL.

This is the per-extraction Δ pattern — every adapter run on the same source content produces new immutable rows; old rows tombstone but stick around for audit + diff.

For q_a_pairs (the curated artefacts): cocoindex Δ does NOT touch them. Promotion is user-driven (UC5 + UC6 RATIFIED). Source-content change to a pair’s source triggers an AI-suggest review per UC6 §6.6, not auto-update.

---

9. MCP retrieval shape

9.1 New MCP tools (3)

Per intended-arch §9.2 critical gaps + S3 §4 Shape B pattern:

9.1.1 q_a_search — list/preview retrieval

// Input
{
  workspace_id: string;        // UUID
  query: string;               // user question or topic
  scope_filter?: {
    include?: string[];        // scope tag whitelist
    exclude?: string[];        // anti-scope tag enforcement
    expression?: string;       // boolean expression (UC9 §6.9.1 4.8.Q8)
  };
  domain_filter?: {
    primary_domain?: string;
    primary_subtopic?: string;
  };
  provenance_filter?: ('EXTRACTED' | 'INFERRED' | 'AMBIGUOUS')[];
  publication_status_filter?: ('draft' | 'in_review' | 'published')[];
                               // default: ['published']
  include_extractions?: boolean;
                               // false (default) = q_a_pairs only;
                               // true = also includes q_a_extractions not yet promoted
  as_of?: string;              // ISO timestamp — temporal slice (Shape A)
  limit?: number;              // default 10, max 50
  offset?: number;             // pagination
}

// Output (list/preview shape)
{
  results: Array<{
    q_a_pair_id?: string;      // UUID, present for q_a_pairs hits
    q_a_extraction_id?: string; // UUID, present for q_a_extractions hits (when include_extractions=true)
    kind: 'curated' | 'extracted';
    question_text: string;     // verbatim short string (full text — no preview truncation needed)
    answer_preview: string;    // first 200 chars + ellipsis
    scope_tags: string[];
    anti_scope_tags: string[]; // surfaced so consumer can confirm filter took effect
    primary_domain: string | null;
    primary_subtopic: string | null;
    similarity: number;        // cosine 0-1
    rank_method: 'vector' | 'fulltext' | 'hybrid';
    provenance: 'EXTRACTED' | 'INFERRED' | 'AMBIGUOUS';
    confidence: number;        // 0-1
    valid_from: string;        // ISO
    valid_to: string | null;
    current: boolean;          // valid_to IS NULL
    publication_status: 'draft' | 'in_review' | 'published';
    source_content_item_id: string | null;  // for click-through
  }>;
  total: number;
  limit: number;
  offset: number;
  query: string;
  taxonomy_version?: string;   // taxonomy version at query time (UC9 cache-key)
}

9.1.2 q_a_get — verbatim retrieval

// Input
{
  q_a_pair_id?: string;        // UUID — one or the other required
  q_a_extraction_id?: string;  // UUID
  workspace_id: string;        // UUID — RLS scope check
  include_citations?: boolean; // default true
  include_history?: boolean;   // default false — version chain
}

// Output
{
  id: string;
  kind: 'curated' | 'extracted';
  workspace_id: string;
  question_text: string;
  alternate_question_phrasings: string[];  // only on curated
  answer_standard: string;
  answer_advanced: string | null;
  scope_tag: string[];
  anti_scope_tag: string[];
  primary_domain: string | null;
  primary_subtopic: string | null;
  secondary_domain: string | null;
  secondary_subtopic: string | null;
  provenance: 'EXTRACTED' | 'INFERRED' | 'AMBIGUOUS';
  confidence: number;
  valid_from: string;
  valid_to: string | null;
  publication_status: 'draft' | 'in_review' | 'published' | 'archived';
  source_content_item_id: string | null;
  source_chunk_id: string | null;
  source_file_basename: string | null;
  origin_kind?: 'extracted_from_corpus' | 'curated_explicit' | 'derived_from_bid_response' | 'imported_legacy';  // q_a_pairs only
  extractor_kind?: string;     // q_a_extractions only
  citations?: Array<{          // present if include_citations=true
    citation_id: string;
    citing_entity: 'bid_response' | 'sales_proposal' | 'q_a_pair' | 'mcp_search_response';
    citing_entity_id: string;
    citation_kind: 'verbatim_quote' | 'paraphrase' | 'derived_inference' | 'q_a_match';
    confidence: number;
    created_at: string;
    invalidated_at: string | null;
  }>;
  history?: Array<{            // present if include_history=true
    version_number: number;
    change_type: string;
    change_reason: string | null;
    changed_at: string;
    changed_by_display_name?: string;
  }>;
  verified_at: string | null;
  verified_by_display_name?: string;
  governance_review_status: string | null;
  governance_review_due: string | null;
  superseded_by_id?: string | null;  // forward pointer
  supersedes_ids?: string[];          // back pointer (computed)
}

9.1.3 q_a_create — curated entry (admin/editor)

// Input — manually create a curated Q&A pair
{
  workspace_id: string;
  question_text: string;
  alternate_question_phrasings?: string[];
  answer_standard: string;
  answer_advanced?: string;
  scope_tag?: string[];
  anti_scope_tag?: string[];
  primary_domain?: string;
  primary_subtopic?: string;
  secondary_domain?: string;
  secondary_subtopic?: string;
  source_content_item_id?: string;  // optional lineage
  promote_from_extraction_id?: string;  // optional — extract → curate
  publication_status?: 'draft' | 'in_review' | 'published';  // default 'draft'
  scope_tag_intent?: string;        // UC9 §6.9.1 4.8.Q4 closed vocabulary
}

// Output
{
  q_a_pair_id: string;
  created_at: string;
  publication_status: string;
  version_number: 1;
}

9.2 Extension to existing search_knowledge_base_strict

The S3-spec’d search_knowledge_base_strict (Shape B) becomes “search the corpus chunks”. It does NOT search q_a_pairs — the new q_a_search is the Q&A-specific surface. However, search_knowledge_base_strict can OPTIONALLY include Q&A-derived chunks if the consumer asks for include_qa=true (default true) — those chunks come tagged so the LLM consumer can prefer them.

9.3 Two-step retrieval rationale (S3 §4.1 pattern)

Per S3 §4.1: list/preview → get/verbatim prevents accidental full-corpus dump. q_a_search returns enough to choose; q_a_get returns full content for the chosen IDs.

For bid-question matching (intended-arch §6.2), the matching RPC (POST /api/bids/[id]/questions/match) calls q_a_search per bid-question, gets top-K Q&A pair IDs, then q_a_get for the top match to compose the bid response. Two-step is preserved end-to-end.

9.4 Anti-tag enforcement (UC9 §6.9.1 4.8.Q8 RATIFIED)

scope_filter.exclude is filter-time exclusion regardless of any caller’s boolean expression. SQL pattern: WHERE NOT (anti_scope_tag && @anti_array) AND (scope_tag && @scope_array). Test integration required (per CX.27 RESOLVED).

9.5 Temporal slicing (Shape A)

as_of parameter on q_a_search runs the query against the temporal slice: rows where valid_from <= as_of AND (valid_to IS NULL OR valid_to > as_of). Enables “what was our PCI-DSS answer as of Q1 2026?” per intended-arch §4.3 notes.

---

10. Migration plan

10.1 Staged migration sequence (4 migrations)

Migration 1 — Create q_a_pairs + history + indexes

supabase/migrations/2026MMDDHHMMSS_qa_001_create_q_a_pairs.sql:

BEGIN;

-- Table
CREATE TABLE q_a_pairs (
  id                          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  workspace_id                UUID NOT NULL REFERENCES workspaces(id) ON DELETE RESTRICT,
  question_text               TEXT NOT NULL,
  alternate_question_phrasings TEXT[] NOT NULL DEFAULT '{}',
  answer_standard             TEXT NOT NULL,
  answer_advanced             TEXT,
  source_content_item_id      UUID REFERENCES content_items(id) ON DELETE SET NULL,
  source_chunk_id             UUID REFERENCES content_chunks(id) ON DELETE SET NULL,
  promoted_from_extraction_id UUID,  -- FK added in migration 2 (forward declaration)
  promoted_from_bid_response_id UUID REFERENCES bid_responses(id) ON DELETE SET NULL,
  origin_kind                 TEXT NOT NULL CHECK (origin_kind IN (
    'extracted_from_corpus', 'curated_explicit', 'derived_from_bid_response', 'imported_legacy'
  )),
  question_embedding          vector(1024),
  primary_domain              TEXT,
  primary_subtopic            TEXT,
  secondary_domain            TEXT,
  secondary_subtopic          TEXT,
  classification_confidence   NUMERIC(3,2) CHECK (classification_confidence BETWEEN 0 AND 1),
  scope_tag                   TEXT[] NOT NULL DEFAULT '{}',
  anti_scope_tag              TEXT[] NOT NULL DEFAULT '{}',
  provenance                  TEXT NOT NULL DEFAULT 'EXTRACTED' CHECK (provenance IN (
    'EXTRACTED', 'INFERRED', 'AMBIGUOUS'
  )),
  confidence                  NUMERIC(3,2) NOT NULL DEFAULT 1.0,
  valid_from                  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  valid_to                    TIMESTAMPTZ,
  publication_status          TEXT NOT NULL DEFAULT 'draft' CHECK (publication_status IN (
    'draft', 'in_review', 'published', 'archived'
  )),
  superseded_by               UUID REFERENCES q_a_pairs(id) ON DELETE SET NULL,
  CHECK (superseded_by IS NULL OR superseded_by <> id),
  verified_at                 TIMESTAMPTZ,
  verified_by                 UUID REFERENCES auth.users(id) ON DELETE SET NULL,
  governance_review_status    TEXT CHECK (governance_review_status IS NULL OR governance_review_status IN (
    'pending', 'approved', 'reverted', 'changes_requested', 'review_overdue'
  )),
  governance_review_due       TIMESTAMPTZ,
  governance_reviewer_id      UUID REFERENCES auth.users(id) ON DELETE SET NULL,
  next_review_date            DATE,
  review_cadence_days         INTEGER CHECK (review_cadence_days IS NULL OR (review_cadence_days BETWEEN 1 AND 1095)),
  created_at                  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at                  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  created_by                  UUID REFERENCES auth.users(id) ON DELETE SET NULL,
  updated_by                  UUID REFERENCES auth.users(id) ON DELETE SET NULL,
  archived_at                 TIMESTAMPTZ,
  archived_by                 UUID REFERENCES auth.users(id) ON DELETE SET NULL,
  archive_reason              TEXT,
  ingest_source               TEXT,
  layer                       VARCHAR REFERENCES layer_vocabulary(key) ON DELETE SET NULL
);

-- History
CREATE TABLE q_a_pair_history (
  id                  UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  q_a_pair_id         UUID NOT NULL REFERENCES q_a_pairs(id) ON DELETE CASCADE,
  version_number      INTEGER NOT NULL,
  question_text       TEXT NOT NULL,
  alternate_question_phrasings TEXT[],
  answer_standard     TEXT NOT NULL,
  answer_advanced     TEXT,
  scope_tag           TEXT[],
  anti_scope_tag      TEXT[],
  primary_domain      TEXT,
  primary_subtopic    TEXT,
  publication_status  TEXT,
  change_type         TEXT NOT NULL CHECK (change_type IN (
    'initial_ingest', 'edit', 'promote_from_extraction', 'promote_from_bid',
    'merge', 'split', 'supersede', 'archive', 'reclassify', 'governance_review_action'
  )),
  change_reason       TEXT,
  changed_by          UUID REFERENCES auth.users(id) ON DELETE SET NULL,
  changed_at          TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  operation_id        UUID,
  UNIQUE (q_a_pair_id, version_number)
);

-- Indexes
CREATE INDEX idx_q_a_pairs_workspace ON q_a_pairs(workspace_id);
CREATE INDEX idx_q_a_pairs_source_content ON q_a_pairs(source_content_item_id);
CREATE INDEX idx_q_a_pairs_question_embedding ON q_a_pairs
  USING hnsw (question_embedding vector_cosine_ops) WITH (m=16, ef_construction=64);
CREATE INDEX idx_q_a_pairs_question_text_fts ON q_a_pairs
  USING gin (to_tsvector('english',
    question_text || ' ' || array_to_string(alternate_question_phrasings, ' ')
  ));
CREATE INDEX idx_q_a_pairs_active_published ON q_a_pairs(workspace_id, valid_from DESC)
  WHERE valid_to IS NULL AND publication_status = 'published' AND archived_at IS NULL;
CREATE INDEX idx_q_a_pairs_governance_due ON q_a_pairs(governance_review_due)
  WHERE governance_review_due IS NOT NULL AND archived_at IS NULL;
CREATE INDEX idx_q_a_pairs_scope_tag ON q_a_pairs USING gin (scope_tag);
CREATE INDEX idx_q_a_pairs_anti_scope_tag ON q_a_pairs USING gin (anti_scope_tag);
CREATE INDEX idx_q_a_pairs_superseded_by ON q_a_pairs(superseded_by) WHERE superseded_by IS NOT NULL;
CREATE INDEX idx_q_a_pair_history_pair ON q_a_pair_history(q_a_pair_id, version_number DESC);
CREATE INDEX idx_q_a_pair_history_operation ON q_a_pair_history(operation_id) WHERE operation_id IS NOT NULL;

-- Triggers — Q-S16-11: full DDL bodies + companion REVOKE EXECUTE FROM anon must accompany
-- this migration (or split into Migration 1b). Names shown below are placeholders; expand
-- pre-apply.
-- enforce_q_a_pair_archive_state_consistency  -- mirrors enforce_archive_state_consistency
-- auto_v1_history_on_q_a_pair_insert          -- writes initial q_a_pair_history row
-- update_q_a_pair_updated_at                  -- BEFORE UPDATE timestamp
-- re_embed_q_a_pair_on_question_change        -- sets question_embedding=NULL for re-embed pickup

-- RLS
ALTER TABLE q_a_pairs ENABLE ROW LEVEL SECURITY;
ALTER TABLE q_a_pair_history ENABLE ROW LEVEL SECURITY;

-- Workspace-scoped read for editors+admins (mirrors content_items pattern)
CREATE POLICY q_a_pairs_select_workspace_member ON q_a_pairs
  FOR SELECT TO authenticated
  USING (
    EXISTS (
      SELECT 1 FROM workspaces w
      WHERE w.id = q_a_pairs.workspace_id
        AND public.get_user_role() IN ('editor', 'admin')
    )
  );

CREATE POLICY q_a_pairs_insert_editor_admin ON q_a_pairs
  FOR INSERT TO authenticated
  WITH CHECK (
    public.get_user_role() IN ('editor', 'admin')
  );

CREATE POLICY q_a_pairs_update_editor_admin ON q_a_pairs
  FOR UPDATE TO authenticated
  USING (public.get_user_role() IN ('editor', 'admin'))
  WITH CHECK (public.get_user_role() IN ('editor', 'admin'));

CREATE POLICY q_a_pairs_delete_admin ON q_a_pairs
  FOR DELETE TO authenticated
  USING (public.get_user_role() = 'admin');

-- Same pattern for q_a_pair_history (read-only for editors; write via trigger only)
CREATE POLICY q_a_pair_history_select_workspace_member ON q_a_pair_history
  FOR SELECT TO authenticated
  USING (
    EXISTS (
      SELECT 1 FROM q_a_pairs p
      WHERE p.id = q_a_pair_history.q_a_pair_id
        AND public.get_user_role() IN ('editor', 'admin')
    )
  );

-- No anon access — no GRANT to anon.

COMMIT;

Migration 2 — Create q_a_extractions + add FK back to q_a_pairs.promoted_from_extraction_id

2026MMDDHHMMSS_qa_002_create_q_a_extractions.sql:

BEGIN;

CREATE TABLE q_a_extractions (
  id                          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  workspace_id                UUID NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE,
  source_content_item_id      UUID NOT NULL REFERENCES content_items(id) ON DELETE CASCADE,
  source_chunk_id             UUID REFERENCES content_chunks(id) ON DELETE SET NULL,
  question_text               TEXT NOT NULL,
  answer_text                 TEXT NOT NULL,
  answer_standard             TEXT,
  answer_advanced             TEXT,
  question_embedding          vector(1024),
  extractor_kind              TEXT NOT NULL CHECK (extractor_kind IN (
    'docx_pattern_a', 'docx_pattern_b', 'docx_pattern_c', 'docx_pattern_positional',
    'markdown_heading_v1', 'yaml_frontmatter_v1', 'llm_extraction'
  )),
  extractor_version           TEXT NOT NULL DEFAULT '1.0',
  source_file_basename        TEXT,
  source_section_path         TEXT[],
  source_table_index          INTEGER,
  source_row_index            INTEGER,
  source_id_token             TEXT,
  provenance                  TEXT NOT NULL DEFAULT 'EXTRACTED' CHECK (provenance IN (
    'EXTRACTED', 'INFERRED', 'AMBIGUOUS'
  )),
  extraction_confidence       NUMERIC(3,2) NOT NULL DEFAULT 1.0,
  promoted_to_pair_id         UUID REFERENCES q_a_pairs(id) ON DELETE SET NULL,
  promoted_at                 TIMESTAMPTZ,
  promoted_by                 UUID REFERENCES auth.users(id) ON DELETE SET NULL,
  invalidated_at              TIMESTAMPTZ,
  invalidation_reason         TEXT,
  created_at                  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  cocoindex_run_id            TEXT,
  metadata                    JSONB DEFAULT '{}'::jsonb
);

-- Forward-declared FK from migration 1
ALTER TABLE q_a_pairs
  ADD CONSTRAINT q_a_pairs_promoted_from_extraction_fk
  FOREIGN KEY (promoted_from_extraction_id) REFERENCES q_a_extractions(id) ON DELETE SET NULL;

-- Indexes
CREATE INDEX idx_q_a_extractions_workspace ON q_a_extractions(workspace_id);
CREATE INDEX idx_q_a_extractions_source ON q_a_extractions(source_content_item_id);
CREATE INDEX idx_q_a_extractions_source_chunk ON q_a_extractions(source_chunk_id);
CREATE INDEX idx_q_a_extractions_question_embedding ON q_a_extractions
  USING hnsw (question_embedding vector_cosine_ops) WITH (m=16, ef_construction=64);
CREATE INDEX idx_q_a_extractions_active ON q_a_extractions(source_content_item_id)
  WHERE invalidated_at IS NULL AND promoted_to_pair_id IS NULL;
CREATE INDEX idx_q_a_extractions_promoted ON q_a_extractions(promoted_to_pair_id)
  WHERE promoted_to_pair_id IS NOT NULL;
CREATE INDEX idx_q_a_extractions_extractor ON q_a_extractions(extractor_kind, extractor_version);

-- RLS
ALTER TABLE q_a_extractions ENABLE ROW LEVEL SECURITY;

CREATE POLICY q_a_extractions_select_workspace_member ON q_a_extractions
  FOR SELECT TO authenticated
  USING (
    EXISTS (
      SELECT 1 FROM workspaces w
      WHERE w.id = q_a_extractions.workspace_id
        AND public.get_user_role() IN ('editor', 'admin')
    )
  );

CREATE POLICY q_a_extractions_insert_pipeline_admin ON q_a_extractions
  FOR INSERT TO authenticated
  WITH CHECK (
    public.get_user_role() = 'admin'
  );

-- Pipeline writes via service-role key; this RLS doesn't block service-role.

COMMIT;

Migration 3 — RPCs + functions (with anon EXECUTE revoke)

2026MMDDHHMMSS_qa_003_rpcs.sql:

BEGIN;

-- Hybrid search RPC for q_a_search MCP tool
CREATE OR REPLACE FUNCTION public.q_a_search(
  p_workspace_id UUID,
  p_query TEXT,
  p_query_embedding vector(1024),
  p_scope_include TEXT[] DEFAULT '{}',
  p_scope_exclude TEXT[] DEFAULT '{}',
  p_provenance_filter TEXT[] DEFAULT '{EXTRACTED,INFERRED,AMBIGUOUS}',
  p_publication_status_filter TEXT[] DEFAULT '{published}',
  p_include_extractions BOOLEAN DEFAULT FALSE,
  p_as_of TIMESTAMPTZ DEFAULT NULL,
  p_limit INTEGER DEFAULT 10,
  p_offset INTEGER DEFAULT 0
) RETURNS TABLE (
  q_a_pair_id UUID,
  q_a_extraction_id UUID,
  kind TEXT,
  question_text TEXT,
  answer_preview TEXT,
  scope_tags TEXT[],
  anti_scope_tags TEXT[],
  primary_domain TEXT,
  primary_subtopic TEXT,
  similarity NUMERIC,
  rank_method TEXT,
  provenance TEXT,
  confidence NUMERIC,
  valid_from TIMESTAMPTZ,
  valid_to TIMESTAMPTZ,
  current_flag BOOLEAN,
  publication_status TEXT,
  source_content_item_id UUID
)
LANGUAGE plpgsql
STABLE
SECURITY INVOKER
SET search_path = public, extensions
AS $$
DECLARE
  v_as_of TIMESTAMPTZ := COALESCE(p_as_of, NOW());
BEGIN
  RETURN QUERY
  -- q_a_pairs hits
  SELECT
    p.id AS q_a_pair_id,
    NULL::UUID AS q_a_extraction_id,
    'curated' AS kind,
    p.question_text,
    LEFT(p.answer_standard, 200) || CASE WHEN LENGTH(p.answer_standard) > 200 THEN '...' ELSE '' END AS answer_preview,
    p.scope_tag AS scope_tags,
    p.anti_scope_tag AS anti_scope_tags,
    p.primary_domain,
    p.primary_subtopic,
    (1 - (p.question_embedding <=> p_query_embedding))::NUMERIC AS similarity,
    'hybrid' AS rank_method,
    p.provenance,
    p.confidence,
    p.valid_from,
    p.valid_to,
    (p.valid_to IS NULL) AS current_flag,
    p.publication_status,
    p.source_content_item_id
  FROM q_a_pairs p
  WHERE p.workspace_id = p_workspace_id
    AND p.archived_at IS NULL
    AND p.valid_from <= v_as_of
    AND (p.valid_to IS NULL OR p.valid_to > v_as_of)
    AND p.publication_status = ANY(p_publication_status_filter)
    AND p.provenance = ANY(p_provenance_filter)
    AND (cardinality(p_scope_include) = 0 OR p.scope_tag && p_scope_include)
    AND NOT (p.anti_scope_tag && p_scope_exclude)
  UNION ALL
  -- q_a_extractions hits (if requested)
  SELECT
    NULL::UUID,
    e.id,
    'extracted',
    e.question_text,
    LEFT(e.answer_text, 200) || CASE WHEN LENGTH(e.answer_text) > 200 THEN '...' ELSE '' END,
    '{}'::TEXT[],
    '{}'::TEXT[],
    NULL::TEXT,
    NULL::TEXT,
    (1 - (e.question_embedding <=> p_query_embedding))::NUMERIC,
    'hybrid',
    e.provenance,
    e.extraction_confidence,
    e.created_at,
    NULL,
    TRUE,
    'extracted',
    e.source_content_item_id
  FROM q_a_extractions e
  WHERE p_include_extractions
    AND e.workspace_id = p_workspace_id
    AND e.invalidated_at IS NULL
    AND e.promoted_to_pair_id IS NULL
    AND e.provenance = ANY(p_provenance_filter)
  ORDER BY similarity DESC
  LIMIT p_limit
  OFFSET p_offset;
END;
$$;

-- Anon-EXECUTE revoke per CLAUDE.md gotcha
REVOKE EXECUTE ON FUNCTION public.q_a_search(UUID, TEXT, vector, TEXT[], TEXT[], TEXT[], TEXT[], BOOLEAN, TIMESTAMPTZ, INTEGER, INTEGER) FROM anon;

-- Helper RPCs: q_a_get, q_a_create — sketches only
-- (Full implementations live in the implementation phase, not this spike.)

COMMIT;

Migration 4 — Backfill from content_items.q_a_pair rows

2026MMDDHHMMSS_qa_004_backfill_legacy_q_a_pairs.sql:

BEGIN;

-- Backfill q_a_pairs from existing content_items rows with content_type='q_a_pair'.
-- Per CX.22 RESOLVED, the pre-launch re-ingest replaces most content. This
-- migration is a safety-net: any q_a_pair rows that survive re-ingest become
-- 'imported_legacy' origin_kind in q_a_pairs.

INSERT INTO q_a_pairs (
  id,
  workspace_id,
  question_text,
  answer_standard,
  answer_advanced,
  source_content_item_id,
  origin_kind,
  question_embedding,
  primary_domain,
  primary_subtopic,
  secondary_domain,
  secondary_subtopic,
  classification_confidence,
  scope_tag,
  provenance,
  confidence,
  valid_from,
  publication_status,
  governance_review_status,
  next_review_date,
  review_cadence_days,
  created_at,
  updated_at,
  created_by,
  updated_by,
  archived_at,
  archived_by,
  archive_reason,
  ingest_source,
  layer
)
SELECT
  c.id,  -- preserve UUID for citation FK continuity
  -- Workspace lookup via content_item_workspaces (first workspace only).
  -- Q-S16-15 + Q-S16-16: workspace scoping deferred + sentinel RLS handling to resolve before apply.
  COALESCE(
    (SELECT ciw.workspace_id FROM content_item_workspaces ciw WHERE ciw.content_item_id = c.id LIMIT 1),
    '00000000-0000-0000-0000-000000000000'::UUID  -- sentinel; Q-S16-16 — admin triage post-migration
  ) AS workspace_id,
  c.title AS question_text,  -- legacy: question stored in title
  -- Q-S16 verification §5.4: `c.content` includes "Q: <question>\n\n" prefix per build_content_record;
  -- strip prefix at implementation time. Working form:
  --   COALESCE(c.answer_standard, regexp_replace(c.content, '^Q: [^\n]*\n+', ''))
  COALESCE(c.answer_standard, c.content) AS answer_standard,
  c.answer_advanced,
  c.id AS source_content_item_id,
  'imported_legacy' AS origin_kind,
  -- Q-S16-12: c.embedding is content-level (title + summary + content), NOT question-level.
  -- Question_embedding here will be semantically wrong for "match this question shape" queries.
  -- Implementation: synchronous re-embed during Migration 4, or background re-embed via trigger.
  c.embedding AS question_embedding,
  c.primary_domain,
  c.primary_subtopic,
  c.secondary_domain,
  c.secondary_subtopic,
  c.classification_confidence,
  -- Q1.7 RESOLVED added content_items.scope_tag column pre-spike; prefer that when present.
  -- Working form:
  --   COALESCE(c.scope_tag, c.user_tags, '{}'::TEXT[])
  COALESCE(c.user_tags, '{}'::TEXT[]) AS scope_tag,
  'EXTRACTED' AS provenance,
  1.0 AS confidence,
  COALESCE(c.created_at, NOW()) AS valid_from,
  c.publication_status,
  c.governance_review_status,
  c.next_review_date,
  c.review_cadence_days,
  c.created_at,
  c.updated_at,
  c.created_by,
  c.updated_by,
  c.archived_at,
  c.archived_by,
  c.archive_reason,
  'qa_import_legacy' AS ingest_source,
  c.layer
FROM content_items c
WHERE c.content_type = 'q_a_pair'
ON CONFLICT (id) DO NOTHING;

-- DO NOT drop content_items.answer_standard / answer_advanced yet.
-- DO NOT remove 'q_a_pair' from content_type CHECK yet.
-- Those cleanups are migration 5 (post-cutover, after MCP tool + UI surface
-- migrate to q_a_pairs).

COMMIT;

10.2 Anon-EXECUTE revoke checklist (per migration)

Per CLAUDE.md gotcha “Supabase auto-grants anon EXECUTE on every new public.* PL/pgSQL function” — every CREATE FUNCTION in migrations 1-4 must have its companion REVOKE EXECUTE ... FROM anon. Migration 3 shows the pattern explicitly. Trigger functions called only via trigger context still get the revoke (belt and braces).

10.3 Migration ordering + risks

Step	Risk	Mitigation
Migration 1 (create q_a_pairs)	None — purely additive	Apply staging first; verify via supabase gen types
Migration 2 (create q_a_extractions + FK back)	None — additive + ALTER ADD CONSTRAINT	Same
Migration 3 (RPCs)	Anon EXECUTE leak if revoke skipped	CI lint via migration-revoke-guard.yml (already exists per CLAUDE.md)
Migration 4 (backfill)	Workspace assignment via content_item_workspaces first-row LIMIT 1 is best-effort; multi-workspace q_a_pairs need manual triage	Pre-cutover query to count NULL workspace_assignments; admin manual fix if non-zero
Migration 5 (post-cutover cleanup — NOT in this spike)	Dropping content_items.answer_standard + answer_advanced breaks any code path still reading them	Knip + grep audit pre-migration 5; gated on MCP + UI surface fully cut over

10.4 Re-ingest plan compatibility

Per CX.22 RESOLVED + S12 §5.5: the pre-launch re-ingest wipes and re-populates relevant tables. New tables (q_a_pairs, q_a_extractions) participate in the re-ingest:

• During re-ingest, the new pipeline orchestrator (post-import_bid_library.py evolution) writes envelopes to q_a_extractions directly (not content_items q_a_pair rows).
• Curated Q&A pairs (the 395 existing legacy rows) either (a) re-ingest via the new pipeline that creates q_a_extractions then promotes to q_a_pairs via UC5/UC6 review flow, OR (b) ride migration 4’s imported_legacy backfill if curation review isn’t feasible at re-ingest time.
• Recommendation: prefer (a) for the high-value curated content; (b) for tail content that doesn’t warrant manual review.

---

11. Open questions for parent session review

#	Question	Resolution path
Q-S16-1	Workspace scoping for q_a_pairs — single-workspace (current design) or workspace-shared via join table?	DEFERRED — wider schema design decision per S231 feedback. Bundled with broader workspace/multi-tenant scoping pass; not resolved at S16 level.
Q-S16-2	alternate_question_phrasings TEXT[] v q_a_question_variants table — at scale, TEXT[] cost may dominate. v1 keeps simple shape; v1.1 evaluates split.	Bake-off measure at S8 with real corpus; default to TEXT[] unless evidence forces split.
Q-S16-3	Form-question-extraction strategy + existing extractor fate. Two coupled questions: (a) How were the existing Standard Selection Questionnaire + Charnwood ITT Services question sets extracted historically? (audit needed — what tooling, what output shape). (b) What’s the intended forward path for extracting questions from forms (Shape F: XLSX/PDF) so they can be matched against q_a_pairs at bid time? (c) Pattern A/B docx extractor (Q3.5 [RATIFY-AT-REVIEW]) — retained as one-shot Phew-content migration helper per §7.3; fate post-migration (retire vs repurpose for form-question-extraction) gates on (a)+(b).	Liam to confirm at review. Forms extraction path was previously mis-tracked as Shape F → “Q-extraction via extract_docx_tables.py cousins”; needs explicit decision.
Q-S16-4	LLM-extraction prompt design — v1 ingestion uses the predetermined canonical shape (§7), so LLM-extraction is NOT a recurring ingestion path. It’s a candidate one-shot migration helper for any Phew content not covered by Patterns A/B + markdown_heading_v1 + Shape E conversion. Scope: needed only if migration coverage gap surfaces.	Defer to migration execution; sub-spike only if coverage gap appears.
Q-S16-5	Citations table migration — intended-arch §4.3 specifies a new polymorphic citations table; this spike inherits the design but doesn’t fully migrate content_citations. Migration plan for the cutover is out of scope.	Separate migration sketch; bundled with citation re-anchor work per UC6 + UC8 §6 ratifications.
Q-S16-6	Promotion UX — how does an extraction become a pair? UC5 has an answer for bid response → pair; UC6 covers user-direct edits. Extraction → pair promotion needs a UI surface spec.	Implementation-phase UX work; out of S16 spike.
Q-S16-7	Q&A pair language — multi-language support? v1 assumes English.	Out of scope; defer to v2 multi-language platform.
Q-S16-8	DRAFT-vs-final pattern detection for q_a_extractions — S10 dedup substrate has resolved to HYBRID (a)+(c) cocoindex @coco.fn chunk-embedding + skill-seekers keyword co-confirmer. Schema (superseded_by, provenance, valid_to) is ready; detection signal originates from the S10 hybrid substrate, with per-tenant rules under WP-DEDUP-RULES (coordinated with OPS-X-SCOPE-TAGS).	RESOLVED via S10 hybrid; no further action at S16 level.
Q-S16-9	Embedding cost projection — re-ingest cost across the Phew corpus is small (recorded in §12.2 cost model).	No action; record in cost model.
Q-S16-10	Sales proposal + future activity reuse — intended-arch §6.5 says q_a_pairs is reusable across activity types. This spike doesn’t add a kind discriminator. Should it?	Recommend NO — citation polymorphism (citations.citing_entity_id + citing_entity) handles “this Q&A pair was used in a sales proposal” without changing q_a_pairs shape.
Q-S16-11	Migration 1 trigger functions — current draft shows trigger names as placeholder comments (enforce_q_a_pair_archive_state_consistency, auto_v1_history_on_q_a_pair_insert, update_q_a_pair_updated_at, re_embed_q_a_pair_on_question_change). Full DDL bodies + companion REVOKE EXECUTE FROM anon must accompany Migration 1 (or be split into Migration 1b).	Implementation-phase work; expand pre-apply.
Q-S16-12	Migration 4 backfill embedding correctness — c.embedding on legacy q_a_pair rows was generated over title + summary + content (whole-pair text), NOT over question text alone. Backfilled question_embedding will be semantically wrong for “match this question shape” queries. Synchronous re-embed during Migration 4, OR background job, OR via the re_embed_q_a_pair_on_question_change trigger pickup?	Implementation-phase decision; recommend synchronous re-embed (small corpus, bounded cost).
Q-S16-13	Migration 5 reader-side cutover audit — before dropping content_items.answer_standard / answer_advanced + removing q_a_pair from content_items.content_type CHECK, run an explicit knip + grep audit of every code path reading those columns (app/api/, lib/bid/, lib/content/, lib/mcp/tools/, hooks/bid/).	Implementation-phase audit; Migration 5 gated on completion.
Q-S16-14	Vector index choice — hnsw chosen per KH house pattern (matches content_chunks + content_items.embedding precedent). Original draft used ivfflat; corrected via S16 verification audit §2.2 A3.	RESOLVED — hnsw across all q_a_pairs + q_a_extractions question_embedding indexes.
Q-S16-15	q_a_pairs.workspace_id ON DELETE RESTRICT vs CASCADE — deviation from KH house pattern (other workspace FKs use CASCADE). RESTRICT is defensible (Q&A pairs are high-value curated artefacts; refuse workspace deletion if pairs exist). Q-S16-1 workspace-scoping resolution may change the answer.	Coupled with Q-S16-1; resolve together.
Q-S16-16	Backfill workspace sentinel UUID handling — Migration 4 uses 00000000-...-0000 sentinel when no content_item_workspaces row exists for a legacy q_a_pair. RLS implications: which workspace policy matches this UUID? Admin manual fix-up post-migration, or fail-loud at Migration 4?	Implementation-phase decision; recommend fail-loud with explicit admin triage.
Q-S16-17	q_a_extractions immutability invariant — §8.4 frames extraction rows as immutable snapshots, but no UNIQUE constraint prevents concurrent flow runs from double-inserting (source_content_item_id, extractor_kind, extractor_version, source_table_index, source_row_index). Add UNIQUE constraint OR document append-only invariant.	Implementation-phase decision; recommend UNIQUE constraint for safety.
Q-S16-18	q_a_search UNION ALL ranking — when include_extractions=true, cosine similarity is the only ranking signal across curated q_a_pairs + uncurated q_a_extractions. Consumers will want curated to rank-prefer. Add kind_priority or confidence_boost term in ORDER BY.	Implementation-phase tuning; spec the boost term during RPC implementation.
Q-S16-19	Mempalace v4-alpha PG-backend availability — Liam asks (S231 feedback): is mempalace’s v4-alpha PostgreSQL backend available to inspect (i.e., exists on the mempalace GitHub repo)? Tracking via S15 (PRs #665 PG drawer + #1337 PG KG) — verify before any commitment.	Liam-flagged for S15 update; not S16-blocking.

---

12. Implementation phasing + operational cost

12.1 Implementation phasing

Implementation breaks into ordered phases. The phases are sequential where dependencies are real (Migrations 2 forward-declares an FK from Migration 1; MCP tools depend on RPC migration; pipeline orchestrator depends on lib/qa/ codebase) and parallelisable otherwise (adapter dev + LLM extractor + test triangulation).

Phase	Scope	Depends on
A — Migrations 1+2 (DDL)	q_a_pairs + q_a_pair_history + q_a_extractions + RLS + indexes + trigger DDL (Q-S16-11)	None
B — Migration 3 (RPCs)	q_a_search + q_a_get RPCs + anon-EXECUTE revokes + SECURITY INVOKER pattern	A
C — lib/qa/ codebase	envelope.ts (Zod) + writer.ts + yaml_frontmatter_v1 (v1 recurring path); migration helper adapters (Pattern A/B, markdown_heading_v1, optional LLM-extraction) under scripts/qa-migration/	A (uses schema)
D — MCP tools	q_a_search + q_a_get + q_a_create MCP tool registrations	B + C
E — Pipeline orchestrator	import_bid_library.py → import_qa_bundle.py envelope-path refactor; one-shot migration runner for Phew’s existing corpus	C
F — Migration 4 (legacy backfill)	Backfill from content_items rows where content_type='q_a_pair'; per Q-S16-12 re-embed correctly; per Q-S16-16 sentinel handling	A + E
G — Tests + docs	Adapter unit + envelope round-trip + RPC + MCP integration + RLS tests; client template + onboarding doc	All preceding
H — Migration 5 (post-cutover cleanup, OUT of S16 scope)	Drop content_items.answer_standard / answer_advanced + remove q_a_pair from content_type CHECK; gated on Q-S16-13 reader-side audit	F + reader cutover complete

12.2 Operational cost projection

Pipeline stage	Per-Q&A cost	Per-corpus cost (500 Q&As)
Adapter dispatch + parse (deterministic)	~0 (CPU-only, microseconds)	~0
Question embedding (OpenAI text-embedding-3-large)	~$0.0013	~$0.65
Classification (existing keyword + Pass 2)	~$0.001	~$0.50
LLM-extraction (one-shot migration helper, only invoked if Phew coverage gap surfaces — Q-S16-4)	~$0.03 per chunk	~$3.00 if ~100 chunks fall through deterministic helpers
Total one-time migration cost		~$4.20 per 500 Q&As (worst-case all-LLM)
Recurring v1 ingest cost per Q&A pair	embedding only (~$0.0013)	classification stays as today

Recurring cost is minimal — predetermined-shape ingestion is parse + embed + insert. LLM-extraction is one-shot migration cost only. Monitor at 5K+ Q&A scale per workspace.

---

13. Decision gate G16 verdict

G16 status: DESIGN-RECOMMENDED — READY-FOR-REVIEW.

Adopted (recommendations):

1. v1 canonical Q&A schema = q_a_pairs (golden) + q_a_extractions (cache) + q_a_pair_history (versions). 2 new tables + 1 history table + 4 migrations (citations migration deferred per Q-S16-5).
2. Provenance + confidence model = S12 ratified shape (CHECK text + numeric NUMERIC(3,2)). hnsw vector index per KH house pattern (Q-S16-14 RESOLVED).
3. Ingest envelope = Zod schema in lib/qa/envelope.ts; the recurring v1 path emits via yaml_frontmatter_v1; migration helpers (Pattern A/B docx, markdown_heading_v1, LLM-extraction) emit envelopes for the one-shot Phew cutover only.
4. Client-input policy = predetermined markdown shape (YAML-frontmatter per pair). KH defines the v1 canonical wire format pre-ingestion. Existing Phew content (5 historical shapes) migrates once via one-shot tooling under scripts/qa-migration/.
5. MCP tool surface = q_a_search (list/preview) + q_a_get (verbatim) + q_a_create (curated entry). Mirrors mempalace two-step Shape B.
6. content_items.answer_standard + answer_advanced migrate to q_a_pairs; q_a_pair value removed from content_items.content_type CHECK post-cutover (Migration 5; gated on Q-S16-13 reader-side audit).

Deferred to implementation phase (post-review):

• LLM-extraction migration helper (scope confirmed only if Phew migration coverage gap appears — Q-S16-4).
• Promotion UX from extraction → pair (Q-S16-6).
• Citation table cutover from content_citations to polymorphic citations (Q-S16-5).
• Multi-language Q&A (v2; Q-S16-7).
• Workspace scoping (Q-S16-1 + Q-S16-15; coupled with broader workspace decision).
• Form-question-extraction strategy (Q-S16-3; coupled with audit of existing Standard Selection Questionnaire + Charnwood ITT Services extraction tooling).
• Mempalace v4-alpha PG-backend availability check (Q-S16-19; tracked at S15).

Hand-off: Liam review at next decision-graph rewrite pass. Open questions in §11 require resolution. No code changes in this spike — design only.

---

End of S16 spike output. No schema, validation, or migration files modified. Client documentation in docs/client-documentation-base/ untouched. Recommendations only.