ID-57 RESEARCH — question_matches + retrieval + scoring

ID-57 {57.1} RESEARCH — question_matches + retrieval + scoring (T10)

Authored: 13/06/2026 (Planner, fresh context — {57.1} RESEARCH dispatch) Task: ID-57/T10 — question_matches table + retrieval RPC + scoring function. Status: spec_needed (ledger). Deps [37, 41] done. Effort ≈1.5 PLAN units. Scope of this doc: current-state inventory + gap analysis + open questions. NOT a product or tech spec — no behaviour decisions are ratified here; this doc routes the blocking ambiguity (OQ-A, the LHS-of-a-match FK target) to the parent before {57.2}.

Method note (de-identification): the v1 pilot tenant is referred to generically as “the client” / “the pilot tenant” throughout. No counterparty name appears.

---

0. Code-intelligence orientation (verbatim outputs)

Per the binding pre-spec-write rule, orientation ran before authoring. Raw outputs cited verbatim (not paraphrased) so the Checker can verify the step.

gitnexus_query({query: 'question_matches q_a_pairs retrieval scoring', repo: '…/subo-id-57'}) returned (top relevant symbols, abbreviated to the load-bearing rows):

• Function:app/api/procurement/[id]/questions/match/route.ts:POST — startLine: 32, endLine: 252, module: "[id]" (process proc_44_post “POST → CreateClient”).
• Function:lib/templates/template-coverage.ts:matchRequirement — startLine: 198, endLine: 346, module: "Scripts".
• Function:lib/source-documents/document-diff.ts:extractStructuredPairs — startLine: 206, endLine: 242 (Q&A-pair extraction, ingest-side; not retrieval).
• No question_matches-bearing symbol returned — the table is unbuilt (confirmed below).

gitnexus_context({name: 'q_a_search', repo: '…/subo-id-57'}) returned: { "error": "Symbol 'q_a_search' not found" }. GitNexus is TS-only; q_a_search is a PL/pgSQL RPC, so absence here is expected, not evidence of non-existence. Per the greenfield-disclaimer discipline, a ccc/grep fallback was run before concluding anything “does not exist”:

• grep -rln 'q_a_search' over supabase/ lib/ app/ types/ → hits ONLY in supabase/migrations/{20260520231524_t6_q_a_search_rpcs.sql, 20260521095209_t6_followup_revoke_public_execute_anon_inherit_fix.sql, 20260520225456_t6_q_a_pairs_full_schema.sql, 20260606130224_id75_reference_search_rpcs.sql}. Zero TS call sites. The RPC is defined but never invoked from application code yet.
• grep -rln 'question_matches' over supabase/ lib/ app/ scripts/ → hits ONLY in supabase/migrations/20260520231524_t6_q_a_search_rpcs.sql (a header comment naming question_matches.embedding_score/fulltext_score as the column-shape source of truth) and scripts/cocoindex_pipeline/extraction.py:322 (a comment referencing question_matches.question_kind per §7.2 — a comment, not a write).

Conclusion of orientation: question_matches is genuinely unbuilt (no DDL, no row writes anywhere — TS, Python, or SQL). The retrieval/scoring substrate exists at the SQL layer (q_a_search, q_a_get_verbatim) but has no caller. This is the precise gap T10 fills.

---

1. Current state — what exists today (file:line citations)

1.1 The corpus: q_a_pairs (BUILT)

supabase/migrations/20260520225456_t6_q_a_pairs_full_schema.sql carries the full shape (squashed-baseline columns + spec §2.1 additions):

• alternate_question_phrasings text[] NOT NULL DEFAULT '{}' (line 53), question_embedding vector(1024) NULL (line 54).
• origin_kind text with CHECK reconciled (lines 64–76; DEFAULT 'curated_explicit'), publication_status text with CHECK reconciled (lines 83–87; 'superseded' value dropped — lineage carried by the superseded_by UUID column).
• scope_tag text[], anti_scope_tag text[] — GIN-indexed: idx_q_a_pairs_scope_tag (line 285), idx_q_a_pairs_anti_scope_tag (line 288).
• Temporal valid_from/valid_to; q_a_pair_history snapshot table (line 153); q_a_extractions derived-cache table (line 103).

Corpus-level shape anchored canonically in 04-workspace-types.md §5. No workspace_id FK — corpus relevance is computed at read time via scope_tag && workspace.scope_tag overlap (and NOT anti_scope_tag), per the 04-workspace-types.md §11 anti-pattern list (idx_q_a_pairs_workspace, q_a_pairs.workspace_id NOT NULL, and the q_a_pair_workspaces M:N junction are all RATIFIED-DO-NOT-BUILD).

1.2 The retrieval/scoring precedent: q_a_search + q_a_get_verbatim (BUILT, uncalled)

supabase/migrations/20260520231524_t6_q_a_search_rpcs.sql — the canonical retrieval/scoring precedent T10 builds on. Key facts:

• q_a_search(p_query text, p_query_embedding vector(1024), p_limit int DEFAULT 20) returns a TABLE with separate embedding_score numeric(5,4) + fulltext_score numeric(5,4) (plus pair_id, preview text, scope_tag, publication_status).
• embedding_score = (1.0 - (qap.question_embedding <=> p_query_embedding))::numeric(5,4) (cosine similarity).
• fulltext_score = ts_rank(to_tsvector('english', question_text || ' ' || COALESCE(answer_standard,'') || ' ' || array_to_string(alternate_question_phrasings,' ')), plainto_tsquery('english', p_query), 2)::numeric(5,4) — the 2 is the ts_rank normalisation flag (divide rank by document length — the linear doc-length divisor; 1+log(ndoc) is flag 1, not flag 2). Note the migration’s inline comment at …231524_…:103 uses the looser wording “divide by 1 + log(ndoc)” — imprecise; flag 2 is the doc-length divisor (consistent with §4). This is the bl-76 calibration anchor and the exact flag-2 semantics its TECH will cite.
• Internal ORDER BY (embedding_score * 0.6 + fulltext_score * 0.4) DESC — the blend is used for ranking but NOT returned; callers receive the raw per-method scores and apply their own blend/display policy (N9 RESOLVED-S236 rationale, header lines).
• Scope filtering is CALLER-SIDE: scope_tag + publication_status are pass-through columns; the RPC filters only WHERE question_embedding IS NOT NULL AND publication_status = 'published'. The caller applies WHERE scope_tag && caller_scope_tags.
• Security: STABLE SECURITY DEFINER SET search_path = public, extensions; explicit REVOKE EXECUTE … FROM anon + GRANT … TO authenticated, service_role (RLS-PATTERN P-4; the pg_default_acl anon auto-grant gotcha). The follow-up migration 20260521095209_t6_followup_revoke_public_execute_anon_inherit_fix.sql hardens this.
• The header comment already names question_matches.embedding_score + question_matches.fulltext_score as the column-shape source of truth and embeds the ts_rank normalisation-option-2 note. So the scoring columns are pre-specified at the SQL-comment layer; ID-57 makes them real.

q_a_get_verbatim(p_pair_id uuid) is the two-step retrieval Step 2 (full row minus question_embedding; no publication_status filter so superseded lineage is resolvable).

1.3 The form-question instance: form_questions (BUILT — pin its LIVE name + shape)

OQ-A-critical pin. The bid→form rename ({64.16}, supabase/migrations/20260609145550_id64_14_bid_to_form_rename.sql:46) renamed bid_questions → form_questions. The live form-question row shape (baseline columns at 20260416102457_pre_squash_reconciliation.sql:3320–3336):

form_questions (workspace-scoped):
  id, workspace_id (FK), section_name, section_sequence, question_sequence,
  question_text NOT NULL, word_limit, evaluation_weight, confidence_posture,
  matched_content_ids uuid[],            -- existing KB-match output (NOT q_a_pairs)
  status, has_variants, assigned_to, created_by, created_at, updated_at,
  template_requirement_id uuid           -- FK → form_template_requirements(id)  ← KEY

The FK form_questions_template_requirement_id_fkey (renamed at migration line 56) and index idx_form_questions_template_requirement_id (line 81) confirm form_questions already carries the link to the Path C catalogue via template_requirement_id. This is the crux of OQ-A (§6).

1.4 The Path C catalogue: form_template_requirements (BUILT)

Renamed from template_requirements (20260520120828_t2_…:209). Baseline shape (20260416102457_…:4142–4170): id, template_name, template_version, template_type, section_ref, section_name, question_number, requirement_text, requirement_type (CHECK policy/statement/evidence/data/narrative/declaration/reference), primary/secondary domain+subtopic, matching_keywords text[], matching_guidance text, requirement_embedding vector(1024), is_mandatory, is_current, sector_applicability text[], word_limit_guidance, display_order. Post-T2, template_type is an FK to form_types(key) (…t2…:695), not an inline CHECK.

1.5 The form-type vocabulary: form_types (BUILT — 8 values)

form_types CV table (20260520120828_t2_…:628, PK key text). Seeded 8 values (…:639–648): 'bid', 'rfp', 'pqq', … (5 of 8 values are procurement-applicable; checklist/questionnaire are cross-application-type; sales_proposal_template is sales-proposal only — see the T2 migration’s form_types INSERTs for applicable_application_types per value). This is the LIVE vocabulary the §7.2 question_kind discriminator must align to.

1.6 Citations: citations (BUILT, ID-58 — distinct from question_matches)

supabase/migrations/20260609192337_id58_citations_polymorphic_replace.sql. Live shape (verify against the brief, which overstated it):

• citing_entity_kind ENUM ('form_response') — one citing value today, not five. The enum is designed for extension but v1 ships a single value (line 26).
• cited_target_kind ENUM ('content_item', 'q_a_pair') (line 25); one-of CHECK enforces exactly one cited FK (cited_content_item_id / cited_q_a_pair_id, lines 60–63).
• The q_a_pair cited path is DORMANT v1 (D1) — present-but-unused, gated by bl-74 (header lines 11–12).

Relationship to characterise (kept distinct in the gap analysis): citations records what cites a q_a_pair (a form-response → q_a_pair provenance edge, dormant in v1). question_matches records ranked candidate q_a_pairs for a form-question — a retrieval/scoring artefact, not a provenance record. They sit on opposite ends of the round-trip: matches feed answer authoring; citations record what authoring consumed.

1.7 The EXISTING (different) matching surface

app/api/procurement/[id]/questions/match/route.ts:POST (lines 32–252) matches form-questions against the KB content corpus (content_items), not against q_a_pairs. It calls supabase.rpc('search_for_form_response', …) (line 124) and writes results to form_questions.matched_content_ids uuid[] (line ~166). lib/ai/match.ts provides MatchResult, assessConfidence, deduplicateResults, MATCH_THRESHOLDS. lib/templates/template-coverage.ts:matchRequirement (line 199) matches form_template_requirements against content_items (cosine + keyword), reading the catalogue at line 466. Neither surface touches q_a_pairs — the Q&A-corpus match path is the gap.

---

2. The gap — what question_matches must capture that nothing does today

2.1 The missing edge

Nothing today records a ranked candidate edge between a form-question and a corpus-level q_a_pair. The existing match route edges form-questions → content_items (matched_content_ids); citations edges form-responses → q_a_pairs (dormant, provenance not candidacy). The q_a_search RPC can compute the ranked q_a_pair candidates but has no persistence target and no caller. question_matches is that target: per §7.2 it “records the ranked match candidates, NOT the final selected answer.”

2.2 The schema shape (from canonical §7, verified)

05-qa-flow.md §7 is the canonical design. Verified facts:

• §7.1: table is question_matches (renamed from bid_question_matches; never built under the old name — confirmed by grep, zero DDL anywhere). 04-workspace-types.md §11 lists bid_question_matches as [RATIFIED-RENAME].
• §7.2: question_kind discriminator aligned to form_types vocabulary (v1: 'bid'/ 'rfp'/'pqq'). A row links a form-question (workspace-scoped) ↔ a q_a_pair (corpus-level). Match-time scope filter = q_a_pairs.scope_tag overlap with workspace scope_tag (and NOT anti_scope_tag).
• §7.3: separate embedding_score NUMERIC(5,4) + fulltext_score NUMERIC(5,4) columns (N9 RESOLVED-S236). A single blended match_score is a §11 [RATIFIED-RENAME] anti-pattern. §7.3 gap flag explicitly delegates the per-method blend function + UI score presentation to the feature spec — i.e. ID-57’s TECH owns the runtime combination policy; the architecture record only fixes the two-column schema shape.

2.3 The retrieval surface — open design question (OQ-C)

Is question_matches a persisted/materialised candidate cache, computed-at-query, or both? The §7.2 language (“records the ranked match candidates”) implies persistence, but q_a_search already computes the same scores on demand. The TECH spec must decide whether question_matches rows are written (and when — OQ-B) or whether the table is a denormalised cache of an otherwise on-demand q_a_search call. The precedent strongly suggests reuse/generalise q_a_search for the scoring core rather than re-implement the cosine + ts_rank logic — but q_a_search lacks (a) a form-question / question_kind input, (b) the workspace scope-overlap filter (currently caller-side), and (c) any write path. OQ-C captures whether T10 adds a new RPC (e.g. question_match_search) or extends q_a_search.

2.4 The scoring surface

Stores per-method embedding_score + fulltext_score (NUMERIC(5,4)), mirroring the q_a_search output columns exactly (the precedent’s header already declares this binding). The blend weight (precedent uses 0.6/0.4 internally, unexposed) and the UI presentation of separate-vs-blended scores are the explicitly-deferred TECH-owned policy.

---

3. How question_matches relates to neighbouring entities

Entity	Relationship	Constraint / note
q_a_pairs (corpus)	RHS of the match edge; candidate answer source	scope_tag overlap filter; publication_status='published'; question_embedding IS NOT NULL. No workspace FK — relevance computed, not stored.
q_a_extractions	Derived cache of q_a_pairs (ingest-side)	Not directly referenced by matches; upstream of q_a_pairs.
citations (ID-58)	Records what a form-response cited (q_a_pair path dormant v1)	Distinct from matches: provenance vs candidacy. Do not conflate.
form_template_requirements (Path C, ID-52)	The catalogue the form-question may derive from (template_requirement_id)	Read shape lands in ID-52.14; T10 consumes it. The OQ-A FK-target question lives here.
form_questions	LHS of the match edge (the instance)	Already carries template_requirement_id FK to the catalogue.
form_types	question_kind discriminator vocabulary	FK target candidate for question_kind.

---

4. bl-76 ts_rank normalisation fold-in (post-cutover calibration riding ID-57)

Per the S299 fold-in register, bl-76 folds into ID-57 at spec time. bl-76 (ledgers/backlog/76.md, parked, track search-quality) is a MEASUREMENT task: measure q_a_search’s ts_rank normalisation choice against a real Q&A corpus and decide between flag 0 (none), 1 (1+log(doc-length)), 2 (linear doc-length divisor — current). The activation recipe: seed realistic q_a_pairs, build A/B queries with known relevance ground truth, run the three flags, compare fulltext_score distributions + NDCG@10, ratify the best.

Characterisation for ID-57: bl-76 is a post-cutover calibration task, NOT a schema/feature blocker. Its only real precondition is a post-T7 real Q&A corpus (the re-ingested pilot-tenant corpus). The brief’s “Cloud Run sidecar” framing is stale — bl-76’s own [S299 correction] block confirms Cloud Run was torn down (S298); the pipeline now runs on IONOS/Coolify, and the sidecar dependency is obsolete. So bl-76 rides ID-57 as a calibration deliverable that runs after the post-cutover re-ingest, separate from the pre-cutover schema slice (§5). bl-76 also notes a benign adjacent nit (trailing space from array_to_string on empty alternate_question_phrasings) — fix only if q_a_search gets a new CREATE OR REPLACE for another reason.

---

5. The pre-cutover schema-slice line (G6 / OQ-64-5)

id-93-pre-reingest-intent-gap/RESEARCH.md §5.1 + §6 establishes the gate. Verified facts:

• The pipeline never writes question_matches (ID-93 §5.1, flow.py:2987–3045; corroborated by §0 grep — only a comment in extraction.py:322). ID-57 is a fully post-ingest / post-cutover Lane-B feature — building it after re-ingest forces no re-run.
• G6 ratification (Liam, 08/06/2026, ID-93 §6 RATIFICATIONS block): OQ-64-5 was ratified as “structurally complete, based on ratified specs” — i.e. T10/T11/T12 schema DDL lands pre-cutover ONLY per ratified ID-57/58/60 specs ({64.8} gate G6). ID-93’s own §5.1 author-recommendation was “minimal”, but Liam over-rode to structurally-complete-if-ratified. The CLOSE-OUT block (11/06/2026) routes “ID-57 schema-slice spec-ratification ({64.8} G6 tail)” as a residual open.

The line ID-57’s spec chain must draw cleanly:

• Pre-cutover gate (schema slice): the question_matches table DDL (columns + FKs + CHECKs + indexes + RLS/grants) must be RATIFIABLE before cutover so it can land in the structurally-complete handover DB per G6. This slice is the load-bearing artefact for the {64.8} G6 tail.
• Post-cutover feature: the retrieval RPC, the scoring/blend policy, row population, UI presentation, and the bl-76 calibration are post-cutover (they need the re-ingested real corpus and ratified runtime policy). They do NOT gate cutover.

Implication for the spec chain: {57.2}/{57.3} should front-load the table DDL as a ratifiable slice (so G6 can consume it) and clearly demarcate the RPC/scoring/population as the post-cutover feature half. The PLAN ({57.4}, if needed) should order the schema Subtask first and independently ratifiable.

---

6. Open questions

OQ-A — LHS-of-a-match FK target (form-question instance vs catalogue requirement). BLOCKING for the schema slice.

The tension. 05-qa-flow.md §7.2 says a question_matches row links a form-question instance (bid_question/form_questions) → q_a_pair. The ID-57 ledger framing (S299) says “question-to-template-requirement matching surface that consumes Path C catalogue rows (ID-52.14)” → i.e. form_template_requirements (catalogue) → q_a_pair. These are different FK targets for the left-hand side.

Evidence that narrows it (but does not fully close it): form_questions already carries template_requirement_id uuid FK to form_template_requirements (§1.3). So a form-question instance transitively reaches its catalogue requirement. This makes a form-question-instance LHS the more likely canonical target (matching the architecture record §7.2 verbatim, and giving workspace scoping for free via form_questions.workspace_id), with the “template-requirement” ledger language reading as “the questions that derive from Path C catalogue rows” rather than a direct catalogue FK.

Why it is still BLOCKING. Three viable shapes remain, and they produce different DDL (which must ratify pre-cutover under G6):

1. Instance LHS: question_matches.form_question_id → form_questions(id). Workspace scope via the instance. Matches §7.2 literally. Rows are per-instance (created when a form-question exists).
2. Catalogue LHS: question_matches.template_requirement_id → form_template_requirements(id). Workspace-agnostic, reusable across workspaces (a requirement’s best q_a_pairs are tenant-independent given scope_tag). Matches the ledger’s “catalogue rows” language. But then “workspace scope_tag overlap” (§7.2) has no workspace to anchor to at match time — contradicts §7.2’s scope filter.
3. Both (discriminated): a question_source/question_kind discriminator with a nullable FK to each. Heaviest; risks the one-of-CHECK polymorphic shape (cf. citations).

Shape 2’s contradiction with §7.2’s workspace-scope filter is a strong signal against catalogue-LHS as the primary edge, but the ledger’s explicit “consumes Path C catalogue rows (ID-52.14)” wording and the ID-52 hand-off framing mean I cannot unilaterally discard it — the parent may intend the catalogue requirement as the matchable unit (so the same matches serve every workspace instantiating that requirement). This determines the FK shape of the pre-cutover schema slice → escalate to the parent. Recommended framing for the parent: “Confirm question_matches LHS = form-question instance (form_questions.id, per architecture §7.2, workspace-scoped) — with the catalogue reachable transitively via form_questions.template_requirement_id — NOT a direct catalogue FK. If catalogue-level reusable matches are wanted, that is a separate (likely post-cutover) shape and should be stated explicitly.”

OQ-B — population timing/trigger of question_matches rows. Non-blocking (post-cutover feature; schema must anticipate).

When are rows written: on form-question create (trigger), on-demand (API call materialises top-N), or batch (post-ingest sweep)? Post-cutover decision, but the schema must anticipate (e.g. a matched_at timestamptz, a uniqueness constraint on (form_question_id, q_a_pair_id) or (…, question_kind), and whether stale rows are recomputed or versioned). Resolve in {57.3} TECH; does not block the G6 schema slice if the table anticipates a populate path.

OQ-C — does scoring live in a new RPC or reuse/generalise q_a_search? Non-blocking (TECH decision; precedent leans reuse).

q_a_search already implements the exact cosine + ts_rank(flag 2) scoring and the separate-column output. It lacks: (a) a form-question/question_kind input, (b) the workspace scope-overlap filter (currently caller-side), (c) a write path. Options: extend q_a_search (risks coupling the MCP search tool to the match path), or add a sibling RPC (e.g. question_match_search) that wraps the same scoring expression with the form-question + scope inputs and a write/upsert into question_matches. The precedent’s header already binds the score columns to question_matches, so reuse of the scoring expression is strongly indicated; the open question is RPC topology, not scoring maths. TECH-owned. The bl-76 normalisation-flag decision (§4) feeds whichever RPC owns the ts_rank(…, flag) call.

OQ-D — question_kind storage: FK to form_types(key) vs inline value. Non-blocking (schema-slice detail; recommend FK).

§7.2 aligns question_kind to the form_types vocabulary. The codebase precedent (form_template_requirements.template_type was migrated from inline CHECK to an FK to form_types(key) at …t2…:695) strongly indicates question_kind should likewise be an FK to form_types(key) (typed cardinality, extension via INSERT not ALTER). Flag for the schema slice; recommend FK to match the established pattern. Pre-cutover-relevant (part of the ratifiable DDL) but mechanical, not blocking.

OQ-E — citations-vs-matches boundary confirmation. Non-blocking (characterisation, resolved here).

Confirmed distinct (§1.6/§3): citations = provenance (what a form-response cited; q_a_pair path dormant v1, gated by bl-74); question_matches = ranked candidates for a form-question. No shared FK, no overlap. Noted so TECH does not accidentally fold one into the other. Resolved — listed for completeness.

---

7. Recommendations to the Orchestrator (routing, not ratification)

• Escalate OQ-A to the parent before {57.2}/{57.3} — it sets the FK shape of the pre-cutover G6 schema slice; cannot be Planner-resolved without intent confirmation.
• Spec-chain tier: PRODUCT is light (the user-facing surface is internal/admin matching; behaviour is mostly the ranked-candidate contract + score presentation). TECH is the load-bearing artefact (schema slice + RPC topology + blend policy + bl-76 calibration plan). Recommend TECH+PLAN tier (or full chain if OQ-A resolution opens product-behaviour ambiguity around catalogue-reuse).
• Order the schema-slice Subtask first and independently ratifiable so {64.8} G6 can consume it ahead of the post-cutover feature half.
• bl-76 rides as a post-cutover calibration Subtask gated on the post-T7 corpus — not on the critical path to the schema slice or the RPC.