ID-57 PRODUCT — question_matches + retrieval + scoring
ID-57 {57.2} PRODUCT — question_matches + retrieval + scoring (T10)
Section titled “ID-57 {57.2} PRODUCT — question_matches + retrieval + scoring (T10)”Authored: 13/06/2026 (Planner, fresh context — {57.2} PRODUCT dispatch; NOT the
{57.1} RESEARCH instance).
Task: ID-57/T10 — question_matches table + retrieval RPC + scoring.
Builds on: ratified {57.1} RESEARCH (id-57-question-matches-retrieval/RESEARCH.md)
and canonical architecture 05-qa-flow.md §7 (+ §11 anti-patterns).
Status: PRODUCT (behaviour invariants + the pre-cutover schema-slice contract). No DDL
or RPC implementation here — {57.3} TECH derives those one-to-one from §A and §C–§E.
De-identification: the v1 pilot tenant is “the client” / “the pilot tenant” throughout.
No counterparty name appears.
Figma: none provided. This is a backend data-model + retrieval-contract feature (table, RPC, scoring); the only “users” are the code/agents that read and write the surface (see Summary). No visual surface in T10. If a future admin matching UI is built, it is a separate spec.
Summary
Section titled “Summary”question_matches is the corpus-match substrate that records the ranked candidate
q_a_pairs for a form-question — the missing edge between a workspace-scoped
form-question instance and the shared Q&A corpus. T10 delivers three surfaces: (1) the
table that stores per-candidate, per-method scores; (2) a retrieval contract that
returns ranked candidates for a form-question in the established two-step list/preview →
verbatim shape; (3) a scoring contract with separate embedding_score +
fulltext_score (never a blended single column). The spec separates a pre-cutover
schema-slice contract (§A — the ratifiable G6 gate) from the post-cutover feature
behaviour (§C–§F — retrieval, scoring, population, calibration).
Problem
Section titled “Problem”Nothing today records a ranked candidate edge between a form-question and a corpus-level
q_a_pair. The existing match route edges form-questions → content_items
(form_questions.matched_content_ids), and citations (ID-58) edges form-responses →
q_a_pairs for provenance (dormant in v1) — neither is candidacy against the Q&A corpus.
The scoring/retrieval maths already exists at the SQL layer (q_a_search) but has no
persistence target and no caller. question_matches is that target.
Code-intelligence orientation (verbatim, fresh {57.2} run against the
subo-id-57 worktree index):
gitnexus_query({query: 'question_matches form_questions q_a_pairs scoring retrieval'})returned execution flows but noquestion_matches-bearing symbol — top match-adjacent symbols wereFunction:app/api/procurement/[id]/questions/match/route.ts:POST(startLine 32, endLine 252, module[id], processproc_44_post“POST → CreateClient”),Function:lib/templates/template-coverage.ts:matchRequirement(startLine 198, endLine 346), andFunction:lib/source-documents/document-diff.ts:extractStructuredPairs(startLine 206, endLine 242). None touchquestion_matchesorq_a_pairsretrieval.gitnexus_context({name: 'q_a_search'})returned{ "error": "Symbol 'q_a_search' not found" }— expected, GitNexus is TS-only andq_a_searchis a PL/pgSQL RPC; not evidence of non-existence.- Per greenfield-disclaimer discipline, a grep fallback ran before concluding anything
“does not exist”:
grep -rln 'question_matches'oversupabase/ lib/ app/ scripts/ types/→ hits ONLY insupabase/migrations/20260520231524_t6_q_a_search_rpcs.sql(a header comment namingquestion_matches.embedding_score/fulltext_scoreas the column-shape source of truth) and a comment inscripts/cocoindex_pipeline/extraction.py— zero DDL, zero row writes anywhere (TS, Python, or SQL).grep -rln 'q_a_search'→ defined in four migrations, zero TS call sites (the RPC has no application caller).
Orientation conclusion: question_matches is genuinely unbuilt — this is not a
greenfield disclaimer (the orientation found the precise SQL-layer precedent it builds on),
it is a confirmed unbuilt-table gap. The retrieval/scoring precedent — q_a_search +
q_a_get_verbatim (supabase/migrations/20260520231524_t6_q_a_search_rpcs.sql, verified
verbatim: separate embedding_score numeric(5,4) + fulltext_score numeric(5,4), cosine
1.0 - (<=>), ts_rank(…, 2), internal-only embedding*0.6 + fulltext*0.4 ORDER BY,
caller-side scope filter, STABLE SECURITY DEFINER, explicit REVOKE-from-anon) — exists
and is uncalled. T10 makes the scoring columns real and gives them a caller.
Goals / Non-goals
Section titled “Goals / Non-goals”Goals. (a) A ratifiable question_matches schema slice that can land in the
structurally-complete handover DB pre-cutover under {64.8} gate G6. (b) A
behavioural retrieval + scoring contract that an Executor can build post-cutover without
re-deriving product intent. (c) Per-method score separation surfaced to every caller.
Non-goals. (a) No admin/reviewer matching UI in T10 (separate spec if ever built). (b)
No blend-function arithmetic, RPC topology, or DDL syntax here — those are {57.3} TECH
(see the explicit hand-offs in §D and §A). (c) No change to citations, q_a_search, or
the existing content_items match route. (d) No pipeline change — the ingestion pipeline
never writes question_matches (RESEARCH §5; grep-confirmed), so the entire feature is
post-ingest Lane-B and forces no re-ingest re-run.
Behavior
Section titled “Behavior”The invariants are grouped: §A the pre-cutover schema-slice contract (the G6 gate — independently ratifiable); §B capture semantics; §C retrieval; §D scoring; §E population/lifecycle; §F the bl-76 calibration fold-in. §A is a hard pre-cutover gate; §B–§F are post-cutover feature behaviour. The whole feature is post-ingest: building it after the ID-45 re-ingest cutover forces no pipeline re-run.
§A — Pre-cutover schema-slice contract (G6 / OQ-64-5) — independently ratifiable
Section titled “§A — Pre-cutover schema-slice contract (G6 / OQ-64-5) — independently ratifiable”This block is the load-bearing pre-cutover artefact. Per {64.8} gate G6, the handover DB
must be “structurally complete based on ratified specs” — so the question_matches table
must exist (DDL applied) in the handover DB before the ID-45 re-ingest cutover, even
though no row is populated and no RPC is called until post-cutover. The SCHEMA SLICE is the
gate; the RPC, scoring runtime, row population, and bl-76 calibration (§C–§F) are
post-cutover feature behaviour. A future TECH author must be able to derive the DDL
one-to-one from A1–A9; the parent ratifies A1–A9 as the G6 pre-cutover gate.
A1 — The table exists. A table named exactly question_matches exists. It is never
created under the legacy name bid_question_matches (05-qa-flow.md §7.1, §11
[RATIFIED-RENAME]). It is a new table — no prior DDL exists anywhere (orientation
grep-confirmed).
A2 — Left-hand side = the form-question instance (OQ-A RESOLVED → instance-LHS). A
question_matches row links a form-question instance to a corpus q_a_pair. The LHS
is a NOT-NULL FK form_question_id → form_questions(id) (the live table renamed from
bid_questions at {64.16}; carries workspace_id since the T2 project_id → workspace_id rename at
20260520120828_t2_combined_pr_intel_shape_b_form_type_split.sql:157). The Path C
catalogue requirement is reachable transitively via the existing
form_questions.template_requirement_id → form_template_requirements(id) FK — so
question_matches “consumes Path C catalogue rows” transitively (satisfies the ledger
framing) while honouring architecture §7.2 (workspace-scoped match + scope_tag filter). The
catalogue requirement is NOT a direct FK target on question_matches. (Catalogue-level
reusable matches, if ever wanted, are a separate, likely post-cutover shape — not T10.)
A3 — Right-hand side = the corpus q_a_pair. A NOT-NULL FK q_a_pair_id → q_a_pairs(id). q_a_pairs is corpus-level with no workspace_id FK (RESEARCH §1.1;
05-qa-flow.md §1.1, §11 — q_a_pairs.workspace_id, the q_a_pair_workspaces junction,
and idx_q_a_pairs_workspace are all RATIFIED-DO-NOT-BUILD). Workspace relevance is the
read-time scope-overlap filter (A8/B5), never a stored FK on the RHS.
A4 — question_kind discriminator. A NOT-NULL question_kind column holds the
form-type discriminator aligned to the live form_types vocabulary (PK key text,
seeded 8 values at
20260520120828_t2_combined_pr_intel_shape_b_form_type_split.sql:639: bid, rfp,
pqq, itt, tender, checklist, questionnaire, sales_proposal_template). The
vocabulary is NOT procurement-only: checklist applies to
procurement/sales_proposal/product_guide, questionnaire to procurement/competitor_research,
sales_proposal_template to sales_proposal (per each row’s applicable_application_types).
Recommendation (OQ-D, schema-slice detail, non-blocking): model question_kind as an
FK to form_types(key) — matching the established precedent where
form_template_requirements.template_type was migrated from an inline CHECK to an FK to
form_types(key) (…t2…:695), giving typed cardinality and extension via INSERT not ALTER.
TECH confirms FK vs inline; the value MUST resolve to a live form_types.key either way.
A5 — Separate per-method score columns (never a blend). The table carries two
separate score columns: embedding_score NUMERIC(5,4) and fulltext_score NUMERIC(5,4). A single blended match_score column is a [RATIFIED-RENAME] anti-pattern
(05-qa-flow.md §11; N9 RESOLVED-S236). The NUMERIC(5,4) shape mirrors the q_a_search
output columns exactly (the precedent’s header already binds these two column names to
question_matches as the source of truth). Open question (A5-null, → TECH): whether
either score column is nullable depends on the population path (§E) — if a row may be
written before one method has scored it (e.g. a q_a_pair lacking question_embedding),
nullability or a sentinel is needed; if rows are only ever written with both methods scored,
both are NOT NULL. TECH decides alongside OQ-B; the schema slice MUST state the chosen
nullability so the DDL is unambiguous.
A6 — Candidate-edge uniqueness. A uniqueness constraint prevents duplicate candidate
edges for the same form-question. The candidate key is (form_question_id, q_a_pair_id) —
optionally extended with question_kind if a single form-question could be matched under
more than one kind (default assumption: one kind per form-question, so question_kind is
derivable and the two-column key suffices). TECH fixes the exact key alongside the
population path (§E); the schema slice MUST carry a uniqueness constraint so the table
cannot accumulate duplicate ranked edges.
A7 — Population-anticipating columns. Even though population timing is post-cutover
(§E / OQ-B), the schema slice MUST anticipate a populate path so the DDL need not change
post-cutover. Minimum: a matched_at timestamptz NOT NULL DEFAULT now() (when the
candidate edge was last scored) plus the standard id uuid PK and created_at/updated_at
audit columns per the KH table convention. Whether stale rows are recomputed-in-place or
versioned is a §E/TECH decision, but the columns must not force an ALTER post-cutover.
A8 — Indexes. The schema slice MUST include indexes supporting the two load-bearing
access paths: (i) retrieve all candidates for a form-question ordered by score — an index
on form_question_id (and the ranking columns as TECH sees fit); (ii) the uniqueness
constraint of A6 (which provides its own index). It MUST NOT build any workspace-
partition index on the RHS corpus (the q_a_pairs scope filter rides the existing
idx_q_a_pairs_scope_tag GIN index, RESEARCH §1.1; building a workspace-partition index
would encode the superseded framing — 05-qa-flow.md §11).
A9 — RLS / REVOKE posture. The table is RLS-enabled and follows the established KH
new-table grant posture — apply grant_standard_public_table_access('public.question_matches'::regclass)
(the helper at 20260514150238_enable_rls_auto_event_trigger_and_grants_pattern.sql:104),
and any RPC that reads/writes the table follows RLS-PATTERN P-4: STABLE/VOLATILE as
appropriate, SECURITY DEFINER SET search_path = public, extensions, explicit REVOKE EXECUTE … FROM anon then GRANT … TO authenticated, service_role (the pg_default_acl
anon auto-grant gotcha — REVOKE FROM PUBLIC alone is a no-op). Access to candidates is
mediated by the caller’s workspace membership of the form-question’s workspace (B6); anon
never reads question_matches.
§B — Capture semantics (what a row means)
Section titled “§B — Capture semantics (what a row means)”B1 — A row is a ranked candidate, not a selected answer. A question_matches row
records that a corpus q_a_pair is a ranked match candidate for a form-question
(05-qa-flow.md §7.2: “records the ranked match candidates, NOT the final selected
answer”). Selecting a candidate as the answer, and recording that an answer cited a
q_a_pair, are downstream concerns (the latter is citations, ID-58 — see B8). A
question_matches row carries no “selected”/“accepted” semantics in T10.
B2 — Cardinality: one form-question ↔ many candidate q_a_pairs. A single form-question
may have many question_matches rows (one per candidate q_a_pair, up to the retrieval
limit). A single q_a_pair may be a candidate for many form-questions. The
(form_question_id, q_a_pair_id) pair is unique (A6) — the same corpus pair is never a
duplicate candidate for the same form-question.
B3 — Each row carries both per-method scores. Every materialised candidate row carries
its embedding_score and fulltext_score for that (form-question, q_a_pair) pair (subject
to the A5-null decision). The scores are the candidate’s per-method relevance to the
form-question’s text — not a global property of the q_a_pair.
B4 — question_kind reflects the form-question’s form-type. A row’s question_kind
reflects the form-type of its form-question, drawn from the live form_types vocabulary
(A4). This keeps cross-application-type matching clean: a bid-kind match and a
sales_proposal_template-kind match are distinguishable without inspecting the
form-question row (05-qa-flow.md §1.3, §7.2). The value MUST be a live form_types.key.
B5 — Candidates honour the workspace scope-overlap filter at match time. Only corpus
q_a_pairs whose scope_tag overlaps the form-question’s workspace scope_tag, and
whose anti_scope_tag does not overlap it, are eligible to become candidates
(05-qa-flow.md §1.1, §7.2):
WHERE q_a_pairs.scope_tag && workspace.scope_tag AND NOT (q_a_pairs.anti_scope_tag && workspace.scope_tag)The workspace is resolved from form_questions.workspace_id (A2). A q_a_pair excluded by
this filter MUST NOT appear as a candidate, regardless of its raw scores. (Where the
filter executes — inside a new RPC vs caller-side, mirroring the q_a_search caller-side
precedent — is TECH/OQ-C; the behavioural invariant is that excluded pairs never surface.)
B6 — Eligibility additionally honours the corpus publication gate. Consistent with the
q_a_search precedent, only q_a_pairs with publication_status = 'published' and a
non-null question_embedding are eligible candidates for the embedding-scored path. (A pair
with no embedding can still be fulltext-eligible — see A5-null and D3.) Draft/in_review/
archived pairs do not surface as candidates.
B7 — Role/workspace visibility. A caller may retrieve question_matches candidates for
a form-question only if they are a member of that form-question’s workspace (resolved via
form_questions.workspace_id), per the standard KH RLS role coupling. Anon never reads the
table (A9). Cross-workspace candidate reads are not permitted — a form-question’s candidates
are scoped to its workspace’s members even though the candidate q_a_pairs are corpus-level.
B8 — Distinct from citations (ID-58). question_matches (candidacy: ranked
candidates for a form-question) is distinct from citations (provenance: what a
form-response cited; the q_a_pair cited path is dormant in v1, gated by bl-74). They
share no FK and MUST NOT be conflated. Matches feed answer authoring; citations record what
authoring consumed (RESEARCH §1.6, §3).
§C — Retrieval contract
Section titled “§C — Retrieval contract”C1 — Retrieve ranked candidates for a form-question. The retrieval surface, given a
form-question, returns its ranked q_a_pair candidates — highest-relevance first — honouring
the scope/publication eligibility filters (B5/B6). The caller supplies the form-question (and
thereby its workspace and question_kind); the surface returns the ranked candidate list.
C2 — Two-step list/preview → verbatim shape (mirrors the q_a_search precedent).
Retrieval follows the established two-step pattern (05-qa-flow.md §7.2; S16 §6.1):
Step 1 (list/preview) returns a ranked list of candidates, each carrying the
q_a_pair_id, a truncated preview of the question + standard-answer text, both
per-method scores (embedding_score, fulltext_score), and the candidate’s scope_tag +
publication_status as pass-through columns. Step 1 does not return embeddings or full
answer bodies (payload discipline). Step 2 (get/verbatim) returns the full q_a_pair
row for a chosen candidate (minus question_embedding), reusing the existing
q_a_get_verbatim shape — no publication_status filter at Step 2 so superseded/archived
lineage is resolvable.
C3 — Ordering is by the default ranking policy. Step 1 candidates are returned in
descending order of the default ranking policy (D4). The ordering is stable for a given
score set: two candidates with identical blended rank order deterministically (tie-break is
a TECH detail, e.g. by q_a_pair_id), so paginated/repeated reads do not reshuffle.
C4 — Bounded result size. Retrieval returns at most a caller-supplied (or
default) limit of candidates, mirroring the q_a_search p_limit DEFAULT 20 precedent. The
limit bounds both the materialised candidate set (§E) and the Step-1 list.
C5 — Empty result is a valid, distinguishable state. A form-question with no eligible candidates (no corpus pair survives the scope/publication filter, or the corpus is empty pre-population) returns an empty ranked list, not an error. The caller can distinguish “no candidates” from “form-question not found” (the latter is an error/permission outcome per B7) and from “not yet scored” (§E population state).
C6 — Caller receives raw per-method scores. Step 1 returns the raw
embedding_score and fulltext_score per candidate (never only a blended figure), so the
caller/UI can surface per-method signal and apply its own display/blend policy. This mirrors
the q_a_search “callers receive raw per-method scores” rationale (N9 RESOLVED-S236) and is
the load-bearing reason the two columns are separate (A5).
§D — Scoring contract
Section titled “§D — Scoring contract”D1 — Two methods, two scores, always separate. Each candidate is scored by two methods:
embedding (vector cosine similarity) and fulltext (Postgres ts_rank). Both are
surfaced as separate NUMERIC(5,4) values (A5/B3/C6). A single blended score is never
the stored or returned representation.
D2 — embedding_score semantics. embedding_score is cosine similarity in [0,1],
higher = more similar, computed as 1.0 - (q_a_pair.question_embedding <=> query_embedding)
against the form-question’s query embedding — mirroring the q_a_search precedent exactly.
D3 — fulltext_score semantics. fulltext_score is the Postgres ts_rank over the
candidate q_a_pair’s question + standard answer + alternate phrasings against the
form-question’s query terms — mirroring the q_a_search precedent. The ts_rank
normalisation flag is 2 (the linear document-length divisor) by default, matching
the precedent (ts_rank(…, 2)); this default is the bl-76 calibration anchor (§F) and may
be re-ratified post-cutover. ts_rank returns 0 (not null) when there is no query match,
so a fulltext-only-eligible candidate is always scorable.
D4 — Both scores are always present and meaningful; a default ranking policy exists.
For every materialised candidate, both per-method scores are present (subject to the
A5-null decision for the embedding path on embeddingless pairs) and meaningful — neither is
a placeholder. A default ranking/blend policy governs the order candidates are returned
(C3). The precedent’s internal default is embedding_score * 0.6 + fulltext_score * 0.4
(embeddings dominate; fulltext breaks ties); T10’s default ranking policy SHOULD mirror this
unless TECH ratifies otherwise. The blend is used for ordering only — it is never stored
as a column and never the sole returned figure.
D5 — The blend is tunable without a schema change. The blend weights / ranking policy
are runtime configuration, not schema. Re-weighting the blend (or swapping the ranking
policy) MUST NOT require an ALTER to question_matches — because the table stores the two
raw per-method scores, any blend is recomputable at read time. (Hand-off — TECH owns: the
exact blend function, the configuration mechanism by which weights are set/tuned, RPC
topology for OQ-C, and where the scope filter executes. PRODUCT fixes only: two raw scores
always present, a default policy mirroring the precedent, and schema-free tunability.)
§E — Population & lifecycle (post-cutover; schema anticipates)
Section titled “§E — Population & lifecycle (post-cutover; schema anticipates)”E1 — Rows are materialised candidate edges, populated post-cutover. question_matches
rows are written by the feature (an RPC/job), never by the ingestion pipeline (RESEARCH §5;
grep-confirmed — the pipeline has only a comment, no write). Population is a post-cutover
activity; the table can exist empty in the handover DB (the G6 gate is structural — §A —
not data).
E2 — Population timing is a TECH decision the schema anticipates (OQ-B, non-blocking).
Whether rows are written on form-question create (trigger), on-demand (an API/RPC call
materialises top-N), or by a batch post-ingest sweep is a {57.3} TECH decision. The
schema slice anticipates whichever path via matched_at (A7) and the uniqueness constraint
(A6), so no ALTER is needed once TECH decides. The behavioural requirement: a retrieval call
(§C) for a form-question with materialised candidates returns them; the “not yet populated”
state is distinguishable from “no eligible candidates” (C5).
E3 — Recompute / staleness behaviour is bounded. When a form-question’s text or the
eligible corpus changes, candidate rows may be recomputed. T10’s behavioural invariant:
recomputation MUST NOT leave duplicate candidate edges (A6 uniqueness holds across
recompute), and matched_at reflects the last scoring time. Whether stale rows are updated
in place or versioned is TECH (E2-adjacent); the invariant is no-duplicates +
last-scored-visible.
§F — bl-76 ts_rank-normalisation calibration fold-in (post-cutover)
Section titled “§F — bl-76 ts_rank-normalisation calibration fold-in (post-cutover)”F1 — The default normalisation flag is the precedent’s, until calibrated. Until bl-76
calibration runs, fulltext_score uses ts_rank normalisation flag 2 (linear
doc-length divisor), mirroring the q_a_search precedent (D3). This is the shipped default
for the post-cutover feature.
F2 — Calibration is measured against the real re-ingested corpus. bl-76 is a
post-cutover calibration task (RESEARCH §4), gated only on a post-T7 real Q&A corpus
(the re-ingested pilot-tenant corpus). It measures the normalisation choice — flag 0
(none), 1 (1 + log(doc-length)), or 2 (current) — against realistic q_a_pairs with
known relevance ground truth, comparing fulltext_score distributions + NDCG@10, and
re-ratifies the best flag. Any “Cloud Run sidecar” framing is stale (S298 teardown;
pipeline runs on IONOS/Coolify) and MUST NOT appear in the feature.
F3 — Re-calibration changes a runtime parameter, not the schema. Adopting a different
normalisation flag post-calibration changes the ts_rank(…, flag) call inside whichever RPC
owns scoring — it does not alter question_matches (the table stores the resulting
fulltext_score, not the flag). So bl-76 calibration never reopens the §A schema gate.
Open questions
Section titled “Open questions”All carried from RESEARCH; OQ-A is now resolved. Tagged blocking/non-blocking for the schema slice (§A is the only pre-cutover gate).
- OQ-A — LHS FK target. RESOLVED → instance-LHS (non-blocking; ratified into A2). The
Orchestrator resolved OQ-A to the form-question instance:
form_question_id → form_questions(id), with the Path C catalogue reachable transitively viaform_questions.template_requirement_id. Catalogue-LHS is not the T10 shape. (Surfaced to the parent as an override-able interpretation; PRODUCT proceeds on instance-LHS.) - OQ-B — population timing/trigger (non-blocking; TECH; schema anticipates via A7/A6).
On-create trigger vs on-demand materialise vs batch sweep —
{57.3}TECH decides; the schema slice does not change either way. - OQ-C — new RPC vs reuse/generalise
q_a_search(non-blocking; TECH; precedent leans reuse). Where the scope filter executes and whether scoring is a newquestion_match_search-style RPC or an extension ofq_a_searchis RPC topology, not scoring maths — TECH-owned. - OQ-D —
question_kindFK toform_types(key)vs inline value (non-blocking; schema-slice detail; recommend FK). Recommendation in A4 is FK toform_types(key)matching theform_template_requirements.template_typeprecedent; TECH confirms. The value MUST resolve to a liveform_types.keyregardless. - OQ-A5-null — score-column nullability (NEW; non-blocking but schema-slice-relevant;
→ TECH alongside OQ-B). Whether
embedding_score(and/orfulltext_score) is nullable depends on whether a candidate row may exist before one method has scored it (e.g. an embeddinglessq_a_pair). TECH MUST state the chosen nullability so the §A DDL is unambiguous. (Raised in A5.) - OQ-E — citations-vs-matches boundary (non-blocking; resolved into B8). Confirmed distinct; no shared FK. Listed for completeness.