ID-120 {120.2} PRODUCT — Q&A dedup across one client's workspaces & forms (intra-tenant cross-workspace question-embedding-cosine PROPOSER + curator-approved merge)

ID-120 {120.2} PRODUCT — Q&A dedup across one client’s workspaces & forms (intra-tenant cross-workspace)

Inputs (read in full): {120.1} RESEARCH.md (Checker-PASS, Liam-RATIFIED) — it carries the S388 locked inputs and the three open questions this PRODUCT resolves. This spec does NOT re-derive RESEARCH; it builds the proposer behaviour + the curator-review UX + the corpus-wide read + operational proposer boundary on top of it.

cross-workspace (defined once, applies throughout this doc): “cross-workspace” = across the one client’s workspaces (intra-tenant), and across forms — NOT across client tenants. Deployment is one Supabase database per client, so the database itself is the tenant boundary; a single client holds multiple workspaces (one per application_type) and each workspace hosts many forms. Reading the whole single-client published-pair corpus is therefore an intra-tenant read, not a tenancy crossing.

LOCKED direction (S388 Decision A + the ratification decision — not re-litigated here): v1 dedup is CROSS-WORKSPACE (intra-tenant, and cross-form); shape = walk-time PROPOSER on the shipped Stage-5 substrate (repoint entity_mentions → q_a_pairs); the merge write fires only on curator approval, reusing the existing retireSupersededPairs-style archive

• superseded_by write (NO auto-merge, NO new write primitive). ID-120 subsumes UC8 ratification (no separate gate). mempalace-KG rejected; hybrid → v1.1.

CORRECTED FRAMING (S391, Liam-directed): the dedup axis is intra-tenant — the proposer reads the one client’s own published q_a_pairs corpus and proposes near-duplicate questions regardless of which workspace OR form they originated from (the primary driver: the same question answered across different forms of the same application — e.g. a procurement question in a PQQ and an ITT). The DB-per-client boundary is the tenant boundary, so RLS USING(true) on q_a_pairs is correct (one client = one DB). OQ-120-2a is now RESOLVED (intra-tenant: the curator reviews the client’s OWN corpus, so they see both questions AND both answers side-by-side — no cross-tenant disclosure trade-off exists). Cross-TENANT dedup (across separate client deployments) is OUT OF SCOPE, never-v1.

Spec-chain link 2 of {120.1 RESEARCH → 120.2 PRODUCT → 120.3 TECH → 120.4 PLAN}.

Summary

A walk-time pass that proposes that two q_a_pairs rows — potentially in different workspaces or different forms of the same client’s corpus — are near-duplicate questions, surfaces the proposal to a curator for review, and on the curator’s approval fires the existing supersession write (archive the non-survivor + set superseded_by to the survivor). It is a PROPOSER, never an auto-merge: the destructive write only fires on a curator’s explicit, per-pair approval. The corpus-wide read is intra-tenant — it reads the one client’s whole published q_a_pairs population (one DB per client; see §B), spanning that client’s workspaces and forms — and is gated by an operational named-proposer boundary (an automated pipeline caller, like Stage-5), not a tenancy wall. The curator-review UX and the survivor-selection policy are this spec’s two behavioural deliverables; the near-duplicate similarity threshold is set here as a target, with the index posture deferred to TECH.

Problem

The one client’s published q_a_pairs corpus accumulates near-duplicate questions — the same procurement question answered in different forms of the same application (e.g. a PQQ and an ITT), the same question raised in two workspaces, or re-ingested across runs — with no dedup pass. This is the primary real driver: a question answered across different FORMS of the SAME application, within the one client. The supersession write and the similarity substrate already exist (RESEARCH §2.3): what is missing is (a) a pass that reads the client’s whole published-pair population (across that client’s workspaces and forms) and proposes near-duplicate merges, (b) a curator surface to approve/reject those proposals with a clear survivor choice, and (c) an operational named-proposer boundary for the corpus-wide read. The read is the whole single-client published-pair population: because the deployment is one Supabase database per client, the database itself is the tenant boundary, so reading the entire corpus inside it is an intra-tenant read — not a tenancy crossing. The only contract needed is an operational marker that the corpus-wide candidate read is performed by the named automated proposer (like Stage-5), not a tenancy wall — the DB’s RLS USING(true) on q_a_pairs (RESEARCH §4(b)) is correct precisely because there is only ever one client behind it.

Code-intelligence orientation (verbatim — repo:'canonical'):

• gitnexus context retireSupersededPairs → Function lib/q-a-pairs/promote-corpus.ts:retireSupersededPairs (startLine 826, endLine 1057); incoming calls: exactly ONE — promoteCorpusExtractions (lib/q-a-pairs/promote-corpus.ts); outgoing call safeErrorMessage (lib/error.ts). This is the merge-write primitive ID-120 reuses unchanged — single caller, so the approval path is the only thing that triggers it.
• gitnexus context promoteCorpusExtractions → Function lib/q-a-pairs/promote-corpus.ts:promoteCorpusExtractions (startLine 234, endLine 525); incoming calls: exactly ONE — POST (app/api/q-a-pairs/promote-corpus/route.ts). The supersession write today reaches the DB only through this one HTTP route.
• gitnexus query 'cross-workspace q_a_pairs deduplication supersession proposer' returned only generic POST → CreateClient / POST → UnauthorisedResponse / POST → ForbiddenResponse processes (no dedicated cross-workspace-dedup flow) — confirming this is a new pass on an existing substrate, not an extension of an indexed flow. gitnexus context q_a_search → {"error":"Symbol 'q_a_search' not found"} — q_a_search is a SQL RPC outside the TS graph (RESEARCH §2.1), so the similarity surface is verified by direct migration read below, not by gitnexus.
• The current supersession-write auth (verbatim, app/api/q-a-pairs/promote-corpus/route.ts): header lines 11–18 — Caller A — this HTTP route (RLS-scoped admin/editor client); Caller B — ID-45 pipeline (service-role client, escalated to parent); INV-14 (auth guard): getAuthorisedClient(['admin','editor']); INV-15 (RLS-scoped): … no service-role escalation; handler line 53 const auth = await getAuthorisedClient(['admin', 'editor']); line 54 if (!auth.success) return authFailureResponse(auth);. So a role-gated admin/editor RLS-scoped write path already exists — the curator-approval write reuses it.
• The corpus-wide read is RLS-open by design (verbatim, squash_baseline.sql): RLS SELECT on q_a_pairs is CREATE POLICY "q_a_pairs_select" ON "public"."q_a_pairs" FOR SELECT USING (true); (line 10904) — no per-workspace row filter, and correct because the DB is one client (the DB is the tenant boundary); q_a_search filters only WHERE qap.question_embedding IS NOT NULL AND qap.publication_status = 'published' (lines 4299–4300) and its COMMENT states Scope filtering is caller-side (scope_tag pass-through) (line 4322).
• Provenance is already preserved by value (verbatim, squash_baseline.sql): q_a_pair_history (CREATE at line 7065) snapshots source_workspace_id (Snapshot of q_a_pairs.source_workspace_id at transition … provenance preserved by value. ID-64.15.) and superseded_by (… lineage preserved by value. ID-64.15.) at every transition — so a merge’s source-workspace and form lineage (within the one client) is recorded automatically by the existing trigger, with no new provenance store.

Goals / Non-goals

Goals: (a) an intra-tenant question-embedding-cosine proposer that surfaces near-duplicate q_a_pairs candidates — across the client’s workspaces AND forms — for review; (b) a curator-review surface showing both questions, a proposed survivor with its basis, and per-pair approve/reject; (c) an operational named-proposer boundary for the corpus-wide candidate read (an automated pipeline marker, not a tenancy wall — the DB-per-client boundary already is the tenant boundary); (d) reuse of the existing supersession write on approval — no new write primitive.

Non-goals (v1): auto-merge of any kind (every merge is curator-approved — INV-9); a hybrid (lexical + vector + KG) proposer (deferred to v1.1 — RESEARCH §3.4); a new lineage/provenance store (q_a_pair_history already mirrors it — INV-16); the vector-index decision itself (brute-force vs. HNSW is a TECH call — INV-20); workspace-scoping enforcement moved into the DB/RLS (v1’s boundary is the operational proposer marker — INV-10); cross-TENANT dedup / a shared platform-level Q&A library across separate client deployments — never v1 (the DB-per-client boundary IS the tenant boundary; there is no v1 or foreseeable requirement to dedup across client DBs); re-scoping or merging unpublished / archived pairs (the proposer reads the published population only — INV-2).

Behavior

A. The proposer (what gets proposed, and when)

1. The proposer runs as a walk-time Stage-5-style post-pass. After a pipeline run’s per-item phase settles, a post-pass reads a population of the client’s published q_a_pairs rows — regardless of which workspace OR form each pair originated from — and computes question-embedding cosine similarity to find near-duplicate candidates. It runs beside the cocoindex walk on the same substrate as the shipped Stage-5 entity-resolution pass (RESEARCH §3.1); it does not run inside a per-file component.

2. The candidate population is the published, embedding-bearing pairs. The proposer considers only q_a_pairs rows with publication_status = 'published' and a non-null question_embedding (the same eligibility q_a_search already applies). Unpublished, draft, or already-archived (publication_status = 'archived', or non-null superseded_by) pairs are never proposed as either survivor or non-survivor.

3. A proposal pairs two distinct published pairs as near-duplicate questions. Each proposal identifies exactly two q_a_pairs rows (pair A, pair B) whose question_embedding cosine similarity meets or exceeds the near-duplicate threshold (INV-19), records the computed similarity score, and nominates a proposed survivor (INV-6). The two pairs may originate from different workspaces and/or different forms of the same client (anchor on the lineage columns source_workspace_id, source_form_response_id, source_question_id) — the dedup key is the question, corpus-wide. A pair is never proposed against itself; a proposal never groups more than two pairs (chains are handled as separate pairwise proposals, INV-7).

4. The proposer never writes to q_a_pairs. Producing a proposal has no effect on the q_a_pairs corpus — no archive, no superseded_by, no publication_status change. The only state a proposer run produces is the set of pending proposals awaiting curator review. A proposer run is idempotent at the proposal level: re-running over an unchanged corpus does not create duplicate pending proposals for the same pair (A, B).

5. A proposal already actioned is not re-proposed. If a curator has already approved or rejected a proposal for a given pair (A, B), or either pair is already archived/superseded, the proposer does not resurface it as a new pending proposal. (A rejected pair MAY be re-surfaced only if the underlying questions materially change — TECH defines the change-detection mechanism; the behavioural invariant is “no nagging re-proposal of an unchanged, already-rejected pair”.)

B. Corpus-wide read + proposer boundary (intra-tenant) (OQ-120-2 — the central decision)

6. The proposer reads the client’s whole published corpus; this is a named, deliberate widening. The candidate-generation read deliberately ignores the per-workspace source_workspace_id filter (and the per-form lineage filters) that ordinary callers apply, so that a near-duplicate spanning two workspaces or two forms of the same client can be found. This is the second deliberate widening of the otherwise-scoped Q&A read discipline — the first being the Stage-5 ID-80.14 op_id-scope exception (RESEARCH §4(b)), which is key-scoped and never blind-whole-table. ID-120’s widening is broader (the client’s whole published population) and so MUST be named and confined to the proposer’s candidate-generation read — it does not licence any other surface to read corpus-wide. The read stays intra-tenant: it reads only the one client’s corpus (one DB per client), never another client’s deployment.

7. The proposer runs under a service-role read, by an automated (non-interactive) caller — not an interactive curator. RLS on q_a_pairs is USING(true) (§Problem), which is correct because the DB is one client — so the read covers the whole client corpus by design. The proposer is an automated pipeline caller using a service-role client (the same posture as the Stage-5 pass and Caller B of the existing promote path), whose sole licensed action is reading question embeddings + question text to compute candidates across the client’s workspaces and forms. No interactive user triggers the corpus-wide read directly. This keeps the widening inside one named operational boundary (the automated proposer, like Stage-5) rather than spread across user-facing clients — it is an operational marker, not a tenancy wall.

8. The named-proposer marker is the operational boundary on the corpus-wide read. The corpus-wide candidate read MUST be gated by an explicit, named marker that asserts “this caller is the automated dedup proposer, performing the candidate-generation read” — an operational boundary on which automated pass does the whole-corpus read, not a tenancy authorisation (there is no tenant wall to breach: the DB is one client). Any caller that is not the named proposer attempting the same whole-corpus read is out of the operational contract. (TECH defines the marker’s concrete form.)

9. Every merge is curator-approved; the merge write reuses the role-gated admin/editor path. The destructive merge write (archive non-survivor + set superseded_by) fires only when a curator explicitly approves a specific proposal. It reuses the existing supersession write, which is role-gated to admin/editor and RLS-scoped (route header INV-14/15, §Problem). The proposer’s service-role read (the corpus-wide candidate read) does not extend to the write: the approval-triggered merge runs under the curator’s own authorised client/role, not service-role.

10. A curator reviewing a merge sees both questions AND both answers side-by-side — it is the client’s OWN corpus. For the curator to make a sound survivor choice they must see both pairs in full side-by-side (INV-12). Because the corpus is the one client’s own data (one DB per client), there is no cross-tenant disclosure trade-off: the curator has full visibility of both questions, both answers, and the survivor-basis fields (source_workspace_id, source_form_response_id, recency, publication_status, confidence) to judge the merge — these are all this client’s records. OQ-120-2a is RESOLVED (Liam, S391): the cross-tenant question-text disclosure trade-off it described does not exist intra-tenant; the (a)/(b)/(c) bounded-disclosure options are moot. The surface still scopes to the specific proposed pairs (the curator reviews the proposals queue, not an arbitrary corpus browser), but that is a UX-focus choice, not a tenancy boundary.

11. A cross-workspace superseded_by link is fine — canonical ownership moves between the client’s workspaces. The q_a_pairs.superseded_by FK targets q_a_pairs(id) with no workspace constraint (RESEARCH §4(c)), so a survivor in workspace X may supersede a non-survivor in workspace Y (or one originating from a different form). This is accepted for v1: after a merge, the superseded pair points to its survivor, and that survivor is the canonical answer for the deduplicated question regardless of which of the client’s workspaces or forms originally held it. The merge therefore moves the surviving canonical pair’s ownership between the client’s workspaces (intra-tenant) — a consequence the curator is accepting on approval (INV-12 surfaces the survivor’s source_workspace_id so the curator sees this before approving). It is an informative ownership move within the one client, not a cross-tenant event.

C. The merge write + provenance (reuse, not new)

12. The proposed survivor is chosen by an explicit, displayed survivor-selection policy. Each proposal nominates one of the two pairs as the survivor by a deterministic policy, and the review surface shows the basis. The v1 policy, in order: (1) publication_status — a published pair always survives over a non-published one (though both are normally published per INV-2, this guards edge cases); (2) confidence / quality where a confidence or quality signal exists on the pair; (3) recency — the more recently updated/created pair survives, on the rationale that it reflects the latest answer. The curator sees the nominated survivor, the losing pair, and the reason (e.g. “survivor: more recent (updated 15/06/2026)”), and MAY override the nomination (INV-13).

13. The curator may override the survivor before approving. The proposed survivor is a default, not a mandate. On the review surface the curator can swap which pair survives, then approve. The merge write then archives the curator’s chosen non-survivor and sets its superseded_by to the curator’s chosen survivor. Rejecting the proposal performs no write.

14. Approval fires exactly the existing supersession write. On approval the surface archives the non-survivor (publication_status = 'archived') and sets its superseded_by = <survivor id>, via the existing retireSupersededPairs-style path (RESEARCH §2.3). No new write primitive, no new table, no schema change to the write path. The survivor pair is unchanged except as a superseded_by target.

15. Approval and rejection are explicit, auditable terminal states. Every proposal ends in exactly one of: approved (merge written), rejected (no write), or remains pending. There is no silent auto-resolution and no state where a merge half-fires. If the merge write fails after the curator approves, the surface reports the failure explicitly (the proposal does not flip to a misleading “approved/merged” state) and the corpus is left unchanged — never a partial archive without a superseded_by, or vice-versa.

16. Cross-workspace/cross-form provenance is recorded automatically by the existing history mirror. The merge write fires the existing q_a_pair_history trigger, which snapshots both source_workspace_id and superseded_by by value at the transition (§Problem). So the lineage of a merge — which of the client’s workspaces/forms the superseded pair came from, and which survivor it now points to — is preserved automatically. v1 adds no new provenance store; the proposal record SHOULD additionally carry the source source_workspace_id (and form lineage source_form_response_id) of both pairs and the computed similarity score so the review + audit trail is self-describing, but the canonical lineage record remains q_a_pair_history.

D. Curator-review UX (OQ-120-1)

17. Proposals are reviewed per-pair, with optional batch triage. The default and authoritative granularity is per-pair approval — each proposal is approved or rejected on its own merits (the survivor choice is per-pair, INV-12). The surface MAY present pending proposals as a reviewable list/queue and MAY offer a batch “reject all” / “approve all selected” convenience, but a batch approve still applies each proposal’s own (possibly curator-overridden) survivor choice, and there is no batch action that merges pairs the curator has not seen the survivor basis for. Approving is never the zero-effort default — the surface does not pre-select approve.

18. The review surface shows both questions side-by-side with the survivor basis. For each proposal the curator sees: both questions’ text and both answers (INV-10) side-by-side; each pair’s source_workspace_id and form lineage (source_form_response_id) so a proposal is visibly cross-workspace and/or cross-form, plus publication_status, last-updated date (DD/MM/YYYY), and any confidence signal; the nominated survivor clearly marked with its selection reason (INV-12); and per-pair approve / reject controls plus the override-survivor affordance (INV-13). A proposal that spans workspaces/forms is visually distinguished from a same-workspace, same-form one (more than colour alone — a label/badge, WCAG 2.1 AA) so the curator knows the approved merge moves canonical ownership within the client (across the client’s workspaces/forms) — not a cross-tenant merge.

19. Empty, loading, and error states are explicit. When there are no pending proposals the surface shows an explicit “no pending duplicate proposals” empty state, not a blank panel. While proposals load, a skeleton/spinner shows. A fetch or merge-write failure surfaces a clear error affordance (INV-15) — never a silent blank or a misleadingly-empty queue. All states use Warm Meridian semantic tokens only (no raw Tailwind colours), UK English, DD/MM/YYYY.

E. Similarity threshold + index posture (OQ-120-3)

20. The near-duplicate threshold targets curator-reviewable precision over exhaustive recall. v1 sets a deliberately conservative cosine-similarity threshold so that what reaches the curator is a high-precision set of genuine near-duplicates (questions a reasonable curator would agree are “the same question”), accepting that some looser paraphrase duplicates are missed in v1 — because a low-precision flood of weak candidates makes the review surface unusable and erodes curator trust. The behavioural target: the curator’s reject-rate on surfaced proposals should be low (most surfaced pairs are genuine duplicates a curator approves); a high reject-rate means the threshold is too loose and MUST be raised. The concrete numeric threshold is set in TECH (informed by sampling the live embedding distribution), but PRODUCT fixes the direction: precision-first, conservative, tunable, with the threshold a single named, adjustable value — not hard-coded across multiple call sites.

21. The vector-index posture (brute-force vs. HNSW) is deferred to TECH. v1 may ship brute-force cosine (consistent with Stage-5’s current pgvector posture and acceptable at the current published-pair corpus size — RESEARCH §4(d)) or add one HNSW/ivfflat migration to bound scan cost as the corpus grows. This is a TECH decision informed by the expected published-pair population size (the one client’s whole corpus); PRODUCT imposes only the behavioural constraint that the proposer run must complete within the pipeline-run window without degrading the rest of the walk, and that the index choice must not change which pairs are proposed (an index is a performance optimisation, not a different candidate set — same threshold, same proposals).

F. Roles, scoping, and AI-invisibility

22. Roles. The corpus-wide proposer read runs under the named automated service-role caller (INV-7); only admin / editor curators may view proposals and approve/reject merges (reusing the existing getAuthorisedClient(['admin','editor']) gate — §Problem), reviewing the client’s own corpus, with failures routed via authFailureResponse(auth). A viewer MUST NOT see the proposal surface. No interactive role gains a general corpus-wide q_a_pairs read through this Task — only the proposal-review surface of INV-10. (Per-workspace RBAC inside one client was considered and is explicitly not a v1 need — do not build it.)

23. AI-invisible-infrastructure. The embedding-cosine similarity that generates candidates is invisible infrastructure: the curator reviews ordinary “these two questions look like duplicates” proposals, not a user-facing “AI similarity” feature; the deduplicated corpus behaves as if the duplicate was never there (inherits ai-visibility-policy.md). The proposal surface does not present similarity scores as an AI confidence headline — at most a plain “match strength” affordance subordinate to the human review.

Empirical verification (pre-ratification, OQ-3)

Symbol / dependency	Class	Method	Result
q_a_pairs.superseded_by FK + publication_status archive (the reused merge write)	internal-Canonical	gitnexus context retireSupersededPairs + migration read (squash_baseline.sql:9889, RESEARCH §4(c))	PRESENT — single caller promoteCorpusExtractions; FK ON DELETE SET NULL; no schema change to write path. (Out of OQ-3 external scope; recorded for completeness.)
q_a_pairs RLS SELECT USING(true) + q_a_search caller-side scoping	internal-Canonical	migration read (squash_baseline.sql:10904, :4299–4300, :4322)	PRESENT / CONFIRMED — RLS-open is correct (one DB per client = the tenant boundary); the corpus-wide read is intra-tenant, gated by the operational named-proposer marker (INV-6..INV-11).
q_a_pair_history snapshots source_workspace_id + superseded_by by value	internal-Canonical	migration read (squash_baseline.sql:7065, column comments ID-64.15/ID-59)	PRESENT — intra-tenant (cross-workspace/cross-form) merge provenance preserved automatically; no new store (INV-16).
pgvector <=> cosine over question_embedding vector(1024)	extension built-in (in-repo)	RESEARCH §6: exercised in shipped q_a_search (squash_baseline.sql:4282)	PRESENT and EXERCISED in shipped SQL — not asserted from prose. No external import-and-call check required.

No external-library API claim drives this PRODUCT (the substrate is internal Stage-5 + retireSupersededPairs + the in-repo pgvector operator/type, per RESEARCH §6). No ABSENT / SIGNATURE_DRIFT to report.

Open questions (collected — for TECH {120.3} and/or Liam)

OQ-120-2a — RESOLVED (Liam, S391): the cross-tenant question-text disclosure trade-off it posed does not exist. Dedup is intra-tenant (one DB per client), so the curator reviews the client’s OWN corpus and sees both questions AND both answers side-by-side (INV-10). Removed from the open list.

• OQ-120-3-threshold (numeric cosine threshold) → TECH: the concrete near-duplicate cosine value, set in TECH by sampling the live question_embedding distribution (PRODUCT fixes precision-first direction + single-tunable-value constraint, INV-20).
• OQ-120-3-index (brute-force vs. HNSW) → TECH: the vector-index posture, deferred to TECH by expected published-pair population size — the one client’s whole corpus (INV-21); behavioural constraints (run-window, identical candidate set) fixed here.
• OQ-120-re-propose (rejected-pair resurfacing) → TECH: the change-detection mechanism that lets a materially changed previously-rejected pair re-surface without nagging on unchanged pairs (INV-5 fixes the behaviour; TECH picks the mechanism).
• OQ-120-proposal-store → TECH: where pending proposals live (a new lightweight proposals table vs. an existing surface). PRODUCT fixes the proposal’s required fields (two pair ids, both source_workspace_ids and form lineage source_form_response_id, similarity score, nominated survivor, terminal state — INV-3/15/16); TECH picks the substrate. Note: this is the proposal record, distinct from the canonical lineage record (q_a_pair_history, INV-16) which is NOT new.

---

End of PRODUCT. Corrected framing (S391, Liam-directed): dedup is intra-tenant — the proposer reads the one client’s whole published corpus across that client’s workspaces and forms (one Supabase DB per client = the tenant boundary; “cross-workspace” = intra-tenant + cross-form, never cross-client). Cross-TENANT dedup is never-v1. OQ-120-2a is RESOLVED (no cross-tenant disclosure trade-off exists; the curator sees both questions and both answers in the client’s own corpus). Resolves OQ-120-1 (per-pair curator review, side-by-side, survivor-basis, override — §D), OQ-120-2 (corpus-wide service-role proposer read + operational named-proposer boundary + full-visibility intra-tenant curator review + accepted intra-tenant superseded_by ownership move — §B), and OQ-120-3 (precision-first conservative tunable threshold; index posture deferred to TECH — §E). Builds on {120.1} RESEARCH (Liam-RATIFIED) + the S388 locked inputs. No implementation, no DDL, no ledger writes performed.