ID-120 {120.1} RESEARCH — Cross-workspace Q&A dedup (UC8 cross-workspace v1 + Spike#10 Stage-5 substrate)
ID-120 {120.1} RESEARCH — Cross-workspace Q&A dedup (UC8 cross-workspace v1 + Spike#10 Stage-5 substrate)
Section titled “ID-120 {120.1} RESEARCH — Cross-workspace Q&A dedup (UC8 cross-workspace v1 + Spike#10 Stage-5 substrate)”Status: RESEARCH complete — recommendations frame v1 scope; the PRODUCT decisions (curator-review UX, op_id-scope exception boundary, proposer threshold) are left OPEN for
{120.2}. RESEARCH-only: no implementation, no DDL, no code edits, no ledger writes.Task: ID-120 — “Cross-workspace Q&A dedup (UC8 cross-workspace v1 + Spike#10 Stage-5 substrate)”. Carries the recovered S388 grounding (Liam Decision A) as locked inputs — this doc does NOT re-derive them; it records them, verifies the four S388 caveats against the live code/schema, and frames the open questions for
{120.2}PRODUCT.Spec-chain link 1 of
{120.1 RESEARCH → 120.2 PRODUCT → 120.3 TECH → 120.4 PLAN}.Grounding sources (locked, do not re-open): S388 Decision A (cross-workspace v1, override of the S229 4.7.Q8 default); Spike#10 (Stage-5 substrate, options 1+4 collapse); the already-shipped cocoindex Stage-5 (ID-53) post-pass; the live
q_a_pairs/q_a_pair_historyschema (supabase/migrations/20260617130000_squash_baseline.sql).
0. CORRECTION (S391) — workspace ≠ tenant; this dedup is INTRA-tenant
Section titled “0. CORRECTION (S391) — workspace ≠ tenant; this dedup is INTRA-tenant”Authoritative correction (Liam-directed, S391). This RESEARCH was authored on a STALE equation
workspace = client/tenant. Under the current (post-T2 migration, S246/S247) model that equation is FALSE. Deployment is ONE Supabase database PER CLIENT, so the DATABASE itself is the tenant boundary. A single client holds MULTIPLE workspaces (one perapplication_typethey use). “Cross-workspace” dedup WITHIN one client’s database is therefore INTRA-TENANT (structurally also cross-application and cross-FORM) — NOT cross-tenant. The body below is preserved as a historical record; read it through this correction. Only the tenancy framing was wrong (plus a missing FORM axis); all file:line evidence, the proposer→approve→reuse-existing-write mechanism, the precision-first threshold direction, the brute-force-vs-HNSW posture, and theq_a_pair_historyprovenance stand unchanged.
Canonical definitions (condensed; verified vs schema + ontology)
Section titled “Canonical definitions (condensed; verified vs schema + ontology)”- tenant / client = one Supabase project (database) per client. The DATABASE is the tenant
boundary and the ONLY cross-tenant boundary. There is no
tenant_idcolumn — the DB is the tenant. (state-of-the-product.md: “one project per client”.) - application / application_type = a use-case CLASS, realised as the reference TABLE
application_types(6 keys: procurement, intelligence, sales_proposal, product_guide, competitor_research, training_onboarding). NOT a container; there is NOapplicationstable. It REPLACED the legacyworkspaces.typeenum viaworkspaces.application_type_idFK (T2 migration, S246/S247) and sits the layer ABOVE workspace. - workspace = a container WITHIN one client’s database, bound N:1 to exactly one
application_type(workspaces.application_type_id). A single client has MULTIPLE workspaces (one per application_type in use).workspaceis NOT synonymous withclient/tenant— that equation is the pre-S246 legacy reading and is now INCORRECT. - form = a submission/questionnaire (
form_templates/form_questions/form_responses), workspace-scoped viaworkspace_id. One workspace hosts MANY forms (form_type: bid/rfp/pqq/itt/tender/questionnaire/…). “bid” is now a form_type, not an application. - q_a_pair = the corpus-level reusable knowledge unit; NOT workspace-partitioned.
source_workspace_id/source_form_response_id/source_question_idare NULLABLE provenance/lineage only. Workspace relevance is computed at query time viascope_tagoverlap, not a scoping FK. It is ONE CLIENT’s corpus, shared across that client’s applications/workspaces/forms.
Resolved decisions (Liam, S391 — both CLOSED)
Section titled “Resolved decisions (Liam, S391 — both CLOSED)”- DECISION A — dedup axis = INTRA-TENANT. The proposer reads the whole single-client published
q_a_pairscorpus and proposes near-duplicate questions REGARDLESS of which workspace OR form they originated from. “Cross-workspace” means “across the one client’s workspaces” (intra-tenant); it is structurally also cross-application and cross-FORM. The PRIMARY real driver: the same question answered across DIFFERENT FORMS of the SAME application (e.g. the same procurement question in a PQQ and an ITT). The dedup key is the QUESTION, corpus-wide; lineage columns are preserved on the survivor. The FORM axis was missing throughout the body and is now in scope — anchor on theq_a_pairslineage columns (source_form_response_id,source_question_id). - DECISION B — cross-tenant dedup is OUT OF SCOPE, NEVER-V1. No v1 (or foreseeable) requirement to dedup across separate client deployments or to maintain a shared platform-level Q&A library across client DBs. The DB-per-client boundary IS the tenant boundary.
Key reconciliation — ID-120 does NOT contradict UC8’s “Never v1 cross-workspace”
Section titled “Key reconciliation — ID-120 does NOT contradict UC8’s “Never v1 cross-workspace””Once terms are aligned, ID-120 does not contradict the UC8-origin “cross-workspace merge: Never
v1”. UC8’s “never cross-workspace” referred to cross-TENANT in its era — when a client ≈ one
workspace (the pre-S246 reading). ID-120’s DECISION-A “cross-workspace” means INTRA-tenant (within
one client DB, across that client’s workspaces and forms). They are different boundaries; the §1 / §2.2
framing of DECISION-A as an “override of the S229 4.7.Q8 / UC8 default” is, post-S391, a terminology
mismatch rather than a genuine reversal. There is no cross-tenant disclosure trade-off inside one
deployment, so the curator reviewing the CLIENT’S OWN corpus sees BOTH questions AND both answers
side-by-side; RLS USING(true) on q_a_pairs is CORRECT precisely because the DB is one client.
1. Summary
Section titled “1. Summary”ID-120 builds v1 cross-workspace Q&A deduplication: a walk-time pass that proposes that
two q_a_pairs rows (potentially in different workspaces) are near-duplicate questions, surfacing
the proposal for curator approval — at which point the existing supersession primitive fires the
merge write (publication_status='archived' + superseded_by=<survivor>). It is a PROPOSER, not
an auto-merge: the destructive write only fires on app-side curator approval.
Four locked decisions shape the v1 direction:
-
Cross-workspace, by deliberate override (S388 Decision A). UC8 v1 dedup reads ACROSS workspaces, overriding the S229 4.7.Q8 default (“cross-workspace: never v1”) and the workspace-scoped sources (id-59 PRODUCT, id-69 PRODUCT, the
content_items.superseded_byschema comment, which explicitly scopes “cross-workspace semantics … out of scope”). The id-59 PRODUCT INV-8-UC8 / PC-20 — previously the cross-workspace outlier — becomes the chosen direction. -
Substrate = the shipped cocoindex Stage-5 post-pass (ID-53), repointed
entity_mentions → q_a_pairs. Stage-5 is already a Postgres/pgvector post-pass running beside cocoindex (not inside it). The pgvector cosine surface overq_a_pairs.question_embeddingalready exists (q_a_search); the supersession write (retireSupersededPairs) already writes thesuperseded_by/archived lineage. One runtime, zero new pending-spike deps. Spike#10 options 1 (cocoindex-post-write) and 4 (Postgres pass) collapse into one because Stage-5 is a Postgres post-pass attached after the cocoindex walk. -
mempalace-KG REJECTED; hybrid DEFERRED to v1.1. mempalace has no similarity surface over
q_a_pairs, and its lineage-strength role is already covered byq_a_pair_history. A hybrid (lexical + vector + KG) approach defers to v1.1. -
Shape = PROPOSER. The walk-time pass proposes; the merge write fires only on curator approval (app-side), reusing the existing primitive — no auto-merge.
This RESEARCH verifies the four S388 caveats against the live code/schema (§4 — all four CONFIRMED, one with a material correction to how “workspace scoping” is enforced) and frames the open questions for PRODUCT (§5).
2. Problem / Context
Section titled “2. Problem / Context”2.1 Code-intelligence orientation (verbatim)
Section titled “2.1 Code-intelligence orientation (verbatim)”GitNexus MCP tools were not available in this dispatch context (no gitnexus_* tool surface, no
gitnexus CLI on PATH). Per the brief’s mandated fallback chain (gitnexus → ccc → grep), and because
the substrate is the Python pipeline (scripts/cocoindex_pipeline/stage_5.py) plus SQL
migrations — neither of which is in the TypeScript graph — the verification was done by direct
grep over the migrations + Python pipeline + lib/q-a-pairs/, plus targeted Read. The GitNexus
hook (which fires on Bash) returned one verbatim relationship fact, cited below.
Verbatim GitNexus hook output (fired on the retireSupersededPairs grep):
[GitNexus] 1 related symbols found:retireSupersededPairs (lib/q-a-pairs/promote-corpus.ts) Called by: promoteCorpusExtractions Calls: safeErrorMessageVerbatim grep evidence anchoring each mandated symbol (file:line, used throughout §4):
stage_5.py—async def _run_stage_5_resolution@scripts/cocoindex_pipeline/stage_5.py:204; attached imperatively atscripts/cocoindex_pipeline/flow.py:3707(afterawait handle.ready()@flow.py:3597).q_a_pairs—CREATE TABLE … "public"."q_a_pairs"@squash_baseline.sql:7138;"question_embedding" "extensions"."vector"(1024)@:7144;"superseded_by" "uuid"@:7148.retireSupersededPairs— def @lib/q-a-pairs/promote-corpus.ts:827; archive UPDATE @:1018–1023(publication_status:'archived',superseded_by:replacementPairId).promoteCorpusExtractions—export async function@lib/q-a-pairs/promote-corpus.ts:235; callsretireSupersededPairs(client)@:513; HTTP entryapp/api/q-a-pairs/promote-corpus/route.ts:56.question_embedding— declared @squash_baseline.sql:7144; cosine<=>use inq_a_search@:4282; no index DDL referencing it anywhere (§4(d)).
2.2 Locked inputs (S388 — recorded, not re-derived)
Section titled “2.2 Locked inputs (S388 — recorded, not re-derived)”| # | Locked input | Source |
|---|---|---|
| DECISION-A | UC8 v1 dedup is CROSS-WORKSPACE, overriding the S229 4.7.Q8 “cross-workspace: never v1” default and the workspace-scoped sources. id-59 PRODUCT INV-8-UC8 / PC-20 becomes the chosen direction. [S391: “cross-workspace” = intra-tenant, across the client’s workspaces & forms — see §0; not an override of cross-TENANT, which stays Never-v1.] | Liam, S388 Decision A |
| SUBSTRATE | The shipped cocoindex Stage-5 (ID-53) Postgres/pgvector pass, repointed entity_mentions → q_a_pairs (scripts/cocoindex_pipeline/stage_5.py). Spike#10 options 1+4 collapse. pgvector cosine over q_a_pairs.question_embedding already exists; retireSupersededPairs already writes the superseded_by/archived lineage. mempalace-KG REJECTED (no similarity surface; lineage covered by q_a_pair_history). Hybrid → defer v1.1. | Spike#10 |
| SHAPE | Walk-time pass = PROPOSER; merge write fires only on curator approval (app-side), reusing the existing primitive (no auto-merge). | S388 |
| RESOLVED-OQ | ID-120 subsumes UC8 ratification entirely — no separate UC8 ratification gate to depend on. | S388 (folded) |
2.3 What is genuinely new vs. what is reuse
Section titled “2.3 What is genuinely new vs. what is reuse”The dedup write is fully reuse — retireSupersededPairs already archives a superseded pair and
sets superseded_by. The dedup substrate is fully reuse — Stage-5 is already a post-pass that
reads a population of rows, runs a pgvector/embedding similarity computation, and issues scoped writes.
What is new in ID-120 is narrow and well-bounded:
- The candidate population changes from
entity_mentions(per-run, op_id-scoped) toq_a_pairs(the published-pair population, read across workspaces). - The similarity surface changes from entity-name resolution to question-embedding cosine
over
q_a_pairs.question_embedding(the surfaceq_a_searchalready exposes). - The action changes from auto-UPDATE-canonical to propose-then-approve: the pass writes a proposal, the merge write fires only on curator approval (SHAPE).
Everything else — the post-pass attach pattern, the asyncpg pool, the op_id-scope discipline, the supersession write — already exists.
3. Substrate analysis — Stage-5 repoint + Spike#10 collapse
Section titled “3. Substrate analysis — Stage-5 repoint + Spike#10 collapse”3.1 Stage-5 as it ships today (the substrate to repoint)
Section titled “3.1 Stage-5 as it ships today (the substrate to repoint)”Stage-5 (scripts/cocoindex_pipeline/stage_5.py) is an entity-resolution post-pass. Its current
shape (verbatim from the module + the attach):
- Reads the run’s
entity_mentionsrows, op_id-scoped (_select_run_entity_mentions,stage_5.py:96–121). - Computes cross-document canonicalisation via cocoindex
resolve_entities+ the KH entity embedder + KH PairResolver (stage_5.py:204–342). - Writes
UPDATE … SET canonical_name+ collision-collapseDELETEs, op_id-scoped (stage_5.py:462–514). - Attaches imperatively at
flow.py:3707, afterawait handle.ready()(flow.py:3597) and before the flow-end webhook — i.e. a Postgres post-pass running BESIDE the cocoindex walk, not inside a per-file@coco.fncomponent (caveat (a), confirmed §4).
The repoint is conceptually a drop-in of the population + similarity surface + action, keeping the
post-pass scaffold: read a population of q_a_pairs rows (cross-workspace), compute question-embedding
cosine similarity, and propose merges instead of issuing canonical UPDATEs.
3.2 Why Spike#10 options 1 + 4 collapse into one
Section titled “3.2 Why Spike#10 options 1 + 4 collapse into one”- Option 1 (cocoindex-post-write) = “do the dedup as a write that happens after the cocoindex per-item phase settles.”
- Option 4 (Postgres pass) = “do the dedup as a SQL/pgvector pass over the q_a_pairs table.”
Stage-5 is both at once: it is an imperative await call (flow.py:3707) that runs after the
cocoindex walk (handle.ready()), and its body is a Postgres/asyncpg pass issuing raw SQL against
the pool (coco.use_context(DB_CTX)). There is no daylight between “cocoindex-post-write” and
“Postgres pass” once the substrate is the Stage-5 attach — they are the same runtime. One pass, one
runtime, zero new pending-spike deps.
3.3 Why mempalace-KG was rejected
Section titled “3.3 Why mempalace-KG was rejected”- mempalace exposes no similarity surface over
q_a_pairs— there is no embedding index or cosine query path from the KG into the Q&A corpus, so it cannot serve as the proposer’s candidate generator. - Its lineage-strength contribution (which pair supersedes which, with provenance) is already
covered by
q_a_pair_history— the append-only mirror that snapshots everysuperseded_bytransition (squash_baseline.sql:7091comment: “append-only snapshot mirror, lineage preserved by value”). Adding a second lineage store would duplicate, not extend.
3.4 Why hybrid defers to v1.1
Section titled “3.4 Why hybrid defers to v1.1”v1 uses the single similarity surface that already ships: question-embedding cosine. A hybrid (lexical full-text + vector + KG-signal) proposer is a precision/recall refinement, not a substrate change — it can layer on after v1 proves the propose-then-approve loop. Deferring it keeps v1 to “one new candidate population + one existing similarity surface + one existing write primitive.”
4. Caveat verification (the four S388 caveats, each against live code/schema)
Section titled “4. Caveat verification (the four S388 caveats, each against live code/schema)”Caveat (a) — Stage-5 is a Postgres/pgvector post-pass BESIDE cocoindex, NOT inside an @coco.fn
Section titled “Caveat (a) — Stage-5 is a Postgres/pgvector post-pass BESIDE cocoindex, NOT inside an @coco.fn”Verdict: CONFIRMED.
Stage-5 is attached imperatively at scripts/cocoindex_pipeline/flow.py:3707:
resolved_count = await _run_stage_5_resolution( meta=FlowRunMeta(op_id=run_op_id, content_items_id=None), db_pool=coco.use_context(DB_CTX), flow_stage_counter=flow_stage_counter,)This call sits after await handle.ready() (flow.py:3597) and before the flow-end webhook
(flow.py:3820). The module header states it directly (stage_5.py:1–11): “Hosts the flow-scope
cross-document canonicalisation phase that runs AFTER the per-item mount_each fan-out has settled …
this post-pass reads the run’s entity_mentions rows … and issues … UPDATEs.” The body issues raw
asyncpg SQL against the pool (stage_5.py:90, 116, 144, 187, 483, 504) — it is not a per-file
@coco.fn component. Resolution direction: the ID-120 dedup pass adopts the same attach pattern
(imperative await post-handle.ready(), asyncpg pool), so cross-record reads across q_a_pairs are
structurally available — exactly the thing you cannot do inside a per-file @coco.fn (which sees one
record at a time). The caveat’s underlying warning (“you cannot query across records inside a per-file
component”) is the reason the post-pass placement is required, and it already exists.
Caveat (b) — the cross-workspace read is a deliberate op_id-scope exception
Section titled “Caveat (b) — the cross-workspace read is a deliberate op_id-scope exception”Verdict: CONFIRMED, with a material correction to where workspace scoping actually lives.
Two scoping layers are at play, and the precision matters for PRODUCT:
-
op_id-scope (the pipeline-run scope). Stage-5’s writes are strictly op_id-scoped (Inv-5): it reads only the in-flight run’s rows (
_select_run_entity_mentions … WHERE op_id = $1,stage_5.py:116–121) and writes only op_id-matching rows (… AND op_id = $2,stage_5.py:493–507). The module already documents one deliberate op_id-scope exception — the ID-80.14 cross-op collision DELETE (stage_5.py:22–34, “the ONE deliberate exception to the otherwise op_id-scoped writes”; widened-predicate DELETEWHERE … op_id IS DISTINCT FROM $2@stage_5.py:483–489). ID-120’s cross-workspace read is a second, broader deliberate widening of this same scope discipline — the dedup proposer must read the whole publishedq_a_pairspopulation, not just the current op’s rows. -
workspace-scope (the tenancy scope) — CORRECTION. The S388 caveat frames this as widening a “normally enforced” op_id/workspace scope on
q_a_pairsreads. The live evidence corrects the mechanism: workspace scoping onq_a_pairsreads is NOT enforced at the RLS or RPC layer —- The RLS SELECT policy is
q_a_pairs_select … FOR SELECT USING (true)(squash_baseline.sql:10904) — RLS permits all authenticated reads, no per-workspace row filter. - The canonical similarity RPC
q_a_searchhas no workspace filter:FROM public.q_a_pairs qap WHERE qap.question_embedding IS NOT NULL AND qap.publication_status = 'published'(squash_baseline.sql:4298–4300); its comment states “Scope filtering is caller-side” (:4322). source_workspace_idexists as a column (squash_baseline.sql:7144, FK→workspacesON DELETE SET NULL @:9884) and is the value callers filter on, but the filter is applied in application code, not the DB read path.promote-corpus.tsis “RLS-scoped throughout” by role (INV-14/15, file header:17–20), not by workspace row-filter.
So the “deliberate exception” is precise: ID-120 widens the application-side
source_workspace_idfilter that callers normally apply (each workspace sees its own pairs) to read across all workspaces for candidate generation. The DB does not block this today (RLS isUSING(true)), which makes this an operational-boundary question (which automated proposer/role runs the read), not a schema constraint — see OQ-120-2. [S391 correction: this is INTRA-tenant — the read is across the one client’s workspaces (and forms), all inside one client DB; the DB-per-client boundary IS the tenant boundary, so RLSUSING(true)is CORRECT and there is no tenant wall for this read to breach. See §0.] Resolution direction: PRODUCT must define the cross-workspace read boundary (which automated proposer/caller, which role) as an OPERATIONAL boundary — across the client’s workspaces (intra-tenant) — since the DB-per-client boundary is the tenant boundary and the DB does not (and need not) row-filter by workspace here. - The RLS SELECT policy is
Caveat (c) — the LIVE q_a_pairs.superseded_by FK is ON DELETE SET NULL; the history mirror is FK-less
Section titled “Caveat (c) — the LIVE q_a_pairs.superseded_by FK is ON DELETE SET NULL; the history mirror is FK-less”Verdict: CONFIRMED (the earlier “no FK” discovery was wrong — corrected here).
- Live table:
ADD CONSTRAINT "q_a_pairs_superseded_by_fkey" FOREIGN KEY ("superseded_by") REFERENCES "public"."q_a_pairs"("id") ON DELETE SET NULL(squash_baseline.sql:9889). The FK exists and isON DELETE SET NULL. (There is also a btree index on it,idx_q_a_pairs_superseded_by,20260619120100_index_unindexed_fks.sql:32.) - History mirror:
q_a_pair_history.superseded_byis FK-less by design — comment: “Snapshot of q_a_pairs.superseded_by at transition (plain uuid, no FK — append-only snapshot mirror, lineage preserved by value). ID-64.15.” (squash_baseline.sql:7091).
Resolution direction: the supersession write the proposer eventually triggers
(superseded_by=<survivor>) is FK-validated on the live table (the survivor must exist), and the
ON DELETE SET NULL semantics + the FK-less history mirror together mean lineage survives even if a
survivor row is later deleted — no schema change needed for the write path. The cross-workspace case
introduces a wrinkle PRODUCT should note: the FK targets q_a_pairs(id) with no workspace constraint,
so a cross-workspace superseded_by link is already structurally permitted by the schema.
Caveat (d) — NO HNSW index on q_a_pairs.question_embedding (brute-force, like Stage-5)
Section titled “Caveat (d) — NO HNSW index on q_a_pairs.question_embedding (brute-force, like Stage-5)”Verdict: CONFIRMED.
question_embeddingisextensions.vector(1024)(squash_baseline.sql:7144) and is used in cosine<=>similarity inq_a_search(:4282,:4338).- There is no
USING hnswand noUSING ivfflatindex anywhere in the migrations (grep over allsupabase/migrations/*.sqlreturned zero rows), and no index named for or referencingquestion_embedding(the only indexes touchingq_a_pairsare gin/btree onscope_tag,anti_scope_tag,source_workspace_id,superseded_by, etc. — none on the embedding). So cosine similarity overquestion_embeddingis a brute-force sequential scan today — exactly the posture Stage-5’s pgvector pass already operates under.
Resolution direction (frame as OPEN, do not decide): v1 has two options —
(A) brute-force, consistent with Stage-5’s current pgvector posture and acceptable at current corpus
size; or (B) add one migration creating an HNSW (or ivfflat) index on question_embedding to bound
the cross-workspace scan cost as the published-pair population grows. This is a TECH/PRODUCT trade-off
(corpus-size threshold vs. one migration); OQ-120-3 carries it — this RESEARCH does not pick.
5. Open questions for {120.2} PRODUCT
Section titled “5. Open questions for {120.2} PRODUCT”-
OQ-120-1 (curator-review UX for cross-workspace merges): the SHAPE is propose-then-approve, but the review surface is undefined. What does a curator see when the proposer surfaces a cross-workspace near-duplicate — both questions side-by-side? which is the proposed survivor and on what basis (recency, publication_status, confidence)? Is approval per-pair or batched? The merge write (
retireSupersededPairs-style archive +superseded_by) is reused, so this is purely the approval UX + the survivor-selection policy PRODUCT must define. -
OQ-120-2 (op_id-scope exception boundary; intra-tenant read): the cross-workspace read is a deliberate widening (§4(b)). [S391: this is INTRA-tenant. The cross-tenant question-text-disclosure sub-question (formerly OQ-120-2a) COLLAPSES — RESOLVED/CLOSED: the curator reviews the CLIENT’S OWN corpus and sees BOTH questions AND both answers side-by-side; there is no cross-tenant disclosure trade-off because the DB-per-client boundary is the tenant boundary, and RLS
USING(true)is correct precisely because the DB is one client. See §0.] Because the DB does not row-filterq_a_pairsreads by workspace (RLSUSING(true),q_a_searchscope-filtering is caller-side), what remains for PRODUCT is an OPERATIONAL boundary: which automated proposer/caller/role runs the read across the client’s workspaces (intra-tenant); what provenance the proposal records (including the FORM lineage —source_form_response_id,source_question_id— since the dedup spans forms as well as workspaces); and confirming that a cross-workspacesuperseded_bylink (already structurally permitted by the FK, §4(c)) simply moves the survivor’s canonical ownership BETWEEN the client’s workspaces — an informative, intra-tenant event, not a cross-tenant merge. This is an operational/UX decision, not a tenancy wall the schema must police: the DB-per-client boundary is the tenant boundary. -
OQ-120-3 (proposer precision/recall threshold + index posture): what cosine-similarity threshold defines a “near-duplicate” candidate (precision vs. recall trade-off for what a curator is willing to review)? Coupled to this is the index posture from §4(d): brute-force (v1, Stage-5-consistent) vs. add-one-HNSW-migration (bounds scan cost as the corpus grows). PRODUCT/TECH decide the threshold; TECH decides the index, informed by the expected cross-workspace population size.
-
OQ — relationship to UC8 ratification— RESOLVED / CLOSED. ID-120 subsumes UC8 ratification entirely (S388, folded into scope). There is no separate UC8 ratification gate to depend on or sequence against. Recorded here as closed so PRODUCT does not re-open it.
6. Recommendation (frame v1 scope; PRODUCT decisions left OPEN)
Section titled “6. Recommendation (frame v1 scope; PRODUCT decisions left OPEN)”-
v1 = cross-workspace question-embedding-cosine PROPOSER on the Stage-5 substrate. Repoint the shipped Stage-5 post-pass pattern (
flow.py:3707attach, asyncpg pool, post-handle.ready()) fromentity_mentionsto a cross-workspaceq_a_pairsread; compute cosine overquestion_embedding(theq_a_searchsurface); propose near-duplicates; fire the merge write (reuseretireSupersededPairs-style archive +superseded_by) only on curator approval. -
No new write primitive. The supersession write already exists and is FK-validated (
q_a_pairs.superseded_by ON DELETE SET NULL, §4(c)); lineage is preserved by the FK-lessq_a_pair_historymirror. The action is propose → approve → reuse-existing-write. -
Cross-workspace read is a deliberate, DB-unenforced widening. It extends the application-side
source_workspace_idfilter (not RLS, which isUSING(true)) — so the read boundary is an OPERATIONAL/PRODUCT decision (which automated proposer/role reads across the client’s workspaces & forms), not a schema constraint (OQ-120-2). [S391: this is INTRA-tenant — within one client DB; the DB-per-client boundary is the tenant boundary, soUSING(true)is correct. See §0.] -
Index posture is deferred to TECH (OQ-120-3). v1 may ship brute-force (Stage-5-consistent) or add one HNSW migration; do not decide in RESEARCH.
-
Right-sizing: PRODUCT + TECH (+ likely PLAN). Author {120.2} PRODUCT (proposer behaviour + curator-review UX + the intra-tenant cross-workspace/cross-form read boundary + survivor-selection policy), then {120.3} TECH (the Stage-5 repoint: candidate query across workspaces, the cosine threshold, the index decision, the propose-record shape, the approval→write wiring reusing the existing primitive). A {120.4} PLAN is likely warranted (the cross-workspace read, the proposal-record substrate, the curator-review surface, and the approval→merge wiring are chain-dependent slices) — confirm at TECH time; well under the 25-subtask ceiling.
-
Empirical-verification note (Q-EX2 / OQ-3): no external-library API claim drives this RESEARCH — the substrate is internal (Stage-5,
retireSupersededPairs) plus the in-repo pgvector<=>operator andvector(1024)type (both stdlib-to-this-repo / extension built-ins, already in live migrations). The pgvector cosine surface is verified present and exercised in shipped SQL (q_a_search,squash_baseline.sql:4282), not from prose. No external import-and-call check was required; noABSENT/SIGNATURE_DRIFTto report.
7. Sources cited
Section titled “7. Sources cited”| Source | Used for |
|---|---|
scripts/cocoindex_pipeline/stage_5.py (:1–56 header, :96–121 op_id read, :204–342 resolve, :462–514 scoped writes, :22–34 + :483–489 the documented op_id-scope exception) | §3.1, §4(a), §4(b) — the substrate to repoint + op_id-scope discipline |
scripts/cocoindex_pipeline/flow.py (:181 import, :3597 handle.ready(), :3707 attach, :3820 webhook) | §3.1, §3.2, §4(a) — the post-pass attach point (beside cocoindex) |
lib/q-a-pairs/promote-corpus.ts (:17–20 RLS-by-role header, :235 promoteCorpusExtractions, :513 calls retireSupersededPairs, :827 def, :1010–1023 archive UPDATE) | §2.3, §3.3, §4(c), §6 — the existing supersession-write primitive (reuse) |
app/api/q-a-pairs/promote-corpus/route.ts (:37 import, :56 call) | §2.1 — HTTP entry to the promote/retire path |
supabase/migrations/20260617130000_squash_baseline.sql (:7138 q_a_pairs CREATE, :7144 question_embedding vector(1024), :7148 superseded_by, :9889 q_a_pairs_superseded_by_fkey ON DELETE SET NULL, :9884 source_workspace_id FK, :7091 history mirror FK-less comment, :10904 RLS SELECT USING(true), :4268–4316 q_a_search body, :4298–4300 no-workspace-filter, :4322 “scope filtering is caller-side”, :4282/:4338 cosine <=>) | §4(b), §4(c), §4(d) — live schema + RLS + RPC evidence |
supabase/migrations/20260619120100_index_unindexed_fks.sql (:30 source_workspace_id idx, :32 superseded_by idx) | §4(c), §4(d) — index inventory (no embedding index) |
supabase/migrations/20260619120000_rls_initplan_wrap_qa.sql (:39–46 q_a_pairs RLS policies) | §4(b) — RLS is role-based, not workspace-row-scoped |
grep over supabase/migrations/*.sql for USING hnsw / USING ivfflat / INDEX … question_embedding (zero rows) | §4(d) — confirms brute-force (no vector index) |
GitNexus Bash hook (verbatim: retireSupersededPairs called-by promoteCorpusExtractions) | §2.1 — code-intel orientation (gitnexus MCP unavailable; hook + grep fallback) |
S388 Decision A; Spike#10; id-59 PRODUCT INV-8-UC8 / PC-20; S229 4.7.Q8; content_items.superseded_by schema comment (squash_baseline.sql:359) | §1, §2.2 — locked grounding inputs |
End of RESEARCH. Records the S388 locked inputs; verifies all four caveats against live code/schema (all CONFIRMED, with a material correction on where workspace scoping is enforced — §4(b)); frames OQ-120-1/2/3 for PRODUCT (UC8-ratification OQ is CLOSED). No implementation, no DDL, no ledger edits performed.