RESEARCH -- Bound the corpus-promotion batch: unparameterised POST, no terminal state for unpromotable extractions (ID-363.1)

RESEARCH — Bound the corpus-promotion batch (ID-363.1)

Task: id-363 — Bound the corpus-promotion batch + give unpromotable extractions a terminal state. Subtask: {363.1} RESEARCH. Author date: 25/07/2026. Provenance: surfaced by the ID-127 {127.38} verification checker while auditing staging state. Filed as a pre-existing defect, not a {127.38} regression. Status: DECISION-SUPPORT. All numbers below are live reads against Platform staging (rbwqewalexrzgxtvcqrh) taken on 25/07/2026. The data is synthetic and the app is not live — this is a pre-launch correctness/cost question, not an incident.

---

0. Headline — the reported premise is false, and what is actually there is worse

The finding as escalated was: “promoteCorpusExtractions takes no parameters, so one POST mints 135 published pairs + 135 OpenAI embeddings in a single call.”

Against current staging data that is not what happens. All 135 unpromoted extractions have extracted_answer_text IS NULL, so every one of them is rejected at step 2 of the loop (INV-7, no_answer_text) before any INSERT, any CAS, and any embedding call. A POST today returns:

Field	Value
considered	140
skipped	135 (all no_answer_text)
proposed	5 (DR-026 awaiting_review diffs)
promoted	0
embed_failed	0
OpenAI calls issued	0
Marginal cost	$0.00

The unbounded-batch defect is real in the code and latent in the data. It fires the moment the walk starts emitting extractions that carry answer text.

Three separate defects fall out of the same read. Two of them are worse than the one that was reported:

• D1 (reported, latent): the batch is unbounded — no limit, no cursor, no server-side concurrency guard. Wall-clock, not spend, is the binding constraint (§3).
• D2 (found, live): the 135-row backlog is not a queue awaiting a click. It is a permanent, self-refreshing skip set — the eligibility RPC re-selects all 135 on every run, forever, because a skipped extraction has no terminal state (§4). skipped[] carries 135 records on every invocation.
• D3 (found, live): 10 of the 25 published q_a_pairs have no record_embeddings row and zero of them are re-selected by the RPC. The documented “self-healing” property does not hold once the linking extraction is gone. 40% of the published corpus is invisible to hybrid_search’s q_a_pair arm and there is no path that fixes it (§5).

D2 and D3 are the reason to do this task. D1 is the reason to do it before the answer-text defect upstream is fixed, not after.

---

1. The call path

1.1 Surface

POST /api/q-a-pairs/promote-corpus — app/api/q-a-pairs/promote-corpus/route.ts

UI: components/governance/promotion-gate/promotion-candidates-panel.tsx:122
      "Run promotion pass" button -> runMutation.mutate()
  -> lib/query/fetchers.ts  postQaPromoteCorpus()   [POST, no body]
  -> app/api/q-a-pairs/promote-corpus/route.ts      [maxDuration = 120]
  -> lib/q-a-pairs/promote-corpus.ts  promoteCorpusExtractions(auth.supabase)
       step 1: rpc('q_a_extractions_promotion_candidates')   -- 140 rows
       per row: INSERT q_a_pairs -> CAS UPDATE q_a_extractions
                -> UPDATE q_a_pairs.source_document_id (link-then-publish)
                -> generateEmbedding()  [OpenAI]
                -> UPDATE q_a_pairs.publication_status='published'
                -> UPSERT record_embeddings
       step 6: retireSupersededPairs()  -- loop-until-dry, cap 10 iterations

1.2 Who can call it

• Not public. The path is absent from proxy.ts publicRoutes (['/login','/auth/callback','/oauth/consent']); unauthenticated callers are redirected by the middleware. Asserted by __tests__/app/api/q-a-pairs/promote-corpus/route.test.ts:288-296.
• Role gate: getAuthorisedClient(['admin','editor']); viewer -> 403 via authFailureResponse. So any editor, not admin-only.
• RLS-scoped: the handler passes auth.supabase (the cookie-based client), never a service-role client. INV-15 holds.
• Not rate-limited. promote-corpus does not appear in the set of routes that reference rateLimit (app/api/embed, app/api/search/*, app/api/ingest/url, … do). There is no server-side guard against repeat or concurrent invocation.

1.3 Callers

Grep for promoteCorpusExtractions across .ts/.py returns exactly one production call site: the route handler. The module header and route header both describe a “Caller B — ID-45 pipeline (service-role client)”; that caller does not exist in the tree. __tests__/.../route.test.ts:234 proves only the shape is assignable (“INV-3 second-caller shape proof”), not that anything calls it. Treat the second-caller claim in the comments as aspirational.

Practical consequence: the only trigger is a human clicking “Run promotion pass” in the governance promotion-gate panel. That is the human gate (BI-39).

---

2. What exists today: limits, timeouts, idempotency

Control	Present?	Detail
Batch-size limit	No	promoteCorpusExtractions(client) takes only the client. The RPC has no LIMIT. The route parses no body — defineRoute validates the response only.
Cursor / pagination	No	The RPC returns SETOF q_a_extractions ordered by created_at, whole set.
Timeout	Partial	export const maxDuration = 120 (Vercel Pro ceiling). No internal deadline check, no early return.
Idempotency key	No	Nothing distinguishes a retry from a new request.
Server-side concurrency guard	No	No advisory lock, no run row, no status flag.
Client-side guard	Weak	disabled={runMutation.isPending} (panel:123) — per-tab, per-session only. Two operators, or one operator in two tabs, both fire.
Re-run safety	Yes	The CAS (UPDATE ... WHERE promoted_to_pair_id IS NULL) plus the uq_q_a_extractions_promoted_to_pair_id partial unique index make duplicate promotion impossible. The loser deletes its orphan pair and counts already_promoted.
Partial-failure recovery	Partial	Failures leave the pair draft; the RPC re-selects it. But see §5 — this only holds while the extraction survives.

Read of the concurrency picture. Correctness under concurrent POSTs is fine — the CAS is the real guard and it works. What is not guarded is waste: two concurrent runs both fetch the same N candidates and both pay for N embedding calls; only one wins each CAS. The retirement pass also runs twice, each doing a full scan of invalidated+published rows.

Read of the timeout picture. On a 120s overrun Vercel kills the function. Nothing is transactional, so committed rows stay committed — re-running converges. What is lost is the PromotionSummary: the client gets a 504 and the operator never sees failures[], proposals[], or any count. There is no server-side persistence of a run — no pipeline_runs row, no log record. A timed-out run is invisible after the fact.

---

3. What a full run actually costs

3.1 Spend — negligible, and not the constraint

Measured from staging:

135 unlinked candidates, SUM(length(extracted_question_text)) = 34,959 chars
  ~= 8,740 tokens at the ~4 chars/token English ratio used by
     lib/ai/embed.ts's own MAX_EMBEDDING_CHARS reasoning

text-embedding-3-large at $0.13/1M input tokens -> **$0.001** for the whole 135-row backlog. Even a 10,000-row corpus lands around $0.08.

Only the question text is embedded (embedAndPublish(..., extraction .extracted_question_text, ...)), never the answer, so input volume stays small. lib/ai/embed.ts also holds a 500-entry / 1-hour in-process cache, so repeat runs within an hour on the same warm lambda re-issue nothing.

Conclusion: cost is not a reason to bound this batch. Any framing of this as a spend risk is wrong by ~3 orders of magnitude. Do not size the fix around spend.

3.2 Wall-clock — this is the constraint

Every step is sequentially awaited; there is no batching and no concurrency. Per newly-promoted row:

Step	Round-trip
INSERT q_a_pairs ... .single()	Supabase REST
CAS UPDATE q_a_extractions	Supabase REST
UPDATE q_a_pairs.source_document_id	Supabase REST
generateEmbedding()	OpenAI
UPDATE q_a_pairs.publication_status	Supabase REST
UPSERT record_embeddings	Supabase REST

= 5 Supabase round-trips + 1 OpenAI call, strictly serial.

At a conservative 40-60 ms Supabase RTT and 150-400 ms for a short-input text-embedding-3-large call, one row costs roughly 0.35-0.70 s. So:

Promotable rows	Projected wall-clock	vs maxDuration = 120
50	18-35 s	comfortable
135	47-95 s	thin margin
200	70-140 s	exceeds at the upper band
350+	120-245 s	deterministic timeout

Plus the retirement pass afterwards: retireSupersededPairs is loop-until-dry with a 10-iteration cap, each iteration a full scan plus a per-candidate replacement lookup and archive UPDATE.

This is a projection, not a measurement — no full-batch run has ever been executed (there is nothing promotable to run it against, §0). It should be measured before the fix is sized. But the arithmetic is not close enough to be comfortable: the batch is O(n) serial network calls inside a hard 120s slot, with no deadline awareness and no way to resume with the summary intact.

---

4. Is the 135-row backlog expected state, or a symptom?

A symptom — of an upstream defect, and it is permanent.

q_a_extractions: 140 rows total
  135  extracted_answer_text IS NULL      <- the entire "backlog"
    5  extracted_answer_text present      <- all already promoted + published
    0  invalidated_at IS NOT NULL
  created 2026-07-10 00:15 .. 2026-07-13 09:49, across 3 distinct source_documents
  source_document_id: 0 NULL, 0 dangling  -- lineage is intact

So the walk minted 135 extractions with a question and no answer. Their lineage is clean; only the answer text is missing.

promoteCorpusExtractions step 2 skips them:

const answerText: string | null = extraction.extracted_answer_text;
if (!answerText || answerText.trim().length === 0) {
  skipped.push({ extractionId, reason: 'no_answer_text' });
  continue;
}

…and nothing writes anything back to the extraction. No invalidated_at, no skip marker, no counter. The RPC’s branch 1 predicate is e.invalidated_at IS NULL AND e.promoted_to_pair_id IS NULL — still true after the skip. Therefore:

Every future promotion run re-selects the same 135 rows, re-skips them, and returns a 135-element skipped[] array. The set never drains and never shrinks. There is no terminal state for an extraction that cannot be promoted, only for one that has been.

This is D2, and it is the more consequential half of this task. The governance promotion-gate UI reads the same RPC (fetchQaPromotionCandidates), so the operator is shown a permanent 140-row “candidates” list of which 135 can never be actioned from that surface — the per-item accept/edit/reject route (/api/governance/promotion-candidates/[extractionId]/accept, id-145 {145.30}) is scoped to awaiting_review only and returns 409 for new and self_healing kinds.

The upstream defect is out of scope for this task but must be recorded: why does the corpus walk emit q_a_extractions rows with a question and a NULL answer? That is a CocoIndex qa_target question (scripts/), not an app question. See §7 OQ-1 — it likely warrants its own task.

---

5. The self-heal guarantee does not hold (D3)

record_embeddings for owner_kind='q_a_pair': 25 rows, 10 of them orphaned (their owner_id names a q_a_pairs row that no longer exists). Netting out:

q_a_pairs                                          25  (all publication_status='published')
  ...with a resolvable record_embeddings row       15
  ...with NO record_embeddings row                 10
    of which re-selected by the eligibility RPC     0
    of which permanently stranded                  10

Ten published pairs carry no vector. record_embeddings is the sole store since ID-131.19 dropped the inline q_a_pairs.question_embedding column, so those ten are invisible to hybrid_search’s q_a_pair arm — published, listed in the UI, unreachable by search.

Why the RPC does not rescue them. Branch 2 — the documented self-heal path — is:

p.id IS NOT NULL AND re_check.found IS NULL

but the row it selects is an extraction, joined via e.promoted_to_pair_id = p.id. If the extraction is gone (CASCADE-deleted with its source_documents parent during the id-45 deletion waves) or invalidated, there is no row to return and the pair is orphaned from the machinery that would re-embed it. Verified: of_which_reselected_by_rpc = 0.

The module header’s claim — “The eligibility RPC re-selects linked-but- unembedded pairs next run -> self-healing” — is therefore conditional on the extraction surviving, and that condition is not stated anywhere in the code or the spec. It is currently false for 10/25 pairs (40%).

Any fix for D1/D2 should carry a pair-anchored (not extraction-anchored) repair path, or this class of stranding recurs on the next deletion wave.

---

6. Options for the right shape

Ranked. These are not mutually exclusive; A+B is the recommended pairing.

Option A — explicit limit parameter (+ deadline awareness)

POST body { limit?: number }, default something safe (50, from §3.2’s comfortable band), hard cap enforced server-side. Thread it into the RPC as a LIMIT argument so the cap is applied in Postgres, not after the fetch. Optionally add a soft deadline: stop the loop at ~90 s elapsed and return the partial summary with truncated: true rather than being killed at 120 s.

• For: smallest diff; the route already has a defineRoute response schema to extend; re-runnability already makes repeated calls safe, so a limit costs the operator only extra clicks; the summary survives.
• Against: operator must click repeatedly; no progress visibility across calls.
• Effort: small. Recommended as the immediate fix.

Option B — terminal state for unpromotable extractions

Give a skipped extraction somewhere to go so the RPC stops re-selecting it. Cheapest shape: a nullable q_a_extractions.skipped_reason text (+ skipped_at) written on the no_answer_text path, and an added AND e.skipped_reason IS NULL to branch 1 of the eligibility RPC. Re-walking the extraction (which UPSERTs the row with fresh text) should clear it, so a fixed upstream walk naturally re-admits the row.

Do not reuse invalidated_at — that is the retirement/supersession signal and overloading it would make retireSupersededPairs treat skipped rows as retirement candidates.

• For: fixes D2 outright; makes the governance panel’s candidate count honest; makes skipped[] meaningful (only newly skipped rows).
• Against: needs a migration (DDL is gated) + an RPC revision.
• Effort: small-medium. Recommended alongside A.

Option C — cursor / resumable batch

Return a next_cursor (last created_at/id processed); caller re-POSTs until exhausted.

• For: drains an arbitrary corpus without operator arithmetic.
• Against: more moving parts than A for the same guarantee, since the CAS already makes naive re-invocation converge. A cursor buys ordering determinism the CAS does not need.
• Verdict: not worth it on top of A.

Option D — admin-only gate

Narrow ['admin','editor'] to ['admin'].

• Verdict: not the fix. Editor access is a deliberate posture and the route is already authenticated, non-public, and human-triggered from a governance surface. Restricting the role addresses neither wall-clock nor the permanent skip set. Note it, do not act on it. If anything is warranted on the auth axis it is a server-side single-flight guard (advisory lock or a run row), which is orthogonal to role.

Option E — job queue

Move promotion off the request path onto a worker.

• For: the correct end-state at real corpus scale; solves wall-clock, observability, and concurrency in one move.
• Against: no queue infrastructure exists for this today; disproportionate pre-launch against a corpus of 140 rows with 0 promotable.
• Verdict: the right shape eventually; explicitly deferred. Record as the target architecture so A+B is understood as an interim.

Recommendation: A + B now, E recorded as the target, C and D declined.

---

7. Open questions

• OQ-1 (upstream, likely its own task). Why does the corpus walk emit q_a_extractions rows with extracted_question_text populated and extracted_answer_text NULL? 135/140 rows, three source documents, 10-13 July 2026. Until this is answered the promotion path has nothing promotable and D1 stays untestable. Owner: the CocoIndex qa_target path in scripts/, not this task.
• OQ-2. Should the 10 stranded published-but-unembedded pairs (§5) be repaired by this task or by id-364 (the record_embeddings FK task)? They are the inverse of id-364’s orphans — pairs missing embeddings rather than embeddings missing pairs — so they need a distinct backfill. Proposed: a pair-anchored re-embed pass, owned here.
• OQ-3. Measure a real full-batch wall-clock before sizing the default limit. §3.2 is arithmetic, not observation. Needs OQ-1 resolved first, or a synthetic seed of answer-carrying extractions.
• OQ-4. Should a timed-out or partial run persist a pipeline_runs row so the summary survives a 504? Currently a killed run leaves no trace.

---

8. Evidence index

All DB figures: Platform staging rbwqewalexrzgxtvcqrh, read 25/07/2026.

Claim	Source
No parameters, no limit	lib/q-a-pairs/promote-corpus.ts:295-306
Auth = admin/editor, RLS-scoped	app/api/q-a-pairs/promote-corpus/route.ts:76-81
maxDuration = 120	app/api/q-a-pairs/promote-corpus/route.ts:32
Not in publicRoutes	__tests__/app/api/q-a-pairs/promote-corpus/route.test.ts:288-296
Single production caller	grep promoteCorpusExtractions over .ts/.py
POST carries no body params	__tests__/lib/query/promotion-candidates-fetcher.test.ts:207
Skip path writes nothing back	lib/q-a-pairs/promote-corpus.ts:457-461
RPC eligibility predicate (3 branches)	pg_get_functiondef('q_a_extractions_promotion_candidates'); supabase/migrations/20260707140000_id138_promotion_candidates_published_diff.sql
140 candidates = 135 unlinked + 5 linked	SELECT ... FROM q_a_extractions_promotion_candidates()
135/140 extractions have NULL answer	SELECT count(*) FILTER (WHERE extracted_answer_text IS NULL) FROM q_a_extractions
34,959 question chars in the backlog	SUM(length(extracted_question_text)) over branch-1 candidates
25 pairs, all published; 10 unembedded; 0 re-selected	q_a_pairs LEFT JOIN record_embeddings / q_a_extractions
Embedding model + cache	lib/ai/embed.ts:1-70
Per-item accept route is awaiting_review-only	app/api/governance/promotion-candidates/[extractionId]/accept/route.ts:17-23