Skip to content
Canonical

id-370 RESEARCH — unanswered-question routing (S511)

id-370 research — routing walk-discovered unanswered questions

Section titled “id-370 research — routing walk-discovered unanswered questions”

Read-only grounding (S511 agent), commissioned when the id-396 board ratified D5 (pull id-370 forward). Owner context honoured throughout: specs/id-127-platform-pipeline/notes/id-127-additional-context-for-item-3.md — two form states: (i) EMPTY (questions only → complete-from-corpus / coverage review); (ii) COMPLETED (historic → seeds q_a pairs; platform-completed → win-rate data). Q&A sidecar watching retired; first client = ONE-TIME bid-library extraction; DB first-class for verified pairs.

1. Current mechanics

Section titled “1. Current mechanics”
  • extract_qa_form runs unconditionally on both walk branches — flow.py:2001-2002 (Path-A mount_each) and flow.py:3196-3197 (off-mount P4); sequenced after classification but reads nothing from it.
  • Null answers are sanctioned end-to-end: prompts.py:113 (“verbatim answer text if present…OR null”; FIELD CONSTRAINTS :124-130 never constrain answer_text); QAPair.answer_text: str | None (extraction.py:378); declared undefended at flow.py:2290 / :3308, keyed uuid5(…, f"qa:{source_document_id}:{idx}").
  • Trap: live RPC 20260707140000_id138_promotion_candidates_published_diff.sql:57-103. Branch 1 (:74-75) = promoted_to_pair_id IS NULL with NO answer predicate — while branch 3 already carries exactly the needed predicate (:92-95 extracted_answer_text IS NOT NULL AND trim(…) <> ''). The defect is an asymmetry; part of the fix is a one-line symmetry restoration.
  • Consumer: lib/q-a-pairs/promote-corpus.ts:457-461 skips no_answer_text (sole SkipReason, :95), writes nothing back → row re-selected forever. Escape hatch is awaiting_review-scoped only (app/api/governance/promotion-candidates/[extractionId]/accept/route.ts:17-19).

2. The three options vs the two-form-state model

Section titled “2. The three options vs the two-form-state model”

The 135 rows are state (i) artefacts being fed to the state (ii) machine (q_a_extractions → q_a_pairs is the historic-completed-form funnel). That is the conflation in one line.

  • Option 1 (do not emit) = the owner’s model as a predicate. Gate the declare on non-empty answer_text: blank forms mint nothing; the one-time bid-library walk (state ii, completed forms) flows unchanged. The only discard is blank-form questions arriving via the walk — a document class DR-014 says should not arrive via the walk at all.
  • Option 3 reverses DR-014, and the owner’s note does not dissolve that. DR-014 verbatim: “Forms enter the system via manual upload (§J Path-A/B/C → the requirement catalogue, now form_requirement_templates), NOT the corpus ingest walk. The production forms corpus-route was retired (shipped BL-392 → id-136, PR #113, S448)…” — exclusive, not additive. The owner’s “manual-upload too” recalls the pre-DR-014 pivot; DR-014 (accepted, not superseded) closed the walk leg. Needs explicit owner confirmation, not inference.
  • Option 3’s four structural blockers (likely fails before the DR argument):
    1. form_instances cannot represent a walked doc — storage_path/filename/ file_size/mime_type NOT NULL; mime_type CHECK = {docx,xlsx,pdf} (squash baseline, kept by 20260712062000_id145_w1c_rename_reshape.sql), while the walk’s FormMetadata.form_format admits html/md (extraction.py:334).
    2. The needed ingest_source value was deliberately deleted — 20260712062000:69-83 re-cut CHECK {pipeline,app_upload} → {app_upload,minted} and migrated historical 'pipeline' rows away.
    3. No provenance column: form_instances has no source_document_id.
    4. Option 3 names the wrong table — questions are Plane 1 (form_questions, 20260712062000:120-136); form_instance_fields is Plane 2 (fillable slots + geometry, 20260716140000_id147).
  • The form lane already has a question producer: scripts/bid_worker.py → analyse_form_job (:550) writes Plane-1 via app/api/internal/procurement/extract-questions/route.ts and Plane-2 via _write_form_instance_fields (:511). The walk’s extract_qa_form is a duplicate Plane-1 extractor in another language writing another table. Option 3 adds a SECOND producer to a lane that has one.
  • Option 2 parks the rows and answers neither state (task file’s own conclusion).
  • Inv-19 (Path A off the form tables) is still test-enforced (scripts/tests/test_cocoindex_flow_write_path.py:1400-1506) — Option 3 reverses it alongside DR-014.

3. id-363 option E / id-364 option A constraints

Section titled “3. id-363 option E / id-364 option A constraints”
  • id-363 Option E (RESEARCH §6 verbatim): “Move promotion off the request path onto a worker… Against: no queue infrastructure exists for this today… Verdict: the right shape eventually; explicitly deferred.” The Against premise is false: processing_queue + scripts/bid_worker.py poller run two job types today (template_fill, analyse_form; auth-context contract lib/queue/envelope.ts). Option E = a third job type on an existing poller. The owner’s override is cheaper than the doc priced it.
  • Interaction: with promotion queued, id-363’s wall-clock argument dissolves; a skip marker remains only for genuinely malformed rows. If id-370 takes Option 1, class-A volume → ~0, resolving the “no two competing markers” AC cleanly.
  • id-364 Option A (four nullable typed columns + CHECK exactly-one + per-column FK CASCADE; owner-chosen over the RESEARCH’s Option C) breaks the record_embeddings LATERAL inside q_a_extractions_promotion_candidates() (20260707140000:64-71).
  • Three-way collision, recorded nowhere before now: id-370 (branch-1 predicate), id-363 (LIMIT + terminal state), id-364-A (LATERAL rewrite) all edit ONE function body. Ordering must be ruled.

4. id-294 — outdated framing, one durable fact

Section titled “4. id-294 — outdated framing, one durable fact”

S415, parked: unhandled asyncpg FK violation on reference_items_source_document_id_fkey during a prod walk. Durable: independently proves the FK is real and enforced. Outdated: its “tolerate the FK miss gracefully” hardening is the compensating-mechanism approach the owner rejected in id-364 F1; its {75.17}/content_item_id references predate the content_items retirement. Recommendation: fold residual verification into {364.1}; close id-294.

5. reference_items.source_document_id — live FK, not residue

Section titled “5. reference_items.source_document_id — live FK, not residue”
  • Schema: NOT NULL (squash_baseline.sql:7232), FK ON DELETE RESTRICT (:9921).
  • Writer: reference_ingest() mints the sd row FIRST, same transaction, explicitly for this FK (20260708160000_id139…:124-152 — comment names it). Live callers: app/api/ingest/url/route.ts, lib/mcp/tools/content.ts.
  • Readers: reference_list() returns it (20260620000000_id111…:20,39 + api wrapper :58); app/reference/[id]/page.tsx:104-131 renders provenance from it; lib/query/fetchers.ts:264,295-296 filters on it. id-392’s M6 retarget moved MORE reads onto reference_items — coupling is increasing.
  • Verdict: live FK — provenance, not identity. The real residual question (“should a reference_item mint a synthetic 1:1 source_documents row at all”) is id-364 F1’s question, not id-370’s sweep.

6. Flags (unresolved)

Section titled “6. Flags (unresolved)”
  • DR-014 exclusivity needs explicit owner confirmation (see board D6).
  • extraction.py:401-404 docstring is stale — cites the ID-136-removed Path-B writer; rider for whoever lands id-370’s code change.
  • Q&A sidecar residue NOT swept for specifically — treat “no sidecar code found” as unverified, not clean.