ID-145 ARCH-REVIEW — Procurement / forms / Q&A data-model architectural review

ID-145 ARCH-REVIEW — Structured architectural review of the procurement / forms / Q&A data model

Status: Owner-directive review (S463 ratification board → “Owner directive” item), gating {145.2} PRODUCT revision + {145.3} TECH. Authored 11/07/2026 on the procurement track (ca-s435-procurement-baseline == main b6b3f6c3), READ-ONLY against code and the live staging DB rbwqewalexrzgxtvcqrh. Every row count, constraint, and call-site below was verified live this session (SQL via MCP execute_sql; code via gitnexus repo:'canonical' + git grep at b6b3f6c3).

Latitude (owner-granted): the app is NOT live — no users, no bookmarks, no retention/backfill duty. Clean-break redesign is in scope; no deprecation aliases.

Companion inputs: RESEARCH.md (S462 dossier), PRODUCT.md (48 BIs, S463 ratification: 34 accepted / 14 amended / 5 annotated), the forms-extraction lineage trace (S463 scratchpad id145-forms-extraction-lineage.md), and the peer workspaces/MCP/id-71 findings (id145-peer-ws-mcp-scope-findings.md). This review cites rather than re-derives where those documents already carry the evidence.

---

0. Verdict in five sentences

The model is three archaeological strata that were never reconciled: the bid-era workspace stratum (S246), the id-130 per-form stratum, and the OKF/corpus stratum (id-131/132/138) — each internally coherent, each documented, and each still live at HEAD, so every core fact (identity, state, questions, provenance) has two homes. The single worst defect is not a missing table but a missing producer: id-136 deleted the only form_template_fields writer (commit c84be63b, {136.5}), stranding five shipped consumer surfaces (fill, auto-map, field-editing, Path-C cataloguing, bid_worker), while the app-side extraction that survived writes only the semantic question list. The Q&A corpus side (q_a_pairs + q_a_extractions + history + dedup + question_matches + citations) is, by contrast, well-designed and near-complete — its failure mode is that the retrieval edge (question_matches) has zero callers and the promotion gate has a finished RPC but no UI. What is needed is one decisive re-anchoring: the form instance becomes the item (per DR-038 and the S463 ratifications), the workspace stratum is deleted rather than migrated, and the extract→match→fill flow gets its two missing engines (a fillable-structure writer and a matching caller). The empirical checks in §4 show every piece of that flow is buildable now: commonforms detected 198 fields on the owner’s real 57-page Standard SQ in 36 s on CPU, the deterministic extractors are cleanly git-recoverable with their deps still pinned, and langextract (Apache-2.0) runs non-Gemini with an existing Anthropic plugin.

---

1. Current-state map (19 tables, live staging rbwqewalexrzgxtvcqrh, 11/07/2026)

Legend: LIVE = has real producers and consumers; ORPHANED = consumers but no producer (or vice versa); REDUNDANT = duplicates another table’s job; MISNAMED = live but its name asserts the wrong concept; SCAFFOLD = created for a future that never arrived.

1.1 form_* (8 tables)

Table (rows)	What it actually is	Verdict
form_templates (392)	A form INSTANCE + the procurement engagement record, not a template. Carries the id-130 engagement spine: form_type FK→form_types, workflow_state (10-state), outcome triad FK→form_outcome_types, deadline, submission_date, issuing_organisation, plus the file identity (filename/storage_path/file_size/mime_type) and the fill-pipeline status axis (uploaded→analysing→analysed→filling→completed + failure states, CHECK verified).	LIVE + MISNAMED (R11 rename ratified)
form_template_fields (1)	Per-instance fillable slots: field_type CHECK {empty_cell, placeholder, highlighted}, table/row/col coordinates, question_id FK→form_questions (SET NULL), mapping_status, fill_status CHECK {pending, filled, skipped, failed}, is_mandatory, word limits.	ORPHANED — zero producers since c84be63b; readers/updaters only (§3 G1). Also template_id has NO FK (§2 C6).
form_template_requirements (0)	The actual reusable template: the global requirement catalogue (Path C target) — template_name/version/type FK→form_types, section refs, requirement text/type, matching keywords + guidance, sector applicability. UNIQUE (template_name, template_version, section_ref, question_number). requirement_embedding already migrated out per DR-036 — the migration LANDED (20260707200000_id130_form_requirement_embedding_migrate.sql:102 DROP COLUMN; column absent live), answering PRODUCT OQ-11’s carried-or-landed question.	LIVE-but-EMPTY on staging (96 rows exist on prod per RESEARCH §2: Standard SQ 66 + Charnwood ITT 30 — staging seed drift). MISNAMED in mirror-image to form_templates.
form_questions (149)	Per-instance answerable questions (semantic units): workspace_id NOT NULL FK→workspaces CASCADE, form_template_id NULLABLE FK→form_templates CASCADE, section/sequence, word_limit, evaluation_weight, status, matched_record_ids (legacy content-era array), UNIQUE (workspace_id, question_text). 119/149 rows have NULL form_template_id (debris; R3 deletes).	LIVE — written by questions/extract/route.ts:236-241 (upsert, stamps both keys via resolveOrMintFormTemplateId :205) + manual question routes. Wrongly workspace-anchored (§2 C2).
form_responses (0)	Drafted answers per question: version, text (+advanced), source_record_ids, review_status, drafted/edited/approved-by. FK→form_questions CASCADE.	LIVE code, EMPTY data — 12 route files touch it (responses/draft, draft-stream, regenerate, restore, [rId] CRUD/history, questions/route.ts, outcome/*, readiness, estimate, fill; verified git grep from('form_responses')), but staging has never held a draft. The zero is a data-state fact (debris workspaces, no real drafting ever run on staging), not a dead-code fact.
form_response_history (0)	Version history per response (UNIQUE response_id+version).	LIVE code (history/restore routes), empty for the same reason.
form_outcome_types (4)	CV: per-stage outcome vocabulary; stage ∈ {shortlist, final_award}; counts_toward_win_rate drives the win-rate engine ({130.7}).	LIVE (outcome route + win-rate RPCs read it).
form_types (8)	CV: bid/rfp/pqq/itt/tender/checklist/questionnaire/sales_proposal_template with applicable_application_types (S246 sub-task 9.1, 56dee035). Read by FormTypePicker via api.form_types; FK target for form_templates.form_type, form_template_requirements.template_type, and question_matches.question_kind.	LIVE.
template_completions (0)	Fill artefacts: completed-document storage_path, fields_filled/skipped/failed, FK→form_templates CASCADE + job FK→processing_queue.	ORPHANED downstream of G1 — writer exists (scripts/bid_worker.py:94 insert, on template_fill jobs enqueued by fill/route.ts:194) but can never run: fill requires mapped fields, and fields have no producer.

The naming inversion (C1/R11): form_templates holds instances (one uploaded/minted form per engagement); form_template_requirements holds the reusable template. The pair must swap vocabularies: instances → form_instances, requirements → the catalogue keeps a template-flavoured name. This inversion is why id-136’s TECH could mistake “a second form_templates writer” for “a second form_template_fields writer” (lineage doc §3) — the name asserts template-ness the table does not have.

1.2 procurement_* (3 tables)

Table (rows)	What it actually is	Verdict
procurement_workspaces (389)	A derived roll-up cache, not a data home: workspace_id UNIQUE FK→workspaces CASCADE + nearest_deadline, overall_outcome, counts_toward_win_rate, rollup_updated_at. Written ONLY by the DB trigger chain (form_templates_recompute_rollup AFTER INS/DEL/UPD-of-outcome-cols ON form_templates → recompute_procurement_rollup(), mig 20260625130000_id130_rollup_fn.sql:122,196-200); read ONLY via api.get_procurement_rollup from procurement/[id]/route.ts:163 (peer JOB 1; {130.29} journal brands it “tactical v1 only”). 389 rows == the 389 workspaces that have ≥1 form.	REDUNDANT — a cache of aggregates over a 1:1-in-practice relationship. Dies with the workspace stratum; aggregates read straight off form_instances.
procurement_vehicles (0)	CV: {framework, dps, dynamic_procurement} — the “procurement vehicle” ontology axis (S246 sub-task 9.2, 56dee035; id-63 PRODUCT.md:61-64 rationale: G-Cloud is a framework instance, not a form; created to keep form_type clean at 8 keys).	SCAFFOLD/ORPHANED — zero code references (sole mention: scripts/check-api-view-coverage.ts:56-57 internal-only allowlist, “future/unused workspace-type scaffolding”); zero inbound FKs from any data table; no api view; seed drift — the S246 migration seeded 3+2 rows on the old staging project (turayklvaunphgbgscat, commit message 56dee035), the current platform staging has 0.
procurement_vehicle_instances (0)	CV: {g_cloud, dos} FK→procurement_vehicles RESTRICT (sub-task 9.3).	SCAFFOLD/ORPHANED — same story.

Do they belong? The concept is real UK-procurement vocabulary and lives properly in the ontology CV docs (30-procurement-vehicle.md / 31-…-instance.md, per id-63; DR-027 puts ontology source-of-truth in repo docs + bundle). The tables have had no consumer in seven weeks of build-out and aren’t even seeded on the current DB. Recommendation: DROP both tables; re-introduce as a form_instances.vehicle_instance CV column (or re-created CV tables) the day a consumer exists. Nothing breaks — verified zero readers.

1.3 q_a_* (4 tables) + question_matches

Table (rows)	What it actually is	Verdict
q_a_pairs (25)	The promoted Q&A corpus (workspace-less by design): question + answer_standard/advanced, scope/anti-scope tags, origin_kind CHECK {extracted_from_corpus, curated_explicit, derived_from_form_response, imported_legacy, manually_authored}, publication_status CHECK (all 25 live rows published), supersession chain, and a four-way provenance fan: source_workspace_id (4 rows non-NULL, SET NULL), source_form_response_id, source_question_id, source_form_template_id ({130.15}), source_document_id. Embeddings externalised: 25/25 rows have record_embeddings owner_kind='q_a_pair' rows (DR-036 verified live). History minted by trigger q_a_pairs_history_on_update; lifecycle facet by trg_record_lifecycle_mint_q_a_pair (ID-131).	LIVE and well-shaped. One wart: source_workspace_id duplicates lineage the form-grain FKs already carry (§2 C8, §5).
q_a_extractions (140)	The staging layer (DR-026: engine writes stop here; promoted records mutate only via proposals): extractor_kind (all 140 llm_extraction), extracted Q/A text, rich per-question metadata (expected_response_kind, evaluation_criteria, evidence_requirements, scope_tags, alternate_question_phrasings), promoted_to_pair_id FK→q_a_pairs SET NULL (5 promoted / 135 unpromoted), invalidated_at, op_id; FK→source_documents CASCADE. Promotion re-selection is a finished 3-branch RPC: q_a_extractions_promotion_candidates() ({138.17}, definition verified live — new candidates ∪ mid-promotion self-heal ∪ re-walk text-drift re-propose).	LIVE (producer: the cocoindex pipeline — flow.py:2500 one-row upsert per extraction, Inv-17; consumer: the promote route + the {138.17} candidates RPC). The 135-unpromoted backlog is the promotion-gate UI gap (§3 G6), not a model defect.
q_a_pair_history (396)	Trigger-written version snapshots (UNIQUE pair+version; changed_by FK→auth.users).	LIVE (audit substrate; 396 versions over 25 pairs shows the trigger works).
q_a_pair_dedup_proposals (4)	Pairwise dedup queue: similarity, proposed survivor + reason, status, fingerprints, resolution audit.	LIVE (4 pending; producer: pipeline qa_dedup_proposer.py) but carries workspace-era columns pair_a/b_source_workspace_id (§2 C8).
question_matches (0)	The id-57/T10 candidacy edge form_question → q_a_pair: separate embedding_score + fulltext_score (never blended — id-57 §A ratified), question_kind FK→form_types (already form-vocabulary!), UNIQUE (form_question_id, q_a_pair_id), CASCADE both sides. Two-RPC contract exists and is hardened (question_match_recompute(p_form_question_id, p_query, p_query_embedding, p_question_kind, p_scope_tag, p_anti_scope_tag, p_limit) + question_match_search, both public+api, verified in pg_proc; {57.9} clamp patched 20260709).	BUILT-BUT-UNWIRED: zero TypeScript callers of either RPC (grep verified, matching PRODUCT.md problem statement), zero rows. The table is DR-038-ready as-is — it never had a workspace column.

1.4 Others

Table (rows)	What it actually is	Verdict
workspaces (536)	The generic container: name/colour/icon, application_type_id NOT NULL FK→application_types, domain_metadata JSONB, status varchar, is_archived. 498 procurement / 34 intelligence / 4 one-offs. 316 rows carry the full bid-era engagement blob in domain_metadata (keys verified live: outcome_notes, tender_document_ids, notes, estimated_value, reference_number, tender_source, submission_date, outcome, buyer, deadline, status — the id-130 “PRE-GO drop” that never executed). status is a second workflow-state home: 487 NULL / 47 ‘drafting’ / 2 ‘matching’, written by questions/extract/route.ts:268-275 and questions/match/route.ts:223 — while [id]/route.ts:381-405 and outcome/route.ts:104-120 transition form_templates.workflow_state (§2 C2). 15 inbound FKs total (verified pg_constraint): the procurement set (form_templates, form_questions, procurement_workspaces, q_a_pairs.source_workspace_id) + the intelligence lane (feed_sources/articles/prompts, si_processing_queue, intelligence_workspaces) + 5 empty app-type satellites + pipeline_runs/source_documents (SET NULL; all 45 source_documents rows already NULL — the corpus layer has already left).	LIVE for the intelligence lane; DEAD-WEIGHT-BY-RULING for procurement (DR-038). Procurement exits; the table itself stays (intelligence migration is a separate task).
application_types (6)	CV: the application catalogue (procurement, intelligence, sales_proposal, product_guide, competitor_research, training_onboarding) with per-app state_machine_config JSONB.	LIVE and KEEP — id-71 OQ-4 ratified it as the umbrella axis (peer JOB 3); it is the discriminator every workspace read uses. Post-ID-145 its procurement consumer moves from workspace-discrimination to app-catalogue duty.
template_completions	(covered in §1.1 — orphaned downstream).
(satellites) intelligence_workspaces, sales_proposal_workspaces, product_guide_workspaces, competitor_research_workspaces, training_onboarding_workspaces (0 each)	The per-app-type satellite pattern DR-038 killed. Even intelligence’s satellite is empty despite 34 live intelligence workspaces (its Shape-B swap at 56dee035 writes it only on create — none created since).	SCAFFOLD — flag to the Curator: DR-038 says these must never be populated; recommend dropping the 4 non-intelligence ones in the ID-145 batch and the intelligence one when that lane is touched.

Adjacent-but-in-flow: citations (0 rows) — id-58 polymorphic provenance. Live CHECKs verified: cited side is q_a_pair | reference_item | source_document | concept — the content_item branch is ALREADY GONE (dropped {131.19} M6); citing side is form_response only. The table COMMENT still says “cited side = content_item|q_a_pair” — stale, misleading, and part of the BI-37 sweep. processing_queue (23) carries the template_fill job lane (lib/queue/envelope.ts:61) polled by scripts/bid_worker.py.

---

2. Coherence analysis — where the model makes no sense

C1 — The template/instance naming inversion. §1.1. form_templates = instances; form_template_requirements = the template. This is not cosmetic: it already caused the id-136 over-deletion (TECH §1.2 conflated the two “writers” — lineage §3), and it makes form_template_fields ambiguous between “slots of THIS uploaded form” (what it is) and “fields of a reusable template” (what its name says). R11 (rename in TECH) is ratified; §5 gives the target vocabulary.

C2 — Every core fact has two homes. (a) Identity: the item is a workspaces row (route [id], MCP ids, question FKs) AND a form_templates row (all engagement facts). (b) State: one 10-state machine, two columns — workspaces.status (extract route :268-275, match route :223) vs form_templates.workflow_state (PATCH :405, outcome :120); staging shows both populated (47+2 workspace-status rows vs 6 non-draft workflow_state rows) and they are never reconciled. (c) Engagement metadata: 316 domain_metadata blobs vs first-class form columns for the SAME keys (deadline, submission_date, outcome, buyer/issuing_organisation — verified key-by-key). (d) Questions: anchored to workspace (NOT NULL FK + UNIQUE(workspace_id, question_text)) with the form key nullable — the inverse of the ratified model. The §4b “Questions 4 + No forms yet” defect is this split rendered visible.

C3 — THREE mechanisms exist to repair ONE null-drift bug. form_questions.form_template_id NULL-drift is patched by (i) a BEFORE-INSERT trigger form_questions_resolve_form_template_id (resolves earliest form, no mint — definition verified live), (ii) the app helper resolveOrMintFormTemplateId (lib/domains/procurement/resolve-form-template.ts) calling (iii) the advisory-locked RPC resolve_or_mint_form_template_id ({130.27}), plus the {130.8} one-time backfill. Four artefacts to keep a nullable FK non-null — the definitive signal the FK should simply be NOT NULL on a form-anchored table. In the target model all of C3’s machinery deletes.

C4 — The workspace umbrella is a cache-of-a-wrapper. procurement_workspaces caches aggregates (nearest_deadline, overall_outcome) of the forms under a workspace whose only contribution is the FK ({130.6} trigger + {130.29} RPC, self-branded “tactical v1 only”). With item==form, every cached value is a column read. Delete, don’t migrate.

C5 — ingest_source is false at the data level. All 392 form_templates rows read ingest_source='pipeline' (verified GROUP BY), yet 389 are docless app/migration mints (file_size=0) — the {130.8} migration mint didn’t set the column, so the DEFAULT 'pipeline' stamped them; only resolveOrMintFormTemplateId’s newer mints and forms/route.ts set 'app_upload'. Post-DR-014 the value 'pipeline' shouldn’t even be mintable. The CHECK {pipeline, app_upload} needs re-cutting to the DR-014 world (app_upload | minted — or drop the column and let file_size>0 + storage_path carry it).

C6 — form_template_fields.template_id has NO foreign key. Verified against pg_constraint: the table’s only FK is question_id→form_questions. A NOT-NULL “belongs to” column with no FK, no CASCADE, on the table that holds extracted slots — deleting a form strands its fields silently. (Also explains how a lone debris row could survive: it happens to link to an existing template, but nothing enforces it.)

C7 — The matching substrate is complete but nothing feeds or reads it. id-57 shipped table + two RPCs with per-method scores, scope-tag filtering, and clamping — and stopped. No route calls recompute (R7 assigns this to question create/update); nothing computes or persists question embeddings (record_embeddings has NO form_question owner_kind — verified; recompute takes p_query_embedding from the caller, so the caller must embed at call time); nothing derives p_scope_tag/p_anti_scope_tag (R7: form_type + tenant config; the mempalace S462 note confirms the intended derivation “scope_tag lives on / derives from the form instance (or its form_type + sector)”). form_questions.matched_record_ids (content-era array) and q_a_extractions.scope_tags sit as adjacent half-answers.

C8 — Corpus tables still carry workspace shadows. q_a_pairs.source_workspace_id (4/25 rows) is LIVE provenance (writers: q-a-pairs/promote/route.ts:192, workspace PATCH route, bulk toolbar — peer JOB 1.3) but duplicates what the form-grain fan already expresses — the SAME promote insert writes source_form_template_id at :191 beside source_workspace_id at :192 (verified), so the workspace column adds no lineage the form grain lacks — and R3’s wholesale workspace delete would SET-NULL it into amnesia. q_a_pair_dedup_proposals.pair_a/b_source_workspace_id are the same shadow on the dedup queue. Resolution in §5 (migrate the 4 legacy rows’ lineage to the form grain — all 4 verified migratable via their workspace’s earliest form — then drop the columns).

C9 — The vehicles tables are ontology hygiene that leaked into the DB. §1.2. Real concept, right home (ontology CV docs), wrong medium (unseeded, unconsumed tables).

C10 — The citations surface says content_items in three places while the DB forbids it. The M6 migration removed the content_item cited-kind; the live writer (lib/mcp/tools/procurement.ts cite_content) correctly writes cited_kind='q_a_pair' (:473-474) — but its ARG is still content_item_id (:426), its description narrates content tracking, get_content_effectiveness maps content_item_id→p_q_a_pair_id (:551,:570), and the citations TABLE COMMENT still advertises the dropped branch. Nothing is broken at runtime; everything lies at the contract level. This is BI-37’s true shape: a vocabulary/contract sweep (arg renames — breaking, allowed by R8’s no-aliases posture — comment fixes, doc fixes), not a data migration.

C11 — The mime CHECK contradicts the owner’s real corpus. form_templates.mime_type CHECK admits exactly {docx, xlsx, pdf} (verified). The owner-provided evaluation set (docs/testing/test-data/templates/) includes .doc and .xls binaries (Charnwood ITT Evaluation Matrix.xls, British Council rfp_-_learning_partners_osch.doc, rfp_onlinetdcops.doc) — real UK tender artefacts in formats the model cannot ingest. HTML likewise has no lane. §4 handles this as a conversion-step decision, and §5 carries the CHECK change.

C12 — The id-71 ↔ ID-145 ratification conflict (MUST be adjudicated before TECH). id-71 OQ-4 RATIFIED: application_type (procurement = the umbrella) and form_type are distinct axes; “procurement is a form-type” is a rejected anti-pattern; the umbrella MCP tools list_active_procurement / get_procurement_detail are KEPT, not renamed (id-71 PLAN.md:64,314; TECH.md:288-291 — peer JOB 3). ID-145 R1/R8/BI-42 rule: item IS one form, route+MCP re-key to form id, HARD rename. These conflict only at the surface level, and the peer’s reconciliation is correct — adopt it: (a) what retires is the per-instance CONTAINER (workspaces rows + procurement_workspaces), NOT the application-umbrella axis — application_types stays as the catalogue; (b) the umbrella tools KEEP their names (satisfying id-71: they enumerate/inspect the procurement application’s items) and their id semantics re-key workspace→form (satisfying ID-145: reads are form-scoped); (c) R8’s “hard rename” applies where names are genuinely wrong (bid-era vocabulary, content_item_id args, show_procurement_dashboard’s form_id-arg-that-takes-a-workspace-id — peer JOB 2), executed breakingly with no aliases. This narrows BI-42 from “rename the procurement tools” to “re-key the procurement tools; rename only false vocabulary.” Recommendation for the PRODUCT revision: restate BI-42 accordingly; record the id-71 cross-reference so the two ratification registers agree.

---

3. Gap analysis — what is missing to bring extract→match→fill fully online

G1 — A fillable-structure producer (the deleted engine). form_template_fields has readers (fill/route.ts:76, auto-map/route.ts:76,156, fields/[fieldId] + fields/bulk-update editors, templates/[templateId]/route.ts:62, Path-C catalogue/from-instance.ts:189, bid_worker.py:118) and NO writer — git grep finds zero .insert(/.upsert( targets at HEAD; the deterministic package (2,235 LOC, 6 files)

• its flow.py writer died in c84be63b and is cleanly recoverable at c84be63b^ = 5494b527 (lineage §2). Note the surviving in-tree cousin: scripts/analyse_template.py (bid-era, DOCX-only, CLI-only — its docstring still claims it “writes … template_fields table” but it returns a dict and nothing invokes it; its route died at {52.15}). R5 ratified restoration; §4 recommends the mechanism.

G2 — A matching caller + scope-tag derivation + question embeddings. C7. Three concrete sub-gaps: (i) the R7 app-route recompute on question create/update; (ii) the scope-tag derivation (form_type + tenant config → p_scope_tag[]/p_anti_scope_tag[]); (iii) an embedding for the question text at recompute time (compute-on-call via the existing embedding service, or persist under a new record_embeddings.owner_kind='form_question' — TECH decision; persistence buys re-match-on-corpus-change without re-embedding).

G3 — The drafting path cannot yet cite the corpus it will draft from. The MCP cite_content write path is q_a_pair-shaped already (C10), but the drafting stack’s writer (responses/draft-stream et al.) cites nothing q_a_pair-shaped and question_matches (the candidate feed) is empty — so the loop question → candidates → draft → cite → outcome → win-rate has no first edge and a dormant fourth. BI-37’s activation is a writer change (RESEARCH §5.4) + G2.

G4 — Form-first creation. POST /api/procurement (route.ts:265-278 insert) mints a workspace + domain_metadata and NO form (verified; comments literally read “Failed to create bid”). Ratified replacement: create = mint the form instance (FormTypePicker BI-8; upload path BI-9-10).

G5 — The DR-014 manual upload UX. Machinery exists (hardened tender/route.ts upload, Claude extraction) but every entry point is workspace-nested tender-document framing (RESEARCH §4b nuance); “upload a form → it becomes the item” has no home.

G6 — The promotion gate has a backend and no front. q_a_extractions_promotion_candidates() is finished (3-branch, {138.17}); a promote route exists (q-a-pairs/promote/route.ts — peer JOB 1.3); 135/140 extractions sit unpromoted; there is NO reviewer surface. BI-38’s annotation (“check what already exists in /review”) resolves as: /review is the content-quality lane; the extraction→pair promotion queue and the 4-row dedup-proposal queue are Governance-zone work; the thin UI (BI-38/39) composes the existing RPC + route rather than new backend.

G7 — Partial fill is not yet a first-class outcome (BI-22). The vocabulary is close: fill_status CHECK {pending, skipped, filled, failed} + template_completions.fields_skipped already model gaps — what is missing is (i) the VALIDITY rule (a completion with skipped/gap fields is a SUCCESS state, not fill_failed), (ii) re-entrancy (a later fill pass over only-the-gaps — bid_worker.py currently one-shots), and (iii) the gap→create-content loop the owner’s prior conversations describe (gap surfaced → user/Claude drafts the missing answer → q_a_pairs origin manually_authored/derived_from_form_response → re-match → re-fill). The enums and origin_kinds for that loop ALL exist; the loop itself is unbuilt.

G8 — Fill writers beyond DOCX, and an undeployed worker. fill_template.py is DOCX-only (run-level API, formatting-preserving — good) with real test depth (test_fill_template.py fixtures). PDF fill (AcroForm write-back) and XLSX fill (openpyxl cell writes) are net-new; §4 covers the PDF half via the commonforms output artefact. Operationally: bid_worker.py appears in no deploy manifest (grep across workflows/compose/Dockerfiles — zero hits; the onprem compose ships only the pipeline image) — it is a manually-run poller. Restoring fill therefore also needs a worker deployment decision (Coolify service beside the pipeline vs folding fill into the existing queue-consumer).

G9 — Format coverage: HTML has no lane; .doc/.xls are CHECK-blocked (C11).

G10 — BI-40’s two UI data-states. (i) Part-drafted forms: the model supports it (responses carry review_status per question; coverage engine renders readiness) — the gap is only that the item page must render mixed states honestly. (ii) Questions the DB cannot answer: question_matches returning zero candidates is representable TODAY (absence of rows), and manual answering exists (drafting stack) — the gap is the join surface: “no-match → answer manually → optionally promote to corpus” as one flow (pairs with G7’s loop).

---

4. Extraction / matching / filling — empirical findings + recommendation

4.1 Empirical verification block (per shared-discipline §Empirical verification)

Date 11/07/2026. Throwaway venv (Python 3.14), packages from PyPI (sandbox-exempted install; this repo pins none of these — versions below are what resolved today).

Candidate	Verified	Result
commonforms 0.2.1	import commonforms → PRESENT; prepare_form(input_path, output_path, *, model_or_path='FFDetr', keep_existing_fields, use_signature_fields, device='cpu', image_size=1024, confidence=0.4, fast, multiline, batch_size) (signature inspected); functional run on the owner’s real standard-selection-questionnaire-ppn-03-24.pdf (57 pp): 198 AcroForm fields written in 35.9 s CPU (fast=True FFDNet-S; pypdf readback: 141 /Tx text + 57 /Btn checkbox). Deps: rfdetr, ultralytics, transformers, onnxruntime, pypdf (heavy vision stack).	PRESENT + WORKS. PDF-only (API takes PDFs; renders pages → detects widgets). Licence BLOCKER: no code licence — GitHub API license: null, PyPI License: None; only the FFDetr model is stated Apache in release notes. 1,106 stars; last push 17/06/2026. Paper arXiv:2509.16506.
langextract 1.6.0	import langextract → PRESENT (Apache-2.0 per metadata); provider registry inspected after load_builtins_once(): builtin patterns = ^gemini*, OpenAI ^gpt-4/-5, and Ollama-style local families (llama/mistral/qwen/deepseek/…). ModelConfig(model_id, provider, provider_kwargs) + create_model(...) factory verified.	PRESENT. Non-Gemini confirmed (OpenAI + local builtins). No Anthropic builtin, but langextract-anthropic 0.2.1 exists on PyPI (pip index verified) and the factory/plugin seam makes a Claude provider thin. Text-only: PDFs must be textified first — it does layout-free, char-offset-grounded extraction (its differentiator: extractions map to exact source spans → aligns with citations’ span anchoring, D-S330-1).
Deterministic id-52 extractors	git ls-tree c84be63b^ scripts/cocoindex_pipeline/form_extractors/ → all 6 files present (lineage §2); deps still pinned at HEAD: requirements.txt:20 pdfplumber==0.11.9, :26 openpyxl==3.1.5, :29 python-docx>=1.1.2, :3 anthropic==0.79.0. Recoverable tests T21/T22/T23 + fixtures intact.	RECOVERABLE at zero dependency cost. Known fidelity gap: the S3xx wave-2 finding “7 of 8 Charnwood tables surface ZERO fields” (mempalace, ledger finding vs {52.12}) was never resolved — recovery must re-run the fixture suite and treat that finding as an open defect, not a regression.
Claude extraction (live)	extractPDFQuestions (lib/domains/procurement/ai/extract-questions.ts:185-189; gitnexus context verified: 1 caller = questions/extract/route.ts POST, calls extractPDFQuestionsWithToolUse) + extractDOCXQuestions (mammoth→HTML→Claude).	LIVE. Writes ONLY form_questions (semantic list) — no structure, no XLSX/HTML.
Headless-agent / “/goal”	id-71 verified (peer JOB 3): NO /goal command exists, NO curated forms dataset, NO form-fill flow; id-71 ratified a goose (Apache-2.0) + Claude-runtime fleet, propose-only writes, human-gated publication, born-evaluable under the ID-104 contract. Form-fill is a propose-write NOT in the launch headless set.	The fleet is the right FUTURE lane for agentic fill, and its propose-only discipline matches DR-026. It is NOT a v1 extraction mechanism.

Eval dataset (OQ-10 answer): it already exists in-repo — docs/testing/test-data/templates/ (owner-provided; in git since 19/05/2026, 7fbe68ad): Charnwood ITT Services.docx + ITT Evaluation Matrix.xls; EFA evaluation-matrix-itt-vol8.xlsx; British Council annex_2_supplier_response.docx + annex_3_pricing_approach.xlsx + two .doc RFPs; Standard SQ PPN 03/24 .pdf — with symlinks under scripts/tests/fixtures/form-extraction/ and per-fixture acceptance expectations in id-52 ACCEPTANCE.md (e.g. Bidder-1/2 dedup, page-limit capture). Real coverage: pdf, docx, xlsx, doc, xls — no HTML sample (source one before building an HTML lane). mempalace surfaced no additional owner-provided forms beyond this set this session (chroma backend segfaults upstream #1665; FTS fallback queried directly).

4.2 The three-plane model (the design insight the current schema already implies)

The schema separates — correctly — three planes that any robust approach must keep apart:

1. Semantic plane (what is being asked): form_questions (per-instance) ←→ form_template_requirements (catalogue) ←→ q_a_pairs (corpus answers). LLM work.
2. Mechanical plane (where answers physically go): form_template_fields — slot coordinates, widget types, fill status. Deterministic/vision work; NO LLM required.
3. Composition plane (which answer fills which slot, with what provenance): auto-map (field→question — today a pure lexical similarity() over field.question_text vs question.question_text with a threshold, auto-map/route.ts:139-149; no LLM, no embeddings), question_matches (question→corpus candidates), citations (response→source), template_completions (the artefact).

id-136’s over-deletion killed plane 2’s producer; id-57’s unwiring left plane 3’s first edge empty. The recommendation below fills exactly those two holes and changes nothing about the (sound) plane separation.

4.3 Recommendation (feeds {145.3} TECH as its default position)

R-A. Questions (plane 1) — keep and extend the live Claude path; do not adopt langextract for v1. The live extractor already produces sectioned questions with word limits + weights across PDF/DOCX; extend it (i) to XLSX (workbook→structured text; the recovered xlsx reader can supply the textification) and (ii) to emit per-question metadata parity with q_a_extractions (expected_response_kind etc.) so instance questions and corpus staging speak one vocabulary. langextract is the right SECOND-generation upgrade when span-grounded question provenance is wanted (its char-offset alignment feeds citations’ span model directly), and it demonstrably runs non-Gemini — record it as a tracked enhancement, not a v1 dependency (one more moving part + a community Anthropic plugin as a prod dependency is not justified while Claude extraction already works).

R-B. Fillable structure (plane 2) — recover the deterministic engine for DOCX/XLSX; adopt the commonforms APPROACH for PDF, gated on licence. Concretely:

• DOCX/XLSX: resurrect form_extractors/{docx,xlsx,shared}.py from c84be63b^ re-homed under an app-side analyse job (enqueue analyse_form on upload → worker writes form_instance_fields; the bid_worker/processing_queue lane already models exactly this). Deps already pinned; tests recoverable; DR-014 untouched (no corpus walk — the engine was coupled to the walk by wiring, not by nature: lineage §3). Re-run the fixture suite and fix the Charnwood 7/8-tables finding as part of recovery.
• PDF: the empirical run proves detector-based field detection works on real UK forms (198 fields, 57 pp, 36 s CPU — deployable on the existing IONOS worker without GPUs). BUT do not take commonforms as a prod dependency while it has no code licence: (1) open an upstream licence request (the model is Apache; the ask is small); (2) if unresolved by TECH-time, replicate the inference step (render→detect→widgets) against the Apache-licensed FFDetr weights with a ~200-line in-repo wrapper (pypdf + onnxruntime), using commonforms only as a dev-side reference implementation for the eval harness. Either way the PDF lane’s output is BOTH a form_instance_fields row set (coordinates from detection boxes) AND a fillable-PDF artefact for the fill step. Mandatory addition the empirical run exposed: detection yields coordinates ONLY — no label text — while auto-map matches lexically on field.question_text. The PDF lane must therefore pair each detected box with its nearby label text (pdfplumber words left/above the box — the recovered pdf.py already extracts positioned words) or PDF auto-map is a structural no-op.
• Legacy .doc/.xls: convert-on-upload (LibreOffice headless on the worker) or reject with guidance — TECH decision; the eval set says real buyers still send them.
• HTML: defer until a real sample exists (no eval fixture; no mime lane today).

R-C. Matching (plane 3, first edge) — wire R7 exactly as ratified. App-route recompute on question create/update (the extract route’s post-insert loop is the natural site — it already holds the question ids); scope tags derived form_type + tenant config; embedding computed at recompute time (persist under owner_kind='form_question' only if re-match-on-corpus-change proves needed); MCP reader on question_match_search. Zero schema work — id-57’s table/RPCs are DR-038-ready as-is (question_kind is already a form_types FK).

R-D. Fill (plane 3, artefact) — keep the queue-worker shape; add per-format writers. DOCX: surviving fill_template.py (formatting-preserving, tested). PDF: pypdf AcroForm value writes into the plane-2 fillable artefact. XLSX: openpyxl cell writes at recorded coordinates. Partial fill is the DEFAULT contract (BI-22): a completion with gaps is completed + fields_skipped>0 (never fill_failed, which is reserved for I/O/engine errors); gaps stay fill_status='skipped'|'pending' and re-fill passes process only gaps; each gap links to the create-content loop (G7): draft answer → promote (manually_authored) → re-match → re-fill.

R-E. Headless-agent fill — the id-71-aligned lane, sequenced after v1. The completing-forms sibling Task should target the goose/Claude fleet with PROPOSE-ONLY writes (draft form_responses + proposed mappings; human approves; publication gates per DR-026) — reconciling R5’s “autofill is critical” with id-71’s launch posture: v1 autofill = deterministic structure + matched corpus answers + human-gated fill (above); agentic end-to-end fill = the fleet’s first procurement recipe, feeding the SAME tables through the SAME propose-gate. No /goal; the recipe pattern (id-71 MVP = O4 briefing recipe) is the precedent to copy.

---

5. Target model + build path (clean-break; no aliases, no backfill duty)

5.1 Target tables (procurement/forms/Q&A after ID-145)

Renames/reshapes (one migration batch, api-views regenerated in the SAME batch per DR-030/DR-032):

Today	Target	Changes
form_templates	form_instances	DROP workspace_id (identity: the instance IS the item; route + MCP key on this id). ADD reference_number text, estimated_value numeric (R4), engagement_group_id uuid NULL (R2 — see below). KEEP both lifecycle axes but RENAME for honesty: status → processing_status (upload/analyse/fill pipeline) vs workflow_state (procurement lifecycle) — two axes, two names, no more colliding “status” vocabulary with the deleted workspaces.status. Re-cut ingest_source CHECK to {app_upload, minted} (C5). Widen/route mime_type per the R-B .doc/.xls decision (C11).
form_template_fields	form_instance_fields	ADD the missing FK form_instance_id → form_instances(id) ON DELETE CASCADE (C6; rename from template_id). Everything else stands — the slot model (coordinates, mapping_status, fill_status) survived review.
form_template_requirements	form_requirement_templates (or requirement_catalogue)	Name now truthfully claims template-ness. Re-seed staging (0 rows vs prod 96 — seed drift). Embedding already externalised (DR-036 verified).
form_questions	form_questions	DROP workspace_id (+ its FK + UNIQUE). form_instance_id NOT NULL FK CASCADE (rename). UNIQUE (form_instance_id, question_text) (the ex-bl-435 re-scope). DROP matched_record_ids (content-era; question_matches is the edge). DROP the C3 repair trigger + helper + RPC (obsolete once NOT NULL).
form_responses, form_response_history, form_outcome_types, form_types, template_completions	unchanged (rename template_completions.template_id→form_instance_id for consistency)	The response/versioning/outcome/CV spine survived review intact.
question_matches, q_a_extractions, q_a_pair_history	unchanged	Sound. Wire, don’t reshape (G2/G6).
q_a_pairs	q_a_pairs	DROP source_workspace_id after migrating the 4 live rows’ lineage onto the form grain (source_form_instance_id already exists as source_form_template_id — rename it with the batch): one UPDATE joining workspace→earliest-form before the workspace delete. This resolves the peer’s provenance warning without keeping the column.
q_a_pair_dedup_proposals	same	DROP pair_a/b_source_workspace_id (C8; fingerprints + form_response ids remain).
citations	same	Fix the stale table COMMENT; activate the q_a_pair path in the drafting writer (G3); rename the cite_content/get_content_effectiveness content_item_id args (BI-37, breaking per R8).

Deletions (owner-ratified latitude):

1. procurement_workspaces + trigger form_templates_recompute_rollup + recompute_procurement_rollup() + api.get_procurement_rollup (C4). Dashboard/list aggregates become direct form_instances queries (BI-30).
2. procurement_vehicles + procurement_vehicle_instances (C9) — concept stays in the ontology CVs; remove the two check-api-view-coverage.ts allowlist entries with it.
3. All 498 procurement workspaces rows (R3/R10: debris purge + the wholesale delete; CASCADE removes the 119 NULL-key questions and the satellite rows) — sequenced AFTER the q_a_pairs lineage UPDATE (above) and AFTER re-anchoring form_instances (the workspace FK drop must land before its referent rows die; order in §5.3). workspaces itself + application_types + the intelligence lane remain (C12 reconciliation; DR-038 governs future app types).
4. The four empty non-intelligence *_workspaces satellites — recommend to the Curator as an adjacent cleanup (same DR-038 logic; not strictly ID-145’s).
5. workspaces.status writes from procurement routes (extract :268-275, match :223) — the state machine’s ONLY home becomes form_instances.workflow_state (C2b).
6. workspaces.domain_metadata procurement keys after the R4 column move (C2c; 316 blobs die with their rows anyway — the reader list in the peer report is the re-point checklist).

Additions (small):

• engagement_groups (R2): id, name, timestamps — plus form_instances.engagement_group_id NULL FK SET NULL. A LINK, not a container: no state, no metadata, no RLS scoping of its own (BI-27/28). (TECH may choose a self-referential sibling_of edge instead; either satisfies R2 — the table is the simpler read.)
• The analyse_form job type on processing_queue (R-B) + per-format fill writers (R-D).
• Optional record_embeddings.owner_kind='form_question' extension (R-C, only-if-needed).

5.2 What this kills, structurally

The item’s identity chain collapses from workspaces → procurement_workspaces → form_templates → form_questions (workspace-keyed, form-nullable) to form_instances → form_questions. Every C2 double-home resolves to one home; C3’s four repair artefacts delete; the §4b defects become unrepresentable states.

5.3 Build path (staging-first, sequenced BEFORE the ID-45 client re-ingest)

• W0 — Adjudications (owner, pre-TECH): C12 resolution (recommended: re-key umbrella tools, rename only false vocabulary — restate BI-42); commonforms licence posture (R-B); .doc/.xls policy (convert vs reject).
• W1 — The migration batch (one push, staging-first; internal order is load-bearing): (1) q_a_pairs lineage UPDATE (workspace→form grain; all 4 rows verified migratable); (2) DELETE the 119 NULL-form_template_id questions (R3 — must precede the NOT NULL); (3) renames + the form_questions/form_instances workspace-column DROPS (severs the CASCADE so surviving forms/questions outlive the workspace delete) + NOT NULL + UNIQUE re-scope + new FK on form_instance_fields + CHECK re-cuts; (4) api-view regen + RPC wrapper regen in the SAME batch (DR-030/032); (5) DELETE the procurement workspaces rows + docless mint debris (R3/R10) + drop procurement_workspaces/vehicles/rollup machinery; (6) re-seed form_requirement_templates + one form-first exemplar from docs/testing/test-data/templates/ (OQ-10 — the exemplar SHOULD be one of the real forms; the Standard SQ PDF now has a verified 198-field detection baseline).
• W2 — Create/upload/extract: form-first POST (G4), DR-014 upload entry (G5), extraction extended to emit structure (R-A/R-B: recovered readers + PDF lane), form_instance_fields writer live.
• W3 — Match + fill: R7 wiring (G2), auto-map un-orphaned against real fields, fill workers per format with partial-fill contract (R-D/G7), citations activation (G3).
• W4 — Surfaces: item page (BI-13..19), dashboard re-wire (BI-30..33), MCP re-key + false-vocabulary renames (C12/BI-42), copy sweep (Unit-F).
• W5 — Governance: thin promotion-gate UI over the existing RPC + dedup queue (G6/BI-38/39).
• completing-forms opens as the sibling Task once W1’s id contract lands (R-E).

---

6. BI impact — concrete PRODUCT-revision inputs (14 amended BIs + 5 annotations)

BI	Revision input from this review
BI-1 (item IS one form)	Strengthen: name the target table (form_instances) and state the two-axis lifecycle split (processing_status vs workflow_state, §5.1) so “lifecycle facts are attributes of the form” is unambiguous about WHICH axis.
BI-2 (addressable by form id)	Unchanged in substance; add: the id survives the R11 rename (PK carried over), so links minted during ID-145 dev don’t break at the rename.
BI-3 (legacy redirect)	Weaken to match R3. With ALL 498 procurement workspaces deleted (not just debris), there are no legacy workspace-id links worth a primary-form lookup on staging/prod — and no users (owner latitude). Recommend: plain 404/410 for unknown ids; drop the redirect machinery from scope. If the redirect stays, it needs a workspace→form mapping TABLE snapshot taken pre-delete — call that out explicitly or cut it.
BI-5 (ref#/value first-class)	Confirmed; columns land in W1 (§5.1). Note: the other 316-blob keys (tender_source, tender_document_ids, notes) — peer’s reader list shows live readers for tender_document_ids in the tender flow; PRODUCT should name which of the four survives as a column vs dies.
BI-6 (10 states verbatim)	Confirmed; add the C2b consolidation as an explicit invariant: “no surface writes any state anywhere except form_instances.workflow_state” — otherwise the extract/match routes silently keep the second home alive.
BI-7 (nothing born formless)	Confirmed; C3 gives the enforcement mechanism for free: form_questions.form_instance_id NOT NULL makes formless questions unrepresentable, and deleting the mint/trigger/helper stack is the proof the invariant holds by construction.
BI-11 (creation admin/editor-gated)	Unchanged; the existing gates carry over.
BI-16 (documents on the item)	Add: the uploaded form file IS form_instances’ file identity (filename/storage_path/mime) — the docless-mint pattern (389 rows, C5) must be impossible for uploaded items; minted-without-file remains only for the manual-create-then-upload path and is honestly labelled ingest_source='minted'.
BI-22 (fill; partial-fill annotation)	Restate per R-D: partial completion is a SUCCESS state (fields_skipped>0, never fill_failed); gaps remain fillable (skipped/pending) across re-fill passes; each gap offers the create-content loop (draft → promote manually_authored → re-match → re-fill). The enums already support all of it (G7) — the BI should mandate the loop, not new vocabulary.
BI-37 (content_items fully removed)	Reshape + re-scope: the TABLE and the citations branch are already gone (M6, verified live — no .from('content_items') query survives anywhere), so nothing is runtime-broken; what remains is contract vocabulary. But the full sweep is 495 occurrences across 126 non-test files (verified git grep) — dominated by certifications/dashboard/guides/cron surfaces that are themselves content-era features outside ID-145. Recommend: BI-37 owns the procurement + citations slice — cite_content.content_item_id arg + description, get_content_effectiveness arg mapping (procurement.ts:426,473,551,570), get_content_win_rate naming, form_questions.matched_record_ids, the stale citations table COMMENT (breaking arg renames allowed per R8) — and the remaining ~120 files route to the Curator as a tracked platform-wide vocabulary-retirement item.
BI-38 (promotion gate; check /review)	Resolved by G6: /review is the content-quality lane and stays; the promotion gate composes the EXISTING q_a_extractions_promotion_candidates() RPC + promote route + dedup-proposals queue into one thin Governance surface. PRODUCT should scope it as read-queue + accept/edit/reject + dedup-resolve, explicitly “no new backend”.
BI-40 (part-drafted + manual answering)	Both states are already representable (G10); revise the BI to mandate the two flows: (i) mixed per-question states render honestly on the item page; (ii) zero-candidate questions offer manual answer → optional corpus promotion (closing the loop with BI-22’s gap flow).
BI-41 (drafting stack secondary)	Unchanged; note the stack re-points to the form id trivially (it keys on [id] only — RESEARCH §2).
BI-42 (MCP re-key; check id-71)	Restate per C12: umbrella tools (list_active_procurement, get_procurement_detail) KEEP their names (id-71 OQ-4 ratified) and re-key workspace→form; hard-RENAME only false vocabulary (content_item_id args, bid-era names, show_procurement_dashboard’s misleading form_id arg); no aliases anywhere (R8). Record the id-71 cross-reference in both registers.

Annotation outcomes: BI-22 ✓ (above). BI-37 ✓ (above). BI-38 ✓ (above). BI-40 ✓ (above). BI-42 ✓ (above; conflict surfaced + reconciliation recommended, C12).

---

7. The top five decisions {145.3} TECH must make

1. C12 adjudication — adopt the re-key-not-rename reconciliation of ID-145-R8/BI-42 vs id-71-OQ-4 (owner sign-off required; both registers updated).
2. PDF structure mechanism — commonforms-as-dependency (needs upstream licence) vs in-repo FFDetr-weights wrapper (recommended default) — plus the .doc/.xls convert-or-reject policy (C11).
3. The W1 migration-batch shape — rename set, FK re-anchor order, q_a_pairs lineage UPDATE before workspace delete, api-view/RPC-wrapper regen, exemplar seed — sequenced before ID-45 and without freezing legacy shapes into the {128.10} snapshot.
4. Question-embedding persistence — compute-on-recompute vs record_embeddings.owner_kind='form_question' (drives re-match-on-corpus-change behaviour and the R7 route’s latency budget).
5. BI-3 redirect scope — drop the legacy-workspace redirect (recommended, per R3 wholesale delete + no-users latitude) or spec the pre-delete mapping snapshot it would require.

End of ARCH-REVIEW. Written read-only; no ledger writes, no TECH authored.