Completing Forms (Procurement) — Workflows

⚠️ BID-ERA HISTORICAL REFERENCE (bannered S462). Last verified S248 — predates the id-61 renames, id-130 per-form domain model, DR-014 (manual form upload), DR-038 (workspace retirement), and DR-041 (three-zone IA). Valuable for flow/mechanism archaeology only; hook names, table names, workspace-level state framing, and the “100s draft-all timeout” (retired S224 — async 202 queue) are all stale. Current model: ID-145 — specs/id-145-procurement-form-first/RESEARCH.md.

Last verified: Session 248 (20 May 2026) — refreshed for S248 T4 procurement umbrella rename (lib/bid → lib/procurement, components/bid → components/procurement, /api/bids → /api/procurement, BID_STATES → PROCUREMENT_WORKFLOW_STATES); ID-71 bid→forms rename + outcome-grouped MCP tool surface (form_questions/form_responses tables, find/get/where_are_we_exposed/whats_in_my_queue tools); prior S210 (29 April 2026) for S195-S209 + prod-readiness S10-S13 Pending updates: None

Completing Forms (Procurement) — Workflows

Overview

Completing Forms is built on three orchestrated workflows: a 10-state form lifecycle machine, a 3-pass AI drafting pipeline that pulls from the publication-status-aware KB, and an export pipeline that converts markdown-native responses to DOCX/XLSX or fills procurement templates. Q&A Library content (content_items rows with answer_standard / answer_advanced) is the upstream feed for drafting; its edit workflow (S198 §1.5) is documented separately because it gates draft quality.

Workflow 1: Form Lifecycle State Machine

Trigger: User actions on /procurement/[id] (mainly via useFormActions.handleStatusTransition). Owner: lib/procurement/procurement-workflow.ts — pure, no side effects; transition guard enforced at the API layer.

State Diagram

draft
  ↓ (questions extracted)
questions_extracted
  ↓ (KB matching kicked off)
matching
  ↓ (matching complete)
drafting  ←──────────┐
  ↓ (send to review)  │
in_review ────────────┤ (revisions requested)
  ↓ (review approved) │
ready_for_export ─────┤ (issues found pre-export)
  ↓ (submitted)       │
submitted ────────────┤ (submission retracted)
  ↓
won │ lost │ withdrawn   ← terminal states (any state can also → withdrawn)

Forward Transitions

Current State	Event	Next State	Side Effects
draft	Tender uploaded + extracted	questions_extracted	form_questions rows populated by extractor
questions_extracted	KB matching kicked off	matching	Matching pipeline begins
matching	Matching complete	drafting	confidence_posture + matched_content_ids set per question
drafting	Responses finalised	in_review	Reviewer landing
in_review	Approved	ready_for_export	Readiness checklist green
ready_for_export	Final export / submitted	submitted	Final history snapshot committed
submitted	Tender outcome recorded	won / lost / withdrawn	ProcurementOutcomeDialog + metadata locked

Back-Transitions

Current State	Event	Next State	Purpose
in_review	Reviewer requests edits	drafting	Return for revisions
ready_for_export	Issues found	in_review	Re-enter review after export check
submitted	Submission retracted	in_review	Allow corrections post-submission

Universal Withdraw

Any non-terminal state can transition to withdrawn (encoded explicitly in VALID_TRANSITIONS in lib/procurement/procurement-workflow.ts).

Terminal States

won, lost, and withdrawn are terminal — no outbound transitions. Metadata is locked. ProcurementOutcomeDialog records the terminal transition; POST /api/procurement/[id]/outcome/integrate optionally folds outcome learnings back into the KB (e.g. “this Q&A led to a win”).

State machine source: lib/procurement/procurement-workflow.ts. Tests: __tests__/lib/procurement/procurement-workflow.test.ts.

---

Workflow 2: 3-Pass AI Drafting Pipeline

Trigger: “Draft” button in /procurement/[id]/session (single question), “Draft All” on /procurement/[id] (batch), or POST /api/procurement/[id]/responses/[rId]/regenerate (redraft). Owner: lib/ai/draft.ts (runDraftingPipeline, analyseQuestion, draftResponse, draftResponseStreaming) + lib/ai/match.ts (matching helpers).

Flow

Trigger → KB Match → Pass 1 (Sonnet, analyse) → Pass 2 (Opus, draft+citations)
       → Pass 3 (Haiku, quality check) → Stream/return → Persist + history snapshot

Detailed Steps

1. KB Match (only on initial draft, not on regenerate)

   • File: app/api/procurement/[id]/questions/match/route.ts
   • Action: generateSearchQueries(question_text) (Claude) → for each query: generateEmbedding(query) (text-embedding-3-large) → supabase.rpc('search_for_form_response', { query_embedding, query_text, limit_count: 5 }).
   • search_for_form_response carries include_superseded BOOLEAN DEFAULT false (S186 WP-B.3). Form drafting does not opt in, so superseded KB content is excluded from match results by design.
   • Output: assessConfidence(matches) returns one of strong_match / partial_match / needs_sme / no_content based on MATCH_THRESHOLDS (strong: 0.7, partial: 0.5, minimal: 0.3). deduplicateResults keeps the highest-similarity row per content ID.
   • Side effect: writes confidence_posture + matched_content_ids (uuid[]) to form_questions.

2. Pass 1 — Question Analysis (Sonnet)

   • File: lib/ai/draft.ts:124 (analyseQuestion)
   • Input: question_text, word_limit, section_name, matched content summaries.
   • Output (structured via output_config: { format: { type: 'json_schema' } }): QuestionAnalysis (primary topic, content types needed, response structure with heading + word allocation, key points, tone).

3. Pass 2 — Response Drafting with Citations (Opus)

   • File: lib/ai/draft.ts:200 (draftResponse) and :330 (draftResponseStreaming for SSE).
   • Input: question, analysis, matched DraftableContent[] with full text.
   • Action: Anthropic Opus with citations enabled. Citations + structured outputs are mutually exclusive in the Anthropic API, hence the two separate API calls (Pass 1 + Pass 2).
   • Output: responseText (markdown), CitationEntry[] (mapped to content IDs).
   • Streaming variant: emits chunks via SSE consumed by useDraftStream hook in the session UI; StreamingPhaseIndicator reflects pipeline phase.

4. Pass 3 — Quality Check (Haiku)

   • File: lib/ai/quality-check.ts (checkResponseQuality)
   • Action: Deterministic checks (citation coverage, word-count vs limit) plus Haiku-driven AI quality scan.
   • Output: overall_score written to form_responses.overall_score; issues recorded in form_responses.metadata.

5. Persist + History Snapshot

   • Side effects:
     • form_responses row updated/created (markdown in response_text, version incremented, drafted_by set, source_content_ids populated, metadata carries pipeline diagnostics).
     • form_response_history append-only snapshot inserted with change_reason (e.g. 'redraft', 'edit', 'restore').
     • citations rows linking each form response (citing_form_response_id) to its matched content item (cited_content_item_id) via cite_content (MCP) or in-line Citation Panel.

Batch Drafting (draft-all)

POST /api/procurement/[id]/responses/draft-all runs all undrafted questions sequentially. A 100-second safety timeout enforced; if exceeded, processing stops and partial results return via warningsEnvelope(). Per-question failures surface in WarningsBanner (components/dashboard/warnings-banner.tsx) so users can identify and retry individually.

Regeneration

POST /api/procurement/[id]/responses/[rId]/regenerate wraps runDraftingPipeline with rate-limit (5/min/user) and maxDuration = 120. Validates body via ResponseRegenerateBodySchema. Re-uses matched_content_ids from the existing form_questions row — does not re-match.

Error Handling

Error Condition	Handling	User Feedback
search_for_form_response failure for a query	Throws → Promise.allSettled records no_content	Confidence posture downgraded; no silent fallback
Pass 1 schema parse failure	Falls through to default QuestionAnalysis	Draft proceeds with degraded structure
Pass 2 streaming abort (network drop, tab close)	useDraftRecovery snapshots last partial draft	DraftRecoveryDialog on remount
Pass 3 quality flag	Response still saved; flagged in metadata	Reviewer sees flag in session UI
Rate-limit hit (regenerate)	rateLimitResponse(rl.resetAt) (HTTP 429)	Toast with reset countdown
Anthropic 5xx	Bubbles up via safeErrorMessage	Toast: “Drafting failed; please retry”

Database Operations

Operation	Table	Columns	RLS / Trigger
INSERT / UPDATE	form_questions	confidence_posture, matched_content_ids	Editor+ via getAuthorisedClient
INSERT	form_responses	response_text, review_status, version, metadata, overall_score, source_content_ids	Editor+; updates version on edit
INSERT	form_response_history	version, change_reason, response_text	Append-only; change_reason mandatory (S153 guard)
INSERT	citations	citing_form_response_id, cited_content_item_id, citation_type	Editor+ via cite_content MCP tool

---

Workflow 3: Q&A Library Field Edit (S198 §1.5)

Trigger: Editor clicks “Edit” on Standard or Advanced answer panel in QAAnswerDisplay (used by both /library row drill-down and /item/[id] for Q&A pairs). Owner: components/qa/qa-answer-display.tsx + useInlineFieldEdit.

Flow

Click Edit (Standard) → Mount ContentEditor → Edit markdown → Save
  → ≥20% shrink? → BLOCK + toast SAVE_SAFETY_BLOCK_MESSAGE
  → Else → PATCH /api/items/[id] { field: 'answer_standard', value, regenerate_embedding? }
  → Server rebuilds content_items.content (Q: + standard + advanced)
  → If regenerate_embedding: re-embed → refresh chunks
  → Return → Editor unmounts → Read mode resumes

Detailed Steps

1. Edit start

   • Action: inlineEdit.startEdit(field, currentValue) where field ∈ {'answer_standard', 'answer_advanced'}.
   • State change: editingField = field, editValue = currentValue, regenerateEmbedding = false.

2. Mount ContentEditor

   • The dynamic-imported ContentEditor (Tiptap) mounts with field as a remount key (<ContentEditor key={field} ...>) so switching from Standard to Advanced gets a fresh editor with a fresh baseline.
   • The editor is the canonical Tiptap+@tiptap/markdown component (components/item-detail/content-editor.tsx) used elsewhere for the content field. Same toolbar, same markdown serialisation.

3. Save-safety guard

   • Pre-PATCH check: shouldBlockSave(currentLength, baselineLength, baselineRatio = 0.8). If new length < 80% of baseline, save is blocked and SAVE_SAFETY_BLOCK_MESSAGE toast surfaces.
   • Editor stays mounted; user can adjust.

4. PATCH

   • Endpoint: PATCH /api/items/[id] with body { field, value, regenerate_embedding? }.
   • Server-side: rebuilds content_items.content:

     parts = []
     if (question) parts.push(`Q: ${question}\n\n`)
     if (answer_standard) parts.push(answer_standard)
     if (answer_advanced) parts.push(answer_advanced)
     content = parts.join('\n\n')

   • If regenerate_embedding=true: re-embeds via text-embedding-3-large and refreshes downstream chunks (document_chunks).

5. Cleanup

   • On success: editingField = null, regenerateEmbedding reset.
   • The reset is critical (S198 verifier H1 fix): the flag must NOT leak to a subsequent save on a different field via the shared useInlineFieldEdit instance.

State Transitions (per field)

State	Event	Next State	Side Effects
idle	startEdit(field)	editing	ContentEditor mounts, baseline captured
editing	cancelEdit	idle	Editor unmounts, regen flag cleared
editing	saveEdit (≥20% shrink)	editing	Toast block message; no PATCH
editing	saveEdit (allowed)	saving	PATCH in flight
saving	success	idle	Editor unmounts, regen flag cleared
saving	failure	editing	Toast error; baseline + regen retained

Database Operations

Operation	Table	Columns	Notes
UPDATE	content_items	answer_standard OR answer_advanced, content (rebuilt)	content_text_hash is GENERATED ALWAYS — never written explicitly
UPDATE	content_items	embedding (when regenerate_embedding=true)	Triggers chunk refresh
INSERT	content_history	version, change_reason	S153 guard requires change_reason
DELETE+INSERT	document_chunks	content, embedding	Only when embedding regenerates

---

Workflow 4: Tender Extraction

Trigger: User uploads tender file via TenderUpload → POST /api/procurement/[id]/questions/extract. Owner: lib/ai/extract-questions.ts + app/api/procurement/[id]/questions/extract/route.ts.

Flow

Upload PDF/DOCX → Store in workspace storage → Extract via Claude
  → Parse structured questions → Insert form_questions rows
  → Optional metadata extraction (TenderMetadataPrompt)

Detailed Steps

1. Upload + persist — POST /api/procurement/[id]/tender stores the file in Supabase storage; metadata recorded in workspaces.domain_metadata.tender_document.
2. Extract questions — Claude PDF/DOCX extraction returns structured Q&A rows with section_name, question_sequence, and question_text.
3. Persist questions — bulk INSERT into form_questions with confidence_posture: NULL (matching not yet run).
4. Optional tender metadata — TenderMetadataPrompt surfaces AI-extracted buyer/deadline/award_basis fields for the user to confirm.

Q&A Library extraction (separate path)

A form-library import path extracts Q&A pairs from client DOCX files and inserts them into content_items. (The earlier standalone Python ingest script was superseded by the CocoIndex pipeline under scripts/cocoindex_pipeline/.) Where a TypeScript extractor handles DOCX conversion, it preserves links, lists, and nested tables beyond plain bold/italic.

---

Workflow 5: Export

Trigger: ProcurementExportMenu action. Owner: lib/procurement/procurement-export-docx.ts, lib/procurement/procurement-export-xlsx.ts, lib/procurement/procurement-export-data.ts.

DOCX flow

1. POST /api/procurement/[id]/export/docx
2. procurement-export-data.ts assembles the export payload (questions, responses, citations, metadata) with sb() for fail-fast.
3. markdownToDocxParagraphs(markdown) converts markdown to docx paragraphs (preserves headings, lists, bold/italic, links, GFM tables). Plain-text fields use stripMarkdown.
4. Returns the binary stream; client triggers a download.

XLSX flow

1. POST /api/procurement/[id]/export/xlsx
2. lib/procurement/procurement-export-xlsx.ts uses exceljs to build sheets per question with plain-text fields (no markdown formatting in cells).
3. Returns the workbook binary.

Template fill

1. POST /api/procurement/[id]/templates/[templateId]/auto-map maps template fields to KB content via lib/templates/template-auto-map.ts.
2. POST /api/procurement/[id]/templates/[templateId]/fill performs the actual write-back. Currently uses a Python python-docx-based path for the final write-back, which is out-of-band relative to pure Vercel deployments (Railway worker or local script execution).
3. template_completions records the storage_path.
4. GET /api/procurement/[id]/templates/[templateId]/completions/[completionId]/download serves the filled file.

---

Cross-cutting: Publication-Lifecycle (S205-S208 §5.2)

content_items.publication_status ('draft' | 'in_review' | 'published' | 'archived', NOT NULL DEFAULT 'published') enforces lifecycle visibility across the KB. Form drafting consumes the same enum:

• search_for_form_response (RPC) joins content_items and inherits the publication-status filter once Phase 3 (RPC visibility flip) ships. Currently, the RPC returns all non-superseded rows; downstream draft prompts are not yet filtered by publication_status='published'.
• The update_publication_status MCP tool + PATCH /api/items/[id]?field=publication_status go through lib/governance/publication-transitions.ts for the role matrix (admin/editor/viewer × draft/in_review/published/archived).
• All 6 production writers target the new column; AC6.5 grep guard test enforces zero write-position regressions.

Phases 3 (RPC visibility flip), 4 (publication-review queue tab, EP2-bundled), and 5 (supersession + cron-exclusion) remain on the roadmap (§5.2 Phase 3+4+5).

Cross-cutting: Recurring Review Cadence (S205-S208 §5.5)

Q&A pairs in the form drafting feed get a next_review_date via the §5.5 Phase 1 backfill (Q&A cohort: 395 items at 180-day cadence). Articles, blogs, and research are intentionally NOT backfilled.

• Daily cron app/api/cron/review-cadence/route.ts (03:45 UTC) flags overdue items as governance_review_status='review_overdue' for items currently NULL or 'approved'.
• The cadenceCompliancePenalty(nextReviewDate, now) helper in lib/quality/quality-score.ts applies a 0/-5/-10/-15/-25/-40 penalty per spec §9.3 schedule. The freshnessRaw() quality-score branch applies the penalty only when next_review_date is non-null.
• search_content_chunks (S208) gained filter_overdue_review + filter_review_due_within_days so MCP clients (including those drafting form responses) can request lifecycle-filtered chunks.

Automated Processes

Process	Schedule	Route / Script	Form-completion relevance
Review cadence cron	Daily 03:45 UTC	app/api/cron/review-cadence/route.ts	Flags overdue Q&A pairs (those used in form drafts)
Quality score cron	Daily	app/api/cron/quality-score/route.ts	Recomputes quality scores for all content (incl. Q&A)
Freshness cron	Daily	app/api/cron/freshness/route.ts	Updates freshness state for form-source content
Form-library import	Manual	scripts/cocoindex_pipeline/ ingest path	Idempotent batch import of Q&A pairs from client DOCX

Integration Points

External System	Direction	Protocol	Purpose
Anthropic SDK	Request	HTTPS	Question extraction; analyseQuestion (Sonnet); draftResponse (Opus); checkResponseQuality (Haiku)
Anthropic SDK	Stream	HTTPS SSE	draftResponseStreaming for the session UI
OpenAI API	Request	HTTPS	Embeddings (text-embedding-3-large) for query embedding + per-Q&A-pair embedding regen
Supabase Storage	Write/Read	REST	Tender file storage (POST/GET /api/procurement/[id]/tender)
Python environment	Local invoke	exec	DOCX table extraction, template write-back (form-library ingest, python-docx)
Sentry	Outbound	HTTPS	Release tagging + correlation-id-tagged error reporting (kh-prod-readiness S10-S13)

Current Limitations

• Template write-back relies on Python — the python-docx-based path is out-of-band relative to pure Vercel deployments. Requires Railway worker or local invocation. No Vercel-native equivalent yet.
• search_for_form_response does not yet filter by publication_status — Phase 3 (RPC visibility flip) is still on roadmap. Drafting may surface 'in_review' or 'draft' KB content. Mitigation: form drafting rarely surfaces non-published content because authors keep newly added Q&A pairs in 'published' by default.
• Form drafting eval baseline deferred — a synthetic gold-standard fixture exists (24 items) but the baseline awaits real form data per SoTP §AI Evaluation.
• Q&A field edit save-safety guard threshold (20% shrink) — fixed in code at SAVE_SAFETY_THRESHOLD = 0.8. No per-field override yet; users occasionally hit false positives when intentionally trimming verbose pre-existing answers.
• Concurrency — no row-level optimistic-concurrency guard on form_responses updates (a stale PATCH wins). Mitigated in practice by per-form editor headcount; problematic only under genuine concurrent edits.
• §1.7 admin dedup review + §1.9 near-duplicate merge dashboard workflows — specs ratified S209 WP2 but not yet implemented. Duplicate Q&A pairs may surface as form matches.