Form-Extraction Subsystem — RESEARCH

Form-Extraction Subsystem — RESEARCH

Type: RESEARCH (investigation + options framing). This document precedes PRODUCT.md. Its job is to arm the PRODUCT-spec author and Liam with the empirical facts, the live-code/schema reality, and — for each genuine product decision — a set of options plus a recommendation. It does not ratify product direction; the genuine product decisions in §6 are Liam’s call via the parent session’s OQ-escalation channel.

Authored: S273, Subtask ID-52.1 (form-extraction spec track). Fresh Planner context. UK English throughout.

Primary sources (read in full):

• docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md (S272 — WINS over older specs) — §2.2, §2.3, §2.4, §5(a)/(c), §7.
• docs/themes/canonical-pipeline/phase-0-investigation/feedback-findings-review.md (S234 — Theme A/B + §5 post-prereq rollup).
• docs/themes/canonical-pipeline/reference/decision-graph.md — OQ10 (line 468), OQ11 (line 469, RESOLVED-DIRECTION-S233).

Empirical corpus inspected: the 4 blank-form template sets in docs/testing/test-data/templates/ (opened + characterised, §2). Live schema verified against: supabase/types/database.types.ts (line anchors cited, §3). Live code traced: scripts/cocoindex_pipeline/{extraction,flow}.py, app/api/procurement/[id]/templates/[templateId]/analyse/route.ts, lib/ontology/* (§4). Pre-ratification empirical import checks: §8 (OQ-3 / Q-EX2 forcing function).

---

1. Problem framing

The form-question subsystem is the unbuilt machinery that turns a blank procurement form (an SQ / ITT / PQQ / checklist that a buyer issues) into structured, matchable question records in the knowledge base. It is Mode-3 in the §5(c) terminology — a blank form whose questions we want to catalogue and later match against the corpus — as distinct from Mode-1 (an answered form, where the answers are themselves citable Q&A facts).

Per sequencing §2.3, extraction is three separate concerns, and only the first is built:

• Path A — Anthropic @coco.fn extraction (BUILT). The general content stage: classify · qa_pairs · entity-mentions, over already-markdown text (extraction.py:63, claude-opus-4-6). Three deliberate calls, not one mega-prompt.
• Path B — Deterministic extractors (UNBUILT). Pulling question text out of blank forms. OQ11 (decision-graph line 469, RESOLVED-DIRECTION-S233) names XLSX via exceljs
  • PDF via pdf-parse/extract_pdf_text.py.
• Path C — AI-cataloguing skill (UNBUILT). A Claude Plugin Skill authoring requirement_type + taxonomy + matching keywords with human confirmation; output is a generated scripts/catalogue-<slug>.ts seed script.

The “form-review skill” = Path B + Path C. This subsystem is v1 (Liam has form examples; form extraction + matching are on the critical path — §5a) and pairs with T10 question-matching. No task record exists yet beyond the parent Task ID-52.

Relation to the built Path A. Path A already runs extract_qa_form() over every ingested document’s markdown and writes Q&A pairs to q_a_extractions (flow.py:748,790). But Path A does not persist form structure (no form_templates / form_template_fields write) and is lossy on the per-question metadata that forms carry (mandatory flags, word limits, evaluation criteria — see §4.1). The form subsystem is therefore not a greenfield build on top of nothing: it must reconcile with a partially-wired Path A that already touches the same Q&A substrate.

---

2. Empirical corpus inventory

Four real UK procurement form sets in docs/testing/test-data/templates/. Every file below was opened and structurally characterised (pdfplumber / openpyxl / xlrd / python-docx). Formats present: PDF, XLSX, legacy XLS, DOCX — and NO HTML.

2.1 SQ — Standard Selection Questionnaire (PDF)

sq-standard-selection-questionnaire/standard-selection-questionnaire-ppn-03-24.pdf

• Format: PDF 1.4. The OS file tool reports 8 pages; pdfplumber reads 57 pages. The 57-page figure is authoritative for extraction (the 8-page header is a container artefact). Correction to the brief’s “8 pages” note.
• Text is extractable (not scanned/image-only) — page.extract_text() returns clean prose, and page.extract_tables() finds tables on the question pages.
• Structure: front-matter PPN guidance (pp.1–16) → Annex B “Standard Selection Questions” (begins ~p17) → Annex C “Selection Questionnaire Template / SPD” (begins ~p30) → Annex D exclusion grounds.
• Question shape (Annex B): a 2-column table where the left cell carries the mandatory flag (M / O) and the right cell carries the question text. pdfplumber’s table extraction correctly associates the M/O flag with the question (e.g. ['M', 'Registered VAT number']). Sub-parts use bullet enumeration (a/b/c…) and nested M/O flags per sub-item.
• Word limits: present as inline tokens in the question text, e.g. 6.2 … in no more than [500] words … (pp.24–25, 45–46). Not a separate column — embedded in prose.
• Hierarchical numbering: 6.1 / 6.2 / 6.3 (section.question).
• Historical hand-catalogue (OQ11): SSQ = 66 reqs. Plausible against the observed Annex B + Annex C density (dozens of M/O rows plus the template fields). Sanity-check: consistent — the document is question-dense; 66 is the right order of magnitude.

2.2 ITT EFA — evaluation matrix (XLSX)

itt-services-efa/evaluation-matrix-itt-vol8.xlsx

• Format: modern XLSX (openpyxl reads cleanly).
• Sheets: Title Sheet, Summary, Bidder 1, Bidder 2.
• Heavily merged: Bidder 1 / Bidder 2 each carry 42 merged-cell ranges.
• Question shape: a “Scoring Matrix” block with Ref | Criteria | … | Weighting | Score | Weighted Score | Remarks columns. Question number is a dedicated cell (2.1, 3.1, 4.1…); criteria text in an adjacent cell; weighting (0.05, 0.35) and a page limit (“Page Limit” column in the General Compliance Check block).
• Section structure: Part 2 — OVERALL APPROACH, Part 3 — DESIGN AND COST MANAGEMENT, etc., as banner rows interleaved with question rows.
• Duplication hazard: the same question set is repeated across Bidder 1 and Bidder 2 (two scoring copies). A naïve sheet-walk would extract every question twice — extraction must dedupe across bidder sheets.
• Summary sheet aggregates per-Part weighted scores — metadata, not questions.

2.3 CSP — Cloud Security Principles Checklist (XLSX)

csp-checklist/Cloud Security Principles Checklist V5_3.xlsx

• Format: modern XLSX. Sheets: Vendor Checklist, Sheet1 (a 3-value lookup list Yes/No/N\A).
• 23 merged-cell ranges.
• Mixed numbering scheme in one sheet: a free-text preamble block keyed by letters (A General Data Security, B GDPR Compliance, … with sub-items B1/, B2/…) followed by numbered NCSC principles (PRINCIPLE 1 … PRINCIPLE 14, sub-items 2.1, 2.2…).
• Embedded hyperlinks: principle rows carry NCSC reference URLs in a cell (https://www.ncsc.gov.uk/collection/cloud-security…). (Verified URL-recall caveat: the S234/S235 Docling bake-off found 14/15 URL recall — a row-39 anchor quirk shared by all tools.)
• Answer-placeholder cells: TYPE RESPONSE HERE>>>> marks every response column — a strong signal of where a blank field sits (relevant to fill_status / field_type).

2.4 ITT Charnwood — evaluation matrix (legacy XLS) + ITT Services (DOCX)

itt-services-charnwood/ITT Evaluation Matrix.xls + itt-services-charnwood/ITT Services.docx

ITT Evaluation Matrix.xls (legacy binary Excel, BIFF):

• CANNOT be read by the installed Python stack out of the box. pandas.read_excel fails (ImportError: Install xlrd >= 2.0.1); openpyxl does not support .xls. I had to install xlrd 2.0.2 to the user-site to read it. This is itself a load-bearing finding for format coverage (§5 item #1): legacy .xls needs a different reader from .xlsx.
• Once read: 13 sheets (Summary, Tender Evaluation Matrix, + 11 Supplier N Evaluator comments sheets).
• CRITICAL — the .xls is a BLANK TEMPLATE, not a 30-requirement question set. Every criteria row is a placeholder: Description of criteria 1 … Description of criteria 10. The supplier sheets are identical 10-row score-capture grids. There are no real question texts in this file. This contradicts the brief’s “Charnwood ITT = 30 reqs” expectation — the 30 reqs from OQ11 were hand-catalogued at UAT S83 from the companion DOCX, not from this .xls scoring tool.

ITT Services.docx (DOCX) — the real question-bearing artefact:

• 1908 paragraphs + 8 tables. Structurally the most heterogeneous form in the corpus.
• Questions/sections live in BOTH paragraphs AND tables: numbered Word paragraphs (1 Preamble, 2 Conditions of Tender, 5 Business Questionnaire, 6 Legal Obligations, 7 Pricing Schedule …) interleaved with Heading 3 / Body Text 3 styled prose, plus 8 tables (timetable, schedule-completion checklist, a blank question-weighting grid Insert question title | Insert %, a 0–10 scoring rubric, a pricing grid).
• Many cells are blank placeholders ([Insert date], Insert question title, [identification of requirement]) — this DOCX is itself a template to be filled by the issuing authority, so the “questions” are partly structural scaffolding rather than fully-authored question text.

2.5 Corpus summary table

Set	File	Format	Reader	Question carrier	Numbering	Mandatory flag	Word/limit	Dedup hazard
SQ	standard-selection-questionnaire-ppn-03-24.pdf	PDF (57pp)	pdfplumber ✓	2-col table (Annex B/C)	6.1/6.2	M/O left cell	inline [500] words	—
ITT EFA	evaluation-matrix-itt-vol8.xlsx	XLSX	openpyxl ✓	Scoring-Matrix rows	2.1/3.1	(via weighting)	“Page Limit” col	Bidder 1≡Bidder 2
CSP	Cloud Security Principles Checklist V5_3.xlsx	XLSX	openpyxl ✓	letter + numbered rows	A/B1/2.1	—	—	preamble vs principles
ITT Charnwood	ITT Evaluation Matrix.xls	legacy XLS	xlrd req’d	(blank placeholders only)	n/a	—	—	11 supplier copies
ITT Charnwood	ITT Services.docx	DOCX	python-docx ✓	paras and tables	1/5/6	—	scoring rubric	template placeholders

Headline empirical findings for the spec:

1. Four distinct formats, four distinct structural idioms. There is no single table-extraction shape that covers PDF-Annex-tables, XLSX-scoring-matrices, XLSX-checklists, and DOCX-paras-plus-tables. A per-format strategy is unavoidable.
2. No HTML in the corpus. OQ11’s XLSX+PDF plus the corpus’s DOCX+XLS is the real v1 surface; HTML is theoretical (Theme A lists it, but no example exists).
3. Legacy .xls is a real coverage gap — a different binary format needing xlrd, not exceljs/openpyxl.
4. Forms carry rich per-question metadata (M/O flags, weightings, word limits, page limits, section hierarchy, evaluation rubrics, embedded URLs) that maps directly onto form_template_fields columns (§3.2) — and that a flat markdown conversion risks losing (§5 item #2).
5. Blank forms are sparse and placeholder-heavy. “Extracting questions” from a blank template is partly “extracting the structure”; some cells are scaffolding (Insert question title), not authored questions.

---

3. Live schema reality (verified against database.types.ts)

All four tables exist at the line anchors below (verified this session; do not re-derive shapes). No barrel imports — consume row shapes via Tables<'x'> off @/supabase/types/database.types.

3.1 The instance-vs-catalogue distinction (the load-bearing nuance)

There are two different shapes in play, and they are not the same:

• Instance side (per ingested form): form_templates + form_template_fields. A specific file a user uploaded into a workspace. Workspace-scoped, file-backed (storage_path, mime_type, file_size). This is the PIPELINE WRITE target (§5a).
• Catalogue side (global, reusable): form_template_requirements. The reusable requirement catalogue — no workspace FK, carries requirement_embedding (pgvector). This is the MATCHING (T10) READ target, not a pipeline write sink.

3.2 form_templates (instance; line 1686) — PIPELINE WRITE target

id, workspace_id (string, NOT NULL → FK workspaces), created_by (nullable FK user_profiles), name, filename, file_size, mime_type, storage_path, structure_path (nullable), description (nullable), field_count (nullable), mapped_count (nullable), status, timestamps.

• FK verified: form_templates_workspace_id_fkey → workspaces; form_templates_created_by_fkey → user_profiles.
• workspace_id NOT NULL is the workspace-resolution forcing function (§5 item #3).

3.3 form_template_fields (per-field instance rows; line 1519) — PIPELINE WRITE target

id, template_id (NOT NULL → FK form_templates), question_id (nullable → FK bid_questions — legacy naming), question_text (nullable), placeholder_text (nullable), field_type, fill_status, fill_error (nullable), mapping_confidence (nullable), mapping_status, row_index / col_index / table_index (nullable), section_name (nullable), sequence, word_limit (nullable), timestamps.

• FK verified: form_template_fields_question_id_fkey → bid_questions (not q_a_pairs); form_template_fields_template_id_fkey → form_templates.
• The row_index / col_index / table_index / word_limit columns are exactly the coordinate + limit metadata that the raw XLSX/PDF/DOCX forms carry (§2) and that a markdown conversion would flatten — direct evidence for §5 item #2.

3.4 form_template_requirements (GLOBAL catalogue; line 1597) — MATCHING (T10) READ target

id, requirement_text, description (string|null), requirement_type (plain string), requirement_embedding (string|null — pgvector serialised), matching_keywords (string[]|null), matching_guidance (nullable), word_limit_guidance (number|null — a T10-side matching hint, distinct from the instance-side form_template_fields.word_limit), template_name, template_type (→ FK form_types.key), template_version (nullable), section_name, section_ref, question_number (nullable), display_order, primary_domain/primary_subtopic/secondary_domain/secondary_subtopic (nullable), sector_applicability (string[]|null), is_mandatory (nullable), is_current (nullable), timestamps.

• Correction to the brief: the FK discriminator is template_type → form_types.key (verified form_template_requirements_template_type_fkey), not requirement_type. requirement_type is a plain string column (conceptually CV-12 requirement_type: 7 values policy/statement/evidence/data/narrative/declaration/reference), with no FK. So the catalogue has two type axes: template_type (FK to form_types, the form kind)
  • requirement_type (free string, the requirement kind).
• No workspace FK (verified — Relationships lists only the template_type FK). This is what makes it the reusable, embedding-bearing catalogue.

3.5 form_types (CV; line 1755)

key, label, applicable_application_types (string[]), provenance, created_at. Relationships: []. The table exists and is live (verified) — see §4.2 for the CV-file drift this exposes.

3.6 Schema relationship picture

   (user uploads a form into a workspace)
              │
   form_templates ──FK workspace_id──▶ workspaces        [INSTANCE, pipeline WRITE, NOT NULL ws]
        │  └─FK created_by──▶ user_profiles
        ▼ FK template_id
   form_template_fields ──FK question_id──▶ bid_questions  [per-field INSTANCE rows, pipeline WRITE]
        (row/col/table_index, word_limit, field_type, fill_status, section_name, sequence)

   form_template_requirements ──FK template_type──▶ form_types   [GLOBAL CATALOGUE, T10 READ]
        (requirement_embedding, matching_keywords, is_mandatory, NO workspace FK)
                              ▲
                              │  matched against (T10)
   form_template_fields.question_text / q_a_extractions / q_a_pairs

The open architectural question this raises (§5 item #4): how does Path C (AI-cataloguing) relate the instance (form_template_fields) to the catalogue (form_template_requirements)? Does Path C write the catalogue (from instances, with human confirmation), and does T10 then match instance fields → catalogue requirements? The schema permits this reading, but no code today writes form_template_requirements from the pipeline — the catalogue is currently seeded by the hand-written scripts/catalogue-*.ts scripts.

---

4. Live code reality

4.1 Mode-3 FormMetadata persistence ORPHAN (more nuanced than the brief states)

extraction.py (Path A, BUILT):

• class FormMetadata (extraction.py:121) exists and is richer than the brief implies: form_type (an 11-value Literal — bid/rfp/pqq/itt/tender/framework/dps/gcloud/ checklist/questionnaire/sales_proposal_template), form_format (docx/xlsx/pdf/html/md), form_title, issuing_organisation, deadline, evaluation_methodology.
• class QAPair (extraction.py:150) carries question_text, answer_text, expected_response_kind (mandatory/optional), evaluation_criteria, evidence_requirements: list[str], scope_tags: list[str].
• class QAFormExtraction (extraction.py:168) bundles form_metadata: FormMetadata + qa_pairs: list[QAPair]; its docstring states it “maps downstream to q_a_extractions (per QAPair) + form_templates (per FormMetadata)”.

flow.py (the drop):

• extract_qa_form(content_text) is called (flow.py:748).
• Only qa_pairs are persisted, and lossy: the loop at flow.py:790–805 writes each pair to q_a_extractions with only extracted_question_text + extracted_answer_text
  • an extraction_metadata JSONB blob. It drops expected_response_kind, evaluation_criteria, evidence_requirements, scope_tags from every pair.
• form_metadata is discarded entirely — form_type, form_title, deadline, evaluation_methodology, issuing_organisation go nowhere.
• There is NO form_templates mount and NO form_template_fields mount. mount_table_target is called for exactly three tables — content_items (877), q_a_extractions (883), source_documents (889) — and nothing else (verified by grep: zero form_templates / form_template_fields references in flow.py).

So the orphan is precise: the type exists and is exercised, but the persistence of form structure is absent, and the Q&A write that does happen is lossy on per-question metadata. Backlog ID-184 (HIGH, “needs Liam pre-first-ingest”) tracks this.

The spec-says-pipeline / code-does-app-side drift (§5a): the only thing that writes form_templates + form_template_fields today is the app-side route app/api/procurement/[id]/templates/[templateId]/analyse/route.ts. It does not itself parse the form — it validates auth (getAuthorisedClient(['admin','editor']), checks auth.success, routes via authFailureResponse), clears existing fields, flips form_templates.status to analysing, and enqueues a processing_queue job (job_type: 'template_analyse', payload: { template_id, workspace_id: procurementId, storage_path }). The actual field-extraction happens in whatever consumes that queue job. Sequencing §5a says the build migrates this ownership to the pipeline.

Fix options (framed in §5 item #5).

4.2 Ontology / CV loading + the failing parity test (ROOT CAUSE FOUND)

• CV loading path: lib/ontology/loader.ts loadOntologyCVs() reads every docs/ontology/*.md (except README.md), parses frontmatter with gray-matter, and validates each against OntologyCVSchema (lib/ontology/schemas.ts). It throws fatally on the first file that fails, with the offending filename. The content-type consumer (lib/ontology/content-type-registry.ts) re-exports a build-time generated tuple (content-type-values.generated.ts) to keep node:fs out of client bundles; the loader runs Node-side (tests, prebuild).

• The form-type CV is docs/ontology/26-form-type.md — 11 baseline values, keyed by the markdown register, status: APPLIED-S246.

• THE FAILING TEST IS A REAL LOADER CRASH, AND IT IS GATED ON THE FORM-TYPE CV. Running bun run test __tests__/lib/ontology/markdown-parity.test.ts yields:

  Error: [ontology/loader] Schema validation failed for 26-form-type.md:
    - status: Invalid option: expected one of "active"|"planned"|"needed"

  The Zod schema pins STATUS_VALUES = ['active','planned','needed'] (lib/ontology/schemas.ts:23), but 26-form-type.md (and three siblings) use APPLIED-S{NNN}. Because the loader throws on the first failure (alphabetically 26-form-type.md), the entire ontology register fails to load — not just form-type. This is not a benign parity drift; it is a hard crash that takes down content_type, taxonomy, requirement_type — everything that calls loadOntologyCVs().

• Blast radius — exactly 4 CV files carry a non-enum status: 26-form-type.md (APPLIED-S246), 30-procurement-vehicle.md (APPLIED-S246), 31-procurement-vehicle-instance.md (APPLIED-S246), 32-q-a-pair.md (APPLIED-S249). Three of the four are form/procurement-subsystem CVs. Whichever fix is chosen (extend the enum, or normalise the four files), it must cover all four.

• Two stale claims inside 26-form-type.md itself (the CV file pre-dates the migration it documents):

  1. “form_types table (new — to be added in Phase 1 step 3; not yet in production schema)” — stale. form_types IS live (database.types.ts:1755, verified §3.5). status: APPLIED-S246 is the accurate marker.
  2. “Adjacent live table: template_requirements.template_type CHECK has 10 values (sq, rfp, eqq, pqq, gcloud, …)” — stale. template_requirements was renamed to form_template_requirements in S246 (verified: no template_requirements / template_fields / templates tables remain in database.types.ts), and its template_type is now an FK to form_types.key. The 10-value free CHECK is gone.

• Python-side CV read: the pipeline does not call the TS loader. The markdown-parity test cross-checks the markdown register against scripts/tests/fixtures/taxonomy_snapshot.json (per the CLAUDE.md taxonomy dual-source note). The Python pipeline reads taxonomy from that snapshot fixture, and FormMetadata’s form_type is a hard-coded Literal in extraction.py — not loaded from the CV at all. So the 11-value form_type exists in three places that can drift: the markdown CV (26-form-type.md), the live form_types table, and the Python Literal.

4.3 Existing deterministic-extractor assets (OQ11 “evaluate for patterns”)

• scripts/extract_tender_questions.py — a DOCX table extractor (not PDF, despite the OQ11 “PDF via pdf-parse” framing). Reads .docx tables, normalises a large set of question/requirement header variants (_QUESTION_HEADERS), captures word limits and section hierarchy, emits structured JSON. Directly reusable pattern for the DOCX path.
• scripts/analyse_template.py — builds on extract_tender_questions.py (_classify_header); detects merged cells (_detect_merged_cells), empty/placeholder cells (_is_empty_or_placeholder), word limits (_extract_word_limit), section headings (_extract_section_headings → table_index → section_name), and tracked-changes (_has_tracked_changes). Classifies question vs answer columns with a populated-column heuristic. This is a working blank-form structure analyser for DOCX — strong prior art for form_template_fields population (it already computes table_index, section_name, word_limit, placeholder detection).
• scripts/extract_pdf_text.py — present (OQ11-named PDF helper).
• scripts/catalogue-standard-sq.ts + scripts/catalogue-charnwood-itt.ts — the hand-written historical seed scripts (UAT S77 / S83). Do not re-run (sequencing §2.3). They are prior art for the catalogue shape (Path C output), not extractors.

Note: these existing extractors are DOCX-centric, whereas OQ11’s forward path names XLSX (exceljs) + PDF (pdf-parse). The corpus’s heaviest question-bearing files are in fact two XLSX + one DOCX + one PDF — so the existing DOCX assets cover only one quarter of the format surface. This reinforces §5 item #1.

---

5. The six must-resolve items

Each item: Findings → Options → Recommendation → Open Question for Liam. Items flagged [RATIFY-BEFORE-PRODUCT] must be decided by Liam before PRODUCT.md can commit; items flagged [PRODUCT-MAY-PROPOSE] can be proposed in PRODUCT.md with Liam ratifying at the PRODUCT gate.

Item #1 — Format coverage (docx/html beyond OQ11’s XLSX+PDF)

Findings. The corpus spans PDF, XLSX, legacy XLS, DOCX — no HTML. OQ11 names only XLSX+PDF. Theme A lists docx+html too. The legacy .xls needs a different reader (xlrd) from .xlsx (openpyxl/exceljs); the OS could not read it without an added dependency. The existing extractor assets are DOCX-only. So the named forward path (XLSX+PDF) and the actual corpus (XLSX+DOCX+XLS+PDF) diverge.

Options.

• A — Corpus-driven v1 = {PDF, XLSX, DOCX}; XLS via one-time pre-conversion; HTML deferred. Cover the three formats that appear as question-bearing files. Treat legacy .xls as a manual “save-as-.xlsx” pre-step (it is rare and the one example is a blank scoring tool, not a question source). Defer HTML until a real HTML form appears.
• B — OQ11-literal v1 = {XLSX, PDF} only; DOCX + XLS + HTML deferred. Narrowest; matches the ratified OQ11 text verbatim. But drops the Charnwood DOCX (a real question-bearing artefact) and the existing DOCX extractor investment.
• C — All-four-plus-HTML now ({PDF, XLSX, XLS, DOCX, HTML}). Matches Theme A’s “cover ALL document types”. Highest cost; builds an XLS reader and an HTML path with no example to validate against.

Recommendation: Option A. Cover {PDF, XLSX, DOCX} for v1 (the formats that actually carry questions in the corpus), with legacy .xls handled by a documented one-time “save-as .xlsx” pre-step rather than a code path, and HTML explicitly deferred until a real HTML form exists to design against. This honours the spirit of OQ11 (deterministic per-format extraction) while reflecting the empirical corpus, and avoids building an unvalidated HTML path.

Open Question for Liam [RATIFY-BEFORE-PRODUCT]: Confirm v1 format scope = {PDF, XLSX, DOCX}, legacy .xls = manual pre-convert, HTML = deferred? Or do you want HTML in v1 scope despite no example (e.g. because real client forms will be web-form exports)?

Item #2 — Raw-format parsing VS converted-markdown surface

Findings. Sequencing §2.2 converts everything→markdown at Stage 2 (Docling for PDF/DOCX/XLSX; pullmd for HTML), and a single type-agnostic Path-A extraction consumes that markdown. But the corpus (§2) shows forms carry coordinate + flag + limit metadata — M/O mandatory flags in a specific cell, row_index/col_index/table_index, word_limit, weightings, page limits — that map directly onto form_template_fields columns (§3.3) and that a flat GFM-table markdown can lose or scramble (especially merged cells: the EFA sheet has 42 merged ranges; CSP has 23). The S234/S235 Docling bake-off found Docling preserves GFM tables and Q-number↔text association well for PDF, but XLSX merged-cell flattening and URL-only-cell drops were flagged surfaces. Conversely, raw parsing means a per-format extractor (exceljs/openpyxl + pdfplumber + python-docx), i.e. more code and the §2.4 reader-fragmentation cost.

Options.

• A — Raw-format parsing for blank-form structure (Path B), markdown for content (Path A). Split by purpose: the deterministic extractor reads the raw XLSX/PDF/DOCX to populate form_template_fields (preserving row_index/col_index/table_index/ word_limit/mandatory flag); Path A’s markdown stage continues to serve general content extraction. The two are different jobs on the same file.
• B — Markdown-only surface. Run the deterministic extractor (or an LLM cataloguer) over the §2.2 Docling markdown. Uniform, single conversion, reuses the built Stage-2 adapter. Loses cell coordinates and risks merged-cell/word-limit fidelity; row_index/ col_index/table_index would be best-effort or NULL.
• C — Hybrid: markdown for question text + a thin raw “coordinate sidecar”. Convert to markdown for the question text and section structure, but run a lightweight raw pass (openpyxl/python-docx) purely to recover the coordinate/limit/flag columns and stitch them back by question number.

Recommendation: Option A (raw-format parsing for Path B structure). The form_template_fields schema was designed around raw coordinates (row_index, col_index, table_index, word_limit) — those columns are meaningless on flattened markdown, and the M/O flag association (SSQ Annex B) and merged-cell handling (EFA/CSP) are exactly what raw parsers preserve and markdown loses. Path A’s markdown surface stays for general content. Caveat to record: this means the form subsystem does not simply reuse the §2.2 markdown pipeline — it adds per-format raw extractors, which has a deployment-footprint and maintenance cost the PRODUCT/TECH specs must own.

Open Question for Liam [RATIFY-BEFORE-PRODUCT]: Confirm the extraction surface for blank-form structure is the raw format (not the converted markdown), accepting a per-format extractor stack? This is the single most architecturally consequential decision and directly contradicts a literal reading of §2.2’s “everything→markdown” if not scoped to “general content extraction only”.

Item #3 — Pipeline-owned WRITE to form_templates + form_template_fields (incl. workspace_id resolution)

Findings. form_templates.workspace_id is NOT NULL → FK workspaces (§3.2). The app-side route resolves it trivially: the workspace IS the procurement workspace the user uploaded into — analyse/route.ts takes the [id] route param (procurementId) and uses it directly as form_templates.workspace_id (verified .eq('workspace_id', procurementId) and payload.workspace_id: procurementId). The user picks the workspace at upload time in the UI. The pipeline has no equivalent upload-context — cocoindex walks a localfs source folder (COCOINDEX_SOURCE_PATH); there is no inherent “which workspace does this file belong to” signal. Sequencing §5a says the build migrates ownership app→pipeline.

Options for workspace resolution.

• A — Folder-to-workspace convention. The localfs source is laid out so a folder path maps to a workspace (e.g. …/<workspace_id>/forms/… or a workspace.json manifest per folder). The pipeline resolves workspace_id from the file’s path. Matches the “external-folder canonical” intent (§2.2) but requires a folder→workspace mapping contract that does not exist yet.
• B — Pipeline writes workspace_id = NULL-equivalent via a sentinel/“ingest inbox” workspace, app assigns later. Since NOT NULL forbids true NULL, route all pipeline-ingested forms to a reserved “unassigned” workspace row, with an app-side step to reassign. Keeps the pipeline ignorant of workspace semantics. Adds a reassignment UC.
• C — Keep the app-side write for forms; pipeline does NOT write form_templates. The pipeline writes content/Q&A/source-docs (as today); form structure stays app-owned via the existing analyse/route.ts + queue worker. Drop the contract claim that the pipeline writes form structure. Cheapest; but contradicts §5a’s “pipeline owns the write”.
• D — Hybrid: app owns the upload + workspace assignment (writes the form_templates row with workspace_id), pipeline owns the field extraction (writes form_template_fields against that template_id). Splits the write along the natural seam: workspace resolution stays where the context lives (the app upload), field extraction moves to the pipeline (where Path B lives).

Recommendation: Option D (split the write along the workspace-context seam). The NOT NULL workspace_id constraint is the forcing function: workspace identity is an upload-time / app-side fact (the user chooses it), whereas field extraction is a pipeline-side fact (Path B parses the file). Having the app create the form_templates row (with workspace_id, storage_path, mime_type) and the pipeline populate form_template_fields + backfill field_count/structure_path/status is the cleanest ownership boundary and requires no folder→workspace convention. This is a refinement of §5a’s “pipeline owns the write” — pipeline owns the field write, app owns the template row + workspace write. Flag to Liam: if §5a intends a fully pipeline-owned write (localfs-driven, no app upload step), then Option A’s folder→workspace convention is needed and must be specified.

Open Question for Liam [RATIFY-BEFORE-PRODUCT]: For workspace resolution under the NOT NULL constraint — Option D (app creates the template row with workspace; pipeline fills fields), or Option A (pipeline resolves workspace from a folder convention and owns the whole write)? This determines whether a folder→workspace mapping contract must be designed.

Item #4 — Hand-off to matching (T10) against form_template_requirements

Findings. form_template_requirements is the global catalogue (no workspace FK, carries requirement_embedding, discriminated by template_type → form_types.key, §3.4). Nothing in the pipeline writes it today — it is seeded by hand (scripts/catalogue-*.ts). T10 (question_matches / matching) is the consumer that will match something against this catalogue. The §2.4 “Wikipedia Principle” (one record, many views) and the corpus-level q_a_pairs (no workspace FK) sit alongside this. There are therefore three candidate “question” sources that could feed matching: instance form_template_fields.question_text, corpus q_a_extractions/q_a_pairs, and the catalogue form_template_requirements.requirement_text.

Options for the instance→catalogue relationship + where Path C writes.

• A — Path C writes the catalogue from instances (human-confirmed); T10 matches instance fields → catalogue. Path B extracts a form into form_template_fields (instance); Path C (the AI-cataloguing skill) reads those fields, authors requirement_type + taxonomy + matching_keywords, and writes form_template_requirements rows (the reusable catalogue) with human confirmation + an embedding. T10 then matches a new form’s instance fields against the catalogue to reuse prior requirement records. This matches sequencing §2.3’s “Path C authors requirement_type + taxonomy + matching keywords with human confirmation” and §5a’s “matching matches extracted fields AGAINST form_template_requirements”.
• B — Instances ARE the matchable unit; the catalogue is only for cross-form reuse. T10 matches form_template_fields → corpus q_a_pairs directly (to find answers), and the catalogue is a separate, optional reuse layer. Path C becomes a “promote this instance requirement to the reusable catalogue” action rather than the primary write path.
• C — Pipeline auto-writes the catalogue (no human confirmation). Path B writes both instance fields AND catalogue requirements in one pass, embeddings included. Fastest to a populated catalogue; loses the human-confirmation gate that §2.3 mandates and risks catalogue pollution.

Recommendation: Option A. It is the reading the schema and §2.3 jointly imply: instances (form_template_fields, workspace-scoped, file-backed) and catalogue (form_template_requirements, global, embedding-bearing) are distinct by design, and Path C is explicitly the human-confirmed authoring step between them. T10 matches instance → catalogue for reuse, and (separately) catalogue/instance → corpus q_a_pairs for answers. This makes the instance-vs-catalogue relationship explicit, which is the brief’s key ask.

Open Question for Liam [PRODUCT-MAY-PROPOSE]: Confirm the instance→catalogue model: Path B writes instance form_template_fields; Path C (human-confirmed skill) authors the global form_template_requirements catalogue from instances; T10 matches instance fields → catalogue (reuse) and → corpus (answers)? PRODUCT.md can propose this; the matching-side detail belongs to the T10 spec (procurement-question-matching), so the boundary between this spec and T10 needs confirming.

Item #5 — Mode-3 FormMetadata persistence ORPHAN (the fix)

Findings (§4.1). FormMetadata/QAFormExtraction exist and are exercised (extraction.py:121,168; flow.py:748), but flow.py (a) writes no form_templates or form_template_fields, (b) discards form_metadata entirely, and (c) writes qa_pairs lossily to q_a_extractions (drops expected_response_kind, evaluation_criteria, evidence_requirements, scope_tags). Backlog ID-184 (HIGH, needs Liam pre-first-ingest). The app-side analyse/route.ts + queue worker is the only form_templates writer today.

Options.

• A — Wire FormMetadata → form_templates + add a form_template_fields mount in flow.py. Make the pipeline persist what Path A already extracts. Requires resolving workspace_id (item #3) and reconciling with the raw-vs-markdown decision (item #2): Path A’s QAPairs come from markdown extraction, which item #2 recommends against as the form-structure surface. So this option as-is would persist markdown-derived (lossy) fields, not raw-parsed (coordinate-rich) fields.
• B — Keep the app-side write; drop the pipeline contract claim. Mode-3 form structure stays app-owned (route + queue worker, now extended by a real Path-B extractor in the worker). The pipeline’s extract_qa_form is repurposed/removed for forms. Simplest reconciliation with item #3 Option C/D; contradicts §5a “pipeline owns the write”.
• C — Hybrid (aligns with item #2-A + item #3-D): the pipeline’s Path A continues to do general Q&A extraction over markdown (Mode-1 answered-form Q&A → q_a_extractions), while a new Path-B raw extractor owns Mode-3 blank-form structure → form_template_fields (against an app-created form_templates row). FormMetadata’s form-level fields (form_type, deadline, evaluation_methodology) are persisted onto the form_templates row by whichever side creates it. This resolves the orphan by splitting Mode-1 vs Mode-3 (the §5c discriminator split) rather than by force-fitting the markdown-derived Path A output into the instance tables.

Recommendation: Option C. It is the only option consistent with the item #2 (raw surface for structure) and item #3 (split write) recommendations and with the §5c Mode-1/Mode-3 split. The orphan is not “Path A forgot to write a table” — it is “Path A’s markdown-extracted Q&A is the wrong source for blank-form structure”. The fix is to route Mode-3 structure through a raw Path-B extractor into form_template_fields, persist form-level FormMetadata onto the form_templates row, and let Path A keep doing Mode-1 answered-Q&A. This also fixes the lossy q_a_extractions write as a side concern (record expected_response_kind etc. or route them to fields).

Open Question for Liam [RATIFY-BEFORE-PRODUCT]: Confirm the orphan fix = Option C (Path B raw extractor owns Mode-3 form_template_fields; Path A keeps Mode-1 Q&A; form-level metadata onto the form_templates row), rather than wiring the existing markdown-derived Path A output into form_templates (Option A)? This is tightly coupled to items #2 and #3 — they should be ratified together.

Item #6 — Ontology / CV dependency (is it foundational/gating?)

Findings (§4.2). The failing markdown-parity test is a hard loader crash: 26-form-type.md (and 3 siblings) use status: APPLIED-S{NNN}, which the Zod STATUS_VALUES = ['active','planned','needed'] rejects, so loadOntologyCVs() throws and the whole register fails to load. The form-type CV is one of the four offenders. The form_type value set lives in three drift-prone places: the markdown CV, the live form_types table, and a hard-coded Literal in extraction.py. The 26-form-type.md file also carries two stale claims (table “not yet in schema” — it is; template_requirements 10-value CHECK — renamed away). Sequencing §7 calls ontology integration “foundational”, likely gating T7 / form-extraction / T10.

Options.

• A — Treat it as a hard gate: fix the loader/CV parity before the form subsystem builds. Either extend STATUS_VALUES to include an applied/APPLIED-* status (and re-baseline the 4 files), or normalise the 4 files to an accepted status. Then reconcile the form_type triple-source (CV ↔ form_types table ↔ Python Literal) — ideally make the Python pipeline read form_type from the snapshot fixture (the existing dual-source mechanism) rather than hard-coding the Literal.
• B — Treat it as adjacent, not gating. Fix the test separately; the form subsystem hard-codes/reads form_type from whatever source is convenient and the CV-loader crash is a parallel cleanup. Risk: the form subsystem ships with a form_type set that silently diverges from the live form_types table (exactly the Q-EX2 drift class).
• C — Scope the fix to form-relevant CVs only (26, 30, 31, 32) as part of this Task, leaving a general ontology-loader hardening to a separate Task.

Recommendation: Option A (gating), scoped pragmatically. The crash directly blocks reading the form_type CV through the canonical path, and §7 already flags ontology as foundational — so the form subsystem should not build on a register that cannot load. The minimal correct fix is (i) extend STATUS_VALUES (or re-baseline the 4 files) so the register loads, (ii) correct the two stale claims in 26-form-type.md, and (iii) decide how extraction.py’s form_type Literal stays in lockstep with the CV + form_types table (recommend: drive the Python side from the taxonomy snapshot fixture, matching the dual-source pattern). Whether the loader hardening is in-Task or a sibling Task is a decomposition question for {52.4}; the parity fix itself is a prerequisite for {52.3} TECH to cite the form_type CV honestly.

Open Question for Liam [RATIFY-BEFORE-PRODUCT]: Confirm ontology CV-loader parity is a gate for form-extraction (fix STATUS_VALUES + the 4 APPLIED-* files + the form_type triple-source before building), not a parallel cleanup? And confirm the preferred STATUS_VALUES fix direction (extend the enum to accept applied, vs re-baseline the 4 files to active)?

---

6. Consolidated Open Questions for Liam

The genuine product decisions, gathered. [RATIFY-BEFORE-PRODUCT] items must be resolved before PRODUCT.md commits; [PRODUCT-MAY-PROPOSE] items PRODUCT.md can propose for Liam to ratify at the PRODUCT gate.

1. OQ-52-FORMATS [RATIFY-BEFORE-PRODUCT] — v1 format scope. Recommend {PDF, XLSX, DOCX}; legacy .xls = manual pre-convert; HTML deferred (no example exists). Confirm, or pull HTML into v1? (§5 item #1)
2. OQ-52-SURFACE [RATIFY-BEFORE-PRODUCT] — extraction surface for blank-form structure. Recommend raw-format parsing (preserves row/col/table_index, word_limit, M/O flags), not the §2.2 converted markdown. This is the most consequential decision and scopes §2.2’s “everything→markdown” to general content extraction only. Confirm? (§5 item #2)
3. OQ-52-WORKSPACE [RATIFY-BEFORE-PRODUCT] — workspace_id (NOT NULL) resolution. Recommend Option D: app creates the form_templates row (with workspace, at upload), pipeline fills form_template_fields. Alternative is a folder→workspace convention (Option A) if a fully pipeline-owned localfs-driven write is intended. Confirm? (§5 item #3)
4. OQ-52-CATALOGUE [PRODUCT-MAY-PROPOSE] — instance→catalogue model. Recommend: Path B writes instance fields; Path C (human-confirmed) authors the global form_template_requirements; T10 matches instance→catalogue (reuse) + →corpus (answers). Confirm the boundary between this spec and the T10 procurement-question-matching spec. (§5 item #4)
5. OQ-52-ORPHAN [RATIFY-BEFORE-PRODUCT] — Mode-3 FormMetadata orphan fix. Recommend Option C: Path B raw extractor owns Mode-3 form_template_fields; Path A keeps Mode-1 Q&A; form-level metadata onto the form_templates row. Tightly coupled to OQ-52-SURFACE + OQ-52-WORKSPACE — ratify together. (§5 item #5)
6. OQ-52-ONTOLOGY [RATIFY-BEFORE-PRODUCT] — is the CV-loader parity crash a gate? Recommend yes: fix STATUS_VALUES (extend to applied OR re-baseline the 4 APPLIED-* files), correct the 2 stale claims in 26-form-type.md, and lockstep the form_type triple-source (CV ↔ form_types table ↔ Python Literal). Confirm gate + fix direction. (§5 item #6)

Secondary (lower-stakes) confirmations PRODUCT/TECH can carry:

• OQ-52-LOSSY — the existing flow.py lossy q_a_extractions write (drops expected_response_kind/evaluation_criteria/evidence_requirements/scope_tags). Fix in-scope of this Task or fold to a Path-A follow-up? (§4.1)
• OQ-52-XLS — confirm legacy .xls is rare enough that manual pre-convert is acceptable (the one corpus example is a blank scoring tool, not a question source). (§2.4)

---

7. Recommended next steps (what PRODUCT.md should commit to vs what stays open)

PRODUCT.md SHOULD commit to (assuming the recommended OQ resolutions):

• The Mode-1 vs Mode-3 split as the organising frame (per §5c of canonical-pipeline-sequencing.md — a previously-resolved direction, not a new ratification by this RESEARCH doc): Mode-3 (blank form → catalogued structure) is this subsystem’s primary scope; Mode-1 (answered-form Q&A) stays with Path A.
• A per-format deterministic extractor behaviour set (Path B), covering {PDF, XLSX, DOCX}, that populates form_template_fields with coordinate + limit + flag metadata (row_index/col_index/table_index/word_limit/mandatory-flag/section_name/ sequence/field_type/placeholder_text) — expressed as numbered, testable behaviour invariants the Checker can verify against the four corpus fixtures (e.g. “SSQ Annex B question N is extracted with its M/O flag”, “EFA Bidder-1 and Bidder-2 questions are not double-counted”, “CSP principle URLs are preserved”).
• The instance-vs-catalogue boundary (§3.1) stated explicitly: Path B → instance fields; Path C (human-confirmed) → catalogue requirements; T10 reads the catalogue.
• The write-ownership split (app creates template row + workspace; pipeline fills fields) per OQ-52-WORKSPACE.
• The CV-loader parity gate as a prerequisite invariant (the form-type CV must load).
• Acceptance fixtures = the four corpus sets in docs/testing/test-data/templates/ (with the corrected counts: SSQ ~66 from Annex B/C; Charnwood reqs come from the DOCX, not the blank .xls; EFA/CSP counts to be measured against the live files).

PRODUCT.md should leave OPEN (route to {52.3} TECH or T10 spec):

• The exact extractor library choices (exceljs vs openpyxl for XLSX; pdfplumber vs pdf-parse for PDF; python-docx for DOCX) — a TECH.md decision. Note pdf-parse is absent from package.json; pdfplumber + docling are present Python-side; exceljs@4.4.0 is present TS-side. The TS-vs-Python placement of Path B (the pipeline is Python; the existing extractors are Python; the app write is TS) is a TECH boundary decision.
• The Path C cataloguing skill internals (skill shape, human-confirmation UX, embedding generation) — likely its own Subtask or a sibling spec.
• The T10 matching scoring/retrieval — owned by procurement-question-matching.
• Whether the ontology-loader hardening is in-Task ({52.x}) or a sibling Task — a {52.4} decomposition question.

---

8. Verification (OQ-3 / Q-EX2 pre-ratification empirical checks)

External-library symbols cited in this RESEARCH, checked against the installed pins this session (UK English, DD/MM/YYYY):

Date	Pin	Symbol path	Result	Note
28/05/2026	cocoindex==1.0.3	cocoindex.ExtractByLlm	ABSENT (hasattr → False)	Confirms sequencing §2.5. Contradicts feedback-findings §5 (S234), which prescribes ExtractByLlm for form classification (Theme B step 4 / OQ-Q11-A closure). The S234 doc surveyed cocoindex 0.3.x; the symbol was removed in 1.0. This is the Q-EX2 drift class. Any spec citing ExtractByLlm for forms must use a custom @coco.fn instead.
28/05/2026	cocoindex==1.0.3	cocoindex.LlmSpec	ABSENT (hasattr → False)	Same 0.3.x→1.0 removal.
28/05/2026	docling>=2.0.0,<3.0.0	docling.document_converter.DocumentConverter	PRESENT (imports cleanly)	The §2.2 markdown-conversion surface is empirically viable (relevant if item #2 ever goes markdown).
28/05/2026	exceljs@4.4.0 (package.json)	exceljs (XLSX, OQ11-named)	PRESENT	TS-side XLSX reader available for Path B.
28/05/2026	package.json	pdf-parse (PDF, OQ11-named)	ABSENT	The OQ11-named PDF extractor is not installed. Python-side pdfplumber==0.11.9 IS present; TECH must choose the PDF reader + its placement (TS app vs Python pipeline).
28/05/2026	package.json	xlsx (SheetJS)	ABSENT	Not available; openpyxl/exceljs are the XLSX options.
28/05/2026	mammoth@^1.12.0, turndown@^7.2.4, docx@^9.6.1 (package.json)	DOCX TS stack	PRESENT	Available if DOCX handling lands TS-side.
28/05/2026	user-site (NOT in requirements.txt)	xlrd (legacy .xls)	ABSENT from pins (I installed xlrd 2.0.2 ad hoc to read the Charnwood .xls)	Legacy .xls has no installed reader — load-bearing for item #1.
28/05/2026	database.types.ts	form_templates / form_template_fields / form_template_requirements / form_types	PRESENT (lines 1686 / 1519 / 1597 / 1755)	All four tables verified live, with FKs as cited in §3.

No ABSENT/SIGNATURE_DRIFT blocks the RESEARCH (RESEARCH surveys options; it does not itself commit to ExtractByLlm). But the ExtractByLlm ABSENT result is a hard escalation flag for {52.3} TECH: TECH must not cite cocoindex.ExtractByLlm for form classification, despite the S234 feedback-findings doc doing so. Recorded here so the PRODUCT/TECH authors inherit the correction.

---

9. Source-doc contradictions found (for the Orchestrator/Liam)

1. ExtractByLlm drift (CRITICAL). feedback-findings-review.md §5 (S234) repeatedly prescribes cocoindex ExtractByLlm for form classification (Theme B step 4; OQ-Q11-A “RESOLVED — cocoindex ExtractByLlm”). Empirically ABSENT in cocoindex==1.0.3 (§8). Sequencing §2.5 (S272) already caught this for Path A; the form-extraction direction in §5 was not updated. The canonical record is the cocoindex investigation (docs/research/cocoindex-1.0.3-extractbyllm-spec-reality-investigation.md per the Q-EX2 precedent). Sequencing (S272) wins; the S234 ExtractByLlm direction is superseded.
2. OQ11 “XLSX exceljs + PDF pdf-parse” vs the actual stack. pdf-parse is absent; the existing extractors are DOCX-centric (extract_tender_questions.py, analyse_template.py), not the XLSX/PDF stack OQ11 names. The forward-path tool list is aspirational, not implemented.
3. “Charnwood ITT = 30 reqs” vs the .xls reality. The .xls is a blank scoring template (placeholder rows only); the 30 reqs were hand-catalogued from the companion DOCX at UAT S83. The brief’s pairing of “30 reqs” with the .xls is misleading — the DOCX is the question source.
4. SSQ PDF page count. file reports 8 pages; pdfplumber reads 57 (authoritative).
5. 26-form-type.md stale claims. (a) “form_types table … not yet in production schema” — it IS live (database.types.ts:1755). (b) “template_requirements.template_type CHECK has 10 values” — template_requirements was renamed to form_template_requirements (S246) and template_type is now an FK to form_types.key.
6. form_template_requirements discriminator. The brief states the FK is on requirement_type; verified it is on template_type (→ form_types.key). requirement_type is a plain (no-FK) string column.
7. requirements.txt comment drift. litellm is commented as “used by cocoindex ExtractByLlm stage” — but ExtractByLlm does not exist; sequencing §2.5/§10 already re-purposes litellm as the LiteLLMEmbedder (Stage-4) dependency. Comment is stale.
8. lib/bid-library-ingest/ does not exist (the brief’s pre-located path) — the app-side form write owner is app/api/procurement/[id]/templates/[templateId]/analyse/route.ts (renamed in the bid→procurement shift). Minor.

---

End of RESEARCH — ID-52.1. Output: docs/specs/id-52-form-extraction/RESEARCH.md. Not committed (Orchestrator commits after the Checker gate). Subtask status unchanged (the Checker sets it).