Form-Extraction Subsystem — PRODUCT

Form-Extraction Subsystem — PRODUCT

Type: PRODUCT (behaviour spec). Companion to RESEARCH.md (same directory) and the forthcoming TECH.md. This document defines what the form-extraction subsystem does, from the perspective of its consumers, as numbered, testable behaviour invariants a Checker can verify against acceptance criteria. It does not prescribe libraries, migrations, or @coco.fn internals — those belong to TECH.md ({52.3}).

Task: ID-52 (form-extraction — the blank-form question subsystem). Subtask: {52.2}. Authored S273 by a fresh Planner context (distinct from the {52.1} RESEARCH author). UK English throughout.

Primary input: docs/specs/id-52-form-extraction/RESEARCH.md (read in full). Schema facts are canonical from supabase/types/database.types.ts (verified live this session — RESEARCH §3; not re-derived here).

Ratified by Liam (S273) — committed below, not re-opened: v1 formats {PDF, XLSX, DOCX}; raw-format parsing surface for blank-form structure; fully pipeline-owned write via folder→workspace convention; CV-loader parity is a prerequisite gate. See “Ratified decisions” below.

---

Summary

The form-extraction subsystem turns a blank procurement form (an SQ, ITT, PQQ, or checklist that a buyer issues) into structured, matchable question records in the knowledge base. The pipeline ingests each blank form from its source folder, resolves the owning workspace from that folder, parses the raw file to recover every question with its coordinates, mandatory/optional flag, word limit, and section, and persists those as per-field instance rows. A separate human-confirmed cataloguing step promotes instance questions into a reusable, embedding-bearing requirement catalogue that downstream matching (Task T10) reads. The desired outcome: a buyer’s blank form becomes a faithful, queryable set of question records with no silent data loss and no manual re-keying.

Problem

Today the platform can extract general Q&A prose from answered documents (the built “Path A”), but it has no machinery to extract question structure from a blank form — the mandatory/optional flags, word limits, cell coordinates, and section hierarchy that a buyer’s form carries. The Python pipeline already builds a FormMetadata record during extraction but drops it before persistence (the Mode-3 orphan; RESEARCH §4.1, backlog ID-184), and the only code that writes form-structure tables is an app-side route that enqueues a job whose extractor was never built. As a result, a blank form cannot be catalogued or matched against the corpus — the feature is unbuilt, and the partial wiring that exists loses per-question metadata. This subsystem closes that gap and is on the v1 critical path alongside T10 question-matching.

Goals / Non-goals

Goals.

• Extract every question from a blank {PDF, XLSX, DOCX} form into per-field instance rows with coordinate, flag, limit, and section metadata preserved.
• Resolve the owning workspace deterministically and loudly from the form’s source folder.
• Provide a human-confirmed path from instance questions to a reusable requirement catalogue that T10 matching can read.
• Reconcile with the partially-wired Path A so no Q&A metadata is silently dropped.

Non-goals.

• T10 matching internals (scoring, retrieval, ranking against the catalogue) — owned by the procurement-question-matching spec. This spec defines only the boundary the catalogue presents to T10.
• Answered-form Q&A extraction (Mode-1) — stays with the built Path A; this spec does not redesign it, only stops it dropping metadata (Inv-18 / Inv-19; OQ-52-LOSSY governs whether the lossy-write fix is in-scope).
• Legacy .xls automated parsing — out of automated scope; manual pre-convert (Inv-3).
• HTML form parsing — deferred until a real HTML form exists (Inv-2).
• Library / reader choices, deployment placement (Python vs TS), embedding generation mechanism — TECH.md decisions.

Ratified decisions (Liam, S273 — committed, not re-opened)

These four are fixed inputs to the behaviour below. They are recorded here so the Behaviour section reads as a single committed contract; the Checker treats them as given, not as open.

• R1 — Formats. v1 automated scope is exactly {PDF, XLSX, DOCX}. Legacy .xls = manual pre-convert (out of automated scope). HTML = deferred (no corpus example).
• R2 — Surface. Blank-form structure is parsed from the raw file format, not from the converted markdown surface. This preserves the coordinate/limit/flag columns (row_index / col_index / table_index / word_limit / mandatory flag) that a flat markdown conversion would lose. The pipeline’s “everything→markdown” conversion is hereby scoped to general content extraction (Path A) only; blank-form structure (Path B) reads raw.
• R3 — Write ownership. The pipeline fully owns the write: it creates the instance template row and fills the per-field rows, resolving the NOT-NULL owning workspace via a folder→workspace convention on the ingest source. This migrates write ownership off the current app-side route. (Note: RESEARCH §5 item #3 recommended Option D — app creates the form_templates row, pipeline fills fields; Liam ratified Option A, the fully pipeline-owned write, instead at S273.)
• R4 — Ontology gate. The form-type controlled-vocabulary (CV) register must load successfully before the form path runs. This is a precondition invariant (Inv-1), not a fix mechanism — the how of the fix is a TECH.md detail.

Terminology

• Blank form. A form a buyer issues for suppliers to complete — question text present, answer cells empty or placeholder-filled. The subject of this subsystem.
• Mode-3 (this subsystem’s primary scope). A blank form whose questions we catalogue and later match. Distinct from Mode-1 (an answered form whose answers are themselves citable Q&A facts, handled by the built Path A). RESEARCH §1 / §5(c).
• Instance. A specific blank form ingested into one workspace. Persisted as one form_templates row + many form_template_fields rows. Workspace-scoped, file-backed.
• Catalogue. The reusable, global requirement set (form_template_requirements). No workspace scope; carries embeddings for matching. The T10 read target.
• Path B. The deterministic raw-format extractor that turns a blank form into instance rows (this spec’s core build).
• Path C. The human-confirmed cataloguing step that promotes instance questions into the catalogue (proposed; see OQ-52-CATALOGUE).
• Field. One extracted question (or structural placeholder) from a form, persisted as one form_template_fields row.

Behaviour

Preconditions and ingest framing

1. Ontology CV register loads before the form path runs (gate). The form-extraction path runs only when the controlled-vocabulary register loads successfully — in particular the form-type CV (form_types) and the requirement-type CV are available to consumers. If the CV register fails to load, the form path does not run and the failure is surfaced loudly (not swallowed). Verifiable: with the register in its current failing state — where a non-enumerated CV status causes the loader to throw — the form path is blocked; once the register loads, the form path proceeds. (RESEARCH §4.2, §5 item #6; ratified gate R4. The fix mechanism — accepting an applied status vs re-baselining the four APPLIED-* CV files — is a TECH.md detail; this invariant asserts only the loadable-register precondition.)

1a. Layer-5 KG-entity controlled-vocabulary files load alongside Layer-1 enumerable CVs (CV-register heterogeneity). The ontology CV register comprises two structurally distinct kinds of file. Layer-1 enumerable CVs (the majority — content_type, platform, form_type, etc.) carry an enumerated baseline_values array: each row IS a vocabulary entry the platform validates against. Layer-5 KG-entity CVs (q_a_pair, and any future siblings such as entity_mention or citation) carry NO enumerable baseline_values — the corpus row IS the data, not a vocabulary entry. Each Layer-5 KG-entity CV instead declares its golden-source pointers (source_of_truth), adjacent CV relationships (related_ontology), and a maintenance timestamp (last_updated). The CV-register loader (loadOntologyCVs) MUST load both kinds without schema rejection: a Layer-5 KG-entity file with no baseline_values and with declared golden-source pointers is a well-formed CV record, NOT a schema violation. Verifiable: loadOntologyCVs() returns N records covering both the Layer-1 set and the Layer-5 KG-entity set; a Layer-5 file with layer: 5 and no baseline_values key parses successfully; a Layer-1 file with layer: 1 and no baseline_values still fails (the gate is per-layer, not blanket). (Resolves OQ-52-WAVE-1-A, ratified S275: Option 3 — spec amendment over schema-touch. Implementation in TECH §2.6c + Subtask {52.5a}; touches lib/ontology/schemas.ts under wp6-ontology-harness’s shared infrastructure with cross-Task ratification recorded in {52.5a}‘s details.)

2. Supported input formats are exactly {PDF, XLSX, DOCX}. A blank form supplied in any of these three formats is eligible for automated extraction. A form in any other format is not automatically extracted. (Ratified R1.)

3. Legacy .xls and HTML are out of automated scope. A legacy binary .xls file is not parsed automatically; it requires a manual one-time pre-conversion to .xlsx before ingest. An HTML form is not parsed; HTML support is deferred. Neither is treated as an extraction error — they are simply not eligible inputs in v1. (Ratified R1; RESEARCH §2.4 established the one corpus .xls is a blank scoring tool, not a question source.)

4. Each ingested form resolves to exactly one workspace via its source folder. The pipeline determines the owning workspace from the form file’s location under the ingest source (the folder→workspace convention). The resolution is deterministic: the same file in the same folder always resolves to the same workspace. (Ratified R3.)

5. Ingest fails loudly when no workspace resolves — never silently NULL or default. If a form’s source folder does not resolve to exactly one existing workspace, the pipeline does not create an instance with a missing, placeholder, or arbitrarily-defaulted workspace. It records an explicit, surfaced failure for that form and extracts no field rows for it. There is no “unassigned” sentinel workspace and no silent skip. Verifiable: a form under an unmapped folder produces a surfaced resolution failure and zero form_template_fields rows, not an instance row with a guessed workspace. (Ratified R3; form_templates.workspace_id is NOT NULL — RESEARCH §3.2.)

The write the pipeline produces (Mode-3, instance side)

6. The pipeline owns the entire instance write. For each successfully resolved blank form, the pipeline creates the instance template record and fills its per-field rows. No app-side step is required to create the instance row before extraction. Verifiable: a form placed in a workspace-mapped ingest folder, with no prior app interaction, results in both an instance template record and its field rows. (Ratified R3; migrates ownership off the app-side analyse route — RESEARCH §4.1.)

7. Form-level metadata lands on the instance template record. The form’s overall attributes — its form type (from the form-type CV value set), its source filename, its stored-file reference and media type and size, and a human-readable name — are recorded on the instance template record. Where the form additionally exposes a deadline, an issuing organisation, or an evaluation methodology, those are captured as form-level attributes rather than discarded. Verifiable: an ingested SQ produces an instance record whose form type is a recognised CV value and whose filename matches the source. (Resolves the form-level half of the Mode-3 orphan — RESEARCH §4.1: FormMetadata’s form_type / form_title / deadline / evaluation_methodology / issuing_organisation were previously dropped.)

8. Each extracted question is one field row carrying its coordinates. Every question (or structural question slot) the extractor recovers from the raw form becomes one field row. Where the source format exposes positional coordinates — a cell’s row and column, a table’s index within the document — those coordinates are recorded on the field row. For formats or regions where a coordinate does not meaningfully apply, the coordinate is left empty rather than fabricated. Verifiable: an XLSX scoring-matrix question row has its row/column/table position recorded; a PDF prose question without a meaningful cell column leaves the column coordinate empty. (form_template_fields.row_index / col_index / table_index — RESEARCH §3.3. These columns are meaningless on flattened markdown, which is why R2 mandates raw parsing.)

9. Question text is captured verbatim; placeholder scaffolding is distinguished from authored questions. A field row records the question’s text as it appears in the form. Where a cell is an unfilled template placeholder rather than an authored question (e.g. Insert question title, [identification of requirement], TYPE RESPONSE HERE>>>>), the field row records the placeholder text and marks the field’s fill status as a placeholder, distinct from an authored question. The extractor does not promote a placeholder string into an authored question, nor drop a genuine question because its answer cell is blank. Verifiable: the Charnwood DOCX Insert question title grid rows are recorded as placeholder-status fields, not as authored questions; the CSP TYPE RESPONSE HERE>>>> cells mark answer slots, not questions. (RESEARCH §2.3, §2.4; form_template_fields.placeholder_text / fill_status / field_type.)

10. Mandatory/optional status is preserved when the form expresses it. Where a form marks a question as mandatory or optional (e.g. the SQ Annex B left-cell M/O flag, or an EFA weighting that distinguishes scored from compliance items), that status is recorded on the field. The status is never colour-only or inferred-by-omission: it is recorded as an explicit value. Where the form expresses no such status, the field records no mandatory flag rather than defaulting to one. Verifiable: the SQ Annex B Part 1 M-flagged schema row (e.g. the left-cell ['M', 'Registered VAT number'] pair) is recorded with its M/O flag; a CSP free-text preamble item with no flag records none. (RESEARCH §2.1, §2.2.)

11. Word and page limits are captured when present, however the form expresses them. A question’s response limit is recorded on the field whether the form states it inline in the question prose (e.g. SQ … in no more than [500] words …) or in a dedicated column (e.g. EFA “Page Limit”). Where no limit is stated, none is recorded. Verifiable: SQ question 6.2’s [500] inline word limit is recorded as that field’s limit; an EFA question with a “Page Limit” cell records that limit. (form_template_fields.word_limit — RESEARCH §3.3.)

12. Section hierarchy and ordering are preserved. Each field records the section it belongs to (e.g. SQ “Annex B”, EFA “Part 2 — Overall Approach”, CSP “Principle 1”) and its position in reading order within the form, so the extracted set can be presented in the form’s original structure. Verifiable: extracted EFA fields carry their “Part N …” section, and field ordering reproduces the form’s question sequence. (form_template_fields.section_name / sequence — RESEARCH §3.3.)

13. Repeated question blocks are extracted once, not once per copy. Where a form repeats the same question set across multiple regions for layout reasons — most notably the EFA matrix repeating its questions across a Bidder 1 and Bidder 2 sheet, and the Charnwood scoring tool repeating an identical grid across eleven supplier sheets — the extractor records each distinct question once. It does not produce duplicate field rows for the same question appearing in a second scoring copy. Verifiable: the EFA form’s Bidder 1 and Bidder 2 question columns do not produce double-counted fields. (RESEARCH §2.2, §2.4 — the dedup hazard.)

14. Embedded reference links carried by a question are preserved. Where a question or its section carries an external reference URL (e.g. the CSP principle rows’ NCSC collection links), that reference is preserved on the corresponding field rather than dropped during extraction. Verifiable: CSP principle reference URLs survive into the extracted fields. (RESEARCH §2.3. The S234/S235 bake-off observed a one-anchor URL-recall quirk shared across tools; faithfulness of the links that ARE recovered is the invariant, not a 100%-recall guarantee against a known tool limitation.)

15. A page-container artefact does not truncate extraction. Extraction reads the form’s true content extent, not a misleading container-reported page count. Verifiable: the SQ PDF, whose container header reports 8 pages but whose true content is 57 pages, has its Annex B/C questions extracted from the full 57-page extent, not the first 8. (RESEARCH §2.1, §9 item 4.)

16. The instance write is idempotent for re-ingest of the same form. Re-ingesting the same form (same file, same resolved workspace) does not accumulate duplicate instance records or duplicate field rows for the same questions; the instance reflects the current form content. Verifiable: ingesting a form, then re-ingesting the unchanged file, yields one instance with a stable field set, not two. (The current app-side route clears existing fields before re-analysis — RESEARCH §4.1; the pipeline-owned write preserves that no-accumulation guarantee. The precise re-ingest mechanics are a TECH.md detail; the no-duplication outcome is the invariant.)

17. Extraction failure for one form does not corrupt or block others. If one form in an ingest batch fails to parse (corrupt file, unreadable structure, unresolved workspace), that form records a surfaced failure and contributes no partial or malformed instance, while other forms in the same batch extract normally. A form is never left in a state where it has an instance record but silently zero fields with no recorded reason. Verifiable: a batch containing one unreadable file and three readable forms produces three complete instances plus one surfaced failure, not a halted batch.

    Two distinct zero-field shapes (ratified S278 — graceful vs strict). The “no recorded reason” prohibition admits two legitimate zero-field outcomes, and forbids only the silent third:

    • Strict-raise (unreadable / empty / zero-sheet). A genuinely broken input — empty bytes, unreadable/corrupt bytes, a workbook with zero worksheets — STAYS strict: the reader raises a typed extraction error and the form-write path records one instance with status = analysis_failed, zero fields, and the surfaced error reason. This is a failure.
    • Graceful-empty-with-recorded-reason. A structurally readable form that simply yields no extractable fields — e.g. a valid XLSX whose sheets match no known archetype — is NOT a failure. The reader returns an empty instance gracefully (no raise) and emits a surfaced, machine-readable reason (a structured log carrying a reason token such as no_archetype_match). The form-write path records one instance with status = analysed, zero fields, and that reason threaded onto the instance provenance (so the row carries why it is empty). This is a deliberate, recorded empty — distinct from the strict-raise failure above.
    • Forbidden (silent zero). What remains prohibited is the third shape: an analysed instance with zero fields and no recorded reason — neither a surfaced log nor a reason on the row. Graceful-empty must always be the recorded-reason shape, never the silent one.

    Verifiable: ingesting a structurally valid XLSX whose sheets match no archetype yields exactly one analysed instance with field_count = 0 whose provenance carries a recorded no-archetype reason and whose extraction emitted a surfaced structured-log reason — not a silent zero-field row, and not a strict analysis_failed raise.

Faithfulness and the Mode-1 reconciliation

18. No silent loss of per-question metadata. The extraction does not silently discard metadata a form carries on a question — mandatory/optional status, word/page limit, section, coordinates, reference links, and placeholder-vs-authored status are all either recorded (per Inv-8–Inv-14) or, where genuinely absent from the form, left empty. The subsystem does not repeat the Mode-3 orphan pattern of building richer metadata than it persists. Verifiable: for each corpus form, every metadata facet the form expresses on a sampled question appears on that question’s field row. (RESEARCH §4.1; this is the behavioural counterpart to closing the orphan.)

19. Mode-1 answered-form Q&A continues unchanged in scope. This subsystem adds Mode-3 blank-form structure extraction; it does not remove or alter the built Path A behaviour that extracts answered-form Q&A into the corpus. A document that is an answered form continues to yield citable Q&A facts as before. The two modes are distinct concerns over the same file substrate. (RESEARCH §1, §5(c) — the organising frame.)

Catalogue and the matching boundary

20. Instance fields and catalogue requirements are distinct record kinds. An instance field (one question of one ingested form, workspace-scoped) is not the same record as a catalogue requirement (a reusable, global requirement carrying matching metadata and an embedding, with no workspace scope). The subsystem maintains this distinction: filling instance fields does not by itself create catalogue requirements. Verifiable: a pipeline ingest that fills instance fields does not, on its own, write catalogue requirement rows. (RESEARCH §3.1, §3.4 — the load-bearing instance-vs-catalogue nuance.)

21. The catalogue is authored through a human-confirmed step, not auto-written by ingest. Promotion of an instance’s questions into the global catalogue happens only via an explicit human-confirmed cataloguing action (Path C). The ingest pipeline does not auto-populate the catalogue, and the catalogue is never silently filled as a side-effect of extraction. Verifiable: catalogue rows appear only after the human-confirmed cataloguing step runs, never from extraction alone. (RESEARCH §5 item #4; PRODUCT-MAY-PROPOSE OQ-52-CATALOGUE below.)

22. The catalogue presents a stable read boundary to matching (T10). The matching subsystem (T10) reads catalogue requirements — their requirement text, requirement type, matching keywords/guidance, mandatory flag, section, and embedding — to match against. This subsystem owns producing and cataloguing requirements; it does not own how T10 scores, retrieves, or ranks them. The boundary: everything up to and including a populated, embedding-bearing catalogue requirement is in scope; matching logic over the catalogue is out of scope (T10’s spec). Verifiable: the catalogue requirement record exposes the matching-relevant fields T10 needs; no scoring/retrieval behaviour is specified here. (RESEARCH §3.4, §5 item #4, §7.)

23. Catalogue requirements are reusable across workspaces. A catalogue requirement, once authored, is not scoped to the workspace whose instance form seeded it; it is available to match against forms ingested into any workspace. (RESEARCH §3.1, §3.4 — no workspace FK on the catalogue, by design.)

Access and surfacing

24. Operations are gated to the existing procurement form roles. Actions that mutate form-extraction state on the app side — initiating or re-running analysis, and confirming a catalogue promotion — remain available only to authorised roles (admin/editor on the procurement surface), and unauthorised callers receive the standard authorisation failure routing rather than a partial action. Pipeline-side ingest (the folder-driven write) runs under the pipeline’s service identity. Verifiable: a viewer-role caller cannot initiate analysis or confirm a catalogue promotion. (The existing route gates on getAuthorisedClient(['admin','editor']) / authFailureResponse — RESEARCH §4.1; this invariant preserves that gate for any retained app-side action under the new pipeline-owned model.)

25. Workspace scoping holds for instances; the catalogue is global by design. A user’s view of ingested form instances is scoped to workspaces they may access (instances carry a workspace). The catalogue, having no workspace scope, is a shared platform resource; its visibility follows the platform’s catalogue-access rules, not per-workspace scoping. (Multi-tenant scoping note per the spec skill; RESEARCH §3.1.)

26. The extracted question set is presented as structured data, not as an AI feature. The extracted fields and catalogued requirements appear to consumers as ordinary structured knowledge-base records — questions with their flags, limits, sections, and references — consistent with the platform’s AI-invisible-infrastructure stance: the platform behaves as though the structured questions were always there, not as a user-facing “AI extraction” feature. (docs/reference/ai-visibility-policy.md.)

Acceptance fixtures

The validation basis is the four corpus form sets in docs/testing/test-data/templates/ (verified present this session). The invariants above are framed to be checkable against them:

Set	File	Format	What the extraction must demonstrate
SQ	sq-standard-selection-questionnaire/standard-selection-questionnaire-ppn-03-24.pdf	PDF (57pp true extent)	Annex B/C questions extracted (~66 reqs, RESEARCH §2.1) each with its M/O flag (Inv-10) and inline [NNN] words limit (Inv-11); full 57-page extent read, not the 8-page container artefact (Inv-15).
ITT EFA	itt-services-efa/evaluation-matrix-itt-vol8.xlsx	XLSX	Bidder 1 and Bidder 2 questions not double-counted (Inv-13); scoring-matrix questions carry row/column/table coordinates (Inv-8) and “Page Limit” limits (Inv-11).
CSP	csp-checklist/Cloud Security Principles Checklist V5_3.xlsx	XLSX	Principle reference URLs preserved (Inv-14); TYPE RESPONSE HERE>>>> cells marked as answer placeholders, not questions (Inv-9); letter-keyed preamble vs numbered principles both sectioned (Inv-12).
ITT Charnwood	itt-services-charnwood/ITT Services.docx (the DOCX is the question source, ~30 reqs)	DOCX	Questions extracted from the DOCX paragraphs and tables (Inv-8, Inv-12); the companion ITT Evaluation Matrix.xls is a blank scoring tool, out of automated scope (Inv-3), and its placeholder Description of criteria N rows are not extracted as questions.

(Exact EFA/CSP counts are to be measured against the live files during implementation; RESEARCH established the SQ ~66 and Charnwood-DOCX ~30 figures and corrected the brief’s mis-pairing of “Charnwood 30 reqs” with the blank .xls.)

Open questions

These are flagged for Liam to ratify at the PRODUCT gate. The ratified-decision set (R1–R4) above is not re-opened here.

• Open question — OQ-52-CATALOGUE [PRODUCT-MAY-PROPOSE]. This spec proposes the instance→catalogue model in Inv-20–Inv-23: Path B writes per-instance fields; Path C (an AI-cataloguing skill, human-confirmed) authors the global catalogue requirements (carrying matching keywords and an embedding); T10 reads the catalogue to match against. It further proposes the boundary between this spec and the T10 procurement-question-matching spec: this spec owns extraction + cataloguing up to a populated catalogue requirement; T10 owns scoring/retrieval/ranking over the catalogue (Inv-22). Confirm the model and the boundary. (RESEARCH §5 item #4.)

• Open question — OQ-52-LOSSY [PRODUCT-MAY-PROPOSE]. The built Path A currently writes answered-form Q&A pairs lossily, dropping each pair’s expected-response kind, evaluation criteria, evidence requirements, and scope tags (RESEARCH §4.1). Inv-18’s “no silent loss” principle argues for fixing this. Proposal: fold the lossy-write fix into this Task (record those four facets where the Q&A pair carries them), or defer it to a Path-A follow-up Task. Confirm in-Task vs deferred. (This is a Mode-1 concern; Inv-19 keeps Mode-1 scope otherwise unchanged.)

• Open question — UI-upload vs folder→workspace tension [PRODUCT-MAY-PROPOSE, surfaced not blocking]. The ratified write model (R3, Inv-4–Inv-6) is folder-driven: the pipeline resolves the workspace from the ingest folder and owns the whole write. The platform also has an interactive app-upload path today — the analyse route takes a user-chosen workspace at upload time (RESEARCH §4.1, §3.2). Under R3 these two paths could both create instances for the same workspace by different routes (folder-driven vs user-upload-driven). Proposal: treat the folder→workspace ingest as the canonical v1 write path for blank forms, and define the interactive upload path as either (a) retired for blank-form structure in v1, or (b) a thin front-end that drops the file into the resolved workspace folder so the same pipeline write applies — rather than a second, independent writer. Confirm which, so Inv-6’s “no app-side step required” and Inv-16’s idempotency hold without a competing app-side write. (Surfaced per the dispatch brief’s instruction to flag this tension as a sub-question, not to block.)

---

Ratifications (S273, Liam)

All seven Open Questions surfaced by RESEARCH.md §6 are now ratified. The first four were ratified mid-session (in-terminal, while RESEARCH was being authored) and are encoded directly into the invariants above; the last three are the PRODUCT-gate ratifications recorded here.

Mid-session ratifications (encoded into invariants)

• OQ-52-FORMATS — v1 = {PDF, XLSX, DOCX}; legacy .xls = manual pre-convert; HTML deferred (no corpus example). (Per worker recommendation.)
• OQ-52-SURFACE — Raw-format parsing for blank-form STRUCTURE (NOT the §2.2 converted markdown). Scopes “everything→markdown” to Path-A general content only. (Per worker recommendation.)
• OQ-52-WORKSPACE_WRITE — Option A: folder→workspace convention, FULLY pipeline-owned write (pipeline creates the form_templates row resolving workspace_id NOT NULL via source folder, AND fills form_template_fields). Migrates ownership off the app-side analyse/route.ts. (Diverged from worker recommendation Option D — app creates row, pipeline fills fields. Liam chose the fully-pipeline-owned route.)
• OQ-52-ONTOLOGY — The CV-loader parity crash IS a prerequisite gate for the form path. (Per worker recommendation. Fix mechanism — extend Zod STATUS_VALUES to accept applied vs re-baseline the four APPLIED-* CV files, plus the form_type triple-source lockstep — deferred to TECH.)

PRODUCT-gate ratifications (recorded here)

• OQ-52-CATALOGUE — Confirm the worker proposal. Path B writes per-instance form_template_fields; Path C (AI-cataloguing skill, human-confirmed) authors the GLOBAL form_template_requirements catalogue (no workspace FK, carries requirement_embedding); T10 (procurement-question-matching, separate spec) reads the catalogue for scoring/retrieval/ranking. This spec ends at a populated catalogue requirement; T10 owns matching.
• OQ-52-LOSSY — Fold to a Path-A follow-up, as a sub-task / new task (NOT a backlog item). The existing flow.py q_a_extractions write drops expected_response_kind / evaluation_criteria / evidence_requirements / scope_tags. Open a dedicated Task (or canonical-pipeline subtask) at integration; out of scope for ID-52.
• OQ-52-UI-UPLOAD-TENSION — Thin front-end drops file into resolved workspace folder. Keep the interactive app upload UI, but it drops the file into the folder→workspace path so the same pipeline write applies — no second, independent writer. Preserves UX without dual-write tension. Inv-6 and Inv-16 hold.