Form-Extraction Subsystem — ACCEPTANCE

Type: Acceptance attestation. Companion to RESEARCH.md, PRODUCT.md (26 numbered behaviour invariants), TECH.md (§3.1 validation matrix) and PLAN.md. Authored for Subtask {52.16} (final acceptance + Inv-26 AI-invisible review). UK English throughout.

This document records the acceptance state of the form-extraction subsystem (ID-52) against the TECH.md §3.1 per-invariant validation matrix and the PRODUCT acceptance-fixtures table. It is the durable completion artefact for {52.16}; it does not re-open any ratified decision.

1. Corpus measurements (against the live files)

The four real corpus forms in docs/testing/test-data/templates/ were run directly through the per-format readers (scripts/cocoindex_pipeline/form_extractors/{pdf,xlsx,docx}.py::extract) — the same deterministic extraction code the pipeline orchestrator (orchestrator.py::extract_form_structure) dispatches to. Counts are measured, not estimated:

Set	Format	Fields	Mandatory (Inv-10)	Word limits (Inv-11)	Reference URLs (Inv-14)	Placeholders (Inv-9)	Full coords (Inv-8)	Sections (Inv-12)
SQ	PDF	113	43	2	26	0	113	10
ITT EFA	XLSX	19	0	0	0	0	19	6
CSP	XLSX	45	0	0	14	42	45	25
ITT Charnwood	DOCX	58	0	0	0	58	7	3

Reading against the PRODUCT acceptance-fixtures table:

SQ (PDF) — Inv-15 full 57-page extent. 113 extracted fields is the deep read of the true 57-page document, not the handful an 8-page container artefact would yield. 43 carry an explicit M mandatory flag (Inv-10); 26 carry reference URLs (Inv-14). The PRODUCT “~66 reqs” figure was a question-count estimate; the reader’s field-level count (questions plus their schema sub-rows) is the measured reality and is internally consistent with the unit tests that pin the page-extent and the specific [500] words / Annex-B M-flag rows.
ITT EFA (XLSX) — Inv-13 dedup. 19 fields, not 2N — the Bidder 1 and Bidder 2 columns are deduped (no double-count). All 19 carry full row/col/table coordinates (Inv-8).
CSP (XLSX) — Inv-9 / Inv-14 / Inv-12. 42 of 45 fields are answer placeholders (TYPE RESPONSE HERE>>>> cells marked, not counted as questions); 14 carry NCSC reference URLs; 25 distinct section names span the letter-keyed preamble and the numbered principles.
ITT Charnwood (DOCX) — Inv-9 / Inv-8. 58 fields extracted from the DOCX paragraphs and tables; the blank-form grid rows surface as placeholders. (The companion ITT Evaluation Matrix.xls is out of automated scope per Inv-3 — a legacy .xls the orchestrator skips.)

2. Per-invariant validation matrix (TECH §3.1)

Status legend: PASS (run) — a test was executed and passed this session; PASS (measured) — verified by direct corpus measurement (§1); PASS (structural) — verified against database.types.ts / source; authored, verified-when-wired — the integration test exists and is correct but env-gated (see §4).

Inv	Verification	Status
Inv-1 (CV gate + triple-source lockstep)	`markdown-parity.test.ts` + `form-type-parity.test.ts`	PASS (run)
Inv-2 (formats {PDF,XLSX,DOCX})	`test_form_extractors.py`	PASS (run)
Inv-3 (`.xls` skip, no raise)	`test_form_extractors.py`	PASS (run)
Inv-4 (workspace resolution deterministic)	`test_workspace_resolver.py`	PASS (run)
Inv-5 (loud resolution failure, no sentinel)	`test_workspace_resolver.py` (unit) + integration	PASS (run, unit) / authored
Inv-6 (pipeline owns the write)	`form-extraction.integration.test.ts`	authored, verified-when-wired
Inv-7 (form-level metadata)	`form-extraction.integration.test.ts`	authored, verified-when-wired
Inv-8 (coordinates)	`test_form_extractors.py` + §1 (full_coords)	PASS (run + measured)
Inv-9 (placeholder vs authored)	`test_form_extractors.py` + §1 (CSP 42 / Charnwood 58 placeholders)	PASS (run + measured)
Inv-10 (mandatory flag)	`test_form_extractors.py` + §1 (SQ 43 mandatory)	PASS (run + measured)
Inv-11 (word limit)	`test_form_extractors.py` + §1 (SQ word limits)	PASS (run + measured)
Inv-12 (section + sequence)	`test_form_extractors.py` + §1 (sections)	PASS (run + measured)
Inv-13 (per-form dedup)	`test_form_extractors.py` + §1 (EFA 19, not 2N)	PASS (run + measured)
Inv-14 (reference URLs)	`test_form_extractors.py` + §1 (CSP 14, SQ 26)	PASS (run + measured)
Inv-15 (full content extent)	`test_form_extractors.py` + §1 (SQ 113 ≫ 8-page)	PASS (run + measured)
Inv-16 (re-ingest idempotency)	`form-extraction.integration.test.ts`	authored, verified-when-wired
Inv-17 (failure isolation)	`form-extraction.integration.test.ts`	authored, verified-when-wired
Inv-18 (no silent loss of metadata)	§1 (each facet present per fixture) + integration	PASS (measured) / authored
Inv-19 (Path-A Mode-1 unchanged)	`test_analyse_template.py` + `test_extract_docx_tables.py` (no regression)	PASS (run)
Inv-20 (instance ≠ catalogue)	pipeline never writes `form_template_requirements` (grep)	PASS (structural)
Inv-21 (human-confirmed catalogue)	`__tests__/lib/catalogue/from-instance.test.ts`	PASS (run)
Inv-22 (T10 read boundary)	`from-instance.test.ts` + schema shape	PASS (structural)
Inv-23 (catalogue has no workspace FK)	`database.types.ts` Relationships = `template_type → form_types.key` only	PASS (structural)
Inv-24 (auth gate on write)	`from-instance.test.ts` (viewer refused)	PASS (run)
Inv-25 (workspace scoping RLS)	`form-extraction-rls.integration.test.ts`	authored, verified-when-wired
Inv-26 (AI-invisible surfacing)	grep over new code + UI surfacing + manual	PASS (grep + manual)

3. Inv-26 — AI-invisible review

The extracted question set must read as ordinary structured knowledge-base records, never as a user-facing “AI extraction” feature (docs/reference/ai-visibility-policy.md).

New cataloguing code (lib/catalogue/, scripts/catalogue-from-instance.ts, .claude/skills/catalogue-form-requirements/): grep for user-facing AI / AI-extracted / AI extraction / artificial intelligence strings — zero matches. No standalone AI token in any of those files.
UI surfacing of form_template_fields (components/procurement/template-field-review.tsx, template-completion-summary.tsx, the templates page): grep for AI tells (AI-powered, AI-generated, LLM, machine learning, …) — zero matches; emoji scan over the rewired templates-page copy — zero matches. Fields render as ordinary structured records (question text, flags, limits, sections, references).
Retirement copy ({52.15}): the rewired app/procurement/[id]/templates/page.tsx surfaces “This template is processed automatically. Refreshing to check for results.” — UK English, no emoji, no AI tells.

4. Known limitation — live-staging e2e is infra-gated (not a failure)

The form-extraction.integration.test.ts and form-extraction-rls.integration.test.ts suites drive the cocoindex pipeline via the fixture-staging service pattern. Their ENABLED gate requires four env vars (COCOINDEX_STAGING_URL, COCOINDEX_SOURCE_PATH, COCOINDEX_FIXTURE_STAGING_URL, live-DB credentials). The three COCOINDEX_* fixture-staging vars are absent on the current staging .env.local, so both suites skip clean — bun run test:integration passes with zero failures, which is the expected and correct state.

The fixture-staging infrastructure is an open backlog item (OQ-53-FIXTURE-STAGING / backlog-191); it is not wired in this environment. The live pipeline run of the four corpus forms against staging — the PRODUCT-acceptance-table assertion of row counts written to the turayklvaunphgbgscat branch — therefore cannot be executed here. It is “verified-when-wired”: the integration tests are the durable artefact that runs once backlog-191 lands. The extraction behaviour those tests assert at the DB-write boundary (Inv-6/7/16/17/25) is corroborated this session at the reader boundary by §1 (measured corpus counts) and the per-format unit suites.

5. Test-suite results (this session)

bun run test (full unit suite): 13760 passed, 6 failed, 33 skipped. The 6 failures are in two files unrelated to ID-52 and not in the {52.15} diff:
- __tests__/lib/mammoth-track-changes.test.ts (5) — ENOENT on a local-only client fixture (docs/client-documentation/docx/DRAFT 2026 … Copy.docx) not checked into the repo, plus a pandoc ground-truth call. Environmental / fixture-availability, pre-existing.
- __tests__/mcp/review-item-prompt.test.ts (1) — governance freshness lifecycle token drift (Procurement-Discovered) in lib/mcp/resources.ts. Pre-existing content drift, separate domain. Every ID-52 form-extraction and {52.15}-retirement test passes.
bun run test:integration (full suite): 193 passed, 0 failed, 119 skipped (79 files: 33 passed, 46 skipped). GREEN. The two ID-52 form-extraction suites are among the env-gated skips (§4).
python3 -m pytest form-extraction set (test_form_extractors.py, test_workspace_resolver.py, test_analyse_template.py, test_extract_docx_tables.py): 170 passed.
bun run lint: clean (0 errors). bun run build: PASS.

6. {52.15} retirement summary

The legacy synchronous app-side analyse path was retired (commit on this branch):

Deleted app/api/procurement/[id]/templates/[templateId]/analyse/route.ts.
Rewired app/procurement/[id]/templates/page.tsx (handleAnalyse → handleRefreshStatus): info-toast “processed automatically” + refresh; the dead poll machinery removed.
Removed template_analyse from the JobType union (lib/queue/envelope.ts), the orphaned TemplateAnalyseBodySchema (lib/validation/schemas.ts), the template_analyse queue-consumer branch + orphaned analyse_template_job (scripts/bid_worker.py), and the corresponding tests.
analyse_template.py is untouched — its helpers are reused by the {52.11} DOCX reader.
Zero remaining template_analyse / analyse-route code references in app/, components/, hooks/, lib/. gitnexus_impact on the route handler returned 0 upstream graph callers (LOW); gitnexus_detect_changes confirmed the change scope.

The DB processing_queue_job_type_check constraint deliberately retains the template_analyse value (narrowing a CHECK requires proving no extant rows hold it); the union being a strict subset is harmless. Documented in lib/queue/envelope.ts.