Form-Extraction Subsystem — PLAN

Form-Extraction Subsystem — PLAN

Type: PLAN (implementation decomposition). Companion to RESEARCH.md, PRODUCT.md (26 numbered behaviour invariants, ratified S273) and TECH.md (§2.1–§2.11 proposed changes, §3 testing matrix). This document decomposes the ratified PRODUCT + TECH pair into {52.5+} implementation Subtasks an Executor can pick up.

Task: ID-52 (form-extraction). Subtask: {52.4}. Authored S274 by a fresh Planner context (distinct from the {52.1}/{52.2}/{52.3} authors per Q-PLANNER-2 fresh-instance rule). UK English throughout.

Sibling-only dependency rule honoured. Every Subtask dependency below is another ID-52 sibling (52.5–52.16). Cross-Task dependencies (T10 read-boundary, ID-50 route coordination, ID-54 lossy follow-up) are surfaced in §3 Sequencing notes, not as Subtask deps. No constraint bent.

---

1. Overview

Build Path B (deterministic per-format readers — PDF/XLSX/DOCX — fully pipeline-owned, Python-side) and Path C (human-confirmed cataloguing Claude Plugin Skill, TS-side) per the ratified PRODUCT spec. Decomposition follows TECH §2’s proposed-changes spine vertically: each slice delivers a working, testable layer — ontology gate first, schema next, per-format readers in parallel, then the cocoindex orchestration glue + idempotent write, the cataloguing skill, and finally the legacy-route retirement.

2. Architecture decisions inherited (do not re-open)

• R3 (folder→workspace, fully pipeline-owned write) — TECH §2.1 commits a JSON manifest at <COCOINDEX_SOURCE_PATH>/.kh-workspace-map.json + longest-prefix match resolver; the app-side analyse/route.ts retires (TECH §4 Follow-ups + slice 8 below).
• R4 (CV-loader Option (b) — re-baseline, NOT enum extension) — TECH §2.6a commits re-baselining the 4 APPLIED-S{NNN} files to status: active; the session-ID provenance survives in core_seed_path (unchanged).
• form_type triple-source lockstep — TECH §2.6b drives Python from the taxonomy_snapshot.json fixture (extending bun run sync:taxonomy to fold in form_types), removes the hard-coded Literal in extraction.py:130-142, adds a Vitest parity test mirroring markdown-parity.test.ts.
• Path B is Python-only (no TS/Python boundary crossing) — TECH §2.2 commits pdfplumber (PDF) + openpyxl (XLSX) + python-docx (DOCX) under scripts/cocoindex_pipeline/form_extractors/. DOCX reader REUSES the _classify_header / _QUESTION_HEADERS / _detect_merged_cells / _is_empty_or_placeholder / _extract_word_limit prior art from scripts/extract_tender_questions.py + scripts/analyse_template.py.
• Inv-16 idempotency mechanism — TECH §2.8 commits deterministic UUID5 (uuid5(_KH_PIPELINE_DOC_NS, "ft:{rel_path}") / uuid5(_KH_PIPELINE_DOC_NS, "ftf:{rel_path}:{sequence}")) + @coco.fn(memo=True) fingerprint substrate + per-file stale-row trim (DELETE WHERE template_id = $1 AND sequence > $2).
• Path C is a Claude Plugin Skill (TS-side) — TECH §2.7 commits .claude/skills/catalogue-form-requirements/SKILL.md + scripts/catalogue-from-instance.ts
  • lib/catalogue/from-instance.ts. Auth gate via getAuthorisedClient(['admin','editor'])
  • authFailureResponse(auth) on the write step. tryQuery() from @/lib/supabase/safe.

3. Sequencing notes — cross-Task coordination (NOT deps)

The Subtask graph below uses sibling-only dependencies. The following cross-Task boundaries are documented inputs/outputs, not dependencies:

• T10 (procurement-question-matching, separate spec / separate Task) READS form_template_requirements (the catalogue Path C writes). Path C’s write shape (requirement_embedding, matching_keywords, requirement_type, template_type FK to form_types.key) is the read boundary T10 consumes. Inv-22.
• ID-50 (OPS-T1 route rollout) — independent of pipeline stages, but the retirement Subtask {52.15} deletes app/api/procurement/[id]/templates/[templateId]/analyse/route.ts. If ID-50’s per-route-group wave touches the procurement/[id]/templates group, sequence retirement BEFORE that wave to avoid colliding on a soon-deleted file. Not a dep — a coordination signal for the Orchestrator.
• ID-54 (Path-A lossy q_a_extractions follow-up) — separate Task opened S273. The existing flow.py:919-934 block stays untouched in ID-52 per PRODUCT OQ-52-LOSSY ratification. {52.13}‘s integration test asserts Path A’s existing q_a_extractions behaviour PASSES unchanged (Inv-19), but the lossy-write fix is ID-54.

4. Task list — phases + Subtasks

Phase 1 — Ontology gate (gates EVERYTHING after)

{52.5} CV-loader re-baseline + stale-claim correction

Description. Re-baseline 4 APPLIED-S{NNN} ontology CV files to status: active and correct the 2 stale claims in 26-form-type.md so loadOntologyCVs() stops throwing.

Acceptance criteria.

• All four CV files load: 26-form-type.md, 30-procurement-vehicle.md, 31-procurement-vehicle-instance.md, 32-q-a-pair.md carry status: active in frontmatter (was APPLIED-S246 / APPLIED-S249).
• 26-form-type.md two stale claims corrected: (a) “table not yet in production schema” → struck (table IS live at database.types.ts:1755); (b) the template_requirements.template_type 10-value CHECK reference → struck (renamed to form_template_requirements, FK to form_types.key).
• core_seed_path retains the migration filename (provenance preserved).

Verification. bun run test __tests__/lib/ontology/markdown-parity.test.ts PASSES; loadOntologyCVs() returns 29 records (RESEARCH §4.2 / TECH §2.6a).

Dependencies. None.

Files likely touched. docs/ontology/26-form-type.md, docs/ontology/30-procurement-vehicle.md, docs/ontology/31-procurement-vehicle-instance.md, docs/ontology/32-q-a-pair.md.

Scope. XS (4 markdown edits + test pass).

{52.5a} Layer-5 KG-entity CV schema shape (cross-Task wp6-ontology-harness touch)

Description. Relax OntologyCVSchema in lib/ontology/schemas.ts per TECH §2.6c so Layer-5 KG-entity CV files (currently only 32-q-a-pair.md; future siblings expected) load alongside the 28 Layer-1 enumerable CVs without schema rejection. Ratifies the Layer-5 KG-entity semantic the 32-q-a-pair.md author captured at S249 (the # baseline_values intentionally omitted — q_a_pair is a Layer 5 KG entity (not a Layer 1 CV) comment becomes structurally honoured rather than silently rejected).

Cross-Task ratification recorded inline: this Subtask touches lib/ontology/schemas.ts — shared infrastructure under wp6-ontology-harness’s ownership (lib/ontology/schemas.ts:11). The touch is RATIFIED S275 under OQ-52-WAVE-1-A Option 3 (sub-orchestrator + task-executor + Liam concur). The wp6 B2 + E1 verifier reports surveyed only the 28 Layer-1 corpus pre-32-q-a-pair.md and recorded no Layer-5 KG-entity treatment because no Layer-5 file existed at C2 time. The wp6 D1 invariants for Layer-1 CVs are preserved unchanged.

Acceptance criteria.

• lib/ontology/schemas.ts OntologyCVSchema refactored per TECH §2.6c pattern (OntologyCVBaseSchema + .superRefine() per-layer).
• loadOntologyCVs() returns 29 records (28 Layer-1 + 1 Layer-5 q_a_pair).
• __tests__/lib/ontology/markdown-parity.test.ts extended with three cases (per TECH §2.6c).
• 28-record Layer-1 assertions remain unchanged (no regression to wp6 D1 contract).
• bun run test, bun lint PASS.
• type OntologyCV = z.infer<typeof OntologyCVSchema> remains consumer-usable.

Dependencies. None (independent of {52.5}). Disjoint surfaces: {52.5} edits 4 .md files in docs/ontology/; {52.5a} edits lib/ontology/schemas.ts + the parity test. MAY run in parallel under separate Executors.

Phase-2 fan-out gating. {52.5a} joins {52.5} as the Phase-1 sequential gate. Phase-2 ({52.6}/{52.7}/{52.8}) MUST NOT dispatch until BOTH {52.5} AND {52.5a} are done — until {52.5a} lands, loadOntologyCVs() continues to throw on 32-q-a-pair.md even after {52.5} re-baselines its frontmatter (the status-enum throw clears but the four structural-shape violations remain).

Files likely touched. lib/ontology/schemas.ts (~30 LOC delta); __tests__/lib/ontology/markdown-parity.test.ts (3 new test cases).

Scope. XS-to-S (≤30 LOC + 3 test cases).

Risk. LOW — schema relaxation is non-destructive to existing Layer-1 files; wp6 D1 R-A “Zod default-strip” invariant preserved (base schema retains .strict()).

Cross-Task signal — surface to Orchestrator at dispatch. Executor MUST run gitnexus_impact({target: "OntologyCVSchema", direction: "upstream"}) first per CLAUDE.md mandate. Expected: only lib/ontology/loader.ts + parity test surface. HIGH/CRITICAL pauses + escalates.

{52.6} form_type triple-source lockstep

Description. Drive Python FormMetadata.form_type from the taxonomy snapshot fixture (mirroring _load_canonical_content_types pattern); extend bun run sync:taxonomy to include form_types; add Vitest parity test asserting 26-form-type.md baseline_values keys == live form_types.key rows.

Acceptance criteria.

• scripts/tests/fixtures/taxonomy_snapshot.json has a form_types array ({key, label}[]) matching the live form_types table.
• bun run sync:taxonomy regen script extended to fold in SELECT key, label FROM form_types (one-line addition).
• scripts/cocoindex_pipeline/extraction.py:121-142 FormMetadata.form_type: Literal[...] replaced with form_type: str + a field_validator asserting the value is in taxonomy_snapshot.json:form_types (matches _load_canonical_content_types pattern at extraction.py:76-86).
• New test __tests__/lib/ontology/form-type-parity.test.ts asserts 26-form-type.md keys == snapshot keys.

Verification. bun run test __tests__/lib/ontology/form-type-parity.test.ts PASSES; python3 -m pytest scripts/tests/test_cocoindex_extraction.py -v PASSES.

Dependencies. {52.5}.

Files likely touched. scripts/tests/fixtures/taxonomy_snapshot.json, scripts/sync-plugin-taxonomy.ts (or whichever script bun run sync:taxonomy invokes), scripts/cocoindex_pipeline/extraction.py, __tests__/lib/ontology/form-type-parity.test.ts (new).

Scope. S.

Checkpoint A — Ontology gate clears

• markdown-parity + form-type-parity tests PASS.
• loadOntologyCVs() no longer throws.
• Python pipeline reads form_type CV from the snapshot fixture.
• Form path can run (Inv-1 satisfied).

Phase 2 — Schema additions

{52.7} Migration M1 — schema additions + CHECK widening

Description. Add is_mandatory (Inv-10) + reference_urls (Inv-14) to form_template_fields; add ingest_source (provenance) to form_templates; widen form_templates.mime_type CHECK from DOCX-only to {PDF, XLSX, DOCX}. Created via Supabase CLI (supabase migration new + db push --env=staging), regen database.types.ts.

Acceptance criteria.

• New migration supabase/migrations/<timestamp>_id52_form_extraction_schema.sql exists and contains the exact SQL in TECH §2.6 M1.
• bash scripts/check-project-ref.sh (or cat supabase/.temp/project-ref) confirms staging branch (turayklvaunphgbgscat) BEFORE push.
• supabase db push --env=staging succeeds; migration-revoke-guard.yml CI workflow PASS (no new anon-EXECUTE grants).
• supabase gen types typescript --project-id turayklvaunphgbgscat --schema public > supabase/types/database.types.ts regenerated; diff includes the 3 new columns + the widened CHECK.
• is_mandatory boolean NULL comment exactly matches TECH §2.6 (Inv-10 substrate semantics).
• reference_urls text[] NULL comment matches TECH §2.6 (Inv-14).

Verification. bun run lint PASSES (no database.types.ts drift errors); bun run test:integration PASS on any test that already touches form_templates / form_template_fields.

Dependencies. {52.5} (Inv-1 gate must clear before any form-touching migration lands).

Files likely touched. supabase/migrations/<timestamp>_id52_form_extraction_schema.sql (new), supabase/types/database.types.ts (regenerated, never hand-edited).

Scope. S.

{52.M1b} Migration M1b — Dedicated form_templates metadata columns

Description. S275 amendment (Liam ratification) — promotes the 4 form-level metadata facets (form_type FK to form_types.key, deadline timestamptz, issuing_organisation text, evaluation_methodology text) to first-class columns on form_templates, instead of v1-packing them into description / structure_path and deferring schema-clean breakout to v1.1. Rationale: v1.1 needs them regardless; deferring forces a later migration + backfill + writer rewrite, all avoidable if columns land before Phase-5 writer ships. Created via Supabase CLI per TECH §2.6d migration SQL.

Acceptance criteria.

• New migration supabase/migrations/<timestamp>_id52_form_templates_dedicated_metadata_columns.sql exists and contains the exact SQL in TECH §2.6d.
• bash scripts/check-project-ref.sh confirms staging branch BEFORE push.
• supabase db push --env=staging succeeds; migration-revoke-guard.yml CI PASS.
• database.types.ts regenerated; diff includes the 4 new columns + the FK constraint + the partial index on form_type.
• form_type column carries REFERENCES public.form_types(key) FK constraint (DB-level referential integrity to the CV).
• All 4 columns COMMENT-documented per TECH §2.6d.
• No data backfill needed (v1 writer ships from day one with values).

Verification. bun run lint PASSES (no database.types.ts drift); migration re-applied is no-op (IF NOT EXISTS guards); psql -c "\\d form_templates" shows the 4 new columns with correct types + comments.

Dependencies. [7] — M1 must land first; M1b extends the same table.

Files likely touched. supabase/migrations/<timestamp>_id52_form_templates_dedicated_metadata_columns.sql (new); supabase/types/database.types.ts (regenerated).

Scope. XS (4 ADD COLUMN + 1 FK + 1 partial index; pure DDL).

Sequencing within ID-52 — load-bearing. MUST land before Phase-5 ({52.12}) writer dispatch so the writer code targets the dedicated columns from day one (per TECH §2.5 step 3 amendment). {52.M1b} → {52.12}+ writer.

Phase 3 — Folder→workspace resolver (parallel with M1 if executors split)

{52.8} Workspace manifest + resolver module

Description. New Python module scripts/cocoindex_pipeline/workspace_resolver.py exporting load_workspace_manifest(path) + resolve_workspace(manifest, rel_path) per TECH §2.1. Pydantic WorkspaceManifest schema-versioned. Longest-prefix match; ambiguous (equal-length) prefixes → ResolutionFailure; missing/unparseable manifest → ManifestLoadError (flow aborts at start). UUIDs NOT verified against workspaces table at load time (FK at INSERT gives canonical error per TECH §2.1).

Acceptance criteria.

• WorkspaceManifest Pydantic model rejects missing schema_version, missing mappings, duplicate identical path_prefix values.
• resolve_workspace(manifest, "phew-procurement/SQ.pdf") returns the mapped workspace_id UUID; same call returns the SAME UUID on repeated invocation (deterministic per Inv-4).
• resolve_workspace(manifest, "unmapped/X.pdf") returns ResolutionFailure (NOT a default; Inv-5).
• Ambiguous resolution: two mappings of equal path_prefix length matching the same rel_path → ResolutionFailure.
• Longest-prefix wins: acme-bids/2026/ beats acme-bids/ for acme-bids/2026/foo.pdf.

Verification. python3 -m pytest scripts/tests/test_workspace_resolver.py -v PASSES all unit cases above.

Dependencies. None (pure module + tests; can run in parallel with {52.7}).

Files likely touched. scripts/cocoindex_pipeline/workspace_resolver.py (new ~80 LOC), scripts/tests/test_workspace_resolver.py (new).

Scope. S.

Phase 4 — Per-format readers (PARALLELISABLE after {52.9} delivers shared.py)

All three readers (PDF / XLSX / DOCX) export the same ExtractedField + ExtractedForm Pydantic shape per TECH §2.2 — covers Inv-2 (supported formats {PDF, XLSX, DOCX}). They are independently testable against the corpus fixtures in docs/testing/test-data/templates/. The extract(raw_bytes, filename) -> ExtractedForm signature is the contract {52.12} consumes. Sequencing within the phase: {52.9} must land first (it creates form_extractors/shared.py with the shared Pydantic models and FormExtractionError); {52.10} and {52.11} import from shared.py and may then run in parallel.

{52.9} PDF reader (pdfplumber)

Description. scripts/cocoindex_pipeline/form_extractors/pdf.py — pdfplumber-based extractor for blank PDF forms. Handles 2-column Annex tables (left cell = M/O flag, right cell = question text), inline word-limit tokens ([NNN] words), hierarchical numbering, 57-page container-artefact bypass per Inv-15. Pin pdfplumber stays at ==0.11.9 (already in requirements.txt).

Acceptance criteria.

• Module exports extract(raw_bytes: bytes, filename: str) -> ExtractedForm async function with the exact Pydantic shape declared in TECH §2.2.
• SQ PDF standard-selection-questionnaire-ppn-03-24.pdf reads as 57 pages (pdfplumber.open(...).pages count), NOT 8 (Inv-15).
• An SQ Annex B Part 1 M-flagged schema row (e.g. the ['M', 'Registered VAT number'] left-cell pair) extracts with is_mandatory=true (M flag) and section_name carrying “Annex B” (Inv-10, Inv-12); a question carrying an inline [500] words token extracts word_limit=500 from that token (Inv-11).
• A PDF prose question (no meaningful column position) has col_index=NULL (Inv-8 “leave empty rather than fabricate”).
• Placeholder-vs-authored distinguished: empty answer cells under authored questions do NOT cause the question to be dropped (Inv-9).
• Reference URLs (if encountered on a question or its section) preserved on the field row (Inv-14).
• Raises FormExtractionError(reason, rel_path, …) typed exception on unrecoverable read failure; never silently returns empty ExtractedForm (Inv-17).

Verification. python3 -m pytest scripts/tests/test_form_extractors.py -v -k pdf PASSES; real-behaviour discipline — NO mocks of pdfplumber internals (per docs/reference/test-philosophy.md).

Dependencies. {52.7} (needs is_mandatory + reference_urls columns in the ExtractedField → schema mapping).

Files likely touched. scripts/cocoindex_pipeline/form_extractors/__init__.py (new), scripts/cocoindex_pipeline/form_extractors/pdf.py (new), scripts/cocoindex_pipeline/form_extractors/shared.py (new — ExtractedField + ExtractedForm + FormExtractionError shared models), scripts/tests/test_form_extractors.py (new), scripts/tests/fixtures/form-extraction/ (new — symlinks or copies of corpus PDF).

Scope. M.

{52.10} XLSX reader (openpyxl) + per-form dedup

Description. scripts/cocoindex_pipeline/form_extractors/xlsx.py — openpyxl-based extractor for blank XLSX forms. Handles heavily merged cells (EFA 42 ranges, CSP 23 ranges), scoring-matrix layouts (Ref | Criteria | Weighting | Page Limit | Remarks), mixed letter/numbered sections (CSP), and the per-form dedup keyed on (section_name, normalise(question_text)) per TECH §2.3 (Inv-13). Add openpyxl as a direct pin in requirements.txt if not already explicit.

Acceptance criteria.

• EFA evaluation-matrix-itt-vol8.xlsx produces N fields, NOT 2N (Bidder 1 ≡ Bidder 2 deduped per Inv-13).
• EFA scoring-matrix question row carries row_index, col_index, table_index populated (Inv-8); “Page Limit” column value → word_limit (Inv-11); section_name carries “Part 2 — OVERALL APPROACH” etc. (Inv-12).
• CSP Cloud Security Principles Checklist V5_3.xlsx reads: TYPE RESPONSE HERE>>>> cells → field_type='placeholder' (Inv-9); NCSC URLs preserved in reference_urls (Inv-14); letter-keyed preamble (A, B1) AND numbered principles (PRINCIPLE 1) both sectioned via section_name (Inv-12).
• Per-form dedup does NOT collapse two genuinely different questions sharing text across different forms (workspace-scoped — catalogue-level reuse is Path-C’s concern per TECH §2.3).
• Same extract() Pydantic shape as {52.9}; same FormExtractionError typed exception surface (Inv-17).

Verification. python3 -m pytest scripts/tests/test_form_extractors.py -v -k xlsx PASSES against EFA + CSP fixtures.

Dependencies. {52.7}, {52.9} (consumes shared ExtractedField + ExtractedForm + FormExtractionError models from form_extractors/shared.py delivered by {52.9}).

Files likely touched. scripts/cocoindex_pipeline/form_extractors/xlsx.py (new), scripts/tests/test_form_extractors.py (XLSX cases), requirements.txt (add explicit openpyxl pin if not present).

Scope. M.

{52.11} DOCX reader (python-docx) — reusing existing prior-art helpers

Description. scripts/cocoindex_pipeline/form_extractors/docx.py — python-docx extractor for blank DOCX forms. REUSES _classify_header, _QUESTION_HEADERS, _detect_merged_cells, _is_empty_or_placeholder, _extract_word_limit, _extract_section_headings, _has_tracked_changes from scripts/extract_tender_questions.py (507 lines) + scripts/analyse_template.py (363 lines) under a thin compatibility wrapper module. The existing 1-50 placeholder regex set in analyse_template.py (covering [Insert…], {{…}}, <<…>>, n/a, dash/ellipsis sentinels) folded in as-is.

Acceptance criteria.

• Charnwood ITT Services.docx (1908 paragraphs + 8 tables) extracts the question set from BOTH paragraphs AND tables (Inv-8 paras+tables, Inv-12 section + sequence).
• Insert question title grid rows → field_type='placeholder', placeholder_text populated, question_text=NULL (Inv-9).
• Blank scoring-rubric “Description of criteria N” placeholder rows in the companion .xls are NOT extracted as questions (this is the legacy .xls skip path in {52.12} — DOCX reader itself just handles the .docx).
• DOCX-side word limits (where present in question prose or a dedicated column) extracted via the reused _extract_word_limit helper (Inv-11).
• Same extract() Pydantic shape as {52.9} + {52.10}; same FormExtractionError surface.

Verification. python3 -m pytest scripts/tests/test_form_extractors.py -v -k docx PASSES against Charnwood fixture; python3 -m pytest scripts/tests/test_analyse_template.py scripts/tests/test_extract_docx_tables.py (existing) continues to PASS (no regression in the reused helpers).

Dependencies. {52.7}, {52.9} (consumes shared ExtractedField + ExtractedForm + FormExtractionError models from form_extractors/shared.py delivered by {52.9}).

Files likely touched. scripts/cocoindex_pipeline/form_extractors/docx.py (new — thin wrapper around the reused helpers), scripts/tests/test_form_extractors.py (DOCX cases). NO changes to scripts/extract_tender_questions.py or scripts/analyse_template.py (imports only).

Scope. M.

Checkpoint B — Per-format readers green

• Each of {52.9}, {52.10}, {52.11} PASSES unit tests against its corpus fixture.
• Shared ExtractedField + ExtractedForm Pydantic shape consumed by {52.12}.

Phase 5 — Path B orchestration + pipeline write

{52.12} Custom @coco.fn orchestrator + mount_table_target wiring + idempotency

Description. Wire the per-format readers into flow.py. Add extract_form_structure custom @coco.fn(memo=True) orchestrator (TECH §2.4 — deterministic, NO LLM call); add two new mount_table_target calls for form_templates

• form_template_fields (TECH §2.5, after line 1029 alongside ci_target / qa_target / sd_target); extend ingest_file’s signature to accept ft_target + ftf_target; add the form-write block per TECH §2.5 steps 1-4 (workspace resolve → extractor call → form_templates declare → trim stale fields per TECH §2.8 → form_template_fields declares per field); load workspace manifest once at flow start per TECH §2.1. Idempotency mechanics per TECH §2.8: uuid5(_KH_PIPELINE_DOC_NS, "ft:{rel_path}") + uuid5(_KH_PIPELINE_DOC_NS, "ftf:{rel_path}:{sequence}") + stale-row trim (DELETE FROM form_template_fields WHERE template_id = $1 AND sequence > $2).

Acceptance criteria.

• Manifest loaded once at app_main start (TECH §2.1); failure → flow aborts with structured manifest_missing / manifest_invalid error via _emit_stage_error_log.
• Resolution failure per file → _emit_stage_error_log(stage='workspace_resolution', ...) + zero form_template_fields rows for that file + zero form_templates rows (Inv-5 — no sentinel).
• Extraction failure per file → _emit_stage_error_log(stage='form_extraction', ...)
  • ONE form_templates row with status='analysis_failed' + zero fields (Inv-17).
• Successful extraction → form_templates row with status='analysed', workspace_id from resolver, created_by=SERVICE_ACCOUNT_UUID (a0000000-0000-4000-8000-000000000001), mime_type from MIME_BY_SUFFIX[suffix], file_size from file.size, field_count=len(fields), mapped_count=0, ingest_source='pipeline', name from form_metadata.form_title or file.stem, description from form_metadata.evaluation_methodology.
• Each field declared with deterministic UUID5; question_text / placeholder_text / field_type / fill_status / row_index / col_index / table_index / section_name / sequence / word_limit / is_mandatory / reference_urls populated from ExtractedField.
• Stale-row trim runs BEFORE field declares (DELETE WHERE template_id = $1 AND sequence > $2).
• .xls files → extract_form_structure returns None + logs form_extractor.skip (Inv-3); no error raised.
• coco.mount_each signature extended (positional args order matches TECH §2.5).
• managed_by=ManagedBy.USER on the two new mounts (preserves “DDL via CLI only”).

Verification. python3 -m pytest scripts/tests/test_cocoindex_flow_write_path.py scripts/tests/test_cocoindex_flow_stage_counts.py -v PASSES; bun run test __tests__/lib/ontology/markdown-parity.test.ts still PASSES (no upstream regression).

Dependencies. {52.7}, {52.8}, {52.9}, {52.10}, {52.11}.

Files likely touched. scripts/cocoindex_pipeline/flow.py (extended — app_main manifest load, two new mounts, ingest_file form-write block, signature change), scripts/cocoindex_pipeline/form_extractors/__init__.py (export the public extract_form_structure orchestrator), scripts/tests/test_cocoindex_flow_write_path.py (extended with form-write assertions).

Scope. M/L (the largest single Subtask; touches the hot path). Executor sizing note: treat as full-session L (touches flow.py hot path; mount_table_target wiring

• form-write block + idempotency stale-row trim block + ingest_file signature change all land together in one Subtask — do not partial-land).

{52.13} Integration test — pipeline-owned write + idempotency + failure isolation + RLS

Description. End-to-end integration test __tests__/integration/form-extraction.integration.test.ts covering Inv-6 (pipeline owns write), Inv-16 (re-ingest idempotency), Inv-17 (failure isolation), Inv-7 / Inv-15 (SQ PDF metadata + full 57-page extent), Inv-19 (Path A q_a_extractions unchanged) + RLS test __tests__/integration/form-extraction-rls.integration.test.ts for Inv-25. Real Anthropic + Supabase per bun run test:integration convention; copies the 4 corpus fixtures into a temp COCOINDEX_SOURCE_PATH.

Acceptance criteria.

• Place SQ.pdf in workspace-mapped folder; run app_main once; assert one form_templates row + N form_template_fields rows; NO app interaction. (Inv-6).
• Same SQ.pdf re-ingested unchanged → still 1 + N rows, same UUIDs (Inv-16 happy path).
• Source mutated to remove field 7 → re-ingest yields N-1 fields; stranded row trimmed (Inv-16 shrink path).
• Batch [corrupt.pdf, sq.pdf, efa.xlsx, charnwood.docx] → 3 successful instances + 1 status='analysis_failed', batch not halted (Inv-17).
• SQ produces form_templates row with mime_type='application/pdf', file_size>0, description populated from FormMetadata.evaluation_methodology, name matches (Inv-7); content extracted from the full 57-page extent (Inv-15).
• Per-question metadata (is_mandatory, word_limit, section_name, reference_urls, row_index / col_index / table_index) populated on form_template_fields rows wherever the source carries it; integration assertions check at least one row per corpus fixture demonstrating each column populated where the source provides it (Inv-18 — no silent loss of per-question metadata).
• RLS test: viewer of workspace A cannot SELECT form_templates rows of workspace B; catalogue rows visible regardless of workspace (Inv-25 — sanity-check the existing RLS on form_templates / form_template_fields).
• Existing Path A q_a_extractions tests (scripts/tests/test_cocoindex_flow_write_path.py against Mode-1 fixtures) PASS unchanged (Inv-19 — Mode-1 not regressed; lossy-fix is ID-54, NOT here).
• Workspace-resolution failure asserts: a form under an unmapped folder produces zero form_templates rows + zero form_template_fields rows + a surfaced _emit_stage_error_log (Inv-5).

Verification. bun run test:integration __tests__/integration/form-extraction.integration.test.ts __tests__/integration/form-extraction-rls.integration.test.ts PASSES.

Dependencies. {52.12}.

Files likely touched. __tests__/integration/form-extraction.integration.test.ts (new), __tests__/integration/form-extraction-rls.integration.test.ts (new), __tests__/fixtures/form-extraction/.kh-workspace-map.json (new — test manifest).

Scope. M.

Checkpoint C — Path B end-to-end green

• Integration test PASSES against the staging Supabase branch.
• Idempotency + failure isolation + RLS verified.
• Inv-1 through Inv-19 (except cataloguing invariants 20-24) covered.

Phase 6 — Path C cataloguing skill

{52.14} Path C — Claude Plugin Skill + generic catalogue-from-instance script + lib helpers

Description. Build the human-confirmed cataloguing path per TECH §2.7. New skill definition at .claude/skills/catalogue-form-requirements/SKILL.md; generic TS template at scripts/catalogue-from-instance.ts (the skill emits per-form copies); shared helpers at lib/catalogue/from-instance.ts (read + Anthropic classify + embed + confirm + tryQuery() write). Auth gate via getAuthorisedClient(['admin','editor']) + authFailureResponse(auth); the write step refuses to call if auth check fails. Reuses the existing _anthropic_retry tenacity pattern + Q_A_FORM_PROMPT extended with requirement_type + matching_keywords + matching_guidance classification. Embedding via the same text-embedding-3-large / dimensions-1024 config as Stage-4 (consistency with T10 read shape). Embedding serialisation per CLAUDE.md JSON.stringify(embedding) for Supabase RPC vector params.

Acceptance criteria.

• Skill SKILL.md exists; describes inputs (form_templates.id UUID, optional template_type override, human-confirmation flag — no default) + output (executable TS file at scripts/catalogue-from-instance-<form_template_id>.ts).
• Generated script reads form_template_fields via Supabase RPC (read-only); calls Anthropic per field; generates requirement_embedding (vector(1024)); presents candidate rows in stdout for explicit y/n per row; halts pending confirmation (Inv-21).
• Confirmation step writes rows to form_template_requirements via tryQuery() from @/lib/supabase/safe; per-row insert in a single transaction wrapper.
• Auth gate: viewer-role caller blocked with authFailureResponse(auth) (Inv-24).
• Catalogue rows carry template_type → form_types.key FK + requirement_type plain string + matching_keywords + matching_guidance + requirement_embedding
  • is_mandatory + section_name (the read shape T10 consumes — Inv-22).
• Catalogue rows have NO workspace_id (reusable global per Inv-23).
• Pipeline ingest does NOT auto-write form_template_requirements; cataloguing only happens via this path (Inv-20).
• No barrel re-exports anywhere in the new code (direct file imports per CLAUDE.md).
• TanStack Query — N/A (this is a CLI script, not a hook); but lib/catalogue/from-instance.ts helpers are direct-imported (no barrel).

Verification. bun run test __tests__/lib/catalogue/from-instance.test.ts PASSES (covers Inv-21 confirmation gate + Inv-24 auth gate, mocks Anthropic at SDK boundary per docs/reference/test-philosophy.md); manual run against a {52.13}-produced form_templates.id produces a catalogue row visible to T10’s read pattern.

Dependencies. {52.12} (the instance write must exist before Path C reads from it).

Files likely touched. .claude/skills/catalogue-form-requirements/SKILL.md (new), scripts/catalogue-from-instance.ts (new), lib/catalogue/from-instance.ts (new), __tests__/lib/catalogue/from-instance.test.ts (new).

Scope. M.

Checkpoint D — Path C populates the catalogue

• A confirmed instance’s questions appear as form_template_requirements rows with embeddings.
• Inv-20 through Inv-24 covered.

Phase 7 — Retirement of the legacy app-side analyse route

{52.15} Retire analyse/route.ts + UI rewire + template_analyse queue job removal

Description. Retire the legacy app/api/procurement/[id]/templates/[templateId]/analyse/route.ts (142 lines) — now redundant under R3 (pipeline-owned write). Run gitnexus_impact({target: "analyseRoute", direction: "upstream"}) FIRST per CLAUDE.md “MUST run impact analysis before editing any symbol”. Then delete the route file; rewire the UI caller in app/procurement/[id]/templates/page.tsx:145 handleAnalyse to a no-op-with-info-toast (the file is now ingested by the pipeline on the next walk tick — no client-initiated analysis); remove 'template_analyse' from the union in lib/queue/envelope.ts:44; remove the POST /api/bids/:id/templates/:templateId/analyse comment reference in lib/validation/schemas.ts:615; check for + delete any processing_queue consumer for job_type: 'template_analyse'.

Acceptance criteria.

• gitnexus_impact run on the route symbol; impact report shows direct callers (templates/page.tsx line 145 + the queue envelope union); no HIGH/CRITICAL risk surfaces beyond the documented callers (or, if it does, this Subtask is paused + escalated to the Orchestrator).
• Route file app/api/procurement/[id]/templates/[templateId]/analyse/route.ts deleted.
• app/procurement/[id]/templates/page.tsx handleAnalyse callback either: (a) removed and the “Analyse” button hidden, OR (b) replaced with a thin info-toast that surfaces “this form is being ingested automatically” + a refresh to fetch the form_templates row when ready. UX decision recorded inline in the PR description.
• lib/queue/envelope.ts:44 'template_analyse' removed from the ProcessingJobType union; any case 'template_analyse' branch in queue consumers deleted.
• lib/validation/schemas.ts:615 analyse route comment reference removed/updated.
• Search confirms no remaining template_analyse or templates/[templateId]/analyse references in app/, components/, hooks/, lib/.
• bun run lint PASSES; bun run test PASSES (no test references the deleted route — verified via earlier audit: no analyse/route.test.ts exists).
• gitnexus_detect_changes() confirms changes affect only the documented symbols.

Verification. bun run lint && bun run test PASS; bun build PASS (Next.js does not error on a missing route since it’s a deletion, but the build verifies no other file imports the deleted symbol).

Dependencies. {52.13} (the integration test must PASS proving the pipeline-owned write is live before we retire the legacy route).

Files likely touched. app/api/procurement/[id]/templates/[templateId]/analyse/route.ts (DELETED), app/procurement/[id]/templates/page.tsx (UI rewire), lib/queue/envelope.ts (union edit), lib/validation/schemas.ts (comment update).

ID-50 coordination. If ID-50’s per-route-group rollout has not yet reached procurement/[id]/templates, sequence this Subtask BEFORE that wave to avoid colliding on the deleted file. Surface to the Orchestrator at dispatch time.

Scope. S.

Phase 8 — Final acceptance + Inv-26 surfacing

{52.16} Acceptance fixtures end-to-end + Inv-26 AI-invisible review

Description. Run all 4 corpus fixtures (SQ PDF / EFA XLSX / CSP XLSX / Charnwood DOCX) end-to-end through the live pipeline against staging; eyeball form_template_fields rows per Inv-7-Inv-15; review UI surfacing of form_template_fields per Inv-26 (“structured records, not AI-extracted”). Lib-level test asserts no copy strings in lib/catalogue/from-instance.ts or any new component reference “AI” or “extraction” to end users.

Acceptance criteria.

• All 4 corpus form sets produce expected form_templates + form_template_fields row sets in the staging branch (counts measured against the live files per PRODUCT Acceptance fixtures table).
• For each corpus form, the per-invariant validation matrix in TECH §3.1 is checked manually OR via the integration test suite — every row green.
• A grep over the new code (lib/catalogue/, scripts/catalogue-from-instance.ts, .claude/skills/catalogue-form-requirements/) shows no user-facing strings containing “AI”, “AI-extracted”, “AI extraction” (Inv-26).
• UI surfacing of the new form_template_fields rows (in app/procurement/[id]/ templates pages, if any update is needed) presents them as ordinary structured knowledge-base records — verified via a manual look at the relevant page.
• bun run test:integration full suite GREEN.
• bun run test full suite GREEN.

Verification. Manual + bun run test + bun run test:integration PASS; Inv-26 grep PASSES.

Dependencies. {52.13}, {52.14}, {52.15}.

Files likely touched. Documentation only (PR description, completion note); no production code changes expected unless Inv-26 grep surfaces a copy string that needs rewording.

Scope. S.

Checkpoint E — Form-extraction v1 ready for ratification

• All 26 PRODUCT invariants verified per TECH §3.1 matrix.
• All 4 corpus fixtures green.
• Inv-26 AI-invisible surfacing confirmed.
• Subsystem ready for ID-52 close-out via Checker gate.

5. Risks and mitigations (carried over from TECH §5; tied to Subtasks here)

Risk	Subtask most affected	Mitigation
CV re-baseline breaks a literal APPLIED-S consumer	{52.5}	Pre-commit grep over lib/** + scripts/** confirms zero literal APPLIED-S matches (TECH §5).
Migration M1 CHECK widening collides at constraint-name level	{52.7}	DROP CONSTRAINT BEFORE ADD CONSTRAINT per TECH §2.6; staging-first per --env=staging; schema-parity.yml guards prod ↔ staging diff.
Workspace manifest UUID stale / workspace deleted	{52.8}, {52.12}	v1 relies on asyncpg INSERT-time FK to surface canonical error (TECH §2.1); workspace-existence pre-check is a follow-up (TECH §4).
Silent Supabase failure in Path C write	{52.14}	tryQuery() from @/lib/supabase/safe; code-review gate via code-review-and-quality skill before merge.
Anthropic 503 / rate limit in Path C classification	{52.14}	Reuse existing _anthropic_retry tenacity wrapper from extraction.py:9-11 (TECH §5).
coco.mount_each positional-arg order misalignment	{52.12}	TECH §2.5 specifies the exact order; integration test {52.13} covers reactive write path.
GitNexus impact analysis surfaces unexpected high-risk caller of analyse/route	{52.15}	Pause Subtask + escalate to Orchestrator per CLAUDE.md “MUST warn the user if impact analysis returns HIGH or CRITICAL risk”.
ID-50 route-rollout collision on the soon-deleted file	{52.15}	Coordination signal in §3; surface to Orchestrator at dispatch — sequence {52.15} before any ID-50 wave touching procurement/[id]/templates.

6. Follow-ups (carried from TECH §4 — tracked elsewhere, NOT in this PLAN)

• op_id column on form_templates + form_template_fields — v1 records on pipeline_runs.items_created[] only.
• UI upload re-wire (OQ-52-UI-UPLOAD-TENSION) — out of scope here; cross-link to ID-50.
• ID-54 — Path-A lossy q_a_extractions fix (already opened S273; see TECH §2.10).
• Mandatory-flag inference for non-flag forms — deferred until corpus evidence shows the heuristic is worth the false-positive cost.
• Path C re-run idempotency — re-running against an already-catalogued instance currently appends rows; T10’s read should already be tolerant via is_current.

7. Open questions

None survive TECH ratification. The three PRODUCT-MAY-PROPOSE items (OQ-52-CATALOGUE, OQ-52-LOSSY, OQ-52-UI-UPLOAD-TENSION) are recorded in PRODUCT §Ratifications and routed via this PLAN as: CATALOGUE → built in {52.14}; LOSSY → out-of-scope ID-54; UI-UPLOAD → out-of-scope ID-50 cross-link.

---

Char-budget verification. Subtask description values ≤250 chars; testStrategy values ≤300 chars (ledger budgets). PLAN.md per-Subtask Description prose blocks are not budgeted — they are spec text, not ledger fields. The sub-orchestrator persists the JSON record set (returned by this Planner) via bun scripts/ledger-cli.ts add-subtask 52 '<json>', which enforces the budget at write time. Defensive note for any future Checker that re-measures the wrong artefact: the canonical char-budgeted strings are the JSON-record description + testStrategy values, not the markdown blocks below their headers.

---

End of PLAN — ID-52.4. Output: docs/specs/id-52-form-extraction/PLAN.md. Subtask records returned to the sub-orchestrator for persistence to docs/reference/task-list.json via ledger CLI. UK English throughout; sibling-only dependencies verified; 12 Subtasks under the 25-Subtask soft cap; vertical-slice discipline honoured.