ID-153 {153.1} RESEARCH — DOCX/XLSX spatial form-fill via PDF convert-then-revert

Tier: RESEARCH-only (this doc + a go/no-go recommendation feeding a follow-on implementation task). No production code changes in this task.

Provenance: promoted from bl-459 (S477); id-147 grounding §4c; DR-064. Verified live: id-145 = in_progress, id-147 = in_progress, id-153 = pending (bun scripts/ledger-cli.ts get task <id> status). bl-459 no longer resolves as a live backlog item — it was consumed by the promotion into id-153.

DR-064 (live text, decision-register.md:191-206) — the ruling this task sits under: Bounding-box geometry is UNBLOCKED. Fill-slots persist commonforms normalised-top-left coords (Option A) and citations derive coords via Option B; spatial fill-slot review is v1, PDF-only. The register closes with: “DOCX/XLSX spatial overlay is out of v1 (see backlog: PDF-convert-then-revert).” This RESEARCH is that backlog investigation.

1. Problem & scope

Spatial fill-slot review (Extend Bounding-Box overlay driven by form_instance_fields geometry, DR-064 Option A) and bounding-box citations (Option B) are PDF-only in v1. The blocker is not the overlay UI — it is that geometry only exists for PDFs. This task investigates whether converting a DOCX/XLSX form to PDF for the fill+review pass, then reverting to the original type for delivery, is a viable way to bring the spatial surface to DOCX/XLSX forms.

Four questions, each answered against the live pipeline below:

Fidelity — does DOCX/XLSX → PDF → (edited) → back preserve layout, fields, and the user’s original file well enough to deliver?
Round-trip safety — where is data/formatting lost; is a true “revert to original type” feasible, or is this convert-forward-only?
LibreOffice-convert reuse — the {145.31} worker image already shells out to LibreOffice; can that pipeline be reused?
Anthropic DOCX/XLSX → PDF document-block route — does an Anthropic document-block path help vs LibreOffice?

Out of scope: the overlay component itself (built under id-147); the PDF fill/geometry mechanism itself (id-145.11/id-147); any product decision on whether buyers accept a returned PDF vs the original template (flagged for the owner in §7).

The two form lanes today (the fact everything turns on)

The pipeline routes by file suffix into two disjoint lanes (scripts/cocoindex_pipeline/form_extractors/orchestrator.py:29-38, :192-226):

Lane	Detection	Geometry	Fill / delivery
PDF	commonforms ML widget detection (`pdf.py`)	Yes — normalised coords computed, currently dropped (DR-064 Option A un-drops them)	Fill a plane-2 `{form_id}/fillable.pdf` via pypdf AcroForm writes (`fill_template.py:336-402`)
DOCX/XLSX	python-docx / openpyxl cell readers (`docx.py`/`xlsx.py`)	No — logical cell coords only (`table_index`/`row_index`/`col_index`)	Edit the original upload in place (python-docx / openpyxl) — `fill_template.py:149-246` (DOCX), `:466-530` (XLSX)

The DOCX/XLSX lane’s delivery is lossless by construction: it opens the user’s own file, writes answer text into the target cells with formatting preserved (_copy_cell_formatting, fill_template.py:45), and saves (:246). The delivered artefact is the user’s original file with cells filled — no conversion, no fidelity loss. This is the asset the convert-then-revert idea puts at risk, and the reason “revert to original type” is the load-bearing question.

2. Investigation — Fidelity (forward: DOCX/XLSX → PDF)

LibreOffice headless renders OOXML → PDF with high visual fidelity for the target corpus. UK procurement forms are overwhelmingly table-and-text layouts; soffice --convert-to pdf reproduces those faithfully. The critical enabling finding:

commonforms detects on FLAT PDFs — it does not need pre-existing form fields. pdf.py:1-19: “Real UK procurement PDFs are FLAT — zero AcroForm fields (verified 5/5) … ML-based widget detection is mandatory. This module wraps commonforms.prepare_form … to render→detect→write AcroForm widgets onto a fillable copy of the input PDF, then reads the detected widgets back (page, rect, field type).”

A LibreOffice-rendered PDF from a DOCX/XLSX form is exactly this: a flat PDF with visual answer regions. It is a valid commonforms input. So the converted PDF flows straight through the existing PDF lane — detection (pdf.py:330-373), geometry (DR-064 Option A, the ~4-line un-drop at pdf.py:354-356), spatial overlay (id-147), and pypdf fill (fill_template.py:336-402) — with no new geometry mechanism required. This is the single biggest reuse win and the reason the forward path is attractive.

Fidelity risks on the forward render (known LibreOffice-headless behaviours, not code-cited):

Font substitution — fonts not installed in the worker image are substituted, which reflows text and shifts cell boundaries. Because commonforms detects answer regions by rendered position, a substitution-induced shift degrades detection accuracy and overlay alignment for precisely the column-laid-out forms that motivated this work.
Wide/multi-sheet XLSX — spreadsheets that print across many pages or hide columns render unpredictably; detected regions may not line up with a reviewer’s mental model.
Determinism — the conversion must be pinned/reproducible so the same upload yields the same rendered geometry across re-runs.

Verdict: forward render fidelity is good enough for a fill+review proxy on typical forms, with font provisioning as the main quality lever.

3. Investigation — Round-trip safety (revert: PDF → original type)

This is where “convert-then-revert to the original type” splits into three materially different designs. Only one is what the task literally names, and it is the one that does not work.

Design C — literal file revert (PDF → DOCX/XLSX file conversion). NOT SAFE / NOT FEASIBLE. LibreOffice can import a PDF, but PDF→DOCX goes through Draw and reconstructs the page as absolutely-positioned text frames and vector shapes — it does not recover the original’s semantic structure (tables, paragraph styles, headers). The delivered DOCX would not match the buyer’s template; for a “complete and return THIS document” procurement requirement that is a correctness failure, not a cosmetic one. PDF→XLSX is worse: there is no cell/formula/sheet model to recover — the result is not a usable spreadsheet. A true type-preserving file round-trip is therefore infeasible.
Design B — value-mapping revert (keep the original; map approved answers back into it). FEASIBLE, with a real cost. Never convert the deliverable. Use the converted PDF only to drive detection + spatial review, capture the human-approved answer values, then write those values into the retained original DOCX/XLSX via the existing lossless in-place writers (fill_template.py:149-246 / :466-530). This sidesteps the lossy PDF→DOCX conversion entirely. The hard part is a positional→logical mapping bridge: commonforms yields visual widgets on the rendered PDF (page + rect); the original has logical cells (table_index/row_index/col_index). There is no guaranteed correspondence, and the existing lexical label-pairing is already imperfect (pdf.py:11-15). A mis-map lands an approved answer in the wrong cell of a submitted bid — a silent, high-consequence error.
Design A — convert-forward-only (deliver the PDF). SAFE, maximal reuse. Convert once at ingest, treat the PDF as the canonical fillable artefact, and deliver the filled PDF. No revert, no mapping bridge, no reconciliation risk — the entire existing PDF lane is reused unchanged. The only cost is that the deliverable type changes (the user uploaded DOCX/XLSX, receives a filled PDF).

Round-trip verdict: a genuine “revert to the original file type” is a trap (Design C). The feasible options are convert-forward-only (Design A, safe, cheap) or value-mapping into the retained original (Design B, preserves type, carries a mapping-bridge build + mis-map risk). The choice between A and B is a product decision, not a technical one (see §7).

4. Investigation — LibreOffice-convert reuse ({145.31})

The worker already shells out to LibreOffice: scripts/bid_worker.py:332-372, _convert_legacy_office_to_ooxml, runs soffice --headless --norestore --convert-to <ext> --outdir … (:350-360). The invocation is generic — swapping the --convert-to target from docx/xlsx to pdf produces a PDF from the same binary. So a _convert_office_to_pdf(raw_bytes) helper is a near-clone of the existing function, and the worker’s format-routing seam (analyse_form_job, bid_worker.py:518-630; fill dispatch, :96-212) is the natural place to route DOCX/XLSX through the PDF lane after conversion.

Three caveats temper the reuse:

LibreOffice is not yet in the image — this is a hard prerequisite. Subtask 145.31 (get task 145.31) is a pending deploy gate: the gate-145-13 Checker found the worker shells soffice but “repo-wide grep = ZERO libreoffice/soffice/apt-packages config” — the whole DR-059 .doc/.xls path is 0% functional in prod until 145.31 provisions LibreOffice. Any id-153 implementation is blocked on 145.31 and shares its font-provisioning fate (§2).
Current use is normalisation-forward, not to-PDF. The existing function converts legacy OLE2 → OOXML so the OOXML readers can parse it. A convert-to-PDF path is a new (small) addition, not a literal reuse of the existing call.
The bigger reuse is commonforms, not the soffice call. Because commonforms detects on flat PDFs (§2), the converted PDF reuses the entire PDF geometry + fill + overlay lane. That is the reuse that makes this cheap — the soffice invocation is the small part.

5. Investigation — Anthropic DOCX/XLSX → PDF document-block route

Checked against the claude-api skill (Document & File Input reference) and id-147 grounding §7.

There is no native DOCX/XLSX document block. The Messages API document block accepts PDF (base64/file) and plain text only. To give Claude a DOCX/XLSX with layout fidelity you must convert it to PDF yourself — Anthropic provides no conversion endpoint. Today the pipeline pre-flattens DOCX/XLSX to HTML client-side (mammoth/SheetJS), which loses the visual grid (grounding §7, id-147 empirical-grounding.md:283: “A DOCX/XLSX→PDF→document-block route would restore layout signal for column-laid-out forms. Weigh vs conversion cost.”).
What the route actually buys, and what it does not. Sending a converted PDF as a native document block improves Claude’s reading fidelity — better extraction, and it feeds the Option-B B2 vision-coordinate fallback (self-rasterised pages). It does not produce AcroForm widget geometry, does not produce a filled deliverable, and does not avoid the LibreOffice conversion (you still convert to PDF first). It is an answer-quality lever, already tracked in grounding §7 as a lower-priority OPP — orthogonal to the spatial fill/review/deliver round-trip this task is about.

Verdict: the Anthropic document-block route is not a substitute for LibreOffice and does not solve id-153. It rides on the same DOCX/XLSX→PDF conversion and only helps the LLM read the form better. Note it as a complementary answer-quality follow-on, not part of the id-153 build.

6. Options compared

#	Design	Deliverable type	Reuses existing PDF lane	New build	Round-trip safety	Verdict
A	Convert-forward-only (deliver filled PDF)	PDF (changed)	Full — convert → commonforms detect → Option-A geometry → id-147 overlay → pypdf fill	soffice→PDF helper (~clone of `bid_worker.py:332`) + route DOCX/XLSX through PDF lane + deliverable-type handling	Safe — no revert, no mapping	GO (v1) — gated on 145.31 + product acceptance of PDF return
B	Value-mapping revert (keep original, map values back)	Original DOCX/XLSX (preserved)	Detection/geometry/overlay reused; delivery uses existing lossless in-place writers	Everything in A plus a positional→logical mapping bridge (PDF widget → DOCX/XLSX cell) + reconciliation + fidelity tests	Conditional — mis-map risk (wrong answer in wrong cell)	Defer (v2) — only if product requires original-type return
C	Literal file revert (PDF → DOCX/XLSX conversion)	Original type (reconstructed)	n/a	LibreOffice PDF→DOCX/Draw import	Unsafe — structure/styles destroyed; XLSX unrecoverable	NO-GO — do not build
—	Anthropic document-block route	n/a (reading only)	n/a	n/a	n/a	Not applicable — improves LLM reading; still needs LibreOffice; doesn’t fill/deliver
—	Status quo (DR-064 v1)	Original type, lossless	—	none	Safe	Baseline — DOCX/XLSX get a list fill surface, no spatial overlay (PRODUCT.md §C, id-147)

7. Recommendation

Conditional GO — but the task’s literal framing (“convert-then-revert to the original type”) is the wrong shape and should not be built as stated. The revert that the name implies (Design C, PDF→DOCX/XLSX file conversion) is infeasible/lossy and is a NO-GO.

Recommend a follow-on scoped to Design A (convert-forward-only) as v1: bolt a soffice → PDF conversion onto the front of the already-built PDF lane and let DOCX/XLSX forms ride the existing commonforms detection + DR-064 Option-A geometry + id-147 spatial overlay + pypdf fill, delivering a filled PDF. This is the low-cost, high-reuse path with no revert and no mapping risk. Design B (value-mapping revert preserving the original type) is the fallback if the product must return the buyer’s own template — defer it to v2 behind an explicit product decision, because it adds the positional→logical mapping bridge and a mis-map hazard.

Hard prerequisites before any implementation dispatch:

145.31 lands — LibreOffice provisioned in the worker image (currently a pending, unshipped deploy gate; everything here is blocked on it) with a font set covering the corpus.
id-147 spatial-fill v1 is merged — Design A reuses the Option-A geometry column, overlay, and pypdf fill that id-147/id-145 deliver; it cannot start before they exist.
Owner product decision (A vs B) — do buyers accept a returned filled PDF (Design A), or must the answers come back in the original DOCX/XLSX template (Design B)? This gates the effort tier and is the single most important input.

Go/No-Go verdict

Design C (literal file revert): NO-GO. Infeasible/lossy — do not build.
Design A (convert-forward-only): GO, gated on 145.31 + id-147 v1 + product acceptance of a PDF deliverable. Effort ≈ Small–Medium (a few days): conversion helper mirrors bid_worker.py:332, routing change, deliverable-type handling, tests; the heavy lifting (detection/geometry/overlay/fill) already exists.
Design B (value-mapping revert): DEFER (v2) — GO only if product requires original-type return. Effort ≈ Large: the positional→logical bridge, mis-map reconciliation, and per-format fidelity tests are the real work.
Anthropic document-block route: not part of id-153 — record as a separate, lower-priority answer-quality OPP (grounding §7).

Top-2 risks

“Revert” is a trap. A literal PDF→original-type file conversion destroys structure and styles (XLSX is effectively unrecoverable). If the product genuinely needs the answers back in the buyer’s template, Design A (deliver PDF) doesn’t satisfy it and Design B’s value-mapping bridge carries mis-map risk — an approved answer in the wrong cell of a submitted bid is a silent, high-consequence error. Resolve the A-vs-B product decision before committing.
Blocked on unshipped provisioning + rendering fidelity. 145.31 (LibreOffice in the worker image) is a pending deploy gate — the whole path is 0% functional until it lands. Even then, font substitution and complex/wide XLSX shift the rendered layout, degrading commonforms detection and overlay alignment for exactly the column-laid-out forms that motivated the work.