{80.12} — Form-instance versioning / supersession model (D6 investigation)

{80.12} — Form-instance versioning / supersession model (D6 investigation)

Status: INVESTIGATION — recommendation only, nothing implemented. Authored 05/06/2026 per the D6 RATIFIED routing (Liam, S314, 05/06/2026): instance-side form versioning is a separate designed item, investigation-first (docs/research/s314-feedback-synthesis-decision-brief.md:158, docs/specs/ID-80-forms-path-b/80.2-forms-content-separation.md:18-20). No code, no DDL, no ledger changes ship with this document. The migration sketch in §6 is illustrative only and MUST NOT be applied.

UK English throughout. Dates DD/MM/YYYY. Every claim is file:line-grounded against the worktree at canonical-pipeline-setup HEAD e71bd3fe; pins from the S314 research docs were re-verified, not trusted blind (several flow.py pins had drifted — fresh line numbers are used below).

---

1. Current state (verified)

1.1 form_templates carries no versioning substrate

The pipeline write schema FORM_TEMPLATES_SCHEMA (scripts/cocoindex_pipeline/flow.py:1245-1267) enumerates 18 columns — id, workspace_id, created_by, name, filename, file_size, mime_type, storage_path, structure_path, description, field_count, mapped_count, status, ingest_source, form_type, deadline, issuing_organisation, evaluation_methodology. There is no template_version, no is_current, no supersession link anywhere in the table’s DDL lineage:

• Pre-squash templates CREATE: supabase/migrations/20260416102457_pre_squash_reconciliation.sql:4173-4193.
• Rename templates → form_templates: 20260520120828_t2_combined_pr_intel_shape_b_form_type_split.sql:207.
• M1 (mime widen + ingest_source): 20260528134712_id52_form_extraction_schema.sql:13-16,38-43.
• M1b (dedicated metadata columns form_type / deadline / issuing_organisation / evaluation_methodology): 20260528151422_id52_form_templates_dedicated_metadata_columns.sql:5-33.
• ID-64 (status_reason): 20260601180102_id64_content_items_fks_and_form_templates_status_reason.sql:33-34.

1.2 What “a new version of a form” does today

The instance PK is deterministic: uuid.uuid5(_KH_PIPELINE_DOC_NS, f"ft:{rel_path}") (flow.py:2002). Three consequences:

1. New file beside the old (charnwood-v2.docx arriving beside charnwood.docx): different rel_path → different uuid5 → a wholly independent, unlinked sibling row. The old row stays status='analysed'; both list in the workspace (GET app/api/procurement/[id]/templates/route.ts:233). Nothing relates them.
2. Same-file byte mutation (same rel_path): UPSERTs the same row in place (flow.py:2088-2109) and trims stale trailing field rows (_trim_stale_form_fields, flow.py:2153-2178, called at :2123). The previous extraction is destroyed, not versioned.
3. Similar form from a different third party: issuing_organisation is captured (flow.py:2106) but there is no family/grouping key relating, say, two councils’ SQ instruments.

Liam’s stated model (S314 feedback) is that third-party instruments do not mutate in place — “change” arrives as a new file. Today that means a new, unlinked row. Versioning/supersession for form instances is net-new design space (docs/research/s314-id80-forms-current-state.md:262-283).

1.3 The no-rename-surface hazard (constrains every option)

No rename affordance and no PATCH/PUT endpoint exists for the form_templates row — every app-side UPDATE is mapped_count or status, never name or metadata (docs/research/s314-id80-forms-current-state.md:93-113; upload-time name at templates/route.ts:155). On the pipeline side the row is re-asserted from the binary: name = form_metadata.form_title or stem (flow.py:2093), and ingest_file is memoised, so a hypothetical DB-side edit to a pipeline-owned column would survive incremental walks but be silently reverted by any byte change to the file or a full_reprocess walk (docs/research/s314-id80-forms-current-state.md:137-145; docs/runbooks/onprem-b1-deploy.md:170-171).

The structural rule that follows: any versioning/supersession state MUST live in columns (or a table) OUTSIDE FORM_TEMPLATES_SCHEMA. The write-schema omission convention already exists for exactly this purpose — the schema comments record that auto/app-owned columns are deliberately omitted (flow.py:1234-1236 for created_at/updated_at; flow.py:1272-1274 for the Path-C-owned field columns), and status_reason (20260601180102:33-34) is a live example of a column the pipeline UPSERT never touches. Supersession state declared outside the write schema survives byte-change re-ingest AND full_reprocess; anything inside it gets clobbered.

---

2. Two distinct problems — do not conflate

The S314 feedback bundles two different relations:

Problem	Example	Nature
VERSIONING	charnwood-v2.docx replaces charnwood.docx	Lineage: same instrument, new revision. A directed chain with one “current” tail.
FAMILY grouping	Leicester City’s SQ vs Charnwood’s SQ	Similarity: different issuers, comparable instrument. An unordered cluster, no “current”.

These have different cardinalities (chain vs set), different write triggers (arrival of a successor vs recognition of similarity), and different consumers (list-currency filtering vs cross-issuer reuse). §9 treats family grouping separately; §§3-7 are about versioning proper.

---

3. Option survey (versioning)

Option (a) — supersession link table

A separate relation, e.g. form_template_supersessions(predecessor_id, successor_id, reason, confirmed_by, created_at).

• Pros: zero columns on form_templates (trivially outside the pipeline write schema); richest audit shape (reason, actor, timestamp per link); naturally extends to N:1 (several old instruments superseded by one consolidated form); the link row IS the human-confirmation record.
• Cons: no in-repo precedent — the shipped content-supersession model is a pointer column, not a link table (§4.1); every “current only” read becomes an anti-join (NOT EXISTS (... predecessor_id = ft.id)), which PostgREST expresses awkwardly from templates/route.ts; two sources of truth risk if a convenience flag is later added anyway.

Option (b) — pointer column(s) on form_templates

superseded_by uuid NULL REFERENCES form_templates(id) (+ optional superseded_at / superseded_reason), set ONLY by a human-confirmed app endpoint. Currency is derived: a row is current iff superseded_by IS NULL. This is the shipped content-items shape (lib/supersession/set.ts:230-246 writes superseded_by on the OLD row).

• Pros: direct in-repo precedent with a battle-tested validation set (self-ref, not-found, already-superseded — set.ts:163-220); “current only” is a single WHERE superseded_by IS NULL predicate, with the include_superseded opt-in pattern already shipped for search RPCs (supabase/migrations/20260421223339_add_include_superseded_to_search_rpcs.sql:59,141); columns omitted from FORM_TEMPLATES_SCHEMA survive re-ingest (§1.3); no version strings invented, so zero interaction with the 52.22 sentinel (§8).
• Cons: lighter audit than a link table (mitigated by superseded_at/superseded_reason columns); chains need a policy decision (§5 — the content model FORBIDS chains, forms NEED them).

Option (c) — catalogue-style template_version text + is_current boolean

The in-repo precedent is the Path-C catalogue form_template_requirements (renamed from template_requirements, 20260520120828:209): it has template_version text (20260416102457:4145), is_current boolean DEFAULT true (:4161), a natural key UNIQUE(template_name, template_version, section_ref, question_number) (:4523) and a partial index on is_current = true (:5111). Consumed by fetchTemplateRequirements (explicit version, else is_current = true — lib/templates/template-coverage.ts:459-478) and listAvailableTemplates (:653-667).

• Pros: familiar shape; supports human-meaningful labels (“2024 revision”); symmetric with the catalogue side.
• Cons (decisive, in my view):
  1. Nobody can mint the label at ingest time. The pipeline knows only rel_path and what the binary says; a version string would be a guess from the filename. The catalogue gets away with version strings because Path-C promotion is human-confirmed; instance rows are pipeline-written.
  2. The dual-field drift is already observable in-repo: catalogue promotion writes is_current: true with no automatic demotion of prior versions (lib/catalogue/from-instance.ts:307; docs/research/s314-id80-forms-current-state.md:296-298) — two rows of the same template can both claim currency. A stored boolean duplicates state the pointer already encodes, and drifts without trigger-level enforcement.
  3. Version strings don’t relate rows. 'v1'/'v2' labels on two sibling rows still need a grouping key to know they are versions of the same thing — i.e. option (c) quietly requires a family key (§9) before it can express lineage at all. Options (a)/(b) encode the relation directly.
  4. Direct collision surface with the ID-52.22 sentinel (§8).

---

4. Precedents weighed

4.1 Content supersession (shipped)

lib/supersession/set.ts — one helper, three callers (UI admin PATCH, MCP tool, Python ingest --auto-supersede; set.ts:4-8). Writes superseded_by + dedup_status='superseded' + archive side-effects on the OLD row only (:230-246); validates SAME_ID / not-found / already-superseded (:163-220); explicitly prevents chains (OLD_ALREADY_SUPERSEDED / NEW_ALREADY_SUPERSEDED, :123-126). Read side: every search RPC takes include_superseded boolean DEFAULT false and filters (include_superseded OR ci.superseded_by IS NULL) (20260421223339:59,141,167,223).

Carries over: the pointer column, the validation set, the default-hidden + opt-in read pattern, human/admin-confirmed writes. Does NOT carry over: the no-chain rule — content dedup re-points a pair; form revisions naturally form v1 → v2 → v3 chains over a procurement cycle.

4.2 Procurement-workspaces Q&A lineage (specced, not yet signed off)

• B-12, version-on-cite (docs/specs/procurement-workspaces/PRODUCT.md:156-167): citations carry a version snapshot of the Q&A pair at cite time via q_a_pair_history; the pair keeps evolving post-citation. Pattern value here: consumers pin a version at use time — relevant if/when form completions need to record which instance revision they filled (the completion’s template_id FK already pins the row; with option (b) the row is immutable-by-supersession, so the pin is free).
• B-15, draft-vs-final heuristic (PRODUCT.md:198-211): import-time filename heuristic surfaces a supersession hint; the user explicitly chooses auto-supersede or manual review — never silent. This is the template for any future “charnwood-v2.docx looks like a successor of charnwood.docx” suggestion: heuristic proposes, human disposes.

Both are a PATTERN (human-confirmed supersede; version snapshots), not a ready-made form model (docs/research/s314-id80-forms-current-state.md:299-305,316-318).

4.3 Catalogue form_template_requirements

Covered as option (c) in §3 — its drift behaviour (is_current never demoted, from-instance.ts:307) is treated as evidence against a stored boolean, and its template_version column is the 52.22 collision surface (§8).

---

5. Recommendation

Option (b): app-owned supersession pointer columns on form_templates, chain-permitting, human-confirmed only, currency derived. Specifically:

1. superseded_by uuid NULL REFERENCES form_templates(id) + superseded_at timestamptz NULL + superseded_reason text NULL — kept OUT of FORM_TEMPLATES_SCHEMA so pipeline re-ingest/full_reprocess can never clobber a human supersession decision (§1.3).
2. No stored is_current. Currency = superseded_by IS NULL, served by a partial index. This avoids the dual-field drift the catalogue already exhibits (§3c.2) and keeps exactly one writable fact.
3. Linear chains allowed (v1 → v2 → v3), unlike setSupersession’s no-chain rule: validations are self-reference forbidden (CHECK), same workspace required (app-level), successor must not be the predecessor’s own ancestor (app-level cycle walk — chains are short, a loop query suffices). The “current” instrument of a chain is its tail (superseded_by IS NULL).
4. Write path is a new human-confirmed app endpoint only (admin/editor), mirroring the setSupersession validation discipline. The pipeline NEVER sets it — at most, a later B-15-style filename heuristic may suggest a link in the UI; never auto-applied.
5. No version strings instance-side. Lineage is the relation itself; display ordinals (“revision 2 of 3”) are derivable by walking the chain. This makes the 52.22 sentinel interaction vacuous by construction (§8).

Why not (a): the link table’s genuine advantages (N:1 merges, richer audit) are not yet evidenced needs for forms; the pointer matches the shipped precedent and the read paths are simpler. If N:1 consolidation later materialises, a link table can be added then — the pointer migrates into it losslessly. Why not (c): §3c reasons 1-4.

---

6. Migration sketch — ILLUSTRATIVE ONLY, NOT TO BE APPLIED

D6 is “recommend, do not implement”. This DDL exists to make the recommendation concrete for review. It has NOT been created as a migration, NOT pushed anywhere, and MUST NOT be applied from this document.

-- ILLUSTRATIVE ONLY (80.12 / D6) — DO NOT APPLY
ALTER TABLE public.form_templates
  ADD COLUMN superseded_by uuid NULL
    REFERENCES public.form_templates(id) ON DELETE SET NULL,
  ADD COLUMN superseded_at timestamptz NULL,
  ADD COLUMN superseded_reason text NULL,
  ADD CONSTRAINT form_templates_no_self_supersede
    CHECK (superseded_by IS NULL OR superseded_by <> id);

-- Currency is derived; partial index serves the default "current only" list.
CREATE INDEX idx_form_templates_current
  ON public.form_templates (workspace_id)
  WHERE superseded_by IS NULL;

Notes for the eventual implementer (not this Subtask):

• ON DELETE SET NULL: deleting a successor (the DELETE endpoint exists, app/api/procurement/[id]/templates/[templateId]/route.ts:235) resurrects the predecessor as current — the least-surprise outcome.
• Same-workspace and acyclicity are app-level validations in the supersede endpoint (CHECK constraints cannot walk chains; a trigger could, but the admin-only + low-volume profile matches the documented TOCTOU acceptance in lib/supersession/set.ts:28-35).
• No PL/pgSQL function is required, so the SET search_path rule is not in play; RLS posture of form_templates is unchanged (columns, not policies).
• Types regen + database-overrides.ts untouched (no JSONB).

---

7. UI / pipeline touch-points (for the eventual implementation Task)

Surface	Change	Grounding
Pipeline flow.py	NONE. New columns stay outside FORM_TEMPLATES_SCHEMA (flow.py:1245-1267), so byte-change re-ingest and full_reprocess leave them intact — the inverse of the no-rename hazard (§1.3).	flow.py:1234-1236, docs/runbooks/onprem-b1-deploy.md:170-171
GET templates list	Default-filter superseded_by IS NULL; add an include_superseded query param mirroring the search-RPC pattern.	templates/route.ts:233; 20260421223339:59,141
Templates page	Superseded rows hidden by default, toggle to reveal; chain badge (“superseded by {name}”) on revealed rows.	app/procurement/[id]/templates/page.tsx:342-369 (list)
New supersede endpoint	Admin/editor, human-confirmed; setSupersession-style validations adapted per §5.3 (chains allowed).	lib/supersession/set.ts:163-220 (validation precedent)
DELETE endpoint	Already exists; ON DELETE SET NULL handles successor deletion (§6).	templates/[templateId]/route.ts:235
MCP	NONE today — no MCP tool reads or writes form_templates / form_template_fields (the three template tools read only the catalogue). Future ID-71 work inherits the include_superseded convention.	docs/research/s314-id80-forms-current-state.md:149-156
Path-C catalogue	No automatic effect on supersede — catalogue rows are global and human-curated; whether to re-promote from the successor is a per-case human call (decision point 6).	lib/catalogue/from-instance.ts:282-309
Fields / completions of a superseded instance	Retained untouched (history). The completion’s template_id FK pins the revision that was filled — the B-12 version-on-cite property for free.	PRODUCT.md:156-167
Operator folder contract	Must state explicitly: new revisions arrive as NEW files (new rel_path). Replacing bytes in place UPSERTs the same ft: row and trims fields (flow.py:2002,2123,2153-2178) — lineage is destroyed before any model can record it. No column design fixes in-place replacement.	flow.py:2088-2109

The no-rename-surface hazard restated for this table: name, form_type, deadline, issuing_organisation, evaluation_methodology are inside the pipeline write schema and are re-asserted from the binary on re-ingest (flow.py:2093-2107) — any future edit affordance for those columns needs a pipeline-respected override column or a declared precedence rule (docs/research/s314-id80-forms-current-state.md:142-145). The supersession columns recommended here deliberately avoid that class of problem by living outside the schema.

---

8. ID-52.22 sentinel-collision check

The {52.22} idempotency design (HOLD, re-review after the 80.2 chain) fixes a real bug: buildCatalogueRow emits template_version: null (lib/catalogue/from-instance.ts:294) and the catalogue natural key is NULLS DISTINCT, so ON CONFLICT never matches — the fix defaults template_version to a non-NULL sentinel 'v1' (docs/specs/id-52-form-extraction/52.22-idempotent-catalogue-design.md:71-72,181; journalled in the ID-52.22 details field, 04/06/2026). The S314 synthesis flags the interaction: the sentinel must not collide with whatever instance-side model D6 lands (docs/research/s314-feedback-synthesis-decision-brief.md:140-147).

Under the §5 recommendation the collision is structurally impossible: option (b) introduces no version strings anywhere, so the catalogue’s 'v1' sentinel remains the only template_version writer and keeps its meaning (“unversioned catalogue default”), unchallenged.

If Liam instead picks option (c), the collision is live and concrete: the propagation channel already exists — buildCatalogueRow accepts templateVersion (from-instance.ts:288,294) and Path-C promotion could plausibly start passing the instance’s version label through. An instance labelled 'v1' would then be indistinguishable from sentinel rows in the catalogue natural key. Required guard in that world: either namespace instance-derived labels (e.g. inst:<label>) before they reach template_version, or reserve 'v1' as forbidden for instance labels, and record whichever rule is chosen in the 52.22 design doc at its re-review.

---

9. Family vs version — possibly different relations

Versioning (§§3-8) answers “is this the same instrument, revised?”. Family grouping answers “are these comparable instruments from different issuers?” — e.g. SQ forms from two councils. Forcing both through one relation produces either fake lineage (council A’s form does not “supersede” council B’s) or a meaningless shared version axis.

Candidate mechanisms for family, in ascending weight:

1. Derive from existing facets — recommended default, costs nothing. form_type (FK to the form_types CV, 20260528151422:5-6; CV table created 20260520120828:628) × issuing_organisation (flow.py:2106) already supports “all SQ instruments, grouped by issuer” without new schema. Weakness: issuing_organisation is a free-text parse from the binary, not a CV.
2. Explicit nullable family_key text (or form_family table) on form_templates — only worth it once a concrete consumer exists (e.g. a cross-issuer comparison surface). Would also need to stay outside FORM_TEMPLATES_SCHEMA if human-assigned, for the §1.3 reason.
3. Treat the Path-C catalogue as the family abstraction. Arguably already true: form_template_requirements clusters reusable requirements under template_name/template_type (20260416102457:4143-4146) independent of which issuer’s instance they were promoted from — the catalogue IS the “this kind of form” layer, while instances stay issuer-specific.

Recommendation: defer family grouping; do not let it ride on the versioning migration. Mechanism 1 covers near-term needs; mechanism 3 is the architecturally honest home for “similar form, different third party” and needs no instance-side schema at all. Revisit when procurement-workspaces firm up (consistent with the D6 routing rationale, s314-feedback-synthesis-decision-brief.md:134-136).

---

10. DECISION POINTS for Liam

1. Model choice: ratify option (b) — pointer columns (superseded_by + superseded_at + superseded_reason), human-confirmed, outside the pipeline write schema? Or do you want the option (a) link table for richer audit / future N:1 consolidation from day one?
2. Currency representation: agree currency is derived (superseded_by IS NULL) with no stored is_current boolean? (The catalogue’s undemoted is_current drift, from-instance.ts:307, is the cautionary precedent.)
3. Chain policy: allow linear chains v1 → v2 → v3 (recommended for forms; diverges deliberately from setSupersession’s no-chain rule for content dedup) — with self-reference and cycles forbidden?
4. Read default: superseded instances hidden by default in the templates list with an include_superseded opt-in (mirroring 20260421223339)?
5. Suggestion heuristic: should a B-15-style filename hint (“charnwood-v2.docx may supersede charnwood.docx”) be in the eventual implementation’s v1 scope, or deferred? (Either way: suggest-only, human confirms — never auto-supersede.)
6. Catalogue interaction on supersede: confirm NO automatic catalogue effect when an instance is superseded (re-promotion from the successor is a separate, human Path-C action)?
7. Family grouping: accept the §9 recommendation to defer (facet-derived grouping now; catalogue as the family abstraction), i.e. no family key in the versioning migration?
8. If option (c) is preferred instead: which 52.22 sentinel guard — label namespacing (inst:<label>) or reserving 'v1' — and recorded where? (§8.)
9. Operator contract: ratify the folder-contract sentence “new revisions arrive as new files; never replace a form’s bytes in place” (in-place replacement destroys lineage at flow.py:2002 before any model can record it)?
10. Timing: does this become an implementation Subtask under ID-80, or a new designed Task once procurement-workspaces firm up (the original D6 routing suggestion, s314-feedback-synthesis-decision-brief.md:134-136)?

---

11. RATIFICATION — Liam, S316 (05/06/2026)

The §10 decision points resolved as follows (Liam feedback, S316 session open):

DP	Decision
1	RATIFIED option (b) — superseded_by + superseded_at + superseded_reason pointer columns on form_templates, human-confirmed app endpoint only, kept OUTSIDE FORM_TEMPLATES_SCHEMA (§5).
2	RATIFIED — currency is derived (superseded_by IS NULL); no stored is_current boolean.
3	RATIFIED — linear chains allowed (v1 → v2 → v3); self-reference and cycles forbidden.
4	RATIFIED — superseded instances hidden by default in the templates list; include_superseded opt-in mirroring 20260421223339.
5	RATIFIED — v1.1 scope — the B-15-style filename hint is deferred to v1.1 of the eventual implementation. Suggest-only, never auto-supersede (deferral register V11-12).
6	RATIFIED — NO automatic catalogue effect on supersede; re-promotion from a successor is a separate, human Path-C action.
7	RATIFIED — mechanism 3 (mechanism 1 remains the free near-term default): the Path-C catalogue IS the family abstraction — per Liam, “this was always the intention as part of the canonical pipeline implementation work”. No family key in the versioning migration.
8	n/a — option (c) not chosen; the §8 sentinel-guard question is vacuous by construction.
9	RATIFIED — operator folder contract: “new revisions arrive as new files; never replace a form’s bytes in place”. Cross-reference (Liam): ID-56.12 (UI thin folder-drop wrapper) is the user-facing mechanism for adding files to the underlying folder — its spec must encode the new-file semantics (never overwrite an existing form’s bytes in place).
10	Derived (Liam: “TBD — based on feedback provided for the other OQs”) — from DP-5’s v1.1 scoping + DP-7’s catalogue framing: implementation routes as a new designed Task in the v1.1 cycle (deferral register V11-11), sequenced when procurement-workspaces firm up — the original D6 routing. NOT an ID-80 implementation Subtask; ID-80 closes on the S1 re-smoke + end-of-task passes with this investigation as the {80.12} deliverable.

Consequences recorded: {52.22} hold RELEASED — under option (b) the template_version='v1' sentinel collision is structurally impossible (§8); the S312 executor brief is dispatchable unchanged (docs/specs/id-52-form-extraction/52.22-idempotent-catalogue-design.md). Deferral-register rows: V11-11 (implementation Task), V11-12 (filename-hint heuristic).