Skip to content

ID-80.2 — Separate forms (Path-B) from standard content uploads (Path-A)

ID-80.2 — Separate forms (Path-B) from standard content uploads (Path-A)

Section titled “ID-80.2 — Separate forms (Path-B) from standard content uploads (Path-A)”

Artefact kind: architecture / design doc (TECH-style) with a short RESEARCH preamble, self-contained per the {80.2} dispatch brief. This is NOT the full {N.1}…{N.4} chain — it is the one separation-architecture deliverable for the remaining open Subtask of Task ID-80.

Parent Task: ID-80 “Productionise cocoindex form-write path (Path-B / forms)” (docs/reference/task-list.json.tasks[] | select(.id=="80")). Sibling Subtasks already DONE: {80.1} _trim_stale_form_fields DB_CTX accessor (bl-224), {80.3} pandoc install, {80.4} form-write audit + coverage, {80.5} Stage-5 intra-doc canonical-collision fix (bl-225).

UK English throughout. Dates DD/MM/YYYY.

Status: RATIFIED with amendments (Liam, S314, 05/06/2026). OQ-80.2-A/B/C all ratified (see the Open questions section); minimal-provenance-row fallback dropped; blank-vs-answered caveat recorded in B.1/B.3; form-instance versioning routed to the {80.12} investigation Subtask. Implementation Subtask records {80.6}{80.12}: docs/research/s314-id80-subtask-records.json.


Part A — RESEARCH preamble (grounded current state)

Section titled “Part A — RESEARCH preamble (grounded current state)”

Per .gitnexus/CLAUDE.md “Always Do” and .ast-dataflow/CLAUDE.md, this spec was oriented with GitNexus + a Python grep sweep (ast-dataflow does not cover the Python pipeline corpus — scripts/cocoindex_pipeline/*.py).

  • gitnexus_query({query: "cocoindex ingest_file form extraction workspace resolution write form_templates content_items fork", repo: "knowledge-hub"}) — returned no single execution flow that already forks forms from content. The matched definitions confirm the fork is INLINE inside one symbol: Function:scripts/cocoindex_pipeline/flow.py:ingest_file (startLine 1389), Function:scripts/cocoindex_pipeline/flow.py:_ingest_file_body (startLine 1560, endLine 2105), Function:scripts/cocoindex_pipeline/flow.py:app_main (startLine 2139), and Function:scripts/cocoindex_pipeline/workspace_resolver.py:resolve_workspace (startLine 173). The form-vs-content decision is NOT a separate process — it is a tail block of _ingest_file_body. This is the finding the separation must act on.

  • gitnexus_context({name: "extract_form_structure", repo: "knowledge-hub"}) — verdict: incoming: {} (zero indexed callers). It is invoked dynamically as a @coco.fn(memo=True) awaited inside the daemon-thread body (flow.py:1960), so GitNexus indexes no static caller. outgoing.calls → per-format readers form_extractors/{docx,pdf,xlsx}.py::extract. Blast radius of changing the call site is therefore confined to _ingest_file_body; the orchestrator + readers are unaffected by a fork-point change.

  • Python grep sweep (scripts/cocoindex_pipeline/flow.py) confirmed the Path-A extractor calls (extract_classification, extract_qa_form, extract_entity_mentions) at lines 1623–1628 run unconditionally for every file, BEFORE the Path-B form-write block at 1879–2106. There is no suffix / manifest gate in front of Path-A today.

  • Re-verification (05/06/2026, post-ratification dispatch): gitnexus_context re-run on the two fork-critical symbols. _ingest_file_body — unchanged: incoming.calls: [ingest_file] (1 caller); outgoing set still includes resolve_workspace, the three Path-A extractors and _trim_stale_form_fields. resolve_workspace — 9 incoming callers (8 test functions in scripts/tests/test_workspace_resolver.py + _ingest_file_body, the only production caller), so the B.2 shim plan protects the entire existing test surface. Line-ref re-pin: ID-61.4 (693f1565) has since landed at HEAD, adding +55 lines to flow.py (_pydantic_error_detail helper at :367; webhook error_detail param at :715). The Part A/B line refs in this doc were pinned pre-61.4 and now drift by ~+44 from flow.py:1434 (ingest_file) onward. Post-61.4 re-pins live in the {80.6}{80.12} Subtask records (docs/research/s314-id80-subtask-records.json) — implementers use those, not the refs below.

cocoindex 1.0.3 (requirements.txt: cocoindex[postgres]==1.0.3, pydantic==2.12.5, asyncpg>=0.30.0, docling>=2.0.0,<3.0.0) drives a single reactive flow. The relevant span:

File / linesRole
flow.py:2140 app_mainLoads workspace manifest once (2206–2230); mounts seven mount_table_target row targets (2253–2302) under managed_by=ManagedBy.USER; localfs.walk_dir(..., live=True, recursive=True) (2307); fans out per-item via coco.mount_each(..., bound_ingest_file, source.items(), <7 targets>) (2370–2381); runs flow-scope Stage-5 (2430–2446).
flow.py:1389 ingest_file@coco.fn(memo=True) wrapper. Skips the manifest file itself (1480). Resolves run context (op_id / counters / manifest) from explicit closure args with ContextVar fallback (1503–1541). Delegates to _ingest_file_body.
flow.py:1560 _ingest_file_bodyThe actual Stage-2→6 work. Path-A runs first, unconditionally; Path-B is an additive tail block.
adapters.py:56 convert_binary_to_markdownStage-2: routes a FileLike by suffix → Docling (pdf/docx/xlsx) / pullmd (html) / passthrough (md/txt).
form_extractors/orchestrator.py:54 extract_form_structurePath-B Stage-3a: deterministic raw-format reader dispatch by suffix; returns `ExtractedForm
workspace_resolver.py:174 resolve_workspaceLongest-prefix folder→workspace resolution; raises UnmappedPath (benign) / AmbiguousResolution (loud).

A.3 The two write targets (the boundary we must keep clean)

Section titled “A.3 The two write targets (the boundary we must keep clean)”
  • Path-A (standard content — workspace-AGNOSTIC canonical layer): content_items, source_documents, content_chunks, q_a_extractions, entity_mentions. Written at _ingest_file_body 1622–1877. Per ID-69 BI-1, content_items has no workspace_id; workspace association is the content_item_workspaces M2M junction owned by ID-69, NOT by ingest (comment at 1880–1890).
  • Path-B (forms — workspace-SCOPED): form_templates, form_template_fields. Written at _ingest_file_body 1879–2106 via ft_target / ftf_target. The form-write consumes a workspace_id from resolve_workspace (forms ARE workspace-scoped). Schemas: FORM_TEMPLATES_SCHEMA (flow.py:1201), FORM_TEMPLATE_FIELDS_SCHEMA (flow.py:1231).

A.4 Three concrete problems with the current shared path

Section titled “A.4 Three concrete problems with the current shared path”
  1. No fork — every file is double-processed. A blank form (e.g. charnwood.docx) today runs the full Path-A LLM pipeline (classification + qa_form + entity_mentions, three Anthropic calls at 1623–1628) AND lands content_items / source_documents / content_chunks / q_a_extractions / entity_mentions rows — THEN ALSO runs the Path-B form-write. A form is classified as knowledge content it is not. This is the cross-contamination the {80.2} brief asks to remove. Symmetrically, a content .md file runs the form-write block, falls through extract_form_structure → None (1960, 2000), and writes nothing — wasteful but harmless. The expensive, wrong direction is content-classifying a form.

  2. “All-or-nothing per walk” — a form failure reports the whole walk failed. The _ingest_file_body form-write block IS per-file-isolated for the two expected failure modes (UnmappedPath soft-warn at 1930–1937; FormExtractionErroranalysis_failed row at 1961–1998). BUT any unexpected exception escaping a per-item ingest_file is not caught at the mount_each site (app_main 2370–2384 has no per-item try/except), so it propagates to the flow-scope except Exception (2451), sets flow_status="failed" (2457), and the terminal webhook reports the entire walk failed (2537–2548). This is exactly the S301 bl-224 cascade recorded in the ID-80 journal: the form-path AttributeError “cascaded and zeroed short-clause’s writes too in the combined walk.” The separation must ensure a form-branch fault cannot flip the content-branch’s reported status.

  3. The fork point is implicit and order-coupled. “Is this a form?” is decided today by the presence of a workspace manifest for the run (1906–1910: if manifest is None: return) combined with whether extract_form_structure returns non-None (2000). That conflates two orthogonal axes — “is the run form-capable” vs “is THIS file a form” — and means Path-A always runs first regardless. There is no single, inspectable fork predicate.

A.5 Factual corrections folded from S314 research (05/06/2026)

Section titled “A.5 Factual corrections folded from S314 research (05/06/2026)”

Folded from docs/research/s314-id80-forms-current-state.md (§1–§2), grounding Liam’s two direct questions in docs/research/user-feedback-ids-52-75-80.md:

  1. Forms read RAW NATIVE BYTES, not Stage-2 Markdown. extract_form_structure hands await file.read() to the per-format readers (form_extractors/orchestrator.py:68-73); it never consumes the Markdown. The Stage-2 Markdown conversion that runs for forms today exists only because Path-A runs unconditionally first — it is Path-A waste the fork eliminates (the form branch in B.3 drops it entirely). Raw bytes are mandatory: fill-back cell coordinates (fill_template.py:158-175), empty-cell/placeholder signals, merged cells, track-changes resolution and highlight markers do not survive a Markdown projection.
  2. HTML is not a form format. The orchestrator dispatches .pdf/.xlsx/.docx only (.xls logged-and-skipped; everything else, including .html, returns None), and the form_templates.mime_type CHECK permits DOCX/PDF/XLSX only (M1, 20260528134712:12-16). HTML is a Path-A content format (pullmd). The B.3 suffix guard’s .md/.txt/.html mis-wire treatment is therefore correct.
  3. No title-rename surface exists in app or MCP. A form’s title is set once at manual upload (optional “Template Name” field) or binary-derived by the pipeline (filename stem; PDF page-1 title detection), and is re-asserted by the UPSERT on any byte change or full_reprocess walk — a hypothetical DB-side rename would be silently reverted (hazard flagged). Every form_templates UPDATE in the app is mapped_count or status; no MCP tool touches the instance tables. Form-tooling UX is routed to ID-71 per Liam’s feedback — not this Task.

This Subtask introduces an explicit, single fork point inside _ingest_file_body so that each walked file is classified once as form or content and routed down exactly one write path, and so that a fault in either branch is isolated to that file (and that branch) without flipping the other branch’s status. No new tables, no DDL, no MCP surface — this is a control-flow + failure-isolation refactor of one Python module (flow.py) plus a thin manifest extension.

Most-relevant code (verbatim line references, GitNexus + grep grounded):

  • scripts/cocoindex_pipeline/flow.py:1622-1628 — Path-A LLM extraction, runs unconditionally today. The fork must gate this.
  • scripts/cocoindex_pipeline/flow.py:1879-2106 — Path-B form-write tail block.
  • scripts/cocoindex_pipeline/flow.py:1906-1910 — the implicit manifest is None → return gate (today’s only “form capable” signal).
  • scripts/cocoindex_pipeline/flow.py:1960,2000extract_form_structure call + None fall-through (today’s only “is a form” signal).
  • scripts/cocoindex_pipeline/flow.py:2370-2384mount_each per-item fan-out with no per-item exception isolation.
  • scripts/cocoindex_pipeline/flow.py:2451-2490 — flow-scope except Exception that turns any escaped per-item fault into flow_status="failed".
  • scripts/cocoindex_pipeline/workspace_resolver.py:98-135WorkspaceManifest / WorkspaceMapping Pydantic models (the fork-config extension point).
  • docs/specs/id-52-form-extraction/{PRODUCT,TECH}.md — Inv-17 (failure isolation: per-file try/except, batch not halted — TECH §2.5 lines 303/336/953), Inv-19 (Path-A q_a_extractions write untouched by the form-write), the Path-A vs Path-B split (PRODUCT lines 86, 104–109).

Behaviour requirements this design must satisfy are the {80.2} acceptance lines in task-list.json: “forms and content paths separable; a form failure does not abort content writes (per-doc isolation).” and the parent Task description’s “separate forms from standard content uploads … forms are independently testable.”

B.1 The fork point — chosen and rejected candidates

Section titled “B.1 The fork point — chosen and rejected candidates”

The brief asks WHERE the fork happens. Five candidates were considered:

#Candidate fork-pointVerdict
1File-shape heuristic (sniff content: does the doc contain form fields / answer blanks?)REJECTED. Non-deterministic, needs the doc already converted/parsed, and is exactly the LLM-classification coupling we are removing. Re-introduces the “run Path-A to decide” ordering trap.
2Workspace TYPE (a workspaces row flag kind ∈ {content, forms})REJECTED for the fork, retained as defence-in-depth. A workspace can legitimately hold both forms and reference content; making the whole workspace one-or-the-other is too coarse and would force operators to silo uploads. Round-trips to the live workspaces table at ingest (the manifest is deliberately table-free per workspace_resolver.py:27-29).
3Run-level manifest presence (today’s implicit manifest is None → return)REJECTED as the sole signal. It only answers “is this run form-capable”, not “is THIS file a form”, and leaves Path-A running first for forms. It is kept as the outer gate (a content-only run has no manifest) but is insufficient alone.
4Folder/workspace MANIFEST mapping carries a per-prefix route tag (extend WorkspaceMapping with route: Literal["forms","content"] = "content")CHOSEN (primary). Deterministic, declared at the source root, no table round-trip, no doc parse, evaluated from rel_path alone — the same input resolve_workspace already consumes. The operator who lays out _held_forms/charnwood/ vs content folders already knows which is which; the manifest is the right place to declare it. Backward-compatible default "content" keeps every existing prefix on Path-A.
5Suffix-only (.pdf/.docx/.xlsx → forms; .md/.txt/.html → content)REJECTED as the sole signal, used as a SECONDARY guard. docx/pdf/xlsx are also valid Path-A content (a PDF white-paper is knowledge, not a form). Suffix cannot disambiguate a content PDF from a form PDF. Retained only as a cheap pre-filter: a .md/.txt/.html file under a route: "forms" prefix is a manifest mis-wire and is surfaced loudly (see B.3).

Decision: candidate 4 — a manifest-declared per-prefix route tag — is the fork point. It puts the routing decision on the same deterministic axis (rel_path → manifest prefix) that already owns workspace resolution, so forms and content are separated by folder placement the operator already controls, with zero new I/O and zero doc inspection. The fork is computed ONCE, BEFORE either write path runs.

RATIFIED — OQ-80.2-B (Liam, S314, 05/06/2026; synthesis brief D4): the manifest per-prefix route tag (candidate 4) IS the fork point. A separate folder ROOT is recorded as compatible future hardening — tag-now/root-later composes (a second mount or second app when forms cadence/ownership genuinely diverges; prefixes are root-relative so manifests port across). No re-design needed.

Blank-vs-answered caveat (ratified with OQ-80.2-A): route:"forms" routes blank form instruments only (ID-52 Mode-3). An answered/completed form is a knowledge container and remains Path-A (Mode-1) — its folder stays under a route:"content" prefix with the client knowledge corpus. The folder contract carries this distinction; the loud suffix guard (B.3) catches mis-wires.

B.2 Manifest schema extension (the fork config)

Section titled “B.2 Manifest schema extension (the fork config)”

Extend WorkspaceMapping in scripts/cocoindex_pipeline/workspace_resolver.py (currently lines 98–104) with a defaulted route discriminator, and add a resolver that returns BOTH the workspace and the route:

workspace_resolver.py
from typing import Literal
RouteKind = Literal["content", "forms"]
class WorkspaceMapping(BaseModel):
model_config = ConfigDict(extra="forbid", frozen=True)
path_prefix: str = Field(...)
workspace_id: UUID = Field(...)
route: RouteKind = Field(default="content") # NEW — fork discriminator
@dataclass(frozen=True)
class Resolution:
workspace_id: UUID
route: RouteKind
def resolve_route(manifest: WorkspaceManifest, rel_path: str) -> Resolution:
"""Longest-prefix-wins; returns the owning workspace AND its route.
Reuses the existing resolve_workspace prefix logic; raises the same
UnmappedPath / AmbiguousResolution subclasses (unchanged contract)."""

Notes:

  • route defaults to "content" so existing manifests (and id-52 fixtures) parse unchanged and every current prefix stays on Path-A — zero behaviour change until an operator opts a prefix into "forms". This preserves Inv-19 (Path-A writes untouched) for every non-forms prefix.
  • schema_version stays 1 (additive optional field; extra="forbid" still rejects typos). No migration — the manifest is a JSON file on the source volume, not a DB object.
  • resolve_route is the new single entry point; keep resolve_workspace as a thin shim (return resolve_route(...).workspace_id) so the {80.4} real-body tests and any other caller keep working (no barrel churn — direct imports per CLAUDE.md).

Restructure the body so the fork is the first decision after rel_path and Stage-2 conversion are known, and each branch is a self-contained, isolated unit:

_ingest_file_body(file, …targets…):
skip manifest file # unchanged (1480)
rel_path = _to_source_relative(...) # unchanged (1606)
# ── FORK (NEW, single point) ─────────────────────────────────────────
manifest = flow_workspace_manifest or current_workspace_manifest()
route = "content" # default when no manifest (Path-A-only run)
workspace_id = None
if manifest is not None:
try:
res = resolve_route(manifest, rel_path)
route, workspace_id = res.route, res.workspace_id
except UnmappedPath: # benign: no prefix owns this file
route = "content" # → treat as content (bl-219 semantics kept)
except ResolutionFailure as exc: # ambiguous: loud, ZERO rows, return
_emit_stage_error_log(stage="workspace_resolution", …); return
if route == "forms":
await _ingest_form_branch(file, rel_path, workspace_id, ft_target, ftf_target, …)
else:
await _ingest_content_branch(file, rel_path, ci/qa/sd/em/cc targets, …)
  • _ingest_content_branch = today’s Stage-2→6 Path-A block (1610–1877), extracted verbatim. It NEVER touches ft_target / ftf_target.
  • _ingest_form_branch = today’s Path-B block (1953–2106) PLUS the Stage-2 conversion it needs (forms still need the source bytes, but NOT the three Path-A Anthropic extractions — those are skipped entirely for forms). It NEVER touches ci/qa/sd/em/cc targets. It keeps the existing UnmappedPath/FormExtractionError/graceful-empty handling (1930–2106).
  • Mutual exclusion is structural — a file goes down exactly one branch, so the two write-target sets can never both be written for one file. This is the decoupling the brief asks for: no cross-contamination of either write path.
  • Secondary suffix guard (candidate 5): inside _ingest_form_branch, a non-form suffix (.md/.txt/.html) under a route:"forms" prefix is a manifest mis-wire → emit a loud cocoindex.stage_error (extraction_validation_failed) and write zero rows, mirroring the AmbiguousResolution treatment. This catches operator error early rather than silently producing an analysis_failed row.
  • Blank instruments only (OQ-80.2-A caveat, ratified 05/06/2026): the form branch is for blank form instruments (Mode-3). Answered/completed forms are knowledge containers (Mode-1) and must live under route:"content" prefixes — they take the content branch and land content_items / q_a_extractions as today. The folder contract carries the distinction; the suffix guard above is the loud backstop for mis-wires.

B.4 Per-item failure isolation (kills “all-or-nothing”)

Section titled “B.4 Per-item failure isolation (kills “all-or-nothing”)”

The separation is incomplete without fixing problem A.4.2. Wrap the per-item call at the mount_each boundary so an unexpected escape from one file’s branch is contained and attributed, not promoted to a whole-walk failure:

# app_main, replacing the bare bound_ingest_file (2355-2365)
async def bound_ingest_file(file, *targets):
try:
return await ingest_file(file, *targets, flow_op_id=run_op_id,)
except Exception as exc: # noqa: BLE001 — per-item containment
# Inv-17: one file's fault must NOT abort the batch or flip flow_status.
flow_item_failure_counter.increment(_branch_of(file)) # forms|content tally
_emit_stage_error_log(
op_id=run_op_id, stage="ingest_item",
error_class=_classify_stage_exception(exc) or type(exc).__name__,
content_items_id=None, error_message=_redact_error_message(str(exc)),
)
return None # swallow → batch continues
  • The flow-end webhook (2537–2548) gains a item_failures field ({forms: n, content: m}) so a partial-success walk is honestly reported as completed-with-failures rather than a blanket failed. The terminal flow_status stays "completed" when only per-item faults occurred; it goes "failed" only for flow-scope faults (manifest load, Stage-5, mount errors) — which are genuinely walk-wide. RATIFIED — OQ-80.2-C (Liam, S314, 05/06/2026; synthesis brief D5): these status semantics are approved exactly as stated. Implementation note: ID-61.4 (693f1565, at HEAD) extended this same webhook payload surface (errorDetail
    • taxonomyMisses); the item_failures field composes additively alongside them — coordinate, don’t clobber (see {80.9} Subtask record).
  • This is the precise inversion of the S301 bl-224 cascade (“a form failure aborts the whole batch … zeroed short-clause’s writes too”). After this change a form-branch exception increments item_failures.forms and the content files in the same walk still report their landed rows.
  • Stage-5 (2430–2446) is flow-scope and stays inside the outer try — an entity- resolution fault is genuinely walk-wide (it operates over the whole op’s rows), so it correctly remains a failed status. The {80.5}/bl-225 fix already made Stage-5 collision-safe; this Subtask does not touch it.

B.5 What does NOT change (ownership boundaries held)

Section titled “B.5 What does NOT change (ownership boundaries held)”
  • No DDL, no new tables, no RLS changes. form_templates / form_template_fields schemas (flow.py:1201/1231) and the content tables are untouched. The managed_by=ManagedBy.USER row-only contract holds.
  • content_item_workspaces M2M stays ID-69’s job (flow.py:1880-1890 comment) — the content branch still writes NO workspace_id onto content_items.
  • Memo / idempotency (@coco.fn(memo=True) on ingest_file, extract_form_structure, convert_binary_to_markdown) is preserved: the fork is pure-path computation on rel_path, deterministic across runs, so the memo key and the deterministic uuid5 PKs (ft:/ftf:/em: seeds) are unchanged. A re-ingest of the same bytes down the same branch UPSERTs the same rows (Inv-16).
  • _trim_stale_form_fields ({80.1} bl-224 fix, coco.use_context(DB_CTX) at flow.py:2127) is called only inside _ingest_form_branch, unchanged.

Both branches keep deterministic uuid5 PKs seeded on rel_path (content: content_items.id via the per-doc namespace; forms: ft:{rel_path} / ftf:{rel_path}:{seq} at 1958/2088). Because the fork is a pure function of rel_path + manifest (no clock, no I/O, no ordering), a file deterministically takes the same branch on every run → same PKs → UPSERT, never duplicate. A file whose manifest prefix is RE-TAGGED contentforms between runs would orphan its prior-branch rows; this is an operator action and is called out in Risks (B.8).

flowchart TD
A[walk_dir item: FileLike] --> B{manifest file?}
B -- yes --> Z[skip]
B -- no --> C[rel_path = to_source_relative]
C --> D{manifest present\nfor this run?}
D -- no --> CONTENT
D -- yes --> E[resolve_route rel_path]
E -- UnmappedPath --> CONTENT[Path-A content branch]
E -- AmbiguousResolution --> ERR[loud stage_error\nZERO rows]
E -- route=content --> CONTENT
E -- route=forms --> FORMS[Path-B form branch]
CONTENT --> CW[(content_items / source_documents /\ncontent_chunks / q_a_extractions /\nentity_mentions)]
FORMS --> FW[(form_templates /\nform_template_fields)]
CONTENT -. exception .-> ISO[per-item catch:\nitem_failures.content++\nbatch continues]
FORMS -. exception .-> ISO2[per-item catch:\nitem_failures.forms++\nbatch continues]
RiskMitigation
Manifest route typo silently routes a form to content (or vice-versa).extra="forbid" + Literal["content","forms"] makes any non-enum value a load-time ManifestLoadError → flow aborts loudly at start (the manifest-load gate at flow.py:2206-2230).
Suffix/route mismatch (.md under route:"forms").Secondary suffix guard (B.3) → loud cocoindex.stage_error, zero rows. Not silent.
Re-tagging a prefix contentforms orphans prior-branch rows.Operator action only; document in the on-prem runbook. Out of automated scope for {80.2}; flag as a follow-up (a sweep that detects branch-flip by storage_path).
Per-item swallow hides a systemic fault (every file fails the same way).item_failures counter surfaces the tally in the flow-end webhook; a 100%-failure walk is visible as completed with item_failures == items_processed — a loud signal, not a silent green. Add a webhook-side alert threshold as a follow-up.
Regression: existing content-only walks change behaviour.route defaults to "content"; no manifest → content branch. Existing flow tests (which mock mount_each and bind no forms route) exercise the content branch unchanged.
Forms no longer get content_items rows (a behaviour change).Intended — a blank form is not knowledge content. RATIFIED (OQ-80.2-A, Liam, S314, 05/06/2026): forms land ZERO content rows. All 11 downstream consumer groups inventoried (docs/research/s314-id80-forms-current-state.md §3) — none requires the row; every effect is neutral-to-improvement. Confirm with Liam whether any downstream expects a content_items row — confirmed not needed; the minimal-provenance-row fallback is DROPPED. Caveat: blank instruments only — answered forms remain Path-A (see B.1/B.3).

Tests verify real behaviour, not the patched seam (per docs/reference/test-philosophy.md — the {80.4} audit’s root finding was seam-patch coverage debt). Each row maps to the {80.2} acceptance criteria and the inherited ID-52 Inv-17/Inv-19.

Behaviour to proveVerification
Fork routes a form to Path-B onlyReal-body test: ingest a route:"forms" .docx form through _ingest_file_body; assert ft_target/ftf_target received declare_row, and ci/qa/sd/em/cc targets received zero calls (no Path-A LLM extractors invoked).
Fork routes content to Path-A onlyReal-body test: ingest a route:"content" (or unmapped) .md; assert content targets written, ft/ftf zero calls.
Per-doc isolation (the {80.2} headline)Batch test through mount_each: [form_that_raises.docx, good_content.md]; assert good_content content rows landed, flow_status == "completed", item_failures.forms == 1. Proves a form fault does NOT abort content writes — the bl-224 cascade inversion.
Inv-17 graceful failure modes preservedFormExtractionError → one analysis_failed form_templates row, zero fields; UnmappedPath → content branch, no form rows; AmbiguousResolution → loud stage_error, zero rows. (Port existing {80.4}/test_cocoindex_flow_write_path.py cases onto the new branch.)
Inv-19 — Path-A writes never touched by form routingAssert the content branch’s q_a_extractions declare is byte-identical to pre-refactor for a content file.
Suffix/route mismatch is loud.md under route:"forms"cocoindex.stage_error, zero rows.
Idempotency across both branchesRe-ingest same bytes twice down each branch; assert identical uuid5 PKs and UPSERT (no duplicate rows).
Manifest backward-compatA manifest WITHOUT route parses; all prefixes resolve route="content" (default). Existing id-52 fixtures unchanged.
Live staging re-smoke (parent-owned)After merge: re-walk charnwood (form) + a content doc in one walk on staging; assert charnwood writes form_templates=1 + fields, the content doc writes content rows, flow status completed, and charnwood writes no content_items row.

Run unit/Python tests via python3 -m pytest scripts/tests/ (the Python pipeline suite). The TS suite (bun run test) is unaffected — no TS files change.

Proposed implementation Subtasks (for the Orchestrator to decompose / append)

Section titled “Proposed implementation Subtasks (for the Orchestrator to decompose / append)”

This is decomposition guidance, NOT ratified {N.5+} records — the Orchestrator owns appending to task-list.json. Each carries the code-intelligence tool-discipline pattern (impact-before-edit on the named Python symbol via gitnexus_impact + grep; gitnexus_detect_changes before commit) per .ast-dataflow/CLAUDE.md Inv 2/3. All are sibling-only (same Task ID-80).

05/06/2026 — authored. Following ratification, the six items below were turned into TM-shape Subtask records {80.6}{80.11}, plus {80.12} (the D6 form-versioning investigation — see the Versioning section below), at docs/research/s314-id80-subtask-records.json for the Orchestrator to append. Line refs in those records are re-pinned post-ID-61.4 (HEAD 693f1565) — implementers use the record pins, not this doc’s pre-61.4 refs (see A.1).

  1. Manifest route extension — add route to WorkspaceMapping + Resolution/resolve_route in workspace_resolver.py; resolve_workspace becomes a shim. Tests: backward-compat parse, route resolution, default. Depends on: none.
  2. Extract _ingest_content_branch + _ingest_form_branch from _ingest_file_body (no behaviour change yet — pure extraction). Depends on: none.
  3. Wire the fork — insert the single fork predicate calling resolve_route; forms skip the three Path-A Anthropic extractions. Depends on: 1, 2.
  4. Per-item isolation + item_failures — wrap bound_ingest_file; add the counter + webhook field; keep flow-scope faults failed. Depends on: 2.
  5. Test sweep — fork-routing, per-doc isolation, Inv-17/Inv-19 ports, idempotency, suffix-guard, backward-compat. Depends on: 3, 4.
  6. Runbook note + staging re-smoke handoff (parent-owned burn). Depends on: 5.

Open questions — ALL RATIFIED (Liam, S314, 05/06/2026)

Section titled “Open questions — ALL RATIFIED (Liam, S314, 05/06/2026)”

Ratification source: Liam’s verbatim feedback (docs/research/user-feedback-ids-52-75-80.md, ID-80.2 section) as resolved through docs/research/s314-feedback-synthesis-decision-brief.md Part 2 (decisions D3/D4/D5/D6), grounded by docs/research/s314-id80-forms-current-state.md and docs/research/s314-ontology-boundary.md §5.

  • OQ-80.2-A (behaviour delta): Today a form ALSO lands a content_items row (+ chunks/q_a/entity_mentions). After separation a form lands form rows ONLY. Is any downstream (search index, content_item_workspaces, MCP retrieval) relying on a content_items row existing per form today? If yes, the fork should optionally still write a minimal source_documents/content_items provenance row for forms — confirm before {80.2} implementation. RATIFIED (D3): forms land ZERO content rows (no content_items / source_documents / content_chunks / q_a_extractions / entity_mentions). The minimal-provenance-row fallback is DROPPED — all 11 downstream consumer groups were inventoried and none requires the row; any downstream that wants forms gets a forms-aware surface (ID-71 scope), not a content_items shim. The ontology already encodes this position (no form value in the closed content_type CV; forms have their own Layer-5 tables + CVs). CAVEAT: only BLANK form instruments route Path-B; ANSWERED/completed forms are knowledge containers and remain Path-A — the folder contract carries the distinction; the loud suffix guard catches mis-wires.
  • OQ-80.2-B (fork authority): RATIFIED (D4): the fork is the manifest per-prefix route tag (candidate 4). A separate folder ROOT is recorded as compatible future hardening — tag-now/root-later composes (a second mount or app when forms cadence/ownership genuinely diverges). See the ratification block in B.1.
  • OQ-80.2-C (status semantics): RATIFIED (D5): a walk with only per-item form failures reports flow_status="completed" with the item_failures tally set; "failed" is reserved for walk-wide faults. The webhook status contract change is approved. See the ratification note in B.4.

Versioning (out of scope — {80.12} investigation Subtask)

Section titled “Versioning (out of scope — {80.12} investigation Subtask)”

D6 (Liam, S314, 05/06/2026): instance-side form versioning is a separate designed item, authored as a Subtask of ID-80 — investigation-first, not necessarily executed this session. Today charnwood-v2.docx arriving beside charnwood.docx lands a wholly unlinked sibling row (new rel_path → new uuid5("ft:{rel_path}")); form_templates has no version / is_current / supersession columns — only the catalogue side (form_template_requirements.template_version + is_current) carries any versioning substrate. {80.12} surveys the option space (supersession link table vs is_current columns vs catalogue-style template_version), weighs the procurement-workspaces Q&A version-on-cite precedent (PRODUCT.md B-12/B-15), and recommends without implementing. This spec stays separation-scoped — the fork ships without inventing versioning under time pressure.

Verification (external-symbol pins — pre-ratification, OQ-3 / Q-EX2)

Section titled “Verification (external-symbol pins — pre-ratification, OQ-3 / Q-EX2)”

Checked against the installed pins on 04/06/2026:

  • cocoindex==1.0.3coco.fn / coco.mount_each / coco.use_context(DB_CTX) / mount_table_target / ManagedBy.USER are the symbols this spec builds on; all are already live in flow.py (proven by the {80.1}/{80.5} staging burns, op 353fe637 status=completed). No NEW cocoindex symbol is introduced by this spec — the fork is pure KH-authored Python over the existing flow. Result: PRESENT (transitively, via the green {80.1}/{80.5} live runs).
  • pydantic==2.12.5BaseModel / Field(default=...) / ConfigDict(extra= "forbid") / Literal discriminator used by the route extension are Pydantic v2 core; already used by WorkspaceManifest (workspace_resolver.py:39,107). Result: PRESENT.
  • No third-party SDK shape is newly cited; asyncpg/docling usages are unchanged from the current flow.py. No ABSENT / SIGNATURE_DRIFT / BEHAVIOUR_DRIFT found → no escalation required.