ID-80.2 — Separate forms (Path-B) from standard content uploads (Path-A)
ID-80.2 — Separate forms (Path-B) from standard content uploads (Path-A)
Section titled “ID-80.2 — Separate forms (Path-B) from standard content uploads (Path-A)”Artefact kind: architecture / design doc (TECH-style) with a short RESEARCH preamble, self-contained per the {80.2} dispatch brief. This is NOT the full
{N.1}…{N.4}chain — it is the one separation-architecture deliverable for the remaining open Subtask of Task ID-80.Parent Task: ID-80 “Productionise cocoindex form-write path (Path-B / forms)” (
docs/reference/task-list.json→.tasks[] | select(.id=="80")). Sibling Subtasks already DONE: {80.1}_trim_stale_form_fieldsDB_CTX accessor (bl-224), {80.3} pandoc install, {80.4} form-write audit + coverage, {80.5} Stage-5 intra-doc canonical-collision fix (bl-225).UK English throughout. Dates DD/MM/YYYY.
Status: RATIFIED with amendments (Liam, S314, 05/06/2026). OQ-80.2-A/B/C all ratified (see the Open questions section); minimal-provenance-row fallback dropped; blank-vs-answered caveat recorded in B.1/B.3; form-instance versioning routed to the
{80.12}investigation Subtask. Implementation Subtask records{80.6}–{80.12}:docs/research/s314-id80-subtask-records.json.
Part A — RESEARCH preamble (grounded current state)
Section titled “Part A — RESEARCH preamble (grounded current state)”A.1 Code-intelligence orientation
Section titled “A.1 Code-intelligence orientation”Per .gitnexus/CLAUDE.md “Always Do” and .ast-dataflow/CLAUDE.md, this spec was
oriented with GitNexus + a Python grep sweep (ast-dataflow does not cover the
Python pipeline corpus — scripts/cocoindex_pipeline/*.py).
-
gitnexus_query({query: "cocoindex ingest_file form extraction workspace resolution write form_templates content_items fork", repo: "knowledge-hub"})— returned no single execution flow that already forks forms from content. The matcheddefinitionsconfirm the fork is INLINE inside one symbol:Function:scripts/cocoindex_pipeline/flow.py:ingest_file(startLine 1389),Function:scripts/cocoindex_pipeline/flow.py:_ingest_file_body(startLine 1560, endLine 2105),Function:scripts/cocoindex_pipeline/flow.py:app_main(startLine 2139), andFunction:scripts/cocoindex_pipeline/workspace_resolver.py:resolve_workspace(startLine 173). The form-vs-content decision is NOT a separate process — it is a tail block of_ingest_file_body. This is the finding the separation must act on. -
gitnexus_context({name: "extract_form_structure", repo: "knowledge-hub"})— verdict:incoming: {}(zero indexed callers). It is invoked dynamically as a@coco.fn(memo=True)awaited inside the daemon-thread body (flow.py:1960), so GitNexus indexes no static caller.outgoing.calls→ per-format readersform_extractors/{docx,pdf,xlsx}.py::extract. Blast radius of changing the call site is therefore confined to_ingest_file_body; the orchestrator + readers are unaffected by a fork-point change. -
Python grep sweep (
scripts/cocoindex_pipeline/flow.py) confirmed the Path-A extractor calls (extract_classification,extract_qa_form,extract_entity_mentions) at lines 1623–1628 run unconditionally for every file, BEFORE the Path-B form-write block at 1879–2106. There is no suffix / manifest gate in front of Path-A today. -
Re-verification (05/06/2026, post-ratification dispatch):
gitnexus_contextre-run on the two fork-critical symbols._ingest_file_body— unchanged:incoming.calls: [ingest_file](1 caller); outgoing set still includesresolve_workspace, the three Path-A extractors and_trim_stale_form_fields.resolve_workspace— 9 incoming callers (8 test functions inscripts/tests/test_workspace_resolver.py+_ingest_file_body, the only production caller), so the B.2 shim plan protects the entire existing test surface. Line-ref re-pin: ID-61.4 (693f1565) has since landed at HEAD, adding +55 lines toflow.py(_pydantic_error_detailhelper at :367; webhookerror_detailparam at :715). The Part A/B line refs in this doc were pinned pre-61.4 and now drift by ~+44 fromflow.py:1434(ingest_file) onward. Post-61.4 re-pins live in the{80.6}–{80.12}Subtask records (docs/research/s314-id80-subtask-records.json) — implementers use those, not the refs below.
A.2 The unified path as it exists today
Section titled “A.2 The unified path as it exists today”cocoindex 1.0.3 (requirements.txt: cocoindex[postgres]==1.0.3,
pydantic==2.12.5, asyncpg>=0.30.0, docling>=2.0.0,<3.0.0) drives a single
reactive flow. The relevant span:
| File / lines | Role |
|---|---|
flow.py:2140 app_main | Loads workspace manifest once (2206–2230); mounts seven mount_table_target row targets (2253–2302) under managed_by=ManagedBy.USER; localfs.walk_dir(..., live=True, recursive=True) (2307); fans out per-item via coco.mount_each(..., bound_ingest_file, source.items(), <7 targets>) (2370–2381); runs flow-scope Stage-5 (2430–2446). |
flow.py:1389 ingest_file | @coco.fn(memo=True) wrapper. Skips the manifest file itself (1480). Resolves run context (op_id / counters / manifest) from explicit closure args with ContextVar fallback (1503–1541). Delegates to _ingest_file_body. |
flow.py:1560 _ingest_file_body | The actual Stage-2→6 work. Path-A runs first, unconditionally; Path-B is an additive tail block. |
adapters.py:56 convert_binary_to_markdown | Stage-2: routes a FileLike by suffix → Docling (pdf/docx/xlsx) / pullmd (html) / passthrough (md/txt). |
form_extractors/orchestrator.py:54 extract_form_structure | Path-B Stage-3a: deterministic raw-format reader dispatch by suffix; returns `ExtractedForm |
workspace_resolver.py:174 resolve_workspace | Longest-prefix folder→workspace resolution; raises UnmappedPath (benign) / AmbiguousResolution (loud). |
A.3 The two write targets (the boundary we must keep clean)
Section titled “A.3 The two write targets (the boundary we must keep clean)”- Path-A (standard content — workspace-AGNOSTIC canonical layer):
content_items,source_documents,content_chunks,q_a_extractions,entity_mentions. Written at_ingest_file_body1622–1877. Per ID-69 BI-1,content_itemshas noworkspace_id; workspace association is thecontent_item_workspacesM2M junction owned by ID-69, NOT by ingest (comment at 1880–1890). - Path-B (forms — workspace-SCOPED):
form_templates,form_template_fields. Written at_ingest_file_body1879–2106 viaft_target/ftf_target. The form-write consumes aworkspace_idfromresolve_workspace(forms ARE workspace-scoped). Schemas:FORM_TEMPLATES_SCHEMA(flow.py:1201),FORM_TEMPLATE_FIELDS_SCHEMA(flow.py:1231).
A.4 Three concrete problems with the current shared path
Section titled “A.4 Three concrete problems with the current shared path”-
No fork — every file is double-processed. A blank form (e.g.
charnwood.docx) today runs the full Path-A LLM pipeline (classification + qa_form + entity_mentions, three Anthropic calls at 1623–1628) AND landscontent_items/source_documents/content_chunks/q_a_extractions/entity_mentionsrows — THEN ALSO runs the Path-B form-write. A form is classified as knowledge content it is not. This is the cross-contamination the {80.2} brief asks to remove. Symmetrically, a content.mdfile runs the form-write block, falls throughextract_form_structure → None(1960, 2000), and writes nothing — wasteful but harmless. The expensive, wrong direction is content-classifying a form. -
“All-or-nothing per walk” — a form failure reports the whole walk failed. The
_ingest_file_bodyform-write block IS per-file-isolated for the two expected failure modes (UnmappedPathsoft-warn at 1930–1937;FormExtractionError→analysis_failedrow at 1961–1998). BUT any unexpected exception escaping a per-itemingest_fileis not caught at themount_eachsite (app_main 2370–2384 has no per-item try/except), so it propagates to the flow-scopeexcept Exception(2451), setsflow_status="failed"(2457), and the terminal webhook reports the entire walk failed (2537–2548). This is exactly the S301 bl-224 cascade recorded in the ID-80 journal: the form-pathAttributeError“cascaded and zeroed short-clause’s writes too in the combined walk.” The separation must ensure a form-branch fault cannot flip the content-branch’s reported status. -
The fork point is implicit and order-coupled. “Is this a form?” is decided today by the presence of a workspace manifest for the run (1906–1910:
if manifest is None: return) combined with whetherextract_form_structurereturns non-None (2000). That conflates two orthogonal axes — “is the run form-capable” vs “is THIS file a form” — and means Path-A always runs first regardless. There is no single, inspectable fork predicate.
A.5 Factual corrections folded from S314 research (05/06/2026)
Section titled “A.5 Factual corrections folded from S314 research (05/06/2026)”Folded from docs/research/s314-id80-forms-current-state.md (§1–§2), grounding
Liam’s two direct questions in docs/research/user-feedback-ids-52-75-80.md:
- Forms read RAW NATIVE BYTES, not Stage-2 Markdown.
extract_form_structurehandsawait file.read()to the per-format readers (form_extractors/orchestrator.py:68-73); it never consumes the Markdown. The Stage-2 Markdown conversion that runs for forms today exists only because Path-A runs unconditionally first — it is Path-A waste the fork eliminates (the form branch in B.3 drops it entirely). Raw bytes are mandatory: fill-back cell coordinates (fill_template.py:158-175), empty-cell/placeholder signals, merged cells, track-changes resolution and highlight markers do not survive a Markdown projection. - HTML is not a form format. The orchestrator dispatches
.pdf/.xlsx/.docxonly (.xlslogged-and-skipped; everything else, including.html, returnsNone), and theform_templates.mime_typeCHECK permits DOCX/PDF/XLSX only (M1,20260528134712:12-16). HTML is a Path-A content format (pullmd). The B.3 suffix guard’s.md/.txt/.htmlmis-wire treatment is therefore correct. - No title-rename surface exists in app or MCP. A form’s title is set once at
manual upload (optional “Template Name” field) or binary-derived by the
pipeline (filename stem; PDF page-1 title detection), and is re-asserted by the
UPSERT on any byte change or
full_reprocesswalk — a hypothetical DB-side rename would be silently reverted (hazard flagged). Everyform_templatesUPDATE in the app ismapped_countorstatus; no MCP tool touches the instance tables. Form-tooling UX is routed to ID-71 per Liam’s feedback — not this Task.
Part B — TECH (separation architecture)
Section titled “Part B — TECH (separation architecture)”Context
Section titled “Context”This Subtask introduces an explicit, single fork point inside
_ingest_file_body so that each walked file is classified once as form or
content and routed down exactly one write path, and so that a fault in either
branch is isolated to that file (and that branch) without flipping the other
branch’s status. No new tables, no DDL, no MCP surface — this is a control-flow +
failure-isolation refactor of one Python module (flow.py) plus a thin manifest
extension.
Most-relevant code (verbatim line references, GitNexus + grep grounded):
scripts/cocoindex_pipeline/flow.py:1622-1628— Path-A LLM extraction, runs unconditionally today. The fork must gate this.scripts/cocoindex_pipeline/flow.py:1879-2106— Path-B form-write tail block.scripts/cocoindex_pipeline/flow.py:1906-1910— the implicitmanifest is None → returngate (today’s only “form capable” signal).scripts/cocoindex_pipeline/flow.py:1960,2000—extract_form_structurecall +Nonefall-through (today’s only “is a form” signal).scripts/cocoindex_pipeline/flow.py:2370-2384—mount_eachper-item fan-out with no per-item exception isolation.scripts/cocoindex_pipeline/flow.py:2451-2490— flow-scopeexcept Exceptionthat turns any escaped per-item fault intoflow_status="failed".scripts/cocoindex_pipeline/workspace_resolver.py:98-135—WorkspaceManifest/WorkspaceMappingPydantic models (the fork-config extension point).docs/specs/id-52-form-extraction/{PRODUCT,TECH}.md— Inv-17 (failure isolation: per-filetry/except, batch not halted — TECH §2.5 lines 303/336/953), Inv-19 (Path-A q_a_extractions write untouched by the form-write), the Path-A vs Path-B split (PRODUCT lines 86, 104–109).
Behaviour requirements this design must satisfy are the {80.2} acceptance lines in
task-list.json: “forms and content paths separable; a form failure does not
abort content writes (per-doc isolation).” and the parent Task description’s
“separate forms from standard content uploads … forms are independently testable.”
Proposed changes
Section titled “Proposed changes”B.1 The fork point — chosen and rejected candidates
Section titled “B.1 The fork point — chosen and rejected candidates”The brief asks WHERE the fork happens. Five candidates were considered:
| # | Candidate fork-point | Verdict |
|---|---|---|
| 1 | File-shape heuristic (sniff content: does the doc contain form fields / answer blanks?) | REJECTED. Non-deterministic, needs the doc already converted/parsed, and is exactly the LLM-classification coupling we are removing. Re-introduces the “run Path-A to decide” ordering trap. |
| 2 | Workspace TYPE (a workspaces row flag kind ∈ {content, forms}) | REJECTED for the fork, retained as defence-in-depth. A workspace can legitimately hold both forms and reference content; making the whole workspace one-or-the-other is too coarse and would force operators to silo uploads. Round-trips to the live workspaces table at ingest (the manifest is deliberately table-free per workspace_resolver.py:27-29). |
| 3 | Run-level manifest presence (today’s implicit manifest is None → return) | REJECTED as the sole signal. It only answers “is this run form-capable”, not “is THIS file a form”, and leaves Path-A running first for forms. It is kept as the outer gate (a content-only run has no manifest) but is insufficient alone. |
| 4 | Folder/workspace MANIFEST mapping carries a per-prefix route tag (extend WorkspaceMapping with route: Literal["forms","content"] = "content") | CHOSEN (primary). Deterministic, declared at the source root, no table round-trip, no doc parse, evaluated from rel_path alone — the same input resolve_workspace already consumes. The operator who lays out _held_forms/charnwood/ vs content folders already knows which is which; the manifest is the right place to declare it. Backward-compatible default "content" keeps every existing prefix on Path-A. |
| 5 | Suffix-only (.pdf/.docx/.xlsx → forms; .md/.txt/.html → content) | REJECTED as the sole signal, used as a SECONDARY guard. docx/pdf/xlsx are also valid Path-A content (a PDF white-paper is knowledge, not a form). Suffix cannot disambiguate a content PDF from a form PDF. Retained only as a cheap pre-filter: a .md/.txt/.html file under a route: "forms" prefix is a manifest mis-wire and is surfaced loudly (see B.3). |
Decision: candidate 4 — a manifest-declared per-prefix route tag — is the fork
point. It puts the routing decision on the same deterministic axis
(rel_path → manifest prefix) that already owns workspace resolution, so forms
and content are separated by folder placement the operator already controls, with
zero new I/O and zero doc inspection. The fork is computed ONCE, BEFORE either
write path runs.
RATIFIED — OQ-80.2-B (Liam, S314, 05/06/2026; synthesis brief D4): the manifest per-prefix
routetag (candidate 4) IS the fork point. A separate folder ROOT is recorded as compatible future hardening — tag-now/root-later composes (a second mount or second app when forms cadence/ownership genuinely diverges; prefixes are root-relative so manifests port across). No re-design needed.Blank-vs-answered caveat (ratified with OQ-80.2-A):
route:"forms"routes blank form instruments only (ID-52 Mode-3). An answered/completed form is a knowledge container and remains Path-A (Mode-1) — its folder stays under aroute:"content"prefix with the client knowledge corpus. The folder contract carries this distinction; the loud suffix guard (B.3) catches mis-wires.
B.2 Manifest schema extension (the fork config)
Section titled “B.2 Manifest schema extension (the fork config)”Extend WorkspaceMapping in scripts/cocoindex_pipeline/workspace_resolver.py
(currently lines 98–104) with a defaulted route discriminator, and add a resolver
that returns BOTH the workspace and the route:
from typing import Literal
RouteKind = Literal["content", "forms"]
class WorkspaceMapping(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) path_prefix: str = Field(...) workspace_id: UUID = Field(...) route: RouteKind = Field(default="content") # NEW — fork discriminator
@dataclass(frozen=True)class Resolution: workspace_id: UUID route: RouteKind
def resolve_route(manifest: WorkspaceManifest, rel_path: str) -> Resolution: """Longest-prefix-wins; returns the owning workspace AND its route. Reuses the existing resolve_workspace prefix logic; raises the same UnmappedPath / AmbiguousResolution subclasses (unchanged contract)."""Notes:
routedefaults to"content"so existing manifests (andid-52fixtures) parse unchanged and every current prefix stays on Path-A — zero behaviour change until an operator opts a prefix into"forms". This preserves Inv-19 (Path-A writes untouched) for every non-forms prefix.schema_versionstays1(additive optional field;extra="forbid"still rejects typos). No migration — the manifest is a JSON file on the source volume, not a DB object.resolve_routeis the new single entry point; keepresolve_workspaceas a thin shim (return resolve_route(...).workspace_id) so the {80.4} real-body tests and any other caller keep working (no barrel churn — direct imports per CLAUDE.md).
B.3 The branch in _ingest_file_body
Section titled “B.3 The branch in _ingest_file_body”Restructure the body so the fork is the first decision after rel_path and
Stage-2 conversion are known, and each branch is a self-contained, isolated unit:
_ingest_file_body(file, …targets…): skip manifest file # unchanged (1480) rel_path = _to_source_relative(...) # unchanged (1606)
# ── FORK (NEW, single point) ───────────────────────────────────────── manifest = flow_workspace_manifest or current_workspace_manifest() route = "content" # default when no manifest (Path-A-only run) workspace_id = None if manifest is not None: try: res = resolve_route(manifest, rel_path) route, workspace_id = res.route, res.workspace_id except UnmappedPath: # benign: no prefix owns this file route = "content" # → treat as content (bl-219 semantics kept) except ResolutionFailure as exc: # ambiguous: loud, ZERO rows, return _emit_stage_error_log(stage="workspace_resolution", …); return
if route == "forms": await _ingest_form_branch(file, rel_path, workspace_id, ft_target, ftf_target, …) else: await _ingest_content_branch(file, rel_path, ci/qa/sd/em/cc targets, …)_ingest_content_branch= today’s Stage-2→6 Path-A block (1610–1877), extracted verbatim. It NEVER touchesft_target/ftf_target._ingest_form_branch= today’s Path-B block (1953–2106) PLUS the Stage-2 conversion it needs (forms still need the source bytes, but NOT the three Path-A Anthropic extractions — those are skipped entirely for forms). It NEVER touchesci/qa/sd/em/cctargets. It keeps the existingUnmappedPath/FormExtractionError/graceful-empty handling (1930–2106).- Mutual exclusion is structural — a file goes down exactly one branch, so the two write-target sets can never both be written for one file. This is the decoupling the brief asks for: no cross-contamination of either write path.
- Secondary suffix guard (candidate 5): inside
_ingest_form_branch, a non-form suffix (.md/.txt/.html) under aroute:"forms"prefix is a manifest mis-wire → emit a loudcocoindex.stage_error(extraction_validation_failed) and write zero rows, mirroring theAmbiguousResolutiontreatment. This catches operator error early rather than silently producing ananalysis_failedrow. - Blank instruments only (OQ-80.2-A caveat, ratified 05/06/2026): the form
branch is for blank form instruments (Mode-3). Answered/completed forms are
knowledge containers (Mode-1) and must live under
route:"content"prefixes — they take the content branch and landcontent_items/q_a_extractionsas today. The folder contract carries the distinction; the suffix guard above is the loud backstop for mis-wires.
B.4 Per-item failure isolation (kills “all-or-nothing”)
Section titled “B.4 Per-item failure isolation (kills “all-or-nothing”)”The separation is incomplete without fixing problem A.4.2. Wrap the per-item call
at the mount_each boundary so an unexpected escape from one file’s branch is
contained and attributed, not promoted to a whole-walk failure:
# app_main, replacing the bare bound_ingest_file (2355-2365)async def bound_ingest_file(file, *targets): try: return await ingest_file(file, *targets, flow_op_id=run_op_id, …) except Exception as exc: # noqa: BLE001 — per-item containment # Inv-17: one file's fault must NOT abort the batch or flip flow_status. flow_item_failure_counter.increment(_branch_of(file)) # forms|content tally _emit_stage_error_log( op_id=run_op_id, stage="ingest_item", error_class=_classify_stage_exception(exc) or type(exc).__name__, content_items_id=None, error_message=_redact_error_message(str(exc)), ) return None # swallow → batch continues- The flow-end webhook (2537–2548) gains a
item_failuresfield ({forms: n, content: m}) so a partial-success walk is honestly reported as completed-with-failures rather than a blanketfailed. The terminalflow_statusstays"completed"when only per-item faults occurred; it goes"failed"only for flow-scope faults (manifest load, Stage-5, mount errors) — which are genuinely walk-wide. RATIFIED — OQ-80.2-C (Liam, S314, 05/06/2026; synthesis brief D5): these status semantics are approved exactly as stated. Implementation note: ID-61.4 (693f1565, at HEAD) extended this same webhook payload surface (errorDetailtaxonomyMisses); theitem_failuresfield composes additively alongside them — coordinate, don’t clobber (see{80.9}Subtask record).
- This is the precise inversion of the S301 bl-224 cascade (“a form failure
aborts the whole batch … zeroed short-clause’s writes too”). After this change a
form-branch exception increments
item_failures.formsand the content files in the same walk still report their landed rows. - Stage-5 (2430–2446) is flow-scope and stays inside the outer try — an entity-
resolution fault is genuinely walk-wide (it operates over the whole op’s rows),
so it correctly remains a
failedstatus. The {80.5}/bl-225 fix already made Stage-5 collision-safe; this Subtask does not touch it.
B.5 What does NOT change (ownership boundaries held)
Section titled “B.5 What does NOT change (ownership boundaries held)”- No DDL, no new tables, no RLS changes.
form_templates/form_template_fieldsschemas (flow.py:1201/1231) and the content tables are untouched. Themanaged_by=ManagedBy.USERrow-only contract holds. content_item_workspacesM2M stays ID-69’s job (flow.py:1880-1890 comment) — the content branch still writes NOworkspace_idontocontent_items.- Memo / idempotency (
@coco.fn(memo=True)oningest_file,extract_form_structure,convert_binary_to_markdown) is preserved: the fork is pure-path computation onrel_path, deterministic across runs, so the memo key and the deterministic uuid5 PKs (ft:/ftf:/em:seeds) are unchanged. A re-ingest of the same bytes down the same branch UPSERTs the same rows (Inv-16). _trim_stale_form_fields({80.1} bl-224 fix,coco.use_context(DB_CTX)at flow.py:2127) is called only inside_ingest_form_branch, unchanged.
B.6 Idempotency across both branches
Section titled “B.6 Idempotency across both branches”Both branches keep deterministic uuid5 PKs seeded on rel_path (content:
content_items.id via the per-doc namespace; forms: ft:{rel_path} /
ftf:{rel_path}:{seq} at 1958/2088). Because the fork is a pure function of
rel_path + manifest (no clock, no I/O, no ordering), a file deterministically
takes the same branch on every run → same PKs → UPSERT, never duplicate. A file
whose manifest prefix is RE-TAGGED content→forms between runs would orphan its
prior-branch rows; this is an operator action and is called out in Risks (B.8).
B.7 End-to-end flow (after separation)
Section titled “B.7 End-to-end flow (after separation)”flowchart TD A[walk_dir item: FileLike] --> B{manifest file?} B -- yes --> Z[skip] B -- no --> C[rel_path = to_source_relative] C --> D{manifest present\nfor this run?} D -- no --> CONTENT D -- yes --> E[resolve_route rel_path] E -- UnmappedPath --> CONTENT[Path-A content branch] E -- AmbiguousResolution --> ERR[loud stage_error\nZERO rows] E -- route=content --> CONTENT E -- route=forms --> FORMS[Path-B form branch] CONTENT --> CW[(content_items / source_documents /\ncontent_chunks / q_a_extractions /\nentity_mentions)] FORMS --> FW[(form_templates /\nform_template_fields)] CONTENT -. exception .-> ISO[per-item catch:\nitem_failures.content++\nbatch continues] FORMS -. exception .-> ISO2[per-item catch:\nitem_failures.forms++\nbatch continues]B.8 Risks and mitigations
Section titled “B.8 Risks and mitigations”| Risk | Mitigation |
|---|---|
Manifest route typo silently routes a form to content (or vice-versa). | extra="forbid" + Literal["content","forms"] makes any non-enum value a load-time ManifestLoadError → flow aborts loudly at start (the manifest-load gate at flow.py:2206-2230). |
Suffix/route mismatch (.md under route:"forms"). | Secondary suffix guard (B.3) → loud cocoindex.stage_error, zero rows. Not silent. |
Re-tagging a prefix content→forms orphans prior-branch rows. | Operator action only; document in the on-prem runbook. Out of automated scope for {80.2}; flag as a follow-up (a sweep that detects branch-flip by storage_path). |
| Per-item swallow hides a systemic fault (every file fails the same way). | item_failures counter surfaces the tally in the flow-end webhook; a 100%-failure walk is visible as completed with item_failures == items_processed — a loud signal, not a silent green. Add a webhook-side alert threshold as a follow-up. |
| Regression: existing content-only walks change behaviour. | route defaults to "content"; no manifest → content branch. Existing flow tests (which mock mount_each and bind no forms route) exercise the content branch unchanged. |
Forms no longer get content_items rows (a behaviour change). | Intended — a blank form is not knowledge content. RATIFIED (OQ-80.2-A, Liam, S314, 05/06/2026): forms land ZERO content rows. All 11 downstream consumer groups inventoried (docs/research/s314-id80-forms-current-state.md §3) — none requires the row; every effect is neutral-to-improvement. content_items row |
Testing and validation
Section titled “Testing and validation”Tests verify real behaviour, not the patched seam (per
docs/reference/test-philosophy.md — the {80.4} audit’s root finding was
seam-patch coverage debt). Each row maps to the {80.2} acceptance criteria and the
inherited ID-52 Inv-17/Inv-19.
| Behaviour to prove | Verification |
|---|---|
| Fork routes a form to Path-B only | Real-body test: ingest a route:"forms" .docx form through _ingest_file_body; assert ft_target/ftf_target received declare_row, and ci/qa/sd/em/cc targets received zero calls (no Path-A LLM extractors invoked). |
| Fork routes content to Path-A only | Real-body test: ingest a route:"content" (or unmapped) .md; assert content targets written, ft/ftf zero calls. |
| Per-doc isolation (the {80.2} headline) | Batch test through mount_each: [form_that_raises.docx, good_content.md]; assert good_content content rows landed, flow_status == "completed", item_failures.forms == 1. Proves a form fault does NOT abort content writes — the bl-224 cascade inversion. |
| Inv-17 graceful failure modes preserved | FormExtractionError → one analysis_failed form_templates row, zero fields; UnmappedPath → content branch, no form rows; AmbiguousResolution → loud stage_error, zero rows. (Port existing {80.4}/test_cocoindex_flow_write_path.py cases onto the new branch.) |
| Inv-19 — Path-A writes never touched by form routing | Assert the content branch’s q_a_extractions declare is byte-identical to pre-refactor for a content file. |
| Suffix/route mismatch is loud | .md under route:"forms" → cocoindex.stage_error, zero rows. |
| Idempotency across both branches | Re-ingest same bytes twice down each branch; assert identical uuid5 PKs and UPSERT (no duplicate rows). |
| Manifest backward-compat | A manifest WITHOUT route parses; all prefixes resolve route="content" (default). Existing id-52 fixtures unchanged. |
| Live staging re-smoke (parent-owned) | After merge: re-walk charnwood (form) + a content doc in one walk on staging; assert charnwood writes form_templates=1 + fields, the content doc writes content rows, flow status completed, and charnwood writes no content_items row. |
Run unit/Python tests via python3 -m pytest scripts/tests/ (the Python pipeline
suite). The TS suite (bun run test) is unaffected — no TS files change.
Proposed implementation Subtasks (for the Orchestrator to decompose / append)
Section titled “Proposed implementation Subtasks (for the Orchestrator to decompose / append)”This is decomposition guidance, NOT ratified {N.5+} records — the Orchestrator
owns appending to task-list.json. Each carries the code-intelligence
tool-discipline pattern (impact-before-edit on the named Python symbol via
gitnexus_impact + grep; gitnexus_detect_changes before commit) per
.ast-dataflow/CLAUDE.md Inv 2/3. All are sibling-only (same Task ID-80).
05/06/2026 — authored. Following ratification, the six items below were turned into TM-shape Subtask records
{80.6}–{80.11}, plus{80.12}(the D6 form-versioning investigation — see the Versioning section below), atdocs/research/s314-id80-subtask-records.jsonfor the Orchestrator to append. Line refs in those records are re-pinned post-ID-61.4 (HEAD693f1565) — implementers use the record pins, not this doc’s pre-61.4 refs (see A.1).
- Manifest
routeextension — addroutetoWorkspaceMapping+Resolution/resolve_routeinworkspace_resolver.py;resolve_workspacebecomes a shim. Tests: backward-compat parse, route resolution, default. Depends on: none. - Extract
_ingest_content_branch+_ingest_form_branchfrom_ingest_file_body(no behaviour change yet — pure extraction). Depends on: none. - Wire the fork — insert the single fork predicate calling
resolve_route; forms skip the three Path-A Anthropic extractions. Depends on: 1, 2. - Per-item isolation +
item_failures— wrapbound_ingest_file; add the counter + webhook field; keep flow-scope faultsfailed. Depends on: 2. - Test sweep — fork-routing, per-doc isolation, Inv-17/Inv-19 ports, idempotency, suffix-guard, backward-compat. Depends on: 3, 4.
- Runbook note + staging re-smoke handoff (parent-owned burn). Depends on: 5.
Open questions — ALL RATIFIED (Liam, S314, 05/06/2026)
Section titled “Open questions — ALL RATIFIED (Liam, S314, 05/06/2026)”Ratification source: Liam’s verbatim feedback
(docs/research/user-feedback-ids-52-75-80.md, ID-80.2 section) as resolved
through docs/research/s314-feedback-synthesis-decision-brief.md Part 2
(decisions D3/D4/D5/D6), grounded by
docs/research/s314-id80-forms-current-state.md and
docs/research/s314-ontology-boundary.md §5.
- OQ-80.2-A (behaviour delta): Today a form ALSO lands a
content_itemsrow (+ chunks/q_a/entity_mentions). After separation a form lands form rows ONLY. Is any downstream (search index,content_item_workspaces, MCP retrieval) relying on acontent_itemsrow existing per form today?If yes, the fork should optionally still write a minimalRATIFIED (D3): forms land ZERO content rows (nosource_documents/content_itemsprovenance row for forms — confirm before {80.2} implementation.content_items/source_documents/content_chunks/q_a_extractions/entity_mentions). The minimal-provenance-row fallback is DROPPED — all 11 downstream consumer groups were inventoried and none requires the row; any downstream that wants forms gets a forms-aware surface (ID-71 scope), not acontent_itemsshim. The ontology already encodes this position (noformvalue in the closedcontent_typeCV; forms have their own Layer-5 tables + CVs). CAVEAT: only BLANK form instruments route Path-B; ANSWERED/completed forms are knowledge containers and remain Path-A — the folder contract carries the distinction; the loud suffix guard catches mis-wires. - OQ-80.2-B (fork authority):
RATIFIED (D4): the fork is the manifest per-prefix
routetag (candidate 4). A separate folder ROOT is recorded as compatible future hardening — tag-now/root-later composes (a second mount or app when forms cadence/ownership genuinely diverges). See the ratification block in B.1. - OQ-80.2-C (status semantics):
RATIFIED (D5): a walk with only per-item form failures reports
flow_status="completed"with theitem_failurestally set;"failed"is reserved for walk-wide faults. The webhookstatuscontract change is approved. See the ratification note in B.4.
Versioning (out of scope — {80.12} investigation Subtask)
Section titled “Versioning (out of scope — {80.12} investigation Subtask)”D6 (Liam, S314, 05/06/2026): instance-side form versioning is a separate
designed item, authored as a Subtask of ID-80 — investigation-first, not
necessarily executed this session. Today charnwood-v2.docx arriving beside
charnwood.docx lands a wholly unlinked sibling row (new rel_path → new
uuid5("ft:{rel_path}")); form_templates has no version / is_current /
supersession columns — only the catalogue side
(form_template_requirements.template_version + is_current) carries any
versioning substrate. {80.12} surveys the option space (supersession link table
vs is_current columns vs catalogue-style template_version), weighs the
procurement-workspaces Q&A version-on-cite precedent (PRODUCT.md B-12/B-15), and
recommends without implementing. This spec stays separation-scoped — the fork
ships without inventing versioning under time pressure.
Verification (external-symbol pins — pre-ratification, OQ-3 / Q-EX2)
Section titled “Verification (external-symbol pins — pre-ratification, OQ-3 / Q-EX2)”Checked against the installed pins on 04/06/2026:
cocoindex==1.0.3—coco.fn/coco.mount_each/coco.use_context(DB_CTX)/mount_table_target/ManagedBy.USERare the symbols this spec builds on; all are already live inflow.py(proven by the {80.1}/{80.5} staging burns, op 353fe637status=completed). No NEW cocoindex symbol is introduced by this spec — the fork is pure KH-authored Python over the existing flow. Result: PRESENT (transitively, via the green {80.1}/{80.5} live runs).pydantic==2.12.5—BaseModel/Field(default=...)/ConfigDict(extra= "forbid")/Literaldiscriminator used by therouteextension are Pydantic v2 core; already used byWorkspaceManifest(workspace_resolver.py:39,107). Result: PRESENT.- No third-party SDK shape is newly cited;
asyncpg/doclingusages are unchanged from the currentflow.py. NoABSENT/SIGNATURE_DRIFT/BEHAVIOUR_DRIFTfound → no escalation required.