Canonical Pipeline — Sequencing & Status

Canonical Pipeline — Sequencing & Status

⚠ SUPERSEDED as forward-map (S299). The live “where are we / what next” surface is now v1-completion-sequence.md. This doc is retained for history + architecture (§0–§6) + the per-stage write model; its forward-looking §3 fold-tables and status rows are point-in-time (pre-S299) — do not treat them as current.

Living reconciliation of docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md (tasks T1–T14) against the task-list, the backlog, and the live code. It is the master “where are we / what comes next” view that the PLAN → task-list transcription never fully captured.

Authored: S272, from a six-agent investigation, then adversarially verified against source and corrected (T12/T13 task labels, the embedder op, the backlog count, the form-templates spec drift). Provenance: /tmp/claude/s272-pipeline-research/.

Last refreshed: S292 — RE-BASELINE (this block is the current, verified forward map; the per-SXXX notes below are HISTORICAL). S292 was a deliberate re-baseline (Liam): map what remains to complete the canonical pipeline, each workstream tagged by execution mode for the next session. Shipped S292: ID-69 v1 wave {69.5}-{69.8} (operator-route contract test + canonical-record no-intrinsic-workspace assertions + READINESS.md; {69.9} deferred v1.1); {68.11} de-ID CODE-SCOPE (scrubbed client confidential worked-examples from lib/ai/skills/classification*.md + parameterised lib/client-config.ts defaults — the canonical pipeline prompts in scripts/cocoindex_pipeline/prompts.py were ALREADY generic, so the pipeline itself never leaked the client); ID-64 decomposed into {64.2}-{64.9}; Vercel cron review; OQ-64-1 RESOLVED.

Completion definition. The canonical pipeline is “complete” when a clean full-corpus re-ingest (ID-45/T7) runs end-to-end on the on-prem cocoindex stack into a fresh non-prod Supabase target; every stage writes correctly (incl. content_chunks, source_documents provenance, and the content_item_workspaces junction for ID-69 v1.1); the schema is migration-complete (ID-64 DDL); and the result is cut over to prod as the client handover DB. ID-46/T14 (retire the OLD scripts/kb_pipeline/) is the final cleanup.

THE RE-INGEST CRITICAL PATH (verified S292) — true blockers, in order:

1. {66.16} live-verify — observe a real pipeline_runs row from the on-prem corpus ({66.19} code fix + {66.20} webhook fix landed but the row has NOT been observed post-fix). THE single infra hard gate. [FOCUS — live; Coolify MCP + Liam confirm redeploy]
2. ID-64 DDL — {64.2} origin_kind rename · {64.3} source_document_id FK · {64.4} remaining FKs + status_reason · {64.9} primary_domain/subtopic NOT-NULL+sentinel. ({64.5} RLS is NOT a gate — role-based confirmed.) [cmux-MONITOR, after Liam resolves the retention OQs]
3. {56.11} app-chunker retire — dual-writer hazard; cocoindex must be the sole content_chunks writer at re-ingest. [cmux-MONITOR]
4. Non-prod Supabase target (fresh preview branch) wired into a Coolify cocoindex env (OQ-64-8); staging Coolify ({66.21}) is the ready vehicle. [FOCUS — Liam secrets from .env.local]
5. {64.7} retention export/re-link → {64.8} readiness surface → types regen → cutover. [FOCUS / planner]

NOT re-ingest gates (defer or parallelise — do NOT let these hold the pipeline): ID-67 namespace canonicalisation (tests-only; ruled out as the FLOW_META_CTX runtime cause; pinned cocoindex 1.0.3 boots ONE scripts. namespace — WIP branch recoverable but needs rebase + full collection-order pytest) · Cloud Run teardown {66.17}/{66.18} (cost-cleanup, gated AFTER {66.16}; Liam confirms Cloud Run was prod-readiness-only — fully tear-downable, and this also resolves the cloudrun/*phew*.yaml de-ID leak) · ID-62 {62.6}-{62.10} fixture-staging live-test harness (backlog-191; durable test tier, not a one-shot-ingest gate) · ID-42 {42.11}-{42.13} HTML-extraction retirement (pullMD already deployed+healthy on-prem at http://pullmd:3000 — nothing to deploy; retirement is dead-code cleanup) · ID-57/58 (question_matches, citations) capture NOTHING at ingestion — both are downstream of q_a_pairs, which the pipeline never writes (it writes q_a_extractions, a derived cache); so re-ingesting before they are built carries zero second-re-ingest risk. §5 backlog: no item is an ingestion-stage data gap.

HTML datapath caveat: if the re-ingest corpus includes the URL/RSS sources (Stage-1 does, per the two-stage runbook), the HTML→pullmd path ({42.10}) must be proven on the SAME live run as {66.16}. PDF/DOCX/MD are in-process (unaffected). NB the two-stage-re-ingestion runbook is a DRAFT written for the OLD pipeline + a stale project ref — reconcile to the on-prem/cocoindex topology before using it as the operator guide.

De-ID / public-flip (ID-68) — repo is NOT yet flip-safe. {68.11} code-scope landed, but remaining de-ID before {68.9} flip: phew.json + public/clients/phew/* untracking (deploy implications — Liam decision), capture_phew_domain_hook migration, the kh.phew.org.uk docs/docs-site/test sweep (~40 files), scripts/export-phew-articles.ts filename, classification-eval-gold-standard.json (still contains phew), cloudrun/*phew*.yaml (via {66.17}/{66.18}). Framing (Liam): the goal is de-identify while staying client-configurable — Phew becomes deploy-injected config, not a hardcode; the per-client branding mechanism (NEXT_PUBLIC_CLIENT_ID → lib/branding/clients/{id}.json, with a classificationDisambiguation slot) already exists (~70% there). This is NOT ID-69 scope (ID-69 = corpus ingest + workspace association; client config explicitly out of its Non-goals). {68.9} flip + {68.10} history purge are irreversible (purge rewrites every SHA, removes refs/original — no in-place undo); REQUIRE explicit Liam GO + a pre-flip mirror-clone + tag + GitHub export backup. [FOCUS — Liam GO]

Vercel crons (S292 review — Liam disabled all). Re-enable now (no ingest writes): freshness-transitions, review-cadence, coverage-alerts, content-gaps. Keep DISABLED through cutover (write content_items/feed_articles → re-ingest collision): intelligence-poll, process-queue, classification-quality, quality-score, intelligence-cleanup — resume at Post-Stage-2 per the two-stage runbook. None Cloud-Run-dependent; none need retiring (OLD-pipeline retirement is ID-46, not these app crons).

RLS reconciliation (S292) — ledger drift corrected. The {64.1} journal’s S279 “Model A (workspace_members junction + is_workspace_member())” ratification is REVERSED: Liam confirmed role-based single-tenant USING(true) is the intended model; membership NOT required; OQ-64-1 CLOSED. Evidence: procurement-workspaces/PRODUCT.md:382, reserved-workspace-seats/TECH.md:160, bl-207 fix caveat, ID-69/RESEARCH.md:155. {64.5} corrects the RESEARCH framing. True multi-tenant isolation = a separate FUTURE Task if ever needed, NOT a pre-re-ingest gate.

Next-session execution-mode tags live in docs/continuation-prompts/continuation-prompt-kh-s293-*.md.

Last refreshed: S291 — {66.19} DONE: the live-ingest blocker is FIXED — the hard technical gate on ID-62 re-ingest is cleared. A cmux sub-orchestrator (TDD, multiple executors+checkers) landed Option A: thread the flow-run context (op_id + retry/stage/ taxonomy counters + workspace manifest) onto ingest_file via functools.partial, drop the 5 bind_* ContextVar context-managers, and re-bind the ContextVars locally on the cocoindex _LoopRunner daemon thread (the boundary cocoindex 1.0.3 does not propagate user ContextVars across). A non-mocked engine-level regression test (app_main live=True against a temp dir + 1 .md + manifest) was RED-proven pre-fix; two Checkers PASS_WITH_NOTES (notes resolved); pytest 1690 passed. The whole class is fixed — workspace-manifest read, Path-B form-write, and stage_counts are all restored in live mode. {42.10} (blocked e2e ingest proof) shared this root cause and is now unblocked. {66.20} webhook RESOLVED (Liam, in-terminal): the defect was a www. mismatch — the client wired the apex kh.phew.org.uk (CNAME→Vercel, resolves) but the webhook + CLAUDE.md “Production URL” used www.kh.phew.org.uk (no DNS record); fix = drop the www (env var + CLAUDE.md updated, redeploy triggered). {66.21} DONE (staging Coolify cocoindex templates + runbook; operator UI stand-up remains operator-only). Integrated onto canonical-pipeline-setup (S291 feature branch): 66-19 6 commits e08f48ed..0a0f024e. What remains for the ID-62 gate is now live VERIFICATION, not a code blocker — assert a real pipeline_runs row lands from the on-prem corpus post-redeploy ({66.17} live-verify + {66.15} datapath monitor).

ID-69 spec chain COMPLETE + ratified (S291). {69.1} RESEARCH → {69.2} PRODUCT (13 invariants BI-1..BI-13) → {69.3} TECH (driver-agnostic junction-write contract; no DDL — schema already correct) → {69.4} PLAN (decomposed to {69.5}-{69.9}) all done. 5 OQs RATIFIED: OQ-69-1 idempotent composite-key junction contract reused unchanged across v1 operator-side + v1.1 ingest-side drivers; OQ-69-2 v1 = narrow ingest-gap core + file-source readiness gating T7, DB sources + breadth = v1.1; OQ-69-3 theme-3 core + theme-5 onboarding slice (NOT theme 7); OQ-69-4 operator-side curation v1, ingest-side skip-not-raise v1.1; OQ-69-5 YES → ID-69 added to ID-45/T7 dependencies (now [28,41,36,69]). Boundary affirmed: association rides content_item_workspaces ONLY (source_documents.workspace_id stays NULL); ontology = read-only validation layer, out of scope. v1 impl wave {69.5}-{69.9} (operator-route contract test, re-ingest upsert regression, BI-1/2/8 negative-invariant assertions, READINESS doc, optional sb()-harden) is pending, deferred behind the ID-62 live-verify.

ID-68 spec chain ratified + reversible prep DONE (S291). {68.3} TECH + {68.4} PLAN authored (PRODUCT skipped, ID-54 precedent). {68.5}-{68.8} integrated: residual non-public CSP removed, client-identity literals in scripts/ parameterised to lib/client-config.ts / CLIENT_ORGANISATION_NAME (NOT deleted), catalogue-charnwood-itt.ts one-shot seed relocated untracked, ghcr build context confirmed clean + still public. Dependabot HIGH-sev gate (OQ-68-6) GREEN — 0 open high. {68.9} repo public-flip + {68.10} git filter-repo history purge remain HELD for explicit Liam GO (irreversible/outward-facing); the {68.10} path inventory is captured. Cross-cutting de-identification beyond scripts/ (DB signup-domain hook, prod domain, cloudrun service files, phew-procurement/ workspace prefix) is logged for a coordinated follow-up Task.

ID-9 docs-site (S291). {9.24} (E2E → Playwright DOM specs) + {9.25} (dangling-refs gate met, zero-op) DONE; docs-site build empirically green (125 pages). Private-preview Vercel deploy glue (workflow + runbook) authored; deploy is private-preview only (ratified). {9.20} (Session A decommission verification) + {9.21} (CLAUDE.md docubot-ref additive half) BLOCKED — need a live 3× docubot workflow_dispatch + Liam manual review (not headless-doable).

Prior refresh (S290) — {66.16} first-ingest smoke RAN and surfaced the real datapath blocker; the ID-62 gate now depends on {66.19}, not just {66.16}. The smoke (TEST corpus staged into the on-prem cocoindex-state volume, COCOINDEX_SOURCE_PATH activated via a compose fix) deployed clean + walked the corpus, then exposed two defects: (1) live-ingest blocker — every file raised ingest_file invoked without an active FLOW_META_CTX binding; run completed with resolved_count: 0. Root cause (investigation, empirical repro): app_main binds the flow-run context (op_id + retry/stage/taxonomy counters + workspace manifest) via stdlib ContextVars, but cocoindex 1.0.3’s engine invokes ingest_file on its own daemon-thread loop and does NOT propagate user ContextVars across the component-dispatch boundary — so current_flow_meta() reads None. WHOLE CLASS affected: in live mode the workspace manifest reads None too (Path-B form-write silently skipped) and stage_counts read 0. Latent until now because COCOINDEX_SOURCE_PATH was always "" (idle). ID-67 (dual-namespace) RULED OUT as the cause (prod boots a single scripts. namespace via python -m; ID-67 is orthogonal test/import hygiene). Tracked + root-caused as {66.19} (fix = thread the flow-run context as explicit mount_each args + a non-mocked engine-level regression test; next-session cmux sub-orchestrator, TDD, multiple executors+checkers). {42.10} (blocked e2e ingest proof) shares this root cause — it never drove a successful live ingest. (2) webhook DNS — PIPELINE_RUN_WEBHOOK_URL=https://www.kh.phew.org.uk/... but the host can’t resolve the www. subdomain (apex kh.phew.org.uk resolves → Vercel); no pipeline_runs row lands regardless → {66.20} (tied to the prod-wipe / preview-branch / cutover posture). The on-prem app is wired entirely to PROD (Supabase rovrymhhffssilaftdwd), so a non-prod smoke path needs {66.21} staging Coolify (Liam-flagged). The §1/§2.2/§8 “{66.16} is the single remaining live milestone” framing is superseded — {66.16} is gated on {66.19}.

New Task ID-69 (client corpus & AI-readiness onboarding — cross-workspace canonical ingest; theme 3), opened S290; {69.1} RESEARCH done. The corpus is a Wikipedia-style canonical record layer, NOT workspace-organised (Liam, ratified): a record exists once, reused across many workspaces/activities. Schema already supports this (content_items has no workspace_id; M2M content_item_workspaces), but cocoindex never populates that junction — it resolves one workspace_id per file, Path-B-only, via the manifest. This ingest gap is ID-69’s core and gates the REAL (non-TEST) ID-45/T7 ingest (OQ-69-5: add ID-69 to T7 deps — pending Liam). Curated-clean-corpus model (content enters only when AI-ready) supersedes “point at the client’s whole doc dump”. 5 OQs pending Liam async ratification before {69.2}. ID-68: proprietary client content HEAD-removed from the tracked tree S290 (filter-repo history purge still gated).

Prior refresh (S289) — deployment-target pivot RESOLVED + two missing tasks added. The S281 “P2-vs-P4 pending” question is decided: P4 / B1 on-prem (ID-66) ratified S282 and now LIVE on IONOS via Coolify (S288) — a 4-service co-located compose (cocoindex + pullmd + playwright + trafilatura) on one host / one network, off Cloud Run, the ~$200/mo Filestore killed. This dissolves the {62.1} cocoindex→pullmd reachability defect (now a host-local http://pullmd:3000 call), so the “HTML datapath BROKEN as deployed” framing throughout §2.2 / §8 is superseded — the remaining live milestone is {66.16} (first-ingest datapath smoke: set COCOINDEX_SOURCE_PATH → ingest → assert a pipeline_runs row), which is the ID-62 re-ingest gate. ID-66 (on-prem pivot) and ID-67 (cocoindex namespace canonicalisation + _coco_api.py private-API façade, promoted from bl-185) were both absent from this doc and are now added (new §3 cross-cutting block + §8). Status corrections vs the live ledger: ID-52 done (all subtasks; {52.15} retire-analyse-route landed), ID-63 done (S282 — was “decomposed / in-flight”), ID-54 done. {56.8} chunking is BUILT (S286). Prior: S287 — §2.5 Text/chunking bullet amended ({56.17} handoff applied: {56.8} chunking BUILT post-ID-63 recovery, [GAP-CMI-004] RESOLVED-(a) keep-nullable). Prior: S284 (re-baseline) — T9–T14 promotion status corrected in §3/§5(b)/§6 (all promoted to ID-57–61/46, spec_needed); new re-ingest blocker bl-206 (ingest-root path drift: client-documentation-base/ empty vs -corpus populated, 33 files) gates T7/ID-45. Prior (S281) — HTML datapath confirmed BROKEN as deployed ({62.1} reachability, ID-62): the “cocoindex→pullmd works today” premise is now config-confirmed FALSE — both Cloud Run services are ingress=internal and the cocoindex caller has ZERO VPC egress (verified on live revision kh-cocoindex-pipeline-phew-00017-w4h), so the §2.2 HTML→markdown call likely never ran end-to-end live. Pivot space: P2 (enable Direct VPC egress, stay Cloud Run) or P4 (co-locate / on-prem, also kills the ~$200/mo Filestore); P1 (Service/route collapse) ruled out. Probe 0.3 (live internal-call confirmation) deferred to next session. Implications doc: docs/research/s281-cloud-run-vs-onprem-implications.md (decision pending). ID-62 remains the HARD live-tier re-ingest gate; impl deferred until the pivot is chosen. ID-63 (ontology/CV) decomposed into impl Subtasks {63.5}-{63.12} (W1∥{5,7,9,10} · W2{6,8,11} · W3{12}); OQ-63-9 ratified (persist BOTH primary_domain + primary_subtopic; NOT-NULL + ‘unclassified’ sentinel via ONE migration {63.11} PRE-re-ingest; sentinel/out-of-taxonomy surfaced via /review tab + dashboard {63.12} tied to Inv-7). ID-56 {56.8} chunking-stage build still PENDING (re-ingest waits on it). ID-64 (re-ingest readiness) remains the cutover gate — GATED by ID-56 + ID-62 + ID-63.

Prior refresh (S277, main@5e03319b): Stage-5 entity resolution LANDED via ID-53 (live files scripts/cocoindex_pipeline/flow.py + scripts/cocoindex_pipeline/stage_5.py; integration tier authored but not yet live — backlog-191); ID-42 pullmd provenance schema-readiness LANDED ({42.9}) but HTML-extraction retirement DEFERRED ({42.10}-{42.13}); ID-56 PLAN ratified — chunking stage reverse-split RATIFIED (cocoindex becomes sole content_chunks writer via RecursiveSplitter; impl pending); ID-46 RESEARCH ({46.1}) re-audited the collapse-list and shrank T14 scope.

Prior refresh (S276, main@a2a6cdfe): Stage-4 LANDED via ID-49.2; Stage-5 rescoped from ID-49.5 to ID-53 with full spec chain committed S274 (Option B ratified PERMANENT); ID-49 closed done S274 with 49.10 fixture-staging carrying forward; form-extraction PRODUCT/TECH/PLAN committed S274 as ID-52 (R3 Option A ratified — pipeline-owned write incl. instance row); ID-55 created S274 absorbing 5 canonical-pipeline-followup backlog items.

Status: §5(a) and §5(c) ratified S272/S273; §9 umbrella promotion complete — the T9 family now has task records (ID-57–61, all spec_needed; S284). Phase-0 work re-anchored on the S274 spec-chain landings.

---

0. Why this exists — the driver

The CocoIndex migration’s purpose is simplification. The bespoke pipeline had grown into a large codebase that two people cannot comfortably maintain. The shift moves core data functionality into CocoIndex, so the platform retains a golden-source record — structured, deduplicated, embedded data consumable by both AI agents and human users across many activities. This document keeps that goal on track by making the remaining work, its sequencing, and its dependencies explicit.

---

1. You are here (current state)

Built and CocoIndex-canonical: source walk → binary→markdown conversion → LLM extraction → Stage-4 embedding (LANDED S273 via ID-49.2) → Postgres upsert.

Stage-5 entity resolution — LANDED via ID-53:

• Stage-5 entity resolution — net-new flow-scope post-fan-out resolution stage ratified S274 (Option B, PERMANENT with cocoindex re-eval gate). Work rescoped from ID-49.5 → ID-53 with full RESEARCH/PRODUCT/TECH/PLAN committed S274 (docs/specs/id-53-stage-5-entity-resolution/); 12 impl Subtasks 53.5-53.16 across 6 waves all complete S277. Live files: scripts/cocoindex_pipeline/flow.py (entity_mentions declare_row in ingest_file) + scripts/cocoindex_pipeline/stage_5.py (_run_stage_5_resolution).
• Stable platform pattern (managed_by=USER row-only → row+update hybrid): PRODUCT.md Inv-2 ratifies the row+update hybrid as KH’s stable platform pattern; cocoindex remains row-only for ingest_file declare_row writes; the Stage-5 post-pass is a KH-owned UPDATE phase scoped to op_id (Inv-5).
• Re-evaluation gate (ratified S274, PERMANENT): re-verify cocoindex incremental cross-doc reactive write API on every cocoindex upgrade past 1.0.6; if a future release publishes proper per-row cross-doc resolution, re-evaluate the row+update hybrid Option-B architecture.

Consequence (residual risk — RESOLVED S277): the earlier “corpus written with unresolved entities” risk is closed — both vector search (Stage-4 live) and cross-document entity dedup (Stage-5 LANDED via ID-53) now run. Outstanding caveat: the cocoindex integration test tier does NOT yet run live (fixture-staging env unwired — backlog-191), so the Stage-5 live stage→poll→assert assertions remain authored-but-unverified-live until that env lands.

Boot status (S273): Cloud Run staging cocoindex GREEN (ID-49.8 + 49.9 done); ID-49 parent closed done S274 with 49.10 (fixture-staging + RFP corpus) the only carrying-forward Subtask, gating ID-53.14.

Form-question subsystem (§2.3 B/C): spec chain committed S274 as ID-52 — RESEARCH/PRODUCT/TECH/PLAN at docs/specs/id-52-form-extraction/; 12 impl Subtasks 52.5-52.16 across 6 waves. R3 ratified Option A — fully pipeline-owned write (folder→workspace convention; retires app-side analyse/route.ts via ID-52.15).

---

2. Verified current architecture

2.1 Pipeline stages (the CocoIndex flow)

Stage	Mechanism	Built?	CocoIndex-canonical?
1. Source walk	connectors.localfs.walk_dir(live, recursive=True) — built-in (flow.py:899)	✅	✅
1b. Fan-out	coco.mount_each — built-in (flow.py:920)	✅	✅
2. Binary → markdown	custom @coco.fn → Docling / pullmd-HTTP / passthrough (adapters.py:47)	✅	✅ (matches the pdf-to-markdown example)
3. LLM extraction	custom @coco.fn × Anthropic SDK + Pydantic, 3 calls (extraction.py:501)	✅	✅ (matches the multi-codebase-summarization example)
4. Embedding	LiteLLMEmbedder("text-embedding-3-large", dimensions=1024) (flow.py:623-625) → vector(1024); declare_vector_index NOT called — split-ownership (pipeline writes vector, migration owns HNSW) ratified PERMANENT S274 with cocoindex 1.0.6 upgrade-watch	✅	✅ (LiteLLMEmbedder)
5. Entity resolution	LANDED via ID-53 (S275+); flow-scope post-fan-out resolution stage per PRODUCT.md Inv-1 Option B; faiss-cpu==1.14.2 pinned per Inv-18. The managed_by=USER row-only contract deliberately broken — PERMANENT with cocoindex re-eval gate. Spec chain docs/specs/id-53-stage-5-entity-resolution/	✅	n/a (custom flow stage, not built-in op) — LANDED via ID-53
6. Postgres upsert	mount_table_target(managed_by=USER) — built-in (flow.py:877)	✅	✅

kb_pipeline/ is the OLD pipeline — zero real cross-imports with cocoindex_pipeline/ (only a docstring reference). The canonical work is cocoindex_pipeline/ only.

2.2 “Everything → markdown” (confirmed design)

All ingestable types convert to markdown at Stage 2, then a single, type-agnostic LLM extraction path consumes that markdown. Routing by extension (adapters.py:39-41,74):

• PDF / DOCX / XLSX → Docling (export_to_markdown())
• HTML / .htm → pullmd over HTTP
• .md / .markdown / .txt → passthrough
• anything else → hard ValueError. Not supported today: pptx, csv, images.

Nuance (verified correct, S272): conversion is inline during the flow (Stage-2 @coco.fn), not pre-converted markdown at rest on the localfs. The localfs holds the originals; populating it is T7 / ID-45’s job. pullmd (ID-42) is the inline HTML→markdown call (adapters.py:60), not a file-staging step. This matches the 01-vision.md:44 “external-folder canonical” intent — the client’s source files are the canonical store and KH re-ingests + derives markdown inline — so inline is the intended design, not a gap. (The “markdown sits on the localfs” recollection conflates the source (originals) with the derivation (markdown).)

pullmd provenance — schema-ready S277 ({42.9} LANDED); HTML-extraction retirement DEFERRED ({42.10}-{42.13}). The Stage-6 source_documents declare_row write now carries extraction_method (pullmd_* / docling / NULL) + pullmd_share_id, fanned out via adapters.extract_source_provenance(file) mirroring the Stage-2 suffix routing (flow.py ingest_file declare_row; commit 9c27497c). The provenance schema is therefore ready now. The RETIREMENT of the app-side HTML-extraction path ({42.10}-{42.13}) is deferred to a future session — {42.10}‘s non-mocked E2E proof needs a deployed worker with VPC access (the pullmd Service is ingress:internal), so it is not worktree-runnable. The operator-run live-proof runbook is journalled on ledger {42.10} (docs/reference/tasks/ID-42.md).

HTML datapath BROKEN as deployed — config-confirmed S281 ({62.1} reachability, ID-62). The §2.2 routing above assumes “HTML / .htm → pullmd over HTTP” works today — that premise is now FALSE in the deployed topology. Both Cloud Run services are ingress=internal and the cocoindex caller has ZERO VPC egress (verified on live revision kh-cocoindex-pipeline-phew-00017-w4h), so the inline adapters.py:60 pullmd call cannot reach the pullmd Service — the HTML→markdown datapath is broken and likely never ran end-to-end live. (PDF/DOCX/XLSX via Docling and .md/.txt passthrough are unaffected — those are in-process, not network calls.) Pivot space: P2 — enable Direct VPC egress on the cocoindex service, stay on Cloud Run; or P4 — co-locate the two services / move on-prem (also eliminates the ~$200/mo Filestore). P1 (collapse the two Services into one route) is ruled out. Probe 0.3 (live internal-call confirmation) is deferred to next session. Decision is pending in docs/research/s281-cloud-run-vs-onprem-implications.md (on-prem = a topology choice, not an enterprise gate). ID-62 is the HARD live-tier re-ingest gate; its impl is deferred until the pivot is chosen.

RESOLVED S289 — P4 / B1 on-prem chosen and LIVE (ID-66). The pivot above is decided: P4 (co-locate / on-prem) was ratified as ID-66 (B1) S282 and the pipeline is deployed + healthy on IONOS via Coolify (S288, off Cloud Run). cocoindex and pullmd now run on a single host on one compose network, so the inline adapters.py:60 HTML→markdown call is a host-local http://pullmd:3000 request — no VPC egress, no ingress:internal hop. The “BROKEN as deployed” / “P2-vs-P4 pending” framing in this section is therefore historical: the {62.1} reachability defect is dissolved by topology, not by VPC config. What remains is empirical proof, not architecture — {66.16} (set COCOINDEX_SOURCE_PATH to a corpus → ingest → assert a pipeline_runs row) is the first-ingest datapath smoke and the live ID-62 gate; as of S288 boot/idle is proven but the datapath has not yet been exercised end-to-end. ID-62 was wholesale re-spec’d to the co-located topology (S282 P4-RECONCILIATION; /stage route + multipart stageFixture landed {62.5}/{62.8}); its remaining impl ({62.6}/{62.7}/{62.9}/{62.10}) runs on the ID-66 host over localhost, and ID-62 now depends on [66].

2.3 Extraction is THREE separate concerns (resolves the earlier confusion)

Anthropic-direct extraction covers neither the cataloguing nor the deterministic form-question extraction:

• (A) Path-A Anthropic @coco.fn extraction — the pipeline’s general content stage (classify · qa_pairs · entity-mentions; claude-opus-4-6, extraction.py:63), over already-markdown text. BUILT. (Deliberately 3 calls, not one mega-prompt.)
• (B) Deterministic extractors — pdfplumber (PDF) / openpyxl (XLSX) / DOCX reader pulling question text out of blank forms (decision-graph.md:469). BUILT (code-complete; not yet live-smoked — v1-completion-sequence.md Spine S1). v1 coverage = {PDF, XLSX, DOCX} per form-extraction PRODUCT.md R1 (ratified S273); HTML deferred (no corpus example); legacy .xls manual conversion. Operates on raw formats (pre-§2.2 markdown conversion) for structural fidelity. Now scoped under ID-52.
• (C) AI-cataloguing skill — a Claude Plugin Skill authoring requirement_type + taxonomy + matching keywords with human confirmation. BUILT (Path-C catalogue write shipped {52.14}; re-run idempotency follow-up {52.22}).

The “form-review skill” = B + C — a separate, now-BUILT subsystem (B+C code-complete; Path-B form-write not yet live-smoked — Spine S1) for blank-form (Mode-3) question cataloguing, distinct from the built Path-A extraction. (NB: the two existing scripts/catalogue-*.ts are hand-written historical seed scripts, not skill output — “Do not re-run”.)

2.4 The Q&A data model (the golden-source shape)

id-131 note (S436): the “predetermined markdown at ingestion” / __qa__ markdown sidecar path below is retired — under the OKF model a Q&A pair is a record (q_a_pairs), never a markdown concept file (DR-014: forms are manual-upload app-side). Read this section as pre-OKF history. See reference/deployment-architecture.md §3 + the id-131 spec.

• origin_kind enum (05-qa-flow.md:55) — four values: extracted_from_corpus, curated_explicit, derived_from_bid_response, imported_legacy. The two cases that matter to your recollection:
  • Ingested ready-made Q&A → curated_explicit, following a canonical structure at ingestion (OQ10 RATIFIED — predetermined markdown). (yaml_frontmatter_v1 is an extractor_kind value, not an origin_kind.)
  • Platform-formed Q&A → derived_from_bid_response (UC5 promotion during usage). (Live CHECK value per migrations; a rename → derived_from_form_response is intended to match the bid→procurement terminology shift — cleanup in §10.)
• Reuse across forms (the “Wikipedia Principle — one record, many views”, 05-qa-flow.md:36): corpus-level q_a_pairs (no workspace FK), joined to forms via question_matches (T10) — one pair serves many forms — with version stability via q_a_pair_history / version-on-cite (backlog ID-74).

2.5 Why not the obvious CocoIndex ops

• Embeddings — LANDED ID-49.2 (S273). LiteLLMEmbedder("text-embedding-3-large", dimensions=1024) live (flow.py:623-625); reverses S265 OQ-B (the ModuleNotFoundError was the absent third-party litellm package, not a missing cocoindex op). Pgvector index ownership is split — pipeline writes the vector; migration owns the HNSW index; declare_vector_index() IS NOT CALLED (cocoindex 1.0.3 _VectorIndexHandler defect byte-equivalent in 1.0.4-1.0.6; ratified PERMANENT S274 with upgrade-watch). Background: docs/research/id49-2-trade-off-analysis-s274.md.
• langextract evaluated + SKIPPED S274 — wrong tool for Path B (text-only, no PDF/XLSX/DOCX parsing); wrong provider stack for Path A (Gemini-first, no Anthropic; few-shot exemplars vs pydantic.TypeAdapter). Watch-item only.
• Text/chunking — chunking stage RATIFIED (reverse-split) and BUILT (ID-56). The earlier “no chunking stage by design / document-granular only” framing is superseded. ID-56 OQ-CMI-56-1(c) reverse-split is ratified S276 and the PLAN ratified S277: cocoindex becomes the sole writer of content_chunks via a budget-driven chunking stage using cocoindex.ops.text.RecursiveSplitter (NOT cocoindex.functions.SplitRecursively, which is absent in cocoindex 1.0.3); the app-side chunker (lib/content/chunking.ts + chunk-store.ts + 8 callers) retires. Whole-document content_items remains document-granular; chunk-granular retrieval lives at content_chunks. {56.8} chunking-stage BUILT (S286 re-land, post-ID-63 recovery) — the spike ({56.5}) ratified Variant-B (2000B/200B/1000B, RecursiveSplitter, TextPosition char_offset), {56.6} landed content_chunks.op_id, and the @coco.fn chunking stage now lives inside ingest_file (scripts/cocoindex_pipeline/flow.py:1294-1341; bare RecursiveSplitter() budget-split, no language=/AST arg — original build 574d0a0f, reverted by ID-63, re-landed 3d5870ed). The build blocker is cleared; corpus re-ingest is now a downstream operational step rather than waiting on the stage. Heading-derived columns — [GAP-CMI-004] RESOLVED-(a) keep-nullable (Liam S286, OQ oq-c1a83ca097293fb6): heading_text / heading_level / heading_path / parent_chunk_id stay in the schema but are NULL/'{}' on every emitted chunk (budget-split preserves no heading boundary; deliberate OMIT at flow.py:1329-1333). Typed-consumer safety (the search_content_chunks RPC Returns nullability — TECH Migration 2 — plus the {56.10} consumer fix-up) is owned by those Subtasks, not the chunking stage. An AST heading-population route is under a NEW go/no-go spike Subtask (NOT v1.1-deferred; OQ oq-fedd4fc673da6349) — retaining the columns under (a) is its precondition. Cross-ref docs/specs/id-56-content-model-invariants/{PRODUCT,TECH,PLAN}.md + heading-cols-disposition-RESEARCH.md. (This bullet reflects the applied {56.17} handoff packet, S286.)
• Transcription: N/A.
• LLM extraction: Anthropic-direct (not litellm) is a ratified choice (commit 326b9b58, S256 Path-A amendments) — CocoIndex 1.0.3 has no ExtractByLlm (a 0.3.x symbol removed in 1.0); the canonical 1.x pattern is a custom @coco.fn. Provider-agnosticism deferred to v1.1.

---

3. PLAN ↔ ledger status (T1–T14)

T	Title	Task	Status	Key deps	Spec
T1–T6	(foundation / schema)	ID-36…ID-41	done	—	✅
—	T8 close-out residual (cocoindex follow-ups)	ID-44	done	—	✅
T7	Full-corpus ingest	ID-45	spec_needed	28, 41, 36	❌ (gated on §5)
T8	CocoIndex sidecar core	ID-28	done	—	✅ — live remainder = ID-49
T9	Per-UC write-back (UC1/4/6/3/5/8) + Diff-UI sidecar re-point	ID-59	spec_needed	T8, T6, T2 (Spike#3/#10, UC8)	concurrent-edit-intent-arbitration ✅ landed
T10	question_matches table + retrieval + scoring	ID-57	spec_needed	T2, T6	conditional
T11	Citations polymorphic enum + version-on-cite	ID-58	spec_needed	T6	—
T12	Mempalace KG integration (entity_mentions temporal + provenance; mempalace_kg_* wrappers)	ID-60	spec_needed	T2, T8 (Spike#4)	Hard-gated on ID-53 done + tier-1 observation window (PRODUCT.md Inv-19, ratified S274)
T13	Pre-launch operational pre-decisions + observability (prompt-cache spike · tracked-changes DOCX regression · per-tool MCP audit · bid→procurement rename · flip the live-tier verification to a scheduled/automatic cadence at launch — ID-62 Inv-29; the on-demand trigger is deploy/onprem/verify/live-verify.sh, the flip is the one-line uncomment at its # FLIP ON AT LAUNCH (ID-62 Inv-29) marker)	ID-61	spec_needed	T2, T8	mcp-tooling-audit (gates the MCP-audit sub-part only)
T14	CocoIndex absorption cleanup	ID-46	spec_needed	45, 28, 42	RESEARCH ✅ ({46.1}, S277)

Effort (PLAN units): T9=3, T10=1.5, T11=1, T12=2, T13=1.5, T14=1.

Headlines: T9–T13 are now promoted (T9→ID-59, T10→ID-57, T11→ID-58, T12→ID-60, T13→ID-61), all spec_needed — task records exist but no spec chain authored yet. Inter-task dependencies were never transcribed (done tasks carry dependencies: []). T7/ID-45 is blocked on §5 and on bl-206 (ingest-root path drift).

Correction note (was wrong in the first draft): T12 is Mempalace KG integration, not q_a_pair_history (that substrate is backlog ID-74; PLAN.md:454 mis-cites it as “ID-24”, which is unrelated). T13 is pre-launch operational + observability, of which the MCP tooling audit is one sub-part.

Cross-cutting — ID-50 (OPS-T1 route rollout, in_progress, own orchestrator; dep ID-32): wraps ~137 API routes with defineRoute + a generic ctx type and migrates the test call-sites. Not a pipeline T, but it touches every API route, so any pipeline task that adds/changes routes (T7 ingest triggers, T9 UC write-back endpoints, T13 MCP) must adopt the defineRoute wrapper and coordinate with ID-50’s per-route-group waves to avoid colliding on the same files. Hard sequencing constraint: ID-52.15 (retire app-side analyse/route.ts + template_analyse queue) MUST land BEFORE any ID-50 wave touching procurement/[id]/templates route group.

Cross-cutting — the deployment-target / datapath chain (ID-66 → ID-62 → ID-64), added S289. These are not T1–T14 rows but they now gate the re-ingest critical path and were absent from this doc:

• ID-66 (on-prem pivot, B1) — in_progress, LIVE. Spec chain + {66.5}–{66.10} done; the 4-service co-located compose is deployed + healthy on IONOS via Coolify (S288), off Cloud Run. Sits upstream of ID-62: it dissolves the {62.1} reachability defect by co-locating cocoindex + pullmd. Pending: {66.16} first-ingest datapath smoke (THE ID-62 gate — set COCOINDEX_SOURCE_PATH, ingest, assert a pipeline_runs row); {66.11} deploy-webhook; {66.12}–{66.15}, {66.17} (retire Cloud Run), {66.18}. Spec dir docs/specs/ID-66-onprem-pivot/.
• ID-62 (HTML datapath / fixture-staging) — in_progress, re-spec’d to B1. Now depends on [66]. /stage route + multipart stageFixture landed ({62.5}/{62.8}); {62.6}/{62.7}/{62.9}/{62.10} run on the ID-66 host over localhost, gated on {66.16}. The old in-VPC Cloud Run Job design is dissolved (see §2.2 RESOLVED S289).
• ID-67 (cocoindex namespace canonicalisation + private-API façade) — pending. Promoted from bl-185 / ID-185 (the §6 fold-target at the ID-185 row). Kills the dual-path cocoindex_pipeline.* import hazard (root cause of the ID-44.5 / ID-177 cross-file ContextKey clash) and adds a _coco_api.py private-API façade. Predecessor ID-55 {55.5}. {67.1}/{67.2} (namespace sweep, ~24 files, collection-order-sensitive full-suite pytest) are WIP on a branch (unverified); {67.3}–{67.5} pending. Tech-debt cleanup, not a re-ingest gate.

---

4. Closed product decisions — these previously gated T7 and the T9–T13 family

• (a) Q&A extraction / form structure — RATIFIED S273/S274 as ID-52.
  • Pipeline writes form structure into form_templates + form_template_fields (live tables, renamed from templates* in migration S246). The decision-graph Q3.11 bid_template_requirements route was superseded (S243) and never built.
  • R3 Option A ratified S273 — pipeline owns the WRITE end-to-end (incl. instance row via folder→workspace convention), overriding RESEARCH’s Option D recommendation. The app-side analyse/route.ts currently writes these tables; pipeline takes ownership and analyse/route.ts is retired under ID-52.15 (collapse-list R3 Option-A; sequencing constraint: 52.15 MUST land before any ID-50 wave on procurement/[id]/templates).
  • Spec chain committed S274: docs/specs/id-52-form-extraction/{RESEARCH,PRODUCT,TECH,PLAN}.md (PRODUCT 21 invariants; 12 impl Subtasks 52.5-52.16 across 6 waves).
  • Matching (T10) still reads form_template_requirements (global requirement catalogue with requirement_embedding).
• (b) Promote T9–T14. DONE (§9 for structure). All promoted: ID-52/53/54/55 (S273/S274) + T9→ID-59, T10→ID-57, T11→ID-58, T12→ID-60, T13→ID-61, T14→ID-46 — the T9–T13 family all spec_needed (verified S284 re-baseline audit; §3 table corrected).
• (c) Mode-1 vs Mode-3. Split confirmed; folded into ID-52 PRODUCT.md invariants.

---

5. Backlog to fold in (canonical-pipeline-followup + adjacent)

These remain backlog as fold-on-promote targets — distinct from the now-promoted T9–T14 (§3). (Full per-item verdict: G research doc.) NB: the S283/S284 re-baseline backlog wave (bl-197–214) is tracked in product-backlog.json, not folded here; bl-206 (ingest-root path drift) is a hard T7/ID-45 re-ingest blocker.

Process: fold each item INTO the relevant T-task when that task is promoted (or into the existing task if it already exists) — do not open a standalone task per backlog item. The “Maps to” column below is the fold target.

Backlog	Concept	Maps to
ID-74	q_a_pair_history superseded_by + source_workspace_id	feeds T11/T12 version-on-cite
ID-75	q_a_pairs (T6 WP3) integration-test hardening	T6 follow-up
ID-76	ts_rank measurement	T10
ID-158	entity-resolution stage counts	CLOSED S299 — resolved (wired stage_5.py:209)
ID-162	num_reprocesses observability	DEFERRED v2 S299 — deferral-register §2 V2-6
ID-165	Pydantic error-class persistence	T13 observability
ID-166	remove unnecessary as PipelineRunStatus cast	T8 hygiene
ID-169	runbook note: pipeline_runs.result.error_class may be stale	T13 / docs
ID-176	operation-value enum-case alignment	T8 hygiene
ID-177	cocoindex pytest cross-file ContextKey clash	CLOSED S299 — killed by ID-67
ID-185	cocoindex private-API surface + dual-path import cleanup	DONE — promoted → ID-67 (S298)
ID-136	(citation-adjacent)	T11
ID-127	entity merge/split (KG-side); see docs/specs/id-53-stage-5-entity-resolution/PRODUCT.md Inv-9 for app-side coexistence semantics	T12 (real T12 = KG)
ID-109 / ID-119	(ingest / sidecar-adjacent)	T7 + T8
ID-108	(schema-adjacent)	T2 (verify already covered)
ID-70	(tooling-adjacent)	T13

---

6. Outstanding specs

Must-land specs:

• docs/specs/mcp-tooling-audit/{PRODUCT,TECH}.md — missing, gates the T13 MCP-audit sub-part.

Already satisfied: concurrent-edit-intent-arbitration/TECH.md — LANDED (S244, 12a20963); it is T9’s required spec, so T9 has one fewer gate.

Conditional specs (write when the gated task starts): qa-promotion-ui (T9/UC5), procurement-question-matching (T10), pipeline-observability (T13).

---

7. Forward sequence (the plan)

Phase 0  S275 LANDED  (43 commits integrated on main, push a8086faf → a2a6cdfe)
  ├─ ID-49.10  fixture-staging + S266 library + RFP corpus DONE     [unblocked ID-53.14]
  ├─ ID-53 Wave A DONE — 53.5/6/7/8 (op_id migration · faiss-cpu==1.14.2
  │                      · canonicalise_entity_name · extractEntityContext port)
  ├─ ID-52 Wave-1 DONE — 52.5 CV-loader + {52.5a}/id=17 Layer-5 KG-entity schema
  │                      relaxation (OQ-52-WAVE-1-A Option 3 ratified) · 52.6/7/8 parallel.
  │                      New: 52.18 ({52.M1b}) M1b dedicated form_templates columns
  │                      (form_type FK + deadline + issuing_organisation +
  │                      evaluation_methodology) — gates Phase-5 ({52.12+}).
  ├─ ID-35 follow-ups DONE (35.27-33 + 35.39 + ledger close)
  └─ S275: ID-54 Path-A lossy fix; ID-48 wave;
                          ID-52 Wave-2+ (Phase-3 onwards gated on M1b); ID-53 Wave B+;
                          new ID-56 content-model-invariants PLAN.

Phase 1  SPEC (remaining)
  ├─ ID-45 / T7 spec (full-corpus ingest)           [gated on §5]
  └─ mcp-tooling-audit / {PRODUCT,TECH}             [gates the T13 MCP-audit sub-part]

Phase 2  BUILD  (T9/T11/T12/T13 promoted → ID-59/58/60/61, all spec_needed — §9)
  └─ T10 (question_matches; partly covered by ID-52 matching hand-off) → T11 (citations) →
     T9 (UC write-back) → T12 (Mempalace KG) → T13 (pre-launch ops)
     deps: T10←T2,T6 · T11←T6 · T9←T2,T6,T8(+spikes) · T12←T2,T8(+Spike#4) · T13←T2,T8

Phase 3  INGEST + CLEANUP
  ├─ ID-45 / T7 full-corpus ingest
  └─ ID-46 / T14 absorption cleanup                 [hard-gated on T7 + T8-stable]

ID-46 / T14 scope (S277 re-audit, {46.1} RESEARCH): ID-46’s RESEARCH re-audited the collapse-list and ID-46’s scope shrank — HTML-extraction retirement moved to ID-42, the app-side chunker to ID-56, and analyse/route.ts to ID-52 ({52.15}); ID-46’s true residual is scripts/kb_pipeline/ (the OLD Python pipeline) + its legacy CLI/dedup/ batch-reclassify residue, contingent on landing order. Cross-ref docs/specs/id-46-cocoindex-absorption-cleanup/RESEARCH.md.

Critical path to “CocoIndex is a usable golden source”: Stage-5 entity resolution LANDED via ID-53 (Wave A-F complete S277; live in flow.py + stage_5.py), joining the already-live Stage-4 embedding — so the corpus is now written with resolved entities. Caveat: the cocoindex integration test tier does NOT yet run live (fixture-staging env unwired — backlog-191), so the Stage-5 + ID-49 live stage→poll→assert bodies remain authored-but-unverified-live until that lands.

Re-ingest cutover gate — ID-64 (S281): the full-corpus re-ingest is gated by three in-flight tasks, all with real outstanding work — ID-56 ({56.8} chunking-stage build, in_progress), ID-62 (HTML-datapath reachability + live-verification infra, in_progress — broken-as-deployed per §2.2, pivot pending), and ID-63 (CV integration, spec-complete and now decomposed). ID-64’s RESEARCH ({64.1}) is done; its dependencies: ['56','62','63'] make it the cutover gate — it cannot proceed until those three land.

Update S289: of the three — ID-63 is now done (S282), ID-56 {56.8} chunking is BUILT (S286; residual = {56.11} app-chunker retire + {56.12} UI + {56.18} AST spike), and ID-62 is the sole live blocker, now unblocked at the topology level by ID-66 going LIVE (§2.2 RESOLVED) and waiting only on the {66.16} datapath smoke. So the cutover gate has narrowed from “three in-flight tasks” to “{66.16} datapath proof → ID-62 live verify”.

Cross-cutting gate — ID-50 (route rollout): independent of the pipeline stages, but any new API-route surface in T7/T9/T13 must use defineRoute and coordinate with ID-50’s waves (see the §3 callout). Treat as a sequencing constraint on those tasks, not a phase of its own.

---