Cocoindex Absorption Cleanup (ID-46 / T14) — RESEARCH

Cocoindex Absorption Cleanup (ID-46 / T14) — RESEARCH

Status: [CURRENT-CANONICAL] for ID-46. This doc re-audits the historic phase-0 findings (2026-05-06/07) against the platform’s CURRENT shape (S277, main@e344bfb1) and brings the collapse-list §5 retirement seed up to date. It does NOT author PRODUCT/TECH/PLAN.

Authored: kh-S277 main, FRESH Planner (Q-PLANNER-2). Provenance: live grep + schema verification on worktree branch (see §10 Verification).

Last verified: 28/05/2026 (S277).

CRITICAL constraint: ID-46 implementation is HARD-GATED on ID-45 (T7 full-corpus reingest) complete + stable + ID-28 stable (docs/reference/task-list.json ID-46 dependencies: [45, 28, 42]; sequencing §8 “Phase 3 INGEST + CLEANUP — ID-46/T14 absorption cleanup [hard-gated on T7 + T8-stable]”). This doc deletes nothing and applies no migration.

---

1. Why this revisit exists (Liam, S277)

When the canonical-pipeline project began, its shape was not clear until ~phase 0.9. Before that, several platform areas were audited (the phase-0-investigation docs). Those audit docs are ALL HISTORIC — they describe the OLD pipeline (the ten TS/Python ingest paths P1–P10), audited 2026-05-06/07, before cocoindex owned the canonical writes and before the key schema migrations landed. The platform has since moved on substantially:

• Cocoindex (scripts/cocoindex_pipeline/, 5,366 LOC) owns the canonical write path: source-walk → binary→markdown → Path-A LLM extraction → Stage-4 embedding → Postgres upsert (sequencing §2.1; Stage-4 LANDED S273 via ID-49.2).
• Migrations landed since the phase-0 audits: bid→procurement rename (S248, dir lib/bid-library-ingest/ → lib/procurement-library-ingest/), templates* → form_templates* rename (S246), digests → change_reports (S251), id42 pullmd provenance (20260526074944), id53 entity_mentions op_id + pair-cache (20260528122543), id52 form-extraction schema (20260528134712) + M1b dedicated form_templates columns (20260528151422), op_id propagation across the corpus (20260521203414).
• IN-FLIGHT spec chains that re-carve the retirement boundary: ID-42 (pullmd, in_progress, must-priority — owns HTML-extraction retirement + npm-dep removal), ID-52 (form-extraction — pipeline-owned form_templates write), ID-53 (Stage-5 entity resolution), ID-56 (content-model invariants reverse-split — cocoindex becomes SOLE writer of content_chunks, app-side chunker retires).

This RESEARCH re-classifies every phase-0 finding STILL-VALID / SUPERSEDED / CHANGED, surfaces NEW findings the originals missed given the new shape, inventories redundant functionality that could confuse future sessions, re-verifies the §5 retirement scope quantitatively, and proposes the collapse-list currency edits. The headline conclusion: ID-46’s scope has shrunk because ID-42 and ID-56 have absorbed large slices of the original §5 / §7 retirement set into their own spec chains.

---

2. Source corpus (read as context; current-truth checked against live code/schema)

Historic context docs (HISTORIC — describe the OLD pipeline, NOT current truth):

Doc	Date	Role
docs/themes/canonical-pipeline/intended-architecture/07-collapse-list.md §5	S274 stamp	The ID-46 retirement seed. Re-verified + currency edits proposed in §8 below.
.../phase-0-investigation/content-items-cleanup-phase-0-investigation.md	2026-05	The master phase-0 plan (10-path audit, schema-wide, knip, FK).
.../phase-0-investigation/0.2a-content-items-column-audit.md	2026-05-06	73-column content_items audit (P1–P10 writer/reader map).
.../phase-0-investigation/0.2b-side-tables-audit.md	2026-05-06	16 side-table audit.
.../phase-0-investigation/0.2.5-build-not-wired.md	2026-05-06	Build-not-wired findings (EP2 schema fields, Pass-2, cron no-op).
.../phase-0-investigation/0.2.9-fk-coverage-audit.md	2026-05-06	FK coverage (9 missing FKs).
.../phase-0-investigation/0.7.5-knip-config-validation.md	2026-05-07	Knip suppression audit (41+15 baseline; 11 DRIFT).

Current-truth sources (verified against in this doc):

• docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md (S276 refresh) — the v1 master / source-of-truth precedence.
• supabase/types/database.types.ts (live generated schema) + supabase/migrations/ (live DDL).
• Live code: scripts/kb_pipeline/, scripts/cocoindex_pipeline/, lib/extraction/, lib/intelligence/content-extractor.ts, lib/procurement-library-ingest/, dedup/reclassify scripts.
• bun run knip output + .knip-baseline.json.
• In-flight spec chains: docs/specs/id-42-pullmd-deploy/{PRODUCT,TECH,PLAN}.md, docs/specs/id-56-content-model-invariants/{PRODUCT,TECH,PLAN}.md.

---

3. Headline conclusions (read first)

1. The §5 LOC figures are still accurate as raw counts: scripts/kb_pipeline/ = 5,797 LOC across 21 .py files (verified wc -l), lib/extraction/ = 969 LOC across 12 .ts files (the “~1,000 LOC” claim holds). All four npm deps still exist (@mozilla/readability ^0.6.0, jsdom ^28.1.0, unpdf ^1.6.2, @mendable/firecrawl-js ^4.23.0).

2. But the retirement OWNERSHIP has fragmented. Three slices that the §5/§7 collapse-list assigns to “cocoindex absorption” are now owned by OTHER in-flight tasks, NOT ID-46:

   • ID-42 (pullmd, in_progress) owns lib/intelligence/content-extractor.ts HTML tiers, lib/extraction/html.ts + url.ts (Surface A + B), AND the npm-dep removal (@mozilla/readability, jsdom, @mendable/firecrawl-js) — {42.11} + {42.13} (both pending). ID-46 inherits these ONLY as residue if ID-42 lands last.
   • ID-56 (content-model invariants, PLAN ratified S276) owns the app-side chunker retirement (lib/content/chunking.ts + lib/content/chunk-store.ts + 8 regenerateChunks callers + scripts/backfill-chunks.ts) under {56.11}. This is the collapse-list §5.1 regenerateChunks() row — now ID-56’s, not ID-46’s.
   • The firecrawl / trafilatura / pdfplumber CV-lockstep prunes (§11.2/§11.3) are T7-gated and partially RSS-coupled (firecrawl) — they do not retire under cocoindex absorption alone.

3. ID-46’s true residual scope (after ID-42 + ID-56 land) is narrower: scripts/kb_pipeline/ (the OLD Python pipeline, 5,797 LOC) + its legacy CLI entrypoints + dependent backfill scripts + lib/procurement-library-ingest/ (the renamed Q&A parser dir) + dedup substrate + batch-reclassify scripts + the lib/extraction/ files NOT claimed by ID-42. The exact residual set depends on landing order (§5 coordination).

4. NEW column-drop findings (cocoindex now owns writes; see §6): the 0.2a drop-candidates (notes, parent_id, source_bid, source_document text) are ALL still in the live schema (none dropped) and are now stronger drop candidates because cocoindex demonstrably does not write them. There are also column-drop candidates that the 0.2a audit could not see because they postdate it (e.g. the legacy telemetry cluster, summary_data-vs-cocoindex, freshness_checked_at). These are largely NOT ID-46’s — they belong to ID-56 (content model) or a separate schema-cleanup task — but ID-46 RESEARCH surfaces them so the Orchestrator can route them.

5. The collapse-list §5 itself needs currency edits (§8): three stale paths/labels (lib/bid-library-ingest/extract-qa-pairs.ts → renamed; “URL-freshness cron” → it’s a route, not a cron; the regenerateChunks() row → now ID-56-owned) plus an ownership-handoff annotation to ID-42/ID-56. I have applied a bounded currency edit to §5 + the “Last verified” stamp (§8.5); the larger judgement-heavy annotations are recorded as a proposed Curator diff.

---

4. Delta table — every original finding re-classified

Legend: STILL-VALID (finding holds against current code); SUPERSEDED (no longer applies — the new shape dissolved it OR another task absorbed it); CHANGED (finding holds but the specifics moved — path, owner, or scope); NEW (a finding the originals could not see).

Recommended-disposition column key: ID-46 (residual ID-46 scope); ID-42 / ID-56 / ID-52 / ID-53 (owned by that in-flight task); SEP-MIGRATION (a separate schema-cleanup migration, NOT ID-46 pipeline retirement); DOC-ONLY (collapse-list / reference-doc currency, no code).

4.1 Collapse-list §5 pipeline-ingestion retires

#	Original finding (collapse-list §5)	Verdict	Current evidence (file:line / migration)	Recommended disposition
1	§5.1 P1 Python URL ingest retire (scripts/ingest.py)	STILL-VALID	scripts/ingest.py EXISTS, imports kb_pipeline (grep: from kb_pipeline); 615/873-LOC CLIs still live.	ID-46 (residual; gated on T7)
2	§5.1 P3 Python markdown ingest (scripts/import_bid_library.py + scripts/ingest_markdown.py)	STILL-VALID	Both EXIST (909 L / 873 L); both import kb_pipeline; import_bid_library.py:main still sole caller of dedup_across_files_by_title (GitNexus: called_by main + 2 tests).	ID-46 (residual)
3	§5.1 P4 TS URL ingest (lib/extraction/)	CHANGED → split-owned	lib/extraction/ still 969 LOC. BUT html.ts + url.ts (the HTML-tier files) are owned by ID-42 Surface B retirement (id-42/TECH.md:154). The non-HTML files (turndown.ts, clean-mdx-tags.ts, markdown-front-matter.ts, markdown-title.ts, og-metadata.ts, pdf.ts, diff-markers.ts, extraction-result.ts, content-type-detect.ts, url-validation.ts) are ID-46 residual IF still unused post-ID-42.	ID-42 (html.ts/url.ts) + ID-46 (remainder)
4	§5.1 P11 staged markdown ingest (scripts/ingest_stage2_markdown.py)	STILL-VALID	EXISTS (615 L); has def main() + argparse (line 303); imports kb_pipeline. One-shot per wave-08 D13.	ID-46 (residual)
5	§5.1 Markdown batch ingest UI	CHANGED → ID-56-adjacent	The EP2 markdown-batch path (lib/ingest/markdown-orchestrator.ts, app/api/ingest/markdown/route.ts) is still live and calls regenerateChunks (site 8 of ID-56 {56.11}). The UI replacement is the cocoindex folder-drop wrapper (ID-56 {56.12} / form-extraction OQ-52-UI-UPLOAD-TENSION precedent).	ID-56 (chunker + UI) + ID-46 (residual orchestrator/route)
6	§5.1 regenerateChunks() (built-not-wired)	SUPERSEDED → ID-56-owned	NOT built-not-wired — regenerateChunks has 8 live non-test callers (verified grep): app/api/upload/route.ts, app/api/ingest/url/route.ts, app/api/items/route.ts, app/api/items/[id]/route.ts (×2), lib/mcp/tools/governance.ts, lib/mcp/tools/content.ts, lib/ingest/markdown-orchestrator.ts. ID-56 {56.11} deletes lib/content/chunking.ts + chunk-store.ts + these 8 callers + scripts/backfill-chunks.ts; cocoindex becomes SOLE content_chunks writer via RecursiveSplitter.	ID-56 (NOT ID-46) — strike from §5.1
7	§5.2 scripts/kb_pipeline/ (~5,797 LOC) retire	STILL-VALID (count confirmed)	`find scripts/kb_pipeline -name ‘*.py’	xargs wc -l= **5,797 total** across 21 files. Zero REAL cross-imports fromcocoindex_pipeline/(only 2 docstring mentions incanonicalisation.py:8+extraction.py:64`).
8	§5.2 lib/extraction/ (~1,000 LOC) retire	CHANGED → 969 LOC + split-owned	wc -l = 969 total across 12 files (the “~1,000” claim holds within rounding). Ownership split with ID-42 per row 3.	ID-42 + ID-46 (split)
9	§5.2 lib/extraction/html.ts (Readability + JSDOM)	SUPERSEDED → ID-42-owned	html.ts (50 L) uses lazy import('jsdom') + Readability + Turndown. ID-42 {42.11} “Retire Surface A/B HTML tiers” owns this (id-42/TECH.md:108,154).	ID-42 (NOT ID-46)
10	§5.2 lib/intelligence/content-extractor.ts (Tier 2/2.5/3)	SUPERSEDED → ID-42-owned	content-extractor.ts (334 L) — Tier 2 fetch, Tier 2.5 jina, Tier 3 firecrawl, Tier 4 summary_fallback. RSS-pipeline-coupled (consumed by lib/intelligence/pipeline.ts + feed-poller.ts). ID-42 {42.11} deletes the HTML tiers (id-42/TECH.md:380-383).	ID-42 (NOT ID-46)
11	§5.2 lib/bid-library-ingest/extract-qa-pairs.ts (Pattern A/B parser)	CHANGED → renamed path	The path is STALE. Dir renamed lib/bid-library-ingest/ → lib/procurement-library-ingest/ at commit 2105404f (S248 bid→procurement). Current path: lib/procurement-library-ingest/extract-qa-pairs.ts. extractQaPairs has 0 production callers (only docx-to-markdown.ts in-dir + 2 tests) — the whole dir is migration-helper/CLI-only, confirming the §5.2 “timing: post-Phew-migration” framing.	ID-46 (residual; CORRECT THE PATH in §5.2)
12	§5.2 scripts/dedup.py::dedup_across_files_by_title	STILL-VALID	dedup.py (248 L) EXISTS; dedup_across_files_by_title defined there; GitNexus: 1 prod caller (import_bid_library.py:main, itself retiring) + 2 test callers. Matches the §5.2 GitNexus claim exactly.	ID-46 (residual)
13	§5.2 lib/dedup-normalise.ts + scripts/dedup_normalise.py	STILL-VALID	lib/dedup-normalise.ts (48 L) + scripts/dedup_normalise.py (53 L) both EXIST. Share fate with dedup_across_files_by_title.	ID-46 (residual)
14	§5.2 Custom batch-reclassify scripts	CHANGED → 3 scripts, name drift	Three TS scripts EXIST: scripts/batch-reclassify.ts, scripts/batch-reclassify-keywords.ts, scripts/reclassify-items.ts. NB: a queue HANDLER lib/queue/handlers/batch-reclassify.ts + the batch_reclassify ProcessingJobType ALSO exist (S225 migration 20260505211806) — those are queue infra, distinct from the CLI scripts; verify whether the handler retires too.	ID-46 (CLI scripts) — flag handler/job-type to OQ
15	§5.2 Custom URL-freshness governance (content_items.freshness_checked_at cron)	CHANGED → it’s a route, not a cron	freshness_checked_at is written by app/api/freshness/calculate/route.ts (a UI/API-triggered route, called from app/api/procurement/route.ts), NOT a cron. The §5.2 “cron” label is imprecise. The column + route are still live; retirement is conditional on cocoindex source-binding handling URL freshness (COCO.8) — but governance freshness (fresh/aging/stale/expired) is a SEPARATE user-facing concept that does NOT retire (sequencing C-6; collapse-list §5.2 already notes this).	SEP-MIGRATION (not core ID-46 pipeline retire) — correct label in §5.2
16	§5.3 Stream-1 items 2/4/8/9/10/12 (absorbed)	STILL-VALID (absorbed)	These dissolve into the cocoindex flow runtime; no code-level retire target — they were Stream-1 line-items. No action for ID-46 beyond confirming the flow runtime covers them post-T7.	ID-46 (verify-only)
17	§5.4 pipeline_failures table do-not-build; Q4.12 cost-tracking retire	STILL-VALID	pipeline_failures confirmed ABSENT from database.types.ts (0.2b verified absent; still absent). recordPipelineRun() rollup remains the substrate.	n/a (do-not-build)

4.2 Collapse-list §7 dependency retires + §11.2/§11.3 CV-lockstep

#	Original finding	Verdict	Current evidence	Recommended disposition
18	§7 @mozilla/readability retire	CHANGED → ID-42-owned + multi-consumer	^0.6.0. Consumers (non-test): lib/extraction/url.ts, lib/extraction/html.ts, lib/extraction/og-metadata.ts, scripts/extract-reader-html.ts, scripts/backfill-reader-html.ts, AND lib/ai/skills/inlined.generated.ts (a GENERATED skill bundle). ID-42 {42.13} removes the dep. NEW caveat: the generated-bundle reference + the 2 reader-html scripts must be cleared first.	ID-42 (dep removal) — flag reader-html scripts to ID-46 residual
19	§7 JSDOM retire	CHANGED → has a NON-extraction consumer	^28.1.0. Consumers (non-test): lib/extraction/html.ts, app/api/ingest/markdown/route.ts, scripts/extract-reader-html.ts, scripts/backfill-reader-html.ts, AND lib/ontology/loader.ts (ontology CV markdown parsing — NOT extraction, NOT retiring). NEW finding: jsdom CANNOT be dropped wholesale even after ID-42 retires the extraction HTML path — lib/ontology/loader.ts keeps it. The §7 row is over-broad.	ID-42 (extraction tier) — but dep stays (ontology); CORRECT §7
20	§7 unpdf retire (Docling primary)	CHANGED → 4 consumers incl. live TS routes	^1.6.2. Consumers (non-test): lib/extraction/url.ts, lib/extraction/pdf.ts, app/api/items/[id]/images/route.ts, app/api/upload/route.ts. NEW finding: dropping unpdf requires the upload + images routes to move off it first — Docling is a server-side Python sidecar; these are live TS routes. Not a clean cocoindex-absorption drop.	SEP (upload/images routes) + ID-46 (extraction) — needs consumer migration first
21	§7 @mendable/firecrawl-js retire	SUPERSEDED → ID-42-owned + RSS-coupled	^4.23.0. Consumers (non-test): lib/intelligence/content-extractor.ts, lib/intelligence/feed-poller.ts (RSS pollWebSource() at feed-poller.ts:336), scripts/backfill-source-url-firecrawl.ts. ID-42 {42.11} deletes the firecrawl tier in BOTH content-extractor.ts AND feed-poller.ts; {42.13} removes the dep + the workflow FIRECRAWL_API_KEY mount. The §11.2 feed_articles.extraction_method='firecrawl' CHECK value is NOT pruned (S237 lockstep, T7-gated — verified 20260526074944_id42_pullmd_provenance.sql:31-37 deliberately keeps firecrawl in the CHECK).	ID-42 (NOT ID-46) — backfill script to ID-46 residual
22	§11.3 scripts/kb_pipeline/extract.py trafilatura + pdfplumber write paths (conditional on Docling)	STILL-VALID + nuanced	extract.py (416 L) — import trafilatura (line 14), extract_with_trafilatura() (line 205), import pdfplumber (line 334). BOTH retire with kb_pipeline. NEW caveat: trafilatura ALSO survives as a pullmd Tier value — source_documents.extraction_method CHECK now admits pullmd_trafilatura (20260526074944:26), so the trafilatura concept persists in pullmd; only the kb_pipeline/extract.py Python write-path retires. pdfplumber==0.11.9 ALSO now used by scripts/cocoindex_pipeline/form_extractors/pdf.py (ID-52 form extraction) — so the pdfplumber PIP DEP does NOT retire even after kb_pipeline goes (form extraction keeps it).	ID-46 (extract.py path) — but pdfplumber dep STAYS (ID-52)

4.3 Collapse-list §11.4 + §6 cross-references

#	Original finding	Verdict	Current evidence	Recommended disposition
23	§11.4 bid_library parameter literal rename (lib/layer-inference.ts:21 + scripts/kb_pipeline/layer_inference.py)	CHANGED → kb_pipeline side retires anyway	The scripts/kb_pipeline/layer_inference.py half retires WITH kb_pipeline (ID-46). The lib/layer-inference.ts half is a separate rename (out of WP6 scope). The §11.4 rename intent is partly mooted on the Python side by ID-46 deletion.	ID-46 (Python half deletes); DOC-ONLY (TS rename separate)
24	§6 analyse/route.ts + template_analyse job-type retire	SUPERSEDED → ID-52-owned	Tracked as ID-52 {52.15} (collapse-list §6 already cites this). Not ID-46. Noted here only because it touches lib/queue/envelope.ts:44 like the batch-reclassify job-type (row 14).	ID-52 (NOT ID-46)

4.4 Phase-0 schema / build-not-wired / FK / knip findings

#	Original finding	Verdict	Current evidence	Recommended disposition
25	0.2a DROP-CANDIDATE content_items.notes	STILL-VALID	`content_items.notes: string	null STILL in live Row (database.types.ts:767`). Not dropped. 0.2a found 0 readers/writers.
26	0.2a DROP-CANDIDATE content_items.parent_id	STILL-VALID	`parent_id: string	null STILL in Row (:769`). 0.2a + 0.2.9 found 0 writers/readers, no FK, no index.
27	0.2a DROP-CANDIDATE content_items.source_bid	STILL-VALID	`source_bid: string	null STILL in Row (:783); FK to workspaces` exists (0.2.9 WIRED-CORRECTLY) but 0 readers/writers.
28	0.2a SUPERSEDED content_items.source_document (text)	STILL-VALID	`source_document: string	null STILL in Row (:784). Migrated to source_file/source_url typed columns (20260428131822`); readers still need rewrite.
29	0.2a telemetry cluster NEVER-WIRED (7 cols)	CHANGED → still NULL; cocoindex doesn’t write either	classification_model/tokens_in/out/cache_*, embedding_model/tokens STILL in Row (:735-741,752-753). The OLD paths discarded these; cocoindex Path-A extraction ALSO does not persist them today (no ai_call_log table — confirmed still absent). Decision unchanged: ship telemetry instrumentation OR accept NULL.	SEP (ai-telemetry track; NOT ID-46)
30	0.2b source_documents 0/617 rows (D2 EP2 parity)	SUPERSEDED → cocoindex owns source_documents now	The D2 “wire EP2 to write source_documents” question is mooted — cocoindex is the canonical writer; ID-42 added source_documents.pullmd_share_id + extraction_method (20260526074944). The OLD-path parity gap dissolves at T7 re-ingest.	n/a (dissolved by cocoindex)
31	0.2b NEVER-WIRED ingestion_quality_log (Python-only)	CHANGED → kb_pipeline writer retires	ingestion_quality_log written by kb_pipeline/store.py:217 (retires with ID-46). Post-T7, cocoindex’s failure-isolation replaces (Stream-1 item 4, §5.3). Reader surfaces (review queue) remain.	ID-46 (writer retires) — verify reader behaviour post-T7
32	0.2b absent tables (ai_call_log, classification_audit_log, item_files, etc.)	STILL-VALID (absent)	Re-confirmed absent from database.types.ts. Nothing to drop; only stale references in data-entry-points.md to scrub.	DOC-ONLY (reference-doc scrub)
33	0.2.9 9 missing FKs (5×P0 on content_items)	STILL-VALID	archived_by/verified_by/governance_reviewer_id/content_owner_id/source_document_id still FK-less (index-only). 0.2.9 says “land before re-ingest”. NB: these are a PRE-re-ingest gate (ID-45/T7 territory), NOT post-re-ingest cleanup (ID-46).	ID-45 (pre-re-ingest) or SEP-MIGRATION — NOT ID-46
34	0.2.5 EP2 build-not-wired (B-1..B-5: auto_supersede/tag/layer/embedding/maxDuration)	CHANGED → EP2 path retires anyway	The EP2 markdown-batch path retires (rows 5). The parsed-but-unused EP2 schema fields (auto_supersede, tag) surface in knip as BatchWideOptionsSchema/PerFileOverrideSchema (still present — see §7). They die with the path.	ID-56/ID-46 (path retire moots them)
35	0.2.5 Pass-2 entity validation dead branch; cron no-args no-op	CHANGED → kb_pipeline retires; cron orthogonal	Pass-2 (classify.ts validate flag) + Python Pass-2 (classify.py:721) — the Python half dies with kb_pipeline. The TS Pass-2 is a separate code-cleanup decision. Cron no-op (B-8) is production-readiness, orthogonal.	ID-46 (Python half) + SEP (TS Pass-2)
36	0.7.5 knip 41+15 baseline / 11 DRIFT	SUPERSEDED → baseline drifted to 105+73	The 0.7.5 baseline (41 exports + 15 types) is fully superseded: current .knip-baseline.json ceiling is exports=105, types=73, deps=3, unlisted=3 (captured 2026-05-26), bloated by UNRELATED work (ID-50 OPS-T1 codemod ResponseSchemas, ast-dataflow public surface, ledger-cli API). The canonical-pipeline-relevant DRIFT items (D-3/D-4/D-5/D-11 EP2 schemas) survive: BatchWideOptionsSchema + PerFileOverrideSchema still in knip output. They self-resolve when the EP2 path retires.	DOC-ONLY (0.7.5 obsolete) + self-resolve at path-retire

---

5. Coordination map — who owns what (landing-order-sensitive)

The §5 collapse-list assumes a single “cocoindex absorption” actor (ID-46). Reality: the retirement set is split across four tasks. ID-46’s residual scope depends on landing order.

ID-42 (pullmd, in_progress, MUST)   — owns: content-extractor.ts HTML tiers, lib/extraction/{html,url}.ts,
  {42.11} retire Surface A/B (pending)   @mozilla/readability + jsdom(extraction) + @mendable/firecrawl-js
  {42.13} remove npm deps (pending)      dep removal + FIRECRAWL_API_KEY workflow mount.
                                          firecrawl CHECK value NOT pruned (S237 lockstep, T7-gated).

ID-56 (content-model, PLAN ratified) — owns: app-side chunker retire — lib/content/chunking.ts +
  {56.11} retire chunker (Wave E)        chunk-store.ts + 8 regenerateChunks callers + backfill-chunks.ts.
  {56.12} folder-drop UI (Wave E)        cocoindex becomes SOLE content_chunks writer (RecursiveSplitter).

ID-52 (form-extraction)              — owns: analyse/route.ts + template_analyse job-type ({52.15});
                                          keeps pdfplumber + openpyxl + python-docx deps (form_extractors/).

ID-53 (Stage-5 entity resolution)    — owns: entity_mentions resolution; keeps faiss-cpu.
                                          NOT a retirement actor for ID-46.

ID-46 (this task, GATED on 45+28+42)  — residual after the above: scripts/kb_pipeline/ (5,797 LOC) +
                                          legacy CLI entrypoints (ingest.py, ingest_markdown.py,
                                          import_bid_library.py, ingest_stage2_markdown.py) +
                                          dependent backfill scripts (backfill_entities*.py,
                                          backfill_chunks_stage2.py, backfill_progressive_depth_qa.py,
                                          cleanup_section_names.py) + lib/procurement-library-ingest/ +
                                          dedup substrate (dedup.py, dedup-normalise.ts, dedup_normalise.py) +
                                          batch-reclassify CLI scripts + the lib/extraction/ files
                                          NOT claimed by ID-42 + reader-html scripts + ~30 kb_pipeline tests.

Coordination rules for the PRODUCT/TECH/PLAN waves (and for the Orchestrator):

1. ID-42 ordering. The ID-46 ledger description’s framing — “Coordinate with ID-42 (whoever lands last owns content-extractor.ts + @mendable/firecrawl-js deletion)” — is CORRECT but understates ID-42’s claim. ID-42 owns the deletion outright via {42.11}/{42.13}. ID-46 should inherit content-extractor.ts / lib/extraction/html.ts / the three HTML npm deps ONLY as residue if ID-42 has not landed when ID-46 runs. The TECH spec must check ID-42’s merge state at authoring time and scope accordingly. OQ-46-1.

2. ID-56 boundary. ID-46 must NOT touch lib/content/chunking.ts, chunk-store.ts, regenerateChunks callers, or scripts/backfill-chunks.ts — those are ID-56 {56.11}. The collapse-list §5.1 regenerateChunks() row must be struck (it is no longer build-not-wired, and it is ID-56-owned).

3. Dependency deps that DON’T retire. pdfplumber, openpyxl, python-docx are used by scripts/cocoindex_pipeline/form_extractors/ (ID-52) — they survive kb_pipeline deletion. jsdom survives (ontology loader). faiss-cpu survives (ID-53). The TECH spec must NOT pip uninstall these. Only trafilatura (the kb_pipeline/extract.py write path) and the kb_pipeline-only imports retire — and even trafilatura persists as a pullmd Tier concept.

4. Backfill-script dependency chain. kb_pipeline deletion breaks ~7 backfill scripts that import it (backfill_entities.py, backfill_entities_qa.py, backfill_chunks_stage2.py, backfill_progressive_depth_qa.py, cleanup_section_names.py, plus the legacy CLIs). These are one-shot tools whose work is subsumed by cocoindex re-ingest (T7). They retire TOGETHER with kb_pipeline. OQ-46-2: confirm none is still needed as a post-T7 remediation tool.

5. Test-file blast radius. ~30 scripts/tests/test_*.py files import kb_pipeline (mock_kb_pipeline_modules etc.). They delete with the pipeline. The pipeline-parity.test.ts guard (CLAUDE.md “Guard tests break on structural changes”) + python3 -m pytest scripts/tests/ must both stay green post-deletion — the TECH spec must enumerate which guard fixtures need updating.

---

6. NEW: additional columns to drop (cocoindex now owns writes)

The 0.2a/0.2b audits ran against the OLD pipeline. Now that cocoindex is the canonical writer and several migrations have landed, the column-drop picture is sharper. Important scope note: column drops are NOT ID-46’s job — ID-46 retires pipeline code, not schema columns. The content-model column disposition is ID-56’s domain (it owns the content_items / content_chunks READ contract), and orphaned-column drops belong in a separate schema-cleanup migration. This section surfaces them so the Orchestrator can route them; it is NOT a mandate for ID-46 to drop columns (ID-46 is gated and deletes no columns).

Format: table.column | why droppable | which consumers must change FIRST | recommended owner.

6.1 content_items orphan columns (STILL-VALID from 0.2a; now stronger)

table.column	Why droppable (current evidence)	Consumers to change first	Owner
content_items.notes	0 writers / 0 readers (0.2a); cocoindex does not write it; still in Row :767.	None (0 readers). Verify prod COUNT(*) WHERE notes IS NOT NULL ≈ 0 first.	SEP-MIGRATION
content_items.parent_id	0 writers / 0 readers / no FK / no index (0.2a + 0.2.9); cocoindex does not write it; superseded_by handles lineage. Still in Row :769.	None. Verify prod row-count ≈ 0.	SEP-MIGRATION
content_items.source_bid	0 writers / 0 readers (0.2a); workspace assoc lives on content_item_workspaces. FK to workspaces must drop first. Still in Row :783.	Drop content_items_source_bid_fkey FK first.	SEP-MIGRATION
content_items.source_document (text)	SUPERSEDED — migrated to source_file/source_url (20260428131822); ~8 component readers + lib/validation/schemas.ts:~1612 + MCP content.ts still reference it (0.2a reader-rewrite list). Still in Row :784.	Rewrite the ~8 readers to source_file ?? source_url; strip the validation/MCP input field.	SEP-MIGRATION (reader rewrite)

6.2 NEW candidates surfaced by the post-cocoindex shape (need decision, NOT ID-46)

These are findings the 0.2a audit could not see, OR that the cocoindex write-contract changes. None is an ID-46 deletion — each needs its own owner-decision. Listed for routing.

table.column	Why a candidate / what changed	Consumers / blockers	Owner / OQ
content_items.classification_model, classification_tokens_in/out, classification_cache_creation_tokens, classification_cache_read_tokens, embedding_model, embedding_tokens (the 7-col telemetry cluster)	NEVER-WIRED in OLD paths; cocoindex Path-A extraction ALSO does not persist them today (no ai_call_log). Either (a) wire telemetry in the cocoindex flow OR (b) drop the 7 columns. The reader is lib/provenance/item-provenance.ts (displays NULL).	lib/provenance/item-provenance.ts reader must change if dropped.	OQ-46-3 → ai-telemetry track / ID-56, NOT ID-46
content_items.summary_data (jsonb)	Cocoindex Path-A extraction — does it populate summary_data? If the canonical pipeline drops the executive-summary step (sequencing §2.5 “no chunking by design; extraction document-granular”), summary_data may become an OLD-path-only column. Needs verification against the cocoindex extraction contract.	app/item/[id]/page.tsx:124 parses it via SummaryDataSchema.	OQ-46-4 → ID-56 content-model contract
content_items.freshness_checked_at	Written by app/api/freshness/calculate/route.ts (route, not cron). Cocoindex source-binding handles ingest-latency freshness natively (COCO.8). The governance freshness columns (freshness, previous_freshness) STAY (C-6). freshness_checked_at may be redundant if the calculate-route retires.	app/api/freshness/calculate/route.ts + cron freshness-transitions readers.	OQ-46-5 → governance track, NOT ID-46
content_chunks.parent_chunk_id, heading_text, heading_level, heading_path	ID-56 {56.9}+{56.10} set these NULL for cocoindex-emitted chunks (RecursiveSplitter budget-split has no heading concept); [GAP-CMI-004] flags the heading-derived-column disposition as keep-nullable-legacy until {56.14}/{56.15}. Eventual drop candidates once the app-side heading chunker is gone.	ID-56 owns; search_content_chunks RPC nullability ({56.7}).	ID-56 (NOT ID-46) — cross-reference only

6.3 Side-table drop candidates (re-verified against 0.2b)

0.2b found NO side table with zero readers AND zero writers — re-confirmed: the tables it flagged (feed_flags, feed_prompts, source_document_diffs) all retain consumers. The absent-from-DB tables (item_files, item_images, content_intelligence_flags, classification_audit_log, classification_telemetry) are already non-existent — nothing to drop; only stale data-entry-points.md references to scrub (DOC-ONLY). No NEW side-table drop candidate surfaced for ID-46. source_document_diffs retirement is governed by the diff-UI decision (collapse-list §12.1 RESOLVED-S243 RETAINED-for-v1) — NOT ID-46.

---

7. Redundant-functionality inventory (could confuse future sessions)

Functionality currently present that is dead / superseded / stale and could mislead a future session into thinking it is load-bearing. ID-46 (or a coordinating task) should clear these.

Item	Location	Why redundant / confusing	Disposition
OLD Python pipeline	scripts/kb_pipeline/ (5,797 LOC, 21 files)	Zero real cross-imports from cocoindex_pipeline/ (only 2 docstrings); cocoindex owns canonical writes. A future session could mistake it for the live pipeline.	ID-46 (delete post-T7)
Legacy ingest CLIs	scripts/ingest.py, ingest_markdown.py, import_bid_library.py, ingest_stage2_markdown.py	Import kb_pipeline; superseded by cocoindex source-walk. ingest.py is already a cron no-op (0.2.5 B-8 — runs with no URLs, exits 1).	ID-46
Renamed Q&A parser dir	lib/procurement-library-ingest/ (4 files, extract-qa-pairs.ts, docx-to-markdown.ts, extract-answer.ts, resolve-question.ts)	extractQaPairs has 0 prod callers (CLI/migration-helper-only per §5.2 timing). Cocoindex ExtractByLlm-equivalent (Path-A @coco.fn) subsumes. Confusing because the dir was renamed from bid-library-ingest (stale collapse-list path).	ID-46 (post-Phew-migration)
Dedup substrate	scripts/dedup.py, lib/dedup-normalise.ts, scripts/dedup_normalise.py	Replaced by cocoindex content-hash UPSERT + S10 cross-record dedup substrate.	ID-46
Batch-reclassify CLIs	scripts/batch-reclassify.ts, batch-reclassify-keywords.ts, reclassify-items.ts	Cocoindex code-hash memoisation gives incremental re-classification for free. NB: the queue HANDLER lib/queue/handlers/batch-reclassify.ts + batch_reclassify job-type are SEPARATE — verify whether they also retire.	ID-46 (CLIs) + OQ for handler
Reader-HTML scripts	scripts/extract-reader-html.ts, scripts/backfill-reader-html.ts	Use jsdom + Readability; superseded by pullmd reader-view. Self-contained (0 external callers).	ID-42-adjacent/ID-46 residual
Firecrawl backfill	scripts/backfill-source-url-firecrawl.ts	One-shot firecrawl backfill; firecrawl retires under ID-42.	ID-42-adjacent / ID-46 residual
Knip 0.7.5 audit doc	docs/themes/canonical-pipeline/phase-0-investigation/0.7.5-knip-config-validation.md	The 41+15 baseline it proposes is obsolete (current 105+73). Could mislead a future knip-tuning session.	DOC-ONLY (mark superseded)
Stale absent-table refs	docs/reference/data-entry-points.md NOT-WRITTEN sections	Reference item_files, content_intelligence_flags, etc. that never existed — future audits re-discover them as “gaps”.	DOC-ONLY (scrub per 0.2b rec)
EP2 dead schema fields	lib/ingest/markdown-batch-schema.ts auto_supersede, tag (parsed-but-unused, 0.2.5 B-1/B-2)	Surface in knip as BatchWideOptionsSchema/PerFileOverrideSchema. Die when EP2 path retires.	ID-56/ID-46 (path retire)

---

8. Collapse-list §5 currency — applied edits + proposed Curator diff

The brief sets “keep the collapse-list current” as an explicit goal. I have applied a bounded, low-judgement currency edit to 07-collapse-list.md (§8.5 below records exactly what). The larger, judgement-heavy annotations (ownership-handoff to ID-42/ID-56, the §11.2/§11.3 re-framing) are recorded here as a proposed Curator diff rather than applied, because they restructure ratified rows and cross-reference other tasks’ specs — that is Curator territory (workflow-curator.md), and the Orchestrator should route it.

8.1 Applied edit 1 — strike the regenerateChunks() §5.1 row (ID-56-owned)

§5.1 lists regenerateChunks() as “built-not-wired … zero callers”. This is now FALSE (8 live callers) and ID-56-owned. Edit applied: annotate the row as SUPERSEDED → ID-56 {56.11}.

8.2 Applied edit 2 — correct the extract-qa-pairs.ts path (§5.2)

lib/bid-library-ingest/extract-qa-pairs.ts → lib/procurement-library-ingest/extract-qa-pairs.ts (S248 rename, commit 2105404f). Edit applied: path corrected + annotation.

8.3 Applied edit 3 — correct the “URL-freshness cron” label (§5.2)

The freshness_checked_at writer is app/api/freshness/calculate/route.ts (a route), not a cron. Edit applied: label corrected to “URL-freshness calculate route”.

8.4 Applied edit 4 — refresh the “Last verified” stamp

Added an S277 entry to the §1 header stamp recording this ID-46.1 re-audit.

8.5 What was applied vs proposed (transparency)

• APPLIED to 07-collapse-list.md: edits 8.1–8.4 above (path/label corrections + the regenerateChunks SUPERSEDED annotation + stamp). These are factual currency fixes (renamed path, mislabelled cron, a now-false build-not-wired claim) — low judgement, safe to apply.
• PROPOSED as Curator diff (NOT applied) — these restructure ratified rows / cross-reference other-task specs and should go through the Curator:
  1. §5.2 ownership-handoff annotations: mark content-extractor.ts, lib/extraction/html.ts, and the lib/extraction/ “~1,000 LOC” row as split-owned with ID-42 (ID-42 {42.11}/ {42.13} own the HTML tiers + dep removal; ID-46 inherits residue only if ID-42 lands last).
  2. §7 dep-row corrections: jsdom does NOT fully retire (lib/ontology/loader.ts keeps it); unpdf has live TS-route consumers (app/api/upload/route.ts, images/route.ts) that must migrate first; @mozilla/readability has a generated-bundle reference (lib/ai/skills/inlined.generated.ts).
  3. §11.3 nuance: trafilatura persists as a pullmd Tier (pullmd_trafilatura CHECK value); pdfplumber PIP DEP stays (ID-52 form_extractors/pdf.py). Only the kb_pipeline/extract.py write-path retires, not the deps wholesale.
  4. A NEW “§5.5 ownership map” mini-table recording the four-task split (ID-42 / ID-52 / ID-56 / ID-46) so future sessions see the boundary at a glance.

Recommendation for the Orchestrator: dispatch the §8.5 proposed-diff items to the Workflow Curator (workflow-curator.md) as a collapse-list currency pass, OR fold them into ID-46.2 PRODUCT authoring as the “scope boundary” section. They are not blocking for ID-46.1.

---

9. Open questions for Liam (decisions needed)

• OQ-46-1 — ID-42 landing-order scope handoff. ID-42 ({42.11}/{42.13}) owns the HTML-tier retirement + @mozilla/readability/jsdom(extraction)/@mendable/firecrawl-js dep removal. Should ID-46’s PRODUCT/TECH be authored assuming ID-42 has ALREADY landed (so ID-46 scopes only the residual kb_pipeline + dedup + Q&A-parser set), or should ID-46 carry a contingency branch that absorbs the HTML surface IF ID-42 slips past T7? (Recommendation: author ID-46 assuming ID-42 landed; ID-42 is must-priority + in_progress, T7 gates ID-46 anyway.)

• OQ-46-2 — backfill-script retirement. kb_pipeline deletion breaks ~7 backfill scripts (backfill_entities*.py, backfill_chunks_stage2.py, backfill_progressive_depth_qa.py, cleanup_section_names.py) + the 4 legacy CLIs. Confirm none is needed as a post-T7 remediation tool (i.e. cocoindex re-ingest fully subsumes their work). If any IS needed, it must be ported off kb_pipeline BEFORE deletion, or carved out as a keep-as-one-shot tool.

• OQ-46-3 — telemetry cluster: wire or drop? The 7 content_items AI-telemetry columns (classification_model etc.) are NULL on every path including cocoindex. Wire telemetry into the cocoindex flow (and keep the columns) OR drop the 7 columns + the lib/provenance/item-provenance.ts reader? This is NOT ID-46’s job — route to the ai-telemetry track or ID-56 — but the decision affects whether a future schema-cleanup migration drops them.

• OQ-46-4 — summary_data under cocoindex. Does the cocoindex Path-A extraction populate content_items.summary_data, or is it an OLD-path-only column now? If cocoindex does not write it, it becomes a drop candidate (consumer: app/item/[id]/page.tsx:124). Route to ID-56 (content-model contract). ID-46 RESEARCH cannot answer this without the ratified cocoindex extraction contract.

• OQ-46-5 — batch_reclassify queue handler/job-type. The CLI scripts retire under ID-46, but lib/queue/handlers/batch-reclassify.ts + the batch_reclassify ProcessingJobType (S225 migration) are queue infra. Do they retire too (cocoindex memoisation replaces re-classification entirely), or stay as a live admin-triggered re-classify path? Same question for the markdown_batch job-type once the EP2 path retires (ID-56).

• OQ-46-6 — orphan-column drops: ID-46 or separate migration? The 0.2a drop-candidates (notes, parent_id, source_bid, source_document text) are schema columns, not pipeline code. ID-46 is scoped as pipeline-code retirement and is gated. Should the column drops ride in a separate schema-cleanup migration (my recommendation — they have independent consumers + prod-row-count verification needs), or be folded into ID-46’s PLAN as a final migration wave? (Note: §6 keeps them OUT of ID-46 by default.)

---

10. Verification (OQ-3 empirical — claims grounded against live code/schema)

Date: 28/05/2026 (S277). Branch: worktree off main@e344bfb1. Method: find/wc/ grep/jq over the live worktree + database.types.ts read + migration-file read. No external third-party library API symbols were cited in this RESEARCH that require import-and-call verification (this doc surveys KH-internal code + schema + in-flight specs; cocoindex symbol verification — RecursiveSplitter PRESENT in cocoindex==1.0.3 — is owned by ID-56’s RESEARCH §5 V-2/V-11 and is cross-referenced, not re-verified here).

Claim	Verification	Result
scripts/kb_pipeline/ = 5,797 LOC / 21 files	`find … -name ‘*.py’	xargs wc -l`
lib/extraction/ = ~1,000 LOC	wc -l on 12 .ts files	CONFIRMED 969 total / 12 files
scripts/cocoindex_pipeline/ LOC	wc -l	5,366 total (the NEW canonical pipeline)
4 npm deps still exist	jq .dependencies package.json	@mozilla/readability ^0.6.0, jsdom ^28.1.0, unpdf ^1.6.2, @mendable/firecrawl-js ^4.23.0 — ALL PRESENT
cocoindex does NOT import kb_pipeline	grep -rn kb_pipeline scripts/cocoindex_pipeline	Only 2 DOCSTRING mentions (canonicalisation.py:8, extraction.py:64) — no real cross-import
kb_pipeline still imported by legacy CLIs + backfills	grep -rln 'from kb_pipeline' scripts	ingest.py, ingest_markdown.py, import_bid_library.py, ingest_stage2_markdown.py, backfill_entities*.py, backfill_chunks_stage2.py, backfill_progressive_depth_qa.py, cleanup_section_names.py + ~30 tests — CONFIRMED wide blast radius
regenerateChunks has 8 live callers (NOT build-not-wired)	grep -rln regenerateChunks lib app (non-test)	8 sites — CONFIRMED; ID-56 {56.11} owns deletion
lib/bid-library-ingest/ renamed	git log -- lib/bid-library-ingest/extract-qa-pairs.ts	Renamed to lib/procurement-library-ingest/ at commit 2105404f (S248) — CONFIRMED stale path
extract-qa-pairs.ts has 0 prod callers	grep -rln extractQaPairs	only docx-to-markdown.ts (in-dir) + 2 tests — CONFIRMED CLI-only
content-extractor.ts is RSS-coupled	grep -rln content-extractor (non-test)	consumed by lib/intelligence/pipeline.ts + feed-poller.ts — CONFIRMED RSS-only consumers; ID-42 owns retire
firecrawl consumers	grep -rln '@mendable/firecrawl-js' (non-test)	content-extractor.ts, feed-poller.ts, backfill-source-url-firecrawl.ts — CONFIRMED; ID-42 {42.11}/{42.13}
jsdom has non-extraction consumer	grep -rln 'jsdom|JSDOM' (non-test)	incl. lib/ontology/loader.ts — CONFIRMED dep cannot drop wholesale
unpdf has live TS-route consumers	grep -rln unpdf (non-test)	incl. app/api/upload/route.ts, app/api/items/[id]/images/route.ts — CONFIRMED
pdfplumber used by cocoindex form extractors	grep -rn pdfplumber scripts/cocoindex_pipeline	form_extractors/pdf.py (line 51) — CONFIRMED dep STAYS (ID-52)
drop-candidate columns still in live schema	read database.types.ts:725-792	notes:767, parent_id:769, source_bid:783, source_document:784, answer_standard:727, answer_advanced:726 — ALL STILL PRESENT (none dropped)
content_items now has op_id (id53)	grep op_id database.types.ts	`op_id: string
id52 M1b form_templates dedicated cols	read 20260528151422_*.sql	form_type, deadline, issuing_organisation, evaluation_methodology — CONFIRMED landed
id42 pullmd provenance migration	read 20260526074944_*.sql	source_documents.pullmd_share_id + extraction_method CHECK (admits pullmd_*+docling; keeps firecrawl) — CONFIRMED
knip baseline drifted from 0.7.5 (41+15)	read .knip-baseline.json + bun run knip	ceiling now exports=105/types=73/deps=3/unlisted=3 (captured 2026-05-26); current 107/75/3/3 — CONFIRMED 0.7.5 obsolete
ID-42 owns HTML retire + dep removal	read id-42/TECH.md:108,154,380-383 + ledger	{42.11} (pending) retire Surface A/B; {42.13} (pending) remove deps + FIRECRAWL mount — CONFIRMED
ID-56 owns chunker retire	read id-56/PLAN.md:10,40-43,178-185	{56.11} deletes chunking.ts + chunk-store.ts + 8 callers + backfill-chunks.ts — CONFIRMED
ID-46 ledger entry	read task-list.json ID-46	status: spec_needed, dependencies: [45,28,42], subtasks: [] — CONFIRMED (this RESEARCH = {46.1})

---

11. Recommendations for {46.2} PRODUCT

1. Scope ID-46 as the residual kb_pipeline-centred retirement, NOT the full §5/§7 set. The PRODUCT spec’s behaviour invariants should assert: cocoindex is the sole canonical writer post-T7; the OLD Python pipeline + its CLIs + dedup substrate + Q&A-parser dir + batch-reclassify CLIs are deleted with zero dead-code residue; bun run knip + python3 -m pytest scripts/tests/

   • bun run test + bun build all green post-deletion.

2. Explicitly carve OUT (cross-reference, do not duplicate): the HTML surface + 3 HTML deps (ID-42), the app-side chunker + content_chunks writes (ID-56), analyse/route.ts + template_analyse (ID-52), Stage-5 (ID-53), orphan-column drops (separate schema migration).

3. Make the gating explicit — every PRODUCT invariant is conditional on ID-45 (T7) complete + stable + ID-28 stable. No invariant should be verifiable before re-ingest.

4. Resolve OQ-46-1 (ID-42 landing order) and OQ-46-2 (backfill scripts) before TECH — they determine the file set the TECH migration plan enumerates.

5. Keep it UK English; semantic tokens / auth helper / sb()-tryQuery() bars do not apply (this is a pure-deletion pipeline-retirement task with no new UI/API surface — the quality bars that matter are: no dead-code leftover, guard-fixture updates, green test/knip/build).

---

End of {46.1} RESEARCH. Next: {46.2} PRODUCT (fresh Planner per Q-PLANNER-2), gated on Liam’s OQ-46-1..6 decisions where they affect scope.

Addendum — S277 OQ rulings + decision-graph resolution

Folds Liam’s S277 rulings (OQ-46-1/2/6, verbatim) with the decision-graph / architecture / ontology resolution of OQ-46-3/4/5 (S277 read-only pass against main@e344bfb1). Routing column key as §4. This addendum supersedes §9 OQ framing where they conflict.

Liam’s rulings (verbatim — do not re-litigate)

• OQ-46-1 (ID-42 landing order). HTML-extraction retirement + the 3 HTML npm deps are covered by ID-42 → no action for ID-46 (residual-only). Author ID-46 PRODUCT/TECH assuming ID-42 has landed; ID-46 inherits the HTML surface only as residue if ID-42 slips past T7.
• OQ-46-2 (backfill scripts + legacy CLIs). All backfill scripts (backfill_entities*.py, backfill_chunks_stage2.py, backfill_progressive_depth_qa.py, cleanup_section_names.py) + the 4 legacy CLIs are fully subsumed by cocoindex re-ingest → retire together with kb_pipeline (ID-46 residual). None retained as a post-T7 remediation tool.
• OQ-46-6 (orphan columns). notes / parent_id / source_bid / source_document (text) → separate schema-cleanup migration, NOT folded into ID-46. (Confirms §6.1 default.)

Findings — OQ-46-3/4/5 (S277 decision-graph resolution)

• OQ-46-3 (7 AI-telemetry columns: classification_model, classification_tokens_in/out, classification_cache_creation_tokens, classification_cache_read_tokens, embedding_model, embedding_tokens — database.types.ts:735-741,752-753). CONFIRMED: telemetry is NOT retained on content_items; it belongs in the cocoindex pipeline layer. 02-data-flow.md §3.3 rules Roadmap §3 + §3.7 telemetry [RATIFIED-RETIRE] (per CX.21 + COCO.8 + Q4.12); collapse-list §5.4 retires the Q4.12 cost-tracking pattern (cocoindex per-stage metrics + pipeline_runs rollup + CocoInsight supersede). Live: TS classify.ts writes none of the 7; cocoindex_pipeline/ writes none; the S204 ai-telemetry-instrumentation-plan.md was superseded by RATIFIED-RETIRE and never shipped (cost_aggregations + ai_call_log + classification_telemetry all absent from schema). Reader: lib/provenance/item-provenance.ts:128-169. Disposition (NOT ID-46): 6 token/cost columns → AI-TELEMETRY TRACK / separate schema-cleanup migration (drop candidates); classification_model + embedding_model → ID-56 (PRODUCT C-3 keeps classification_model as cocoindex-populated provenance — ID-56 rules the 2-vs-6 split). Citations: 02-data-flow.md §3.3; 07-collapse-list.md §5.4; decision-graph CX.21 + Q4.12 + Q1.2; id-56 PRODUCT.md C-3.

• OQ-46-4 (content_items.summary_data jsonb — database.types.ts:792). CORRECTION to the “layers supersede it” framing: summary_data is NOT superseded by the layers model. Per docs/ontology/03-layer-vocabulary.md §Notes (line 73, INV-1 §6.2): there are three orthogonal axes — layer is the audience axis; brief/detail/reference/summary_data are the depth axis; “Do NOT collapse these into one column.” summary_data is the structured depth-summary (executive/detailed/takeaways), distinct from the plain-text summary (ID-56 C-4). Live: written ONLY by the OLD path lib/ai/summarise.ts::generateSummary() (summarise.ts:165-172,232); cocoindex Path-A writes NEITHER summary NOR summary_data (grep empty) — so it is currently OLD-path-only under cocoindex. 0.2a (:108) flagged P3/P8/P9 never generated it. Readers: app/item/[id]/page.tsx:124, item-detail-client.tsx:48, lib/ai/change-reports.ts:309-325. Disposition (NOT ID-46) → ID-56 content-model contract: ID-56 must rule whether the cocoindex ExtractByLlm step emits a depth-summary (the docs are silent — see NEW-OQ below). If not, summary_data becomes a separate-schema-cleanup drop (4 readers rewrite first). Citations: 03-layer-vocabulary.md line 73; id-56 PRODUCT.md C-4; 0.2a-content-items-column-audit.md:108,265; 02-data-flow.md §3.1.

• OQ-46-5 (batch_reclassify queue job-type / handler). KEPT as a live admin re-classify path today; the decision-graph is SILENT on retiring the queue handler — treat as UNDECIDED, do NOT fold into ID-46. The CLI scripts (scripts/batch-reclassify*.ts, reclassify-items.ts) ARE collapse-list §5.2 [RATIFIED-RETIRE] → ID-46 residual. The QUEUE handler is separate and LIVE: JobType union has 'batch_reclassify' (envelope.ts:46), DB CHECK widened by 20260505211806_s225_*, handler lib/queue/handlers/batch-reclassify.ts dispatched at dispatch.ts:199, and 'batch_reclassify' is a canonical ingest_source (ID-56 C-5). It is an S225 admin-only build that POST-DATES the cocoindex-eval §5.10 (“memoisation eliminates custom batch-reclassify scripts” — refers to scripts, not the queue infra). decision-graph + collapse-list + id-31 PLAN are all silent on the handler/job-type. Disposition (NOT ID-46): route the queue-handler keep-or-retire to a queue/admin (or ID-56-adjacent) decision; ID-46 retires only the CLI scripts. Citations: 07-collapse-list.md §5.2; 0.8.2-cocoindex-evaluation.md §5.10:332; background-queue-infra-plan.md (job-type provenance); lib/queue/envelope.ts:46 + dispatch.ts:199 + id-56 PRODUCT.md C-5.

NEW OQs surfaced (docs silent / ambiguous — flagged, not guessed)

• NEW-OQ-A (telemetry column split). Decision-graph retires the telemetry pattern/dashboards but never explicitly says “drop these 7 columns”; ID-56 C-3 keeps classification_model as an intended cocoindex write. Confirm the 2-keep (model-name provenance, ID-56) vs 6-drop (token/cost, telemetry-track) split.
• NEW-OQ-B (cocoindex summary step). The architecture (02-data-flow.md §3.1) lists classification / Q&A / entity extraction as the cocoindex ExtractByLlm outputs — it is silent on a depth-summary step. Whether cocoindex Path-A emits summary/summary_data is the real unresolved question behind summary_data’s fate; it belongs to ID-56’s extraction contract.
• NEW-OQ-C (queue job-type retirement). decision-graph + collapse-list are silent on retiring the batch_reclassify queue handler/job-type (and, symmetrically, markdown_batch once the EP2 path retires per ID-56). Both need an explicit keep-or-retire ruling on a queue/admin track.

NEW-OQ rulings (S277 — Liam)

• NEW-OQ-A (telemetry 2-keep/6-drop split): Agreed. classification_model + embedding_model kept as cocoindex provenance (ID-56 contract); the 6 token/cost columns drop via the separate schema-cleanup migration (backlog-189).
• NEW-OQ-B (cocoindex summary step / summary_data fate): Deferred to ID-56 for a decision (content-model extraction contract). A summary_data drop, if any, rides backlog-189 once ID-56 rules.
• NEW-OQ-C (batch_reclassify queue handler/job-type): Separate investigation next session — backlog-188. Liam links it to docs/specs/queue-operational-dashboard-spec.md; scope = the user value of batch_reclassify + how the handler/job-type fit the cocoindex setup (the dashboard spec lists items that change/retire). ID-46 retires only the CLI scripts.

---

Addendum — S348 scope-refresh (2026-06-13): core retirement already landed under ID-68.23

Status: [CURRENT-CANONICAL] for ID-46 as of 13/06/2026. FRESH Planner (Q-PLANNER-2), dispatched against canonical-pipeline-setup HEAD e8e981c0a. This addendum DOES NOT delete the prior body — the S277 findings + Liam’s S277/NEW-OQ rulings above remain the decision record. It CORRECTS the single load-bearing premise the S277 body was built on: that scripts/kb_pipeline/

• its CLIs + backfills were ID-46’s residual core. That premise is now FALSE — the core was deleted under ID-68.23 (50b3b2096, 08/06/2026), three months ahead of ID-46’s gate. Where this addendum and the S277 body conflict on the current residual set, this addendum is canonical; the S277 routing rulings (carve-outs, OQ dispositions) stand unchanged.

Method: live filesystem state + git show --stat 50b3b2096 + grep/Python-importer sweeps + ast-dataflow references/importers/dead-exports (type-checker-resolved) on the worktree subo-id-46 @ e8e981c0a. Code-intel evidence cited verbatim per finding (§B/§G).

A. Headline correction — the S277 core residual is DONE

The S277 body (§3.3, §5 coordination box, §7 row 1, §11.1) framed scripts/kb_pipeline/ (then 5,797 LOC / 21 .py) + its 4 legacy CLIs + ~7 backfill scripts + ~30 py tests as ID-46’s residual core — the thing ID-46 existed to delete, hard-gated behind ID-45 (T7). That entire set was removed on 08/06/2026 under ID-68.23, commit 50b3b2096 (“chore(scripts): remove obsolete kb_pipeline + its consumers and guards”), with git show --stat confirming the full deletion:

• scripts/kb_pipeline/ — all 21 modules gone (incl. dedup.py = the dedup_across_files_by_title source, extract.py = the trafilatura/pdfplumber write-path, classify.py, chunk.py, embed.py, store.py, supersede.py, layer_inference.py, …).
• 9 manual CLI/backfill consumers (commit’s own count): ingest.py, ingest_markdown.py, ingest_stage2_markdown.py, import_bid_library.py, backfill_entities.py, backfill_entities_qa.py, backfill_chunks_stage2.py, backfill_progressive_depth_qa.py, cleanup_section_names.py — all confirmed absent (filesystem check, 13/06).
• 32 kb_pipeline-exclusive py test modules (commit’s count; S277’s “~30” estimate was close).
• 3 coupled TS guards: __tests__/validation/pipeline-parity.test.ts, __tests__/validation/content-history-change-reason.test.ts, __tests__/lib/extraction-result-parity.test.ts (+ its py twin test_extraction_result_parity.py).

The commit body records the rationale (kb_pipeline role-obsolete since the S298 Cloud Run teardown; canonical pipeline = scripts/cocoindex_pipeline/) and that pytest scripts/tests/ (1202 passed)

• bun run test + tsc + knip were all green post-removal. Liam-ratified full removal.

Net effect on ID-46: the OQ-46-2 ruling (backfills + CLIs subsumed by cocoindex re-ingest → retire with kb_pipeline) was executed early by ID-68.23 rather than by ID-46. ID-46 has collapsed from “delete the OLD Python pipeline + dependents (≈7k LOC)” to a thin standalone-script + dead-TS residual cleanup (≈3.2k LOC of standalone scripts + a handful of dead lib/ exports). The hard T7 gate that justified ID-46’s heavy framing no longer bites the core — the core is gone, and its deletion did not wait on T7 (kb_pipeline had no auto-trigger post-S298, so deletion was safe without re-ingest). The residual that remains is not T7-gated for safety (none of it is on a canonical write path); it is gated only on its own per-file reachability + the ID-42 landing order.

Recommendation (for the sub-orchestrator / Liam — NOT a decision): ID-46 as a standalone Task is now borderline. The residual is real but small, mostly mechanical pure-deletion, and splits cleanly into (i) self-contained standalone scripts with zero live callers and (ii) ID-42/ID-56 carve-out boundaries it must respect. It would be reasonable EITHER to keep ID-46 as a lean single-wave cleanup Task, OR to fold the residual into a broader “post-cocoindex dead-code sweep” alongside the ID-42/ID-56 retirement waves. See §E for the chain-depth implication. (Recorded as OQ-46-9.)

B. Refreshed residual inventory (every CURRENT candidate, re-verified 13/06)

Disposition key: RETIRE = ID-46 deletes it (0 live callers verified); RETIRE-WHEN-DEAD = ID-46 deletes it after a named upstream lands (ID-42); CARVE-OUT = owned by another task, ID-46 must NOT touch; KEEP = live, never a target. All “GONE” rows are the ID-68.23 deletion — listed only to subtract them from the S277 set. T7-gating note: none of the residual below is T7-gated (the T7 gate attached to the now-deleted core); each is gated only as noted.

Candidate (file)	LOC	Reachability verdict + code-intel evidence	Disposition	Gating
scripts/kb_pipeline/ (21 modules)	—	GONE — ls → No such file; git show --stat 50b3b2096 deletes all 21.	n/a (DONE ID-68.23)	—
4 legacy CLIs + 5 backfills + cleanup_section_names.py	—	GONE — filesystem check all 9 absent; in 50b3b2096.	n/a (DONE ID-68.23)	—
scripts/cocoindex_pipeline/ (27 .py, 11,899 LOC)	11,899	KEEP — the NEW canonical pipeline (find … -name '*.py' | wc -l = 27; wc -l = 11,899). Never a target.	KEEP	—
lib/procurement-library-ingest/extract-qa-pairs.ts	15.8 KB	DEAD (prod) — ast-dataflow references --symbol …:extractQaPairs = 17 refs, of which 1 is the in-file def (:469) and all 16 others are __tests__/** (parity.integration + extract-qa-pairs.test). 0 production callers.	RETIRE	self (dead)
lib/procurement-library-ingest/docx-to-markdown.ts	2.4 KB	DEAD (prod) — only in-repo importer is extract-qa-pairs.ts (grep docx-to-markdown → 1 hit, the Q&A parser). Retires as a pair with extract-qa-pairs.	RETIRE (with extract-qa-pairs)	self (dead)
lib/procurement-library-ingest/resolve-question.ts (resolveQuestionForRebuild)	1.2 KB	LIVE — grep: caller app/api/items/[id]/route.ts (outside dir). S277 row 11 “0 production callers / whole dir migration-helper-only” is WRONG.	KEEP	—
lib/procurement-library-ingest/extract-answer.ts (extractAnswerFromContent)	1.2 KB	LIVE — grep: callers app/api/items/batch/route.ts + lib/mcp/tools/content.ts.	KEEP	—
lib/dedup-normalise.ts (normaliseTitleForDedup)	48	DEAD (prod) — ast-dataflow importers --module @/lib/dedup-normalise = 1, and it is __tests__/lib/dedup-normalise.test.ts. 0 production importers. (NB: distinct from LIVE lib/dedup.ts / lib/entities/entity-dedup.ts — those KEEP.)	RETIRE	self (dead)
scripts/dedup.py (248) + scripts/dedup_normalise.py (53)	301	DEAD — standalone __main__ CLIs; dedup.py:21 imports dedup_normalise; no live (non-test) python importer of either (grep import dedup non-test = empty); not in package.json/ci.yml. Their prod consumer import_bid_library.py was deleted by ID-68.23. NB dedup_normalise is referenced ONLY by test_dedup_normalise.py.	RETIRE	self (dead)
scripts/batch-reclassify.ts (1320)	1320	RESIDUAL — imports { isExcludedEntity, validateDomain } from @/lib/ai/classify (legacy classify path). collapse-list §5.2 [RATIFIED-RETIRE].	RETIRE	self
scripts/batch-reclassify-keywords.ts (319)	319	RESIDUAL — imports classifyContent from @/lib/ai/classify. §5.2 RATIFIED-RETIRE.	RETIRE	self
scripts/reclassify-items.ts (193)	193	RESIDUAL — thin classifyContent({force,validate:false}) wrapper. §5.2 RATIFIED-RETIRE.	RETIRE	self
lib/ai/classify.ts (classifyContent) (1845)	1845	LIVE — classifyContent has many live callers (app/api/items/route.ts, upload/route.ts, cron/classification-quality/route.ts, items/[id]/classify/route.ts, MCP tools, markdown-orchestrator.ts, …). Tied to the UNDECIDED batch_reclassify queue handler (backlog-188).	KEEP (not a target)	—
lib/queue/handlers/batch-reclassify.ts (1125) + batch_reclassify job-type	1125	CARVE-OUT → backlog-188 (UNDECIDED) — live S225 admin path; envelope.ts:55 union, dispatch.ts registered, canonical ingest_source (ID-56 C-5). NOT the CLI scripts.	CARVE-OUT (backlog-188)	—
scripts/extract-reader-html.ts (51)	51	DEAD (standalone) — imports @mozilla/readability + jsdom; only in-repo importer is backfill-reader-html.ts; not in package.json/CI.	RETIRE (with backfill-reader-html)	self / coord §C
scripts/backfill-reader-html.ts (324)	324	DEAD (standalone) — jsdom+Readability one-shot backfill of the reader_html column; not in package.json/CI. (The reader_html COLUMN stays — read by item-detail-client.tsx, review/queue/route.ts, schemas.ts; the SCRIPTS are spent populators.)	RETIRE	self / coord §C
scripts/backfill-source-url-firecrawl.ts (302)	302	DEAD (standalone) — imports firecrawl; one-shot backfill; not in package.json/CI. ID-42 PRODUCT explicitly EXCLUDES it from the {42.13} gate and DISCLAIMS deleting it (PRODUCT.md:235 “separate clean-up, not part of the live cascade … If Liam wants it removed in ID-42, that is an OQ”).	RETIRE (ID-46 owns)	coord §C / OQ-46-7
lib/extraction/html.ts	(50)	CARVE-OUT → ID-42 {42.11} (Surface A/B HTML tiers). 0 consumers outside dir.	CARVE-OUT (ID-42)	ID-42 {42.10} prod proof
lib/extraction/url.ts	—	CARVE-OUT → ID-42 {42.12} (repoint HTML branch at pullmd; KEEP PDF branch). Consumes unpdf + og-metadata.	CARVE-OUT (ID-42)	ID-42 {42.10}
lib/extraction/og-metadata.ts	—	RETIRE-WHEN-DEAD — sole consumer is lib/extraction/url.ts:13 (extractOgMetadata); 0 external. Fate coupled to url.ts post-{42.12}; if url.ts drops the OG branch, og-metadata is dead.	RETIRE-WHEN-DEAD	after ID-42 {42.12}
lib/extraction/extraction-result.ts (PipelineExtractionResult, createPipelineExtractionResult)	—	DEAD — 0 external consumers (grep outside dir = empty); its parity guard was deleted by ID-68.23. Strong RETIRE candidate.	RETIRE-WHEN-DEAD (verify no in-dir use post-ID-42)	after ID-42
lib/extraction/turndown.ts	—	KEEP — 5+ live consumers incl. app/api/upload/route.ts + lib/content/html-to-markdown.ts (survive ID-42; only content-extractor.ts/html.ts consumers retire).	KEEP	—
lib/extraction/pdf.ts	—	KEEP — consumed by app/api/upload/route.ts; unpdf PDF path.	KEEP	—
lib/extraction/{clean-mdx-tags,markdown-front-matter,markdown-title,diff-markers,content-type-detect,url-validation}.ts	—	KEEP (1 live consumer each) — per-file grep each shows ≥1 external non-test consumer. Re-verify post-ID-42 if any consumer was an HTML-tier file.	KEEP (re-verify post-ID-42)	—
lib/ingest/markdown-orchestrator.ts {phase:'import'} leg (orchestrateMarkdownBatch)	—	CARVE-OUT boundary — callers: app/api/ingest/markdown/route.ts + lib/queue/handlers/markdown-batch.ts (queue-backed, dispatch.ts:361 case 'markdown_batch'). The chunker calls inside = ID-56 {56.11}; the markdown_batch job-type symmetry = backlog-188/NEW-OQ-C. ID-46 owns ONLY the residual route/orchestrator shell IF/when ID-56 retires the chunker + the queue track rules the job-type.	CARVE-OUT (ID-56 + backlog-188)	after ID-56
npm/pip deps pdfplumber, openpyxl, python-docx, faiss-cpu, jsdom, unpdf	—	KEEP — survive kb_pipeline deletion: form_extractors (ID-52), ID-53, lib/ontology/loader.ts, upload/images routes.	KEEP	—

C. Coordination map (refreshed, 13/06/2026)

The S277 four-task split (ID-42 / ID-52 / ID-56 / ID-53) stands. What changed: ID-46’s own column in the map has shrunk to standalone scripts + dead-TS exports (the kb_pipeline core that dominated the S277 box is gone). The sharp edge is the reader-html/firecrawl scripts.

ID-42 (pullmd; Task in_progress, subtasks {42.5–42.13} pending, gated on {42.10} prod proof)
  {42.11} retire Surface A HTML tiers (content-extractor.ts) ── owns lib/extraction/html.ts
  {42.12} retire Surface B HTML branch (url.ts; KEEP unpdf PDF) ── og-metadata coupled here
  {42.13} remove npm deps (@mendable/firecrawl-js, @mozilla/readability, jsdom; KEEP unpdf)
          + grep gate Inv-15 SCOPED TO `lib/ app/` ONLY (NOT scripts/) — see OQ-46-7.
          ID-42 PRODUCT explicitly EXCLUDES + DISCLAIMS scripts/backfill-source-url-firecrawl.ts.

ID-56 (content-model) — app-side chunker retire {56.11}; cocoindex SOLE content_chunks writer;
          owns the chunker calls inside orchestrateMarkdownBatch + the markdown_batch UI fate.

ID-52 {52.15} — analyse/route.ts + template_analyse job-type; KEEPS pdfplumber/openpyxl/python-docx.
ID-53 — Stage-5 entity resolution; KEEPS faiss-cpu.

backlog-188 (CONFIRMED, status needs_research) — batch_reclassify QUEUE handler/job-type (UNDECIDED)
          + markdown_batch job-type symmetry. ID-46 retires only the CLI scripts.
backlog-189 (REFERENCED by S277/NEW-OQ-A as the home for the 6 telemetry-column drops + summary_data
          drop) — `ledger-cli show backlog 189` → record-not-found. STALE reference. See OQ-46-8.

ID-46 (this task; ledger deps [45,28,42]) — RESIDUAL after the above:
  RETIRE (0-caller, verified): extract-qa-pairs.ts + docx-to-markdown.ts (pair), lib/dedup-normalise.ts,
    scripts/dedup.py + dedup_normalise.py, 3 batch-reclassify CLI scripts,
    extract-reader-html.ts + backfill-reader-html.ts (pair), backfill-source-url-firecrawl.ts.
  RETIRE-WHEN-DEAD (after ID-42): lib/extraction/{og-metadata, extraction-result}.ts.
  CARVE-OUT (do NOT touch): lib/extraction/{html,url}.ts (ID-42); chunker + markdown_batch (ID-56/188);
    batch_reclassify queue handler (188); classify.ts (live); orphan/telemetry columns (schema migration).

Sharp ordering note (OQ-46-7) — corrected from the dispatch brief’s premise. The dispatch brief stated ID-42 {42.13}‘s zero-grep gate “will FAIL if these scripts survive”. Verified FALSE as the gate is currently authored: ID-42 TECH Inv-15 + PRODUCT acceptance scope the grep to lib/ app/ only (grep -rIn -E 'firecrawl|FIRECRAWL|jina\.ai|extractMainContentHtml' lib/ app/ --include='*.ts') and PRODUCT.md explicitly notes “scripts/ is intentionally NOT in the gate’s path set”. So the 3 scripts in scripts/ do not trip ID-42’s gate. The real conflict is the inverse: ID-42 has disclaimed deleting backfill-source-url-firecrawl.ts (and never names the reader-html pair at all), leaving an ownership gap — nobody currently owns deleting the 3 scripts. ID-46 is the natural owner. The residual risk is only that ID-42 might opportunistically absorb backfill-source-url-firecrawl.ts via its own flagged OQ; ID-46 TECH should check ID-42’s merge state and de-dupe. (This is an execution/ordering question, not a spec blocker.)

D. New Open Questions (flag, don’t guess — none blocks spec authoring)

• OQ-46-7 — reader-html/firecrawl scripts ownership (refined). The 3 standalone scripts (extract-reader-html.ts, backfill-reader-html.ts, backfill-source-url-firecrawl.ts) are dead one-shot tools. ID-42 {42.13}‘s grep gate is lib/ app/-scoped and does NOT catch them, and ID-42 PRODUCT explicitly disclaims deleting backfill-source-url-firecrawl.ts. Two options (both documented; pick at execution): (a) ID-46 owns all 3 deletions as residual (recommended — they are pure dead scripts, no live cascade) and ID-46 TECH checks ID-42 merge state to avoid double-delete; OR (b) ID-42 absorbs backfill-source-url-firecrawl.ts via its own OQ and ID-46 owns only the reader-html pair. Either way the reader-html pair is unambiguously ID-46’s (ID-42 never names it). Not a blocker — TECH enumerates the final set at authoring time.

• OQ-46-8 — backlog-189 missing (ledger discrepancy). S277/NEW-OQ-A routes the 6 telemetry token/cost column drops (and a possible summary_data drop, NEW-OQ-B) to backlog-189, but bun scripts/ledger-cli.ts show backlog 189 → {"ok":false,"error":"record-not-found"}. The reference is STALE. Two options: (a) re-create the backlog-189 carve-out target (the separate schema-cleanup migration for the 6 telemetry columns + orphan columns notes/parent_id/source_bid/source_document) so the S277 routing has a home; OR (b) correct the S277/NEW-OQ-A references to point at whatever record now owns the schema-cleanup migration. This is a ledger-curation question for the Workflow Curator, surfaced for routing — it does NOT block ID-46 (ID-46 retires pipeline CODE, not columns; the column drops were always carved OUT of ID-46 per OQ-46-6).

• OQ-46-9 (strategic, optional) — is a standalone ID-46 still warranted? With the core deleted by ID-68.23, ID-46’s residual is ≈3.2k LOC of standalone scripts + a few dead lib/ exports, all mechanical pure-deletion, none on a canonical write path, none T7-gated for safety. Options: (a) keep ID-46 as a lean single-wave cleanup Task; (b) fold the residual into a broader post-cocoindex dead-code sweep alongside the ID-42/ID-56 retirement waves; (c) demote to a backlog cleanup item. Recommendation in §A/§E: keep it lean OR fold — the sub-orchestrator/Liam decides.

E. Chain-depth recommendation

Given the collapse, the Full chain (RESEARCH → PRODUCT → TECH → PLAN) the S277 framing implied is now over-spec’d. The residual is a bounded pure-deletion set with verified 0-caller evidence (§B) and clear carve-out boundaries (§C). Recommended tier: PRODUCT + combined TECH/PLAN (skip a separate {46.4} PLAN) —

• {46.2} PRODUCT is still warranted: it must assert the behaviour invariants (cocoindex remains sole canonical writer; the named residual scripts/exports are deleted with zero dead-code residue; bun run knip + python3 -m pytest scripts/tests/ + bun run test + bun build all green post-deletion) and pin the carve-out boundaries (ID-42/ID-56/backlog-188/189) so the Executor never strays into another task’s file set.
• {46.3} TECH can carry the decomposition inline (the deletion set is one wave; sibling-only deps are trivial — the RETIRE-WHEN-DEAD lib/extraction/* rows simply depend on ID-42 having merged, which is a cross-Task gate recorded as a BLOCKED-ON note, not a sibling dep). A separate {46.4} PLAN would be ceremony — the residual is well under the >2h / compound-invariant / multi-migration threshold that triggers a standalone PLAN.

If the sub-orchestrator instead folds ID-46 into a broader sweep (OQ-46-9 option b), the residual set in §B drops straight in as a labelled deletion slice with no further spec authoring needed.

F. Refreshed recommendations for {46.2} PRODUCT (supersedes S277 §11 where it cites the now-done core)

1. Scope ID-46 as the thin standalone-residual cleanup, NOT the kb_pipeline core (DONE, ID-68.23). PRODUCT behaviour invariants should assert deletion of the verified-dead set: the extract-qa-pairs.ts+docx-to-markdown.ts pair, lib/dedup-normalise.ts, scripts/dedup.py+ scripts/dedup_normalise.py, the 3 batch-reclassify CLI scripts, and the reader-html/firecrawl script set (per OQ-46-7) — each with a “0 production callers” acceptance check the Checker can run (ast-dataflow references/importers, grep for python/CLI).
2. Carry the RETIRE-WHEN-DEAD lib/extraction/{og-metadata,extraction-result}.ts as ID-42-gated rows — delete only after ID-42 {42.11}/{42.12} merge and a fresh 0-caller check confirms them dead (record as BLOCKED-ON ID-42, not a behaviour invariant verifiable before ID-42 lands).
3. KEEP, explicitly named so the Executor does not over-delete: lib/ai/classify.ts (classifyContent is live), resolve-question.ts + extract-answer.ts (live callers), lib/extraction/{turndown,pdf,clean-mdx-tags,markdown-front-matter,markdown-title,diff-markers, content-type-detect,url-validation}.ts, scripts/extract_docx_tables.py (consumed by cocoindex form_extractors/docx.py), and the unpdf/jsdom/pdfplumber/faiss-cpu deps.
4. Carve OUT (cross-reference, do not duplicate): lib/extraction/{html,url}.ts + 3 HTML deps (ID-42); chunker + content_chunks writes + markdown_batch UI (ID-56); batch_reclassify queue handler/job-type (backlog-188); analyse/route.ts + template_analyse (ID-52); Stage-5 (ID-53); orphan + 6 telemetry columns + summary_data (a separate schema-cleanup migration — the backlog-189 reference is stale, OQ-46-8).
5. Gating correction: the S277 “every invariant conditional on ID-45 (T7)” framing is no longer right for the residual — the T7 gate attached to the now-deleted core. The residual is gated on (a) its own per-file 0-caller verification and (b) ID-42 merge for the two RETIRE-WHEN-DEAD lib/extraction/* rows. PRODUCT should state this corrected gating, not blanket-T7.
6. UK English; semantic-token/auth-helper/sb()-tryQuery() bars do NOT apply (pure-deletion, no new UI/API surface). The quality bars that matter: zero dead-code residue, green knip/pytest/bun run test/bun build, and not straying across the §C/§4 carve-out boundaries.

G. Verification (S348 — claims grounded against live code @ e8e981c0a)

Date: 13/06/2026 (S348). Branch: worktree subo-id-46 @ e8e981c0a (canonical-pipeline-setup HEAD). Method: git show --stat + filesystem ls/wc + grep (python/SQL/string-literal) + ast-dataflow references/importers (type-checker-resolved). No external-library API symbols newly cited that require import-and-call verification (this addendum surveys KH-internal code + the ID-68.23 deletion commit + in-flight specs; cocoindex symbol currency is owned by ID-56/ID-49 RESEARCH, cross-referenced not re-verified).

Claim	Verification	Result
kb_pipeline + 9 consumers + 32 py tests + 3 TS guards removed	git show --stat 50b3b2096	CONFIRMED — 21 modules + ingest*.py/import_bid_library.py/backfill_*.py/cleanup_section_names.py + tests + 3 guards, all in the diffstat
scripts/kb_pipeline/ gone	ls scripts/kb_pipeline	CONFIRMED “No such file or directory”
9 legacy CLIs/backfills gone	filesystem loop	CONFIRMED all 9 → GONE
scripts/cocoindex_pipeline/ = NEW canonical	find -name '*.py' | wc -l; wc -l	27 files / 11,899 LOC — KEEP
extract-qa-pairs.ts::extractQaPairs prod-dead	ast-dataflow references --symbol …:extractQaPairs	CONFIRMED 17 refs = 1 in-file def (:469) + 16 __tests__/**; 0 prod callers
docx-to-markdown.ts coupled to extract-qa-pairs	grep docx-to-markdown (non-test)	CONFIRMED sole in-repo importer = extract-qa-pairs.ts
resolve-question.ts LIVE (S277 row 11 WRONG)	grep resolveQuestionForRebuild	CONFIRMED caller app/api/items/[id]/route.ts
extract-answer.ts LIVE	grep extractAnswerFromContent	CONFIRMED callers app/api/items/batch/route.ts + lib/mcp/tools/content.ts
lib/dedup-normalise.ts prod-dead	ast-dataflow importers --module @/lib/dedup-normalise	CONFIRMED 1 importer = __tests__/lib/dedup-normalise.test.ts; 0 prod
scripts/dedup.py/dedup_normalise.py dead	grep python importers (non-test)	CONFIRMED only dedup.py:21 self-imports dedup_normalise; no live consumer; not in package.json/CI
batch-reclassify CLIs import legacy classify	grep imports	CONFIRMED batch-reclassify.ts (isExcludedEntity,validateDomain), *-keywords.ts + reclassify-items.ts (classifyContent)
lib/ai/classify.ts LIVE (NOT a target)	grep classifyContent callers	CONFIRMED many live routes/MCP/orchestrator
reader-html/firecrawl scripts standalone	grep package.json/.github/workflows	CONFIRMED zero CI/package refs; backfill-reader-html.ts imports extract-reader-html.ts
ID-42 {42.13} grep gate scope = lib/ app/ (NOT scripts/)	read id-42/TECH.md:450 + PRODUCT.md:247-260	CONFIRMED — gate lib/ app/-scoped; PRODUCT explicitly excludes + disclaims backfill-source-url-firecrawl.ts
lib/extraction/ per-file reachability	grep external consumers per file	og-metadata→url.ts only; extraction-result→0; html→0(outside dir); turndown→5+ (KEEP); pdf→upload (KEEP); rest ≥1 (KEEP)
unpdf has live TS-route consumers	grep unpdf (non-test)	CONFIRMED app/api/upload/route.ts, app/api/items/[id]/images/route.ts — dep STAYS
extract_docx_tables.py LIVE (cocoindex) CORRECTED → RETIRE (dead standalone) — see §H	grep importers	CONFIRMED imported by form_extractors/docx.py WRONG. Re-verified S350 @ e8e981c0a: form_extractors/docx.py:84,89,90 imports analyse_template/docx_utils/extract_tender_questions — NOT extract_docx_tables. The sole non-self/non-test ref to extract_docx_tables is a docstring at extract_tender_questions.py:3 (“Adapted from…”), not an import; extract_docx_tables.py is a standalone __main__ CLI (:500). 0 live importers → RETIRE (ID-46 {46.10}); extract_tender_questions.py + docx_utils.py are the LIVE files (KEEP).
orchestrateMarkdownBatch queue-backed	grep callers + dispatch.ts	CONFIRMED callers app/api/ingest/markdown/route.ts + lib/queue/handlers/markdown-batch.ts; dispatch.ts:361 case 'markdown_batch'
content-extractor.ts RSS-coupled (ID-42)	grep consumers (non-test)	CONFIRMED sole consumer lib/intelligence/pipeline.ts
backlog-188 EXISTS (batch_reclassify queue)	ledger-cli show backlog 188	CONFIRMED status needs_research, details confirm CLI-vs-handler boundary + markdown_batch symmetry
backlog-189 MISSING (stale S277 ref)	ledger-cli show backlog 189	CONFIRMED record-not-found — OQ-46-8

End of S348 addendum. The S277 body above remains the decision record for routing/carve-outs; this addendum supersedes it on the current residual set and corrects the kb_pipeline-core premise.

H. S350 correction — extract_docx_tables.py reclassified KEEP→RETIRE

Status: [CURRENT-CANONICAL] correction to §B / §G. Surfaced during {46.3} TECH authoring and confirmed on the {46.3} Checker F-2 finding (S350 fix-Planner pass, worktree subo-id-46 @ e8e981c0a). Supersedes the §B/§G “extract_docx_tables.py LIVE/KEEP” rows.

The S348 §B (KEEP row implicit in the lib/extraction / cocoindex-dep set) and §G verification table wrongly classified scripts/extract_docx_tables.py as LIVE/KEEP on the premise that cocoindex form_extractors/docx.py imported it. That premise is FALSE. Re-verified @ e8e981c0a:

• grep -rn "extract_docx_tables" scripts --include='*.py' (non-self, non-test) → single hit scripts/extract_tender_questions.py:3, a docstring (“Adapted from extract_docx_tables.py — reads tables from .docx files…”), NOT an import.
• scripts/extract_docx_tables.py is a standalone CLI (if __name__ == "__main__" at :500, calling extract_qa_from_docx defined :424; imports docx_utils at :27).
• cocoindex form_extractors/docx.py imports scripts.analyse_template (:84), scripts.docx_utils (:89), scripts.extract_tender_questions (:90) — none is extract_docx_tables.
• The only .github/workflows/ci.yml:177 reference is a comment naming the TEST file test_extract_docx_tables.py as a pandoc-quality-shard member — not a live source invocation.

Corrected disposition: extract_docx_tables.py is the dead OLD precursor to the LIVE extract_tender_questions.py, superseded by cocoindex form extraction — 0 live importers, test-only consumer → RETIRE (ID-46 {46.10} / TECH slice S-f). PRESERVE scripts/extract_tender_questions.py + scripts/docx_utils.py (both imported by cocoindex form_extractors/docx.py). This file was not in the original briefed residual set; it is a {46.3}-surfaced scope addition. The §B/§G rows above are struck/corrected accordingly.