Skip to content
Canonical

Cocoindex Absorption Cleanup (ID-46 / T14) — PRODUCT

Cocoindex Absorption Cleanup (ID-46 / T14) — PRODUCT

Section titled “Cocoindex Absorption Cleanup (ID-46 / T14) — PRODUCT”

Summary

Section titled “Summary”

ID-46 retires the residual dead-code left behind after cocoindex (scripts/cocoindex_pipeline/) became the sole canonical ingest/writer path and after the OLD Python pipeline core (scripts/kb_pipeline/ + its CLIs + backfills) was already deleted under ID-68.23 (50b3b2096, 08/06/2026). The desired outcome: a named set of verified-dead standalone scripts and dead lib/ exports is deleted with zero dead-code residue, the explicitly-preserved live modules and the reader_html column are left untouched, and bun run knip + python3 -m pytest scripts/tests/ + bun run test + bun build are all green afterwards — without straying across the ID-42 / ID-56 / ID-52 / ID-53 / backlog carve-out boundaries.

Problem

Section titled “Problem”

The S277 framing treated ID-46 as the task that would delete the OLD Python pipeline (scripts/kb_pipeline/, then ~5,797 LOC across 21 modules) + its 4 legacy CLIs + ~7 backfill scripts + ~30 py tests, hard-gated behind ID-45 (T7 full-corpus re-ingest). That premise is now false: the entire core was removed early under ID-68.23 (commit 50b3b2096, “chore(scripts): remove obsolete kb_pipeline + its consumers and guards”), three months ahead of ID-46’s gate, with pytest scripts/tests/ (1202 passed) + bun run test + tsc + knip all green post-removal (RESEARCH §A, §G). What remains is a thin, bounded, mechanical pure-deletion residual: standalone scripts with zero live callers and a handful of dead TypeScript exports — none on a canonical write path, none T7-gated for safety. This PRODUCT is the retirement contract for that residual: it pins exactly what ID-46 deletes, what it must preserve, and what it must not touch.

Code-intelligence orientation (spot-verified by this Planner against worktree subo-id-46 @ e8e981c0a, citing tool output verbatim — RESEARCH §G did the deep sweep, these pins were re-confirmed before authoring):

  • extractQaPairs (lib/procurement-library-ingest/extract-qa-pairs.ts): gitnexus_impact upstream → impactedCount: 0, risk: LOW, 0 direct callers; ast-dataflow references → 17 refs = 1 in-file def (:469) + 16 __tests__/**, 0 production callers. The Pattern A/B Q&A parser is prod-dead.
  • docx-to-markdown.ts: grep docx-to-markdown (non-test) → sole in-repo importer is extract-qa-pairs.ts — retires as a pair with it.
  • lib/dedup-normalise.ts (normaliseTitleForDedup): ast-dataflow importers --module @/lib/dedup-normalise → 1 importer, and it is __tests__/lib/dedup-normalise.test.ts — 0 production importers. (Distinct from the LIVE lib/dedup.ts / lib/entities/entity-dedup.ts, which are NOT targets.)
  • scripts/dedup.py + scripts/dedup_normalise.py: grep python importers (non-test) → only dedup.py:21 self-imports dedup_normalise; no live consumer; neither in package.json / .github/workflows. Their prod consumer import_bid_library.py was deleted by ID-68.23.
  • resolveQuestionForRebuild (resolve-question.ts): grep → live caller app/api/items/[id]/route.ts. extractAnswerFromContent (extract-answer.ts): grep → live callers app/api/items/batch/route.ts + lib/mcp/tools/content.ts. Both LIVE — the dir is NOT deleted wholesale (corrects S277 row 11’s “0 callers / whole dir” claim).
  • classifyContent (lib/ai/classify.ts): grep (non-test) → 15+ live callers across app/api/items/*, upload/route.ts, cron/classification-quality/route.ts, MCP tools, markdown-orchestrator.ts, the batch_reclassify queue handler. LIVE — KEEP; ID-46 deletes only its three CLI-script callers, not the module.
  • reader_html column readers: grep (non-test) → item-detail-client.tsx, app/api/review/queue/route.ts, lib/validation/schemas.ts. Column STAYS — ID-46 retires the spent populator scripts, not the column.
  • lib/extraction/og-metadata.ts: grep → sole non-test consumer is lib/extraction/url.ts (which ID-42 {42.12} rewrites). lib/extraction/extraction-result.ts: grep → 0 external consumers. Both are RETIRE-WHEN-DEAD, coupled to ID-42 landing.
  • reader-html / firecrawl scripts: grep package.json / .github/workflows → zero CI / package refs; backfill-reader-html.ts imports extract-reader-html.ts. Standalone one-shot tools, safe to delete.
  • scripts/extract_docx_tables.py: grep importers → imported by scripts/cocoindex_pipeline/form_extractors/docx.py + scripts/extract_tender_questions.py. LIVE (cocoindex form-extraction) — KEEP.

Goals / Non-goals

Section titled “Goals / Non-goals”

Goal: delete the verified-dead residual set with zero leftover dead imports or references, leaving the canonical cocoindex pipeline and all preserved live modules intact, and the full regression gate green.

Non-goals (carved OUT — owned elsewhere, ID-46 must NOT touch):

  • lib/extraction/{html,url}.ts and the 3 HTML npm deps (@mozilla/readability, jsdom, @mendable/firecrawl-js) → ID-42 {42.11}/{42.12}/{42.13}.
  • The app-side chunker (lib/content/chunking.ts + chunk-store.ts + the regenerateChunks callers + scripts/backfill-chunks.ts), content_chunks writes, and the markdown_batch UI/orchestrator fate → ID-56 {56.11}/{56.12}.
  • The batch_reclassify queue handler (lib/queue/handlers/batch-reclassify.ts) + batch_reclassify job-type (and the markdown_batch job-type symmetry) → backlog-188 (UNDECIDED). ID-46 retires only the CLI scripts, not the queue infra.
  • analyse/route.ts + template_analyse job-type → ID-52 {52.15}.
  • Stage-5 entity resolution → ID-53.
  • Orphan columns (content_items.notes / parent_id / source_bid / source_document text) + the 6 AI-telemetry token/cost columns + a possible summary_data drop → a separate schema-cleanup migration. The S277/NEW-OQ-A “backlog-189” home for this is a stale reference (ledger-cli show backlog 189 → record-not-found — open cross-ref OQ-46-8). ID-46 retires pipeline CODE, never schema columns.

Behaviour

Section titled “Behaviour”

All invariants below are scoped to the worktree at canonical-pipeline-setup HEAD and are verified by the named command / code-intel probe. Invariants 1–11 are the unconditional deletion + preservation contract; invariant 12 records the corrected gating (the residual is NOT blanket-T7-gated); invariant 13 is the conditional RETIRE-WHEN-DEAD set.

Retirement (the residual set is deleted with zero dead-code residue)

Section titled “Retirement (the residual set is deleted with zero dead-code residue)”

  1. The Q&A parser pair is deleted. lib/procurement-library-ingest/extract-qa-pairs.ts and lib/procurement-library-ingest/docx-to-markdown.ts no longer exist. Their only references were the in-file definition and __tests__/** (0 production callers, verified: gitnexus_impact extractQaPairs → impactedCount 0; ast-dataflow references → 16/16 non-def refs in __tests__/). The associated test files (__tests__/lib/procurement-library-ingest/extract-qa-pairs.test.ts, the parity.integration.test.ts Q&A-parser cases) are removed alongside, so no test imports a deleted module.

  2. The TS dedup-normalise helper is deleted. lib/dedup-normalise.ts (normaliseTitleForDedup) no longer exists. Its only importer was __tests__/lib/dedup-normalise.test.ts (verified: ast-dataflow importers @/lib/dedup-normalise → 1 test importer, 0 prod), which is removed alongside. The unrelated LIVE modules lib/dedup.ts and lib/entities/entity-dedup.ts are untouched and still present.

  3. The standalone Python dedup scripts are deleted. scripts/dedup.py and scripts/dedup_normalise.py no longer exist. They were standalone __main__ CLIs with no live (non-test) Python importer (verified: only dedup.py:21 self-imported dedup_normalise); neither was wired into package.json or .github/workflows; their former prod consumer import_bid_library.py was already deleted under ID-68.23. The test_dedup_normalise.py test (its only test reference) is removed alongside.

  4. The three legacy batch-reclassify CLI scripts are deleted. scripts/batch-reclassify.ts, scripts/batch-reclassify-keywords.ts, and scripts/reclassify-items.ts no longer exist. These were the collapse-list §5.2 [RATIFIED-RETIRE] CLI scripts that imported the legacy classify surface (classifyContent / isExcludedEntity / validateDomain from @/lib/ai/classify). Cocoindex code-hash memoisation provides incremental re-classification in their place.

  5. The reader-html populator scripts are deleted. scripts/extract-reader-html.ts and scripts/backfill-reader-html.ts no longer exist. They were jsdom + @mozilla/readability one-shot tools to populate the reader_html column; backfill-reader-html.ts was the only importer of extract-reader-html.ts; neither was in package.json / CI. They are unambiguously ID-46’s to delete — ID-42 never names them (RESEARCH §C, OQ-46-7).

  6. The firecrawl backfill script is deleted by ID-46. scripts/backfill-source-url-firecrawl.ts no longer exists. It was a one-shot firecrawl backfill not wired into package.json / CI. ID-42 PRODUCT explicitly excludes and disclaims deleting it (its {42.13} grep gate is scoped to lib/ app/ only — it does NOT cover scripts/), leaving an ownership gap that ID-46 fills. Open question (OQ-46-7): if ID-42 opportunistically absorbs this script via its own flagged OQ before ID-46 runs, ID-46’s TECH must check ID-42’s merge state and de-dupe (delete-if-present, not assume-present) rather than double-delete.

  7. Zero dead-code residue. After the deletions in invariants 1–6, no surviving file contains a dead import, dead re-export, or dangling reference to any deleted module or script. Specifically: no surviving .ts imports extract-qa-pairs, docx-to-markdown, dedup-normalise, or any deleted script path; no surviving .py imports the deleted dedup modules; no package.json script entry, ci.yml step, or docs reference points at a deleted path. (Verified by bun run knip clean-or-baseline + grep sweeps over .ts / .py / .sql / package.json / .github/.)

Preservation (the explicitly-named live set is untouched)

Section titled “Preservation (the explicitly-named live set is untouched)”

  1. The live members of lib/procurement-library-ingest/ survive. resolve-question.ts (resolveQuestionForRebuild, called from app/api/items/[id]/route.ts) and extract-answer.ts (extractAnswerFromContent, called from app/api/items/batch/route.ts

    • lib/mcp/tools/content.ts) still exist and their callers still compile and pass. The directory is not deleted wholesale — only the dead extract-qa-pairs + docx-to-markdown pair within it.

  2. lib/ai/classify.ts survives intact. classifyContent (and isExcludedEntity / validateDomain) remains, with its 15+ live callers unchanged. ID-46 removes only the three CLI-script callers (invariant 4); the module itself, the routes, MCP tools, queue handler, and orchestrator that consume it are untouched and still compile/pass.

  3. The reader_html column and the governance-freshness concept are untouched. The content_items.reader_html column remains in the schema and its readers (app/item/[id]/item-detail-client.tsx, lib/validation/schemas.ts, components/reader/source-metadata.tsx, components/item-detail/reader-view.tsx, components/item-detail/editor-view.tsx) are unchanged. (NB app/api/review/queue/route.ts is NOT a reader — its REVIEW_COLUMNS select explicitly excludes reader_html, per the comment at route.ts:23.) ID-46 retires only the spent populator SCRIPTS (invariant 5), never the column. No migration is authored by ID-46. The user-facing governance-freshness concept (fresh / aging / stale / expired) is likewise unaffected — ID-46 touches no governance code.

  4. The remaining lib/extraction/ files and the surviving deps are untouched. lib/extraction/{turndown,pdf,clean-mdx-tags,markdown-front-matter,markdown-title, diff-markers,content-type-detect,url-validation}.ts (each with ≥1 live consumer or PDF / unpdf coupling) still exist and compile. scripts/extract_docx_tables.py (consumed by cocoindex form_extractors/docx.py) survives. No dependency is uninstalled by ID-46: unpdf (upload / images routes), jsdom (ontology loader — survives even after ID-42’s extraction-tier removal), pdfplumber / openpyxl / python-docx (ID-52 form_extractors), faiss-cpu (ID-53) all remain in package.json / requirements.txt.

Gating

Section titled “Gating”
  1. Execution stays gated on T7 + ID-28 + ID-42, per the Task ledger and brief. ID-46 execution is conditional on (a) ID-45 (T7 full-corpus re-ingest) complete + stable, (b) ID-28 stable (cocoindex proven as the sole canonical ingest/writer path), and (c) ID-42 merged — as fixed by the Task ledger (dependencies: [45, 28, 42]; “Gated on T7 full-corpus (ID-45) complete + stable + ID-28 stable”) and by the sub-orchestrator brief (“execution stays T7-gated”; “Author now, execute post-cutover”; execution subtasks imp_deferred / pending “since T7-gated”). This spec is authored now and executes post-cutover; no invariant in this spec is verifiable before T7 + ID-28 + ID-42, and no invariant writes or migrates a schema column. The deletion-safety rationale — that the residual in invariants 1–6 sits on no canonical write path and each target has 0 production callers (RESEARCH §A/§B/§G) — explains why the deletions are low-risk once the gate opens, but it is evidence, not grounds to ungate: a Planner does not resolve a sequencing question the Task and brief have already fixed (see OQ-46-10). At execution time the Executor MUST, per target, re-run the §Problem code-intel probes as a fresh 0-production-caller re-verify check (the gate having opened does not excuse skipping the per-file reachability confirmation); invariant 13 additionally requires the ID-42 merge-state check. Cocoindex (scripts/cocoindex_pipeline/, 27 modules / 11,899 LOC) remaining the sole canonical writer is the standing ID-28 precondition; ID-46’s deletions do not touch the cocoindex pipeline.

Conditional retirement (RETIRE-WHEN-DEAD, gated on ID-42)

Section titled “Conditional retirement (RETIRE-WHEN-DEAD, gated on ID-42)”
  1. lib/extraction/og-metadata.ts and lib/extraction/extraction-result.ts are deleted only after ID-42 has merged and a fresh 0-caller check confirms them dead. og-metadata.ts’s sole non-test consumer is lib/extraction/url.ts (which ID-42 {42.12} rewrites to drop the OG-metadata branch); extraction-result.ts already has 0 external consumers (its parity guard was deleted by ID-68.23). This invariant is not verifiable before ID-42 lands and is recorded as BLOCKED-ON ID-42, not a standing acceptance criterion. At execution time: if ID-42 has merged AND a fresh grep / ast-dataflow check confirms 0 production consumers, delete both; if ID-42 has NOT merged, ID-46 leaves both in place and records the carry-over (it does not pre-delete a still-consumed module, and it does not block on ID-42 to ship the invariants 1–11 set). See §Edge cases for the not-yet-merged branch.

Edge cases

Section titled “Edge cases”

  • ID-42 not yet merged at execution time. Invariants 1–11 are independent of ID-42 and ship regardless. The HTML-surface residue (lib/extraction/og-metadata.ts, lib/extraction/extraction-result.ts) under invariant 13 is left in place — og-metadata.ts still has a live in-dir consumer (url.ts) until ID-42 {42.12} rewrites it, so deleting it early would break the build. ID-46 records these as a BLOCKED-ON ID-42 carry-over rather than forcing the deletion. lib/extraction/{html,url}.ts themselves are never ID-46’s to delete in any branch — they are ID-42’s carve-out.

  • backfill-source-url-firecrawl.ts double-ownership. ID-42 disclaims it but might absorb it via its own OQ. ID-46’s deletion of it (invariant 6) must be delete-if-present (tolerant of the file already being gone), so a prior ID-42 merge that deleted it does not turn into an ID-46 failure. The reader-html pair (invariant 5) is unambiguously ID-46’s in every branch — ID-42 never names it.

  • lib/ai/classify.ts caller-count drop when the CLI scripts go. Deleting the three batch-reclassify CLI scripts (invariant 4) removes three callers of classifyContent / isExcludedEntity / validateDomain. This is expected and must NOT trigger over-deletion: the module still has 15+ live callers (invariant 9) and stays. A knip “now-unused export” signal on any classify symbol after the CLI deletion would be a false positive to investigate, not a licence to delete the symbol — classifyContent is live via routes / MCP / queue handler.

  • Test-file co-deletion. Each deleted source module’s dedicated test(s) (the Q&A-parser tests, dedup-normalise.test.ts, test_dedup_normalise.py) are deleted in the same change so that bun run test and python3 -m pytest scripts/tests/ do not fail on imports of removed modules. No test that exercises a preserved module (invariants 8–11) is removed.

  • Knip baseline. bun run knip post-deletion should be clean or match the acknowledged pre-existing baseline (.knip-baseline.json, bloated by unrelated ID-50 / ast-dataflow / ledger-cli surface per RESEARCH §4.4 row 36) — the deletions should only ever reduce the knip surface, never add a new unlisted entry. If a deletion surfaces a previously-suppressed dead export as newly-orphaned, that is in-scope to clear in the same change.

Open cross-references

Section titled “Open cross-references”
  • OQ-46-7 — reader-html / firecrawl scripts ownership: recommended that ID-46 owns all three deletions (invariants 5–6) as residual; ID-46 TECH checks ID-42 merge state to avoid double-delete. The reader-html pair is unambiguously ID-46’s; backfill-source-url-firecrawl.ts is ID-46’s unless ID-42 absorbs it via its own OQ.
  • OQ-46-8 — backlog-189 is a stale reference (ledger-cli show backlog 189 → record-not-found). The S277/NEW-OQ-A routing of the 6 telemetry-column drops + a possible summary_data drop to backlog-189 has no live home. This is a ledger-curation question for the Workflow Curator; it does NOT block ID-46 (column drops were always carved OUT of ID-46 per OQ-46-6). Surfaced here for routing.
  • OQ-46-9 (strategic, optional) — whether a standalone ID-46 is still warranted given the core is done (ID-68.23). The residual is ~3.2k LOC of standalone scripts + a few dead lib/ exports. Sub-orchestrator / Liam decides: keep lean, fold into a broader post-cocoindex dead-code sweep, or demote to backlog. Recorded; not a spec blocker.
  • OQ-46-10 (sequencing) — should ID-46 execution be ungated from T7 given the core already landed under ID-68.23? The S348 RESEARCH addendum (§A) argues the residual is not T7-gated for safety (none of it sits on a canonical write path; each target has 0 production callers). The Task ledger (dependencies: [45, 28, 42]) and the sub-orchestrator brief (“execution stays T7-gated”) say it is gated. This is a sequencing question the Planner cannot resolve — the orchestrator / Liam decides. Until that ruling, this spec records the briefed T7 + ID-28 + ID-42 gate (invariant 12); the dead-code evidence is preserved as deletion-safety rationale, not as grounds to ungate. (Mirrors the surface-don’t-resolve humility of RESEARCH §A / OQ-46-9.)