DR-152: The pipeline is rebuilt on upstream cocoindex, not repaired in place

Context

The S564 owner architecture review measured the pipeline against the upstream library it is built on. scripts/cocoindex_pipeline/ is 30,721 lines of Python (55% comment prose) — roughly the size of the entire upstream cocoindex library at our pin — while calling 12 of upstream’s 88 top-level exports. Five measured duplications of installed upstream capability (pair resolution, declare_row targets, retry stacks, ID generation, the connector/source API); the producer package makes exactly one cocoindex call in 9,905 lines; a second machinery layer (mock LLM server, memo identity, writer fences, fault injection) exists only because every walk to date has been synthetic-corpus + mock-tier — the platform has never run real extraction over a real corpus end to end. The upstream vendored example (meeting_notes/main.py, 442 lines) performs an equivalent-shaped job with upstream primitives alone. The owner’s standing frame (“the correct approach is using cocoindex correctly”, S516) and the two-person operability constraint make repair-in-place the expensive option.

Decision

Owner-ruled (S564 board, D1-B — the offered proof-spike interposition was declined): the pipeline is rebuilt on upstream cocoindex primitives at the root-level pipeline/ (DR-142’s ratified destination), and the current tree is replaced, not repaired. The rebuild is two small flows — ingestion (sources → provenance register + staged records) and producer (promoted records → OKF bundle) — using upstream connectors, targets, ops (entity-resolution, LiteLLM, splitting) and identity primitives directly. Ratified product contracts port as acceptance tests, never as ported machinery: the two-gate model and retention classes (corpus reframe R1/R2, DR-025), entity naming stability (DR-140/DR-147), per-retention-class survival (DR-148), mention anchoring (DR-135), unpublished-never-cited (DR-143). The mock-tier apparatus does not port — LiteLLM provider selection replaces the base-URL mock tier, and testing pivots to recorded fixtures plus small real-tier smokes. The in-flight burn sequence (id-434 probes, id-453, id-462’s gate semantics, id-464 burn + publish) is held (owner D2-A) and disposed by the rebase. Execution owned by id-465 under initiative 12.

Alternatives Considered

• Repair in place (continue burn → publish → hygiene on the current tree) — rejected: it polishes the machinery the evidence identifies as the problem, and every session spent there deepens the bespoke layer.
• Proof spike first, then decide (recommended on the board) — declined by the owner in favour of direct commitment; the spike’s content (real data, upstream-style, end to end) is folded into the rebase’s first phase rather than run as a separate gate.
• Adopt an external producer wholesale (OpenWiki) — deferred to its own evaluation (owner D3-A); it does not replace the ingestion flow or the gate.

Consequences

• id-465 is minted as the executing task; initiative 12 (platform rebase) is the owning home and its charter extends from wiring-census repair to rebuild.
• The first rebase phase runs on real data (the staged meeting-note corpus and a real client-corpus slice) with a real LLM tier from day one.
• Machinery-specific decision rulings (e.g. DR-150 memo identity, DR-146 writer topology, DR-133 fixture identity) are retired or re-homed en bloc as the rebase lands, each with retired_reason — not before, since the old tree remains deployed until replaced.
• Model strategy is a config decision (LiteLLM): GLM-5.2 availability is evaluated per stage during the rebase (owner S564 annotation); any whole-app provider question is out of scope here.
• DR-151’s emission plan folds into the rebased producer design (owner D4-B): author/last_modified are source-derived enrichment duties populated when honestly known; admitted_by is not required.