Skip to content

DR-121: Pipeline runtime assets leave the test tree — scripts/cocoindex_pipeline/data/

DR-121 — Pipeline runtime assets leave the test tree

Section titled “DR-121 — Pipeline runtime assets leave the test tree”

scripts/cocoindex_pipeline/extraction.py:129 reads scripts/tests/fixtures/taxonomy_snapshot.json at module import, in production, and raises when it is absent or malformed — it is the runtime content_types / form_types validator source, named as the HARD-reject gate in ontology/04-content-type.md. canonicalisation.py:286 reads entity_aliases_snapshot.json from the same directory and fails soft to baseline-only.

Neither is a test fixture. Both are production runtime assets filed under a test tree, and both paths are assembled segment-wise (Path(__file__).parent.parent / "tests" / "fixtures" / …), so no path grep finds them — the same class of invisibility that nearly broke two consumers during the S524 relocation.

This is load-bearing now rather than tidy-later. DR-119 replaces the buildpacks build with a checked-in Dockerfile whose COPY set is explicit. Today the image carries these files only because the build context is cp -R scripts wholesale. A Dockerfile that copies scripts/cocoindex_pipeline/ alone kills extraction.py at import — in the live deploy path that Platform staging, Platform prod, the client boxes and bid-worker-platform all pull.

taxonomy_snapshot.json moves to scripts/cocoindex_pipeline/data/, and the move lands BEFORE id-416 narrows the build context. DR-117 files fixture trees by owning domain and explicitly rejects scripts/tests/; this extends the same rule to runtime data, which has a stronger claim still. data/ rather than fixtures/ because these are not fixtures and must not join the manifest’s declared fixture roots.

entity_aliases_snapshot.json is NOT moved with it. Its generator, scripts/generate-entity-aliases-snapshot.ts, has zero callers — absent from package.json scripts (sync:taxonomy chains four scripts and this is not one), from every workflow, and from deploy/onprem/*. The “generated at deploy (PI-15)” story in canonicalisation.py:122-140 is self-justification; the same comment concedes the wiring is a follow-up. Do not repoint a path nothing writes: either wire the generator into deploy or delete generator + loader branch, as its own decision.

The two remaining JSONs under scripts/tests/fixtures/ stay puturl_normalisation_parity.json and canonicalisation_parity.json are genuine cross-language parity case tables read only by tests. (url_normalise.py names the file in a docstring at :19 and never opens it.)

  • Wait for the planned scripts/services/ move. Rejected: the files travel inside the package either way, so waiting buys nothing and leaves id-416 blocked on a live deadline.
  • scripts/cocoindex_pipeline/fixtures/. Rejected: that root is anchored by DR-117 and the corpus manifest for fixture trees. A runtime validator source filed there invites exactly the input/output category slip that put a blank extraction form in sixteen content tests.
  • Leave them and special-case the Dockerfile COPY. Rejected: encodes the wrong shape into the new build recipe on day one.
  • ~12 code consumers repoint, plus 9 live doc filesontology/README.md (the ID-133 Decision-A authority chain), ontology/01, 04, 05, runbooks/taxonomy-change-runbook.md, and four under product-functionality/. Under DR-104 the doc is the authority, so moving the file without them makes the docs wrong; they land in the same commit. Specs for task ids below ~130 (id-36, id-52, id-63, id-68) are presumed stale per DR-106 and are deliberately NOT churned.
  • The “Taxonomy dual-source” rationale in CLAUDE.md and extraction.py:123-126 is wrong in both directions and is corrected in the same commit: content-type-registry.ts re-exports a generated tuple, and the ratified human source is the docs-site register markdown. TS and the snapshot are peer DB-derived mirrors, not one derived from the other.
  • Ordering constraint on id-416: this move is a precondition of the build-context narrowing, recorded in tasks/id-416.md.