DR-119: The sidecar leaves buildpacks, and the test lane pulls the image it claims to test

Context

The cocoindex sidecar image is built by Cloud Native Buildpacks using gcr.io/buildpacks/builder:google-22. That builder is inherited verbatim from the pre-Coolify cloudbuild-cocoindex.yaml — onprem-deploy.yml says so in four places, including “byte-for-byte faithful to the original”. The owner confirmed at S524 that Cloud Run was configured but never actually ran, so there is no reversibility to preserve; the platform deploys to a self-hosted VPS via Coolify and has no Google Cloud anywhere.

Three defects converged to force the question, all measured rather than argued:

The lock does not control the image. The google-22 Python buildpack runs pip install --requirement with no --no-deps, so pip re-resolves every pin’s own metadata at build time. Removing opencv-python from requirements.lock changed nothing (id-413), and it is the same mechanism behind the {145.13} commonforms/pypdf gap. What ships is decided by pip, not by us.
The lock does not describe Linux. It is resolved on macOS/arm64 and installed on linux-x86_64. It carries no nvidia-*/CUDA distributions yet pins torch, so a multi-gigabyte dependency set is installed at build time and recorded nowhere. A developer machine’s view of the package graph is committed as the production manifest.
We already run two build systems for one artefact. The OS packages (git, openssh-client, LibreOffice) arrive via a Dockerfile written by heredoc at build time and docker build-ed on top of the pack build output. Buildpacks has not spared us a Dockerfile; it has hidden one.

Buildpacks also actively obstructs the work: python3 is not on PATH in a docker build RUN over a CNB image (verified S524 — the image PATH is /cnb/process:/cnb/lifecycle:…, the interpreter is at /layers/google.python.runtime/python/bin/python3, and the run image ships no system python), and app dependencies sit in a separate google.python.pip layer. id-413’s fix needs an ls -d /layers/*/*/bin/python3 glob and manual site-packages surgery for that reason alone.

Separately, cocoindex-nightly.yml built its own image rather than pulling the published one, because there is no :latest tag and onprem-deploy’s detect-changes job gates the publish, so a checked-out commit may have no matching sha- tag. The consequence: from the lane’s creation (2026-07-08) until S524 it exercised an image missing four packages the deployed one carries — a lane whose stated purpose is dogfooding the deployed artefact, testing something else, for about four weeks, undetected.

Decision

The sidecar image is built from a checked-in Dockerfile, from a lock generated in CI for the deployment target and installed with --no-deps; and the nightly pulls the published image rather than building its own. Owned by id-416.

Four parts, none of which stands alone:

Dockerfile. Debian-slim + Python 3.13, explicit apt-get for the LibreOffice / git / openssh-client set, pip install --no-deps -r requirements.lock, non-root user. gcr.io/buildpacks/builder:google-22 leaves the repo.
CI-generated lock. Resolved on a linux-x86_64 runner (or inside the target base image), never on a developer machine, with a CI check that fails when the committed lock differs from a fresh resolve. uv pip compile resolves rather than compiles C — nothing was ever built on the Mac — but the resolver’s view of the world is what gets committed, and it must be the target’s view.
--no-deps. Only safe once the lock is a real Linux closure, which is why these are one decision and not two. It converts a future lock gap from a silent backfill into a build failure. That is the intended trade.
One image. onprem-deploy.yml builds and publishes sha-<sha>; the nightly pulls it via a new moving :main tag. The deploy composes keep pinning sha-, so the no-:latest rollback discipline is untouched — it governs deploys, not test lanes. A local build survives only as the branch-dispatch fallback, through the same shared recipe.

This supersedes the standing “buildpacks stays, gated on id-404” position recorded in tasks/id-413.md. That gate was discharged at S523 anyway: soffice has been in the image since ID-145 {145.31} / DR-059, via the post-pack apt layer, so buildpacks demonstrably can add apt packages here. The move proceeds on the reasons above, not on soffice.

Alternatives Considered

Keep buildpacks; fix the lock only. Rejected. It leaves the Google builder, the two-build-system shape, and the layer-globbing in place, and --no-deps on pack build (PIP_NO_DEPS=1) was already measured and rejected at S523 — it would ship torch incomplete while the lock stays macOS-resolved.
Keep the nightly building its own image. Rejected. It is what produced the four-week divergence. S524 restored parity by duplicating the post-pack layer into the nightly, with a reciprocal “edit both in the same commit” warning — an explicitly interim state, because two copies of a recipe drift and this lane is the proof.
Narrow the build context now (it currently ships 87 test_*.py / 209 files / 9.2 MB of scripts/tests into the image). Deferred into this work rather than done separately: narrowing is an image-shape change to the live deploy path and it interacts with the fixture-staging question, so both are settled once, here.

Consequences

This is the live deploy path. Platform staging, platform prod and the client boxes all pull this image, and bid-worker-platform runs the same one via a command: override. Both images must be built side by side and diffed (pip list, soffice --version, import cv2, size) before any cutover, staging first.
CNB rebase is lost — free base-image CVE patching goes with it. Replace it with a scheduled rebuild.
The CUDA set becomes visible, and therefore removable. It is already being installed by pip’s backfill, so an honest lock does not add weight; it exposes weight that can then be dropped. Nothing in this pipeline uses a GPU, so CPU-only torch is the obvious lever. A real build measured at S524 came to 7.03 GB on disk against a 5.3 GB budget line — different measure (the budget step reads compressed registry layers), but the direction is clear.
id-413’s post-pack opencv layer may become redundant once --no-deps holds. It is not removed without a build proving cv2 still resolves to headless; the build-time import cv2 guard stays either way.