PRODUCT — On-prem deployment pivot: B1 co-locate cocoindex + pullmd on one host (ID-66.2)

PRODUCT — On-prem deployment pivot: B1 co-locate cocoindex + pullmd on one host (ID-66.2)

Task: ID-66 — On-prem deployment pivot — B1 co-locate cocoindex + pullmd on one host. Subtask: {66.2} PRODUCT. Author date: 29/05/2026. Predecessors (ratified): RESEARCH.md ({66.1}), OQ-CLOSURE.md (OQ-66-1/2/3/4/6 closed). This spec consumes both as given and does not re-litigate them.

Filename convention (ID-48.4): canonical {N.2} artefact is PRODUCT.md. The companion {66.3} TECH.md (validation + implementation detail) and {66.4} PLAN.md (decomposition) are separate fresh-Planner instances per Q-PLANNER-2.

Summary

B1 re-platforms the ingestion pipeline only — cocoindex plus pullmd (and pullmd’s Playwright/Trafilatura sidecars) — off Google Cloud Run onto one co-located host (an IONOS VPS managed by Coolify), so cocoindex reaches pullmd over localhost/a compose alias rather than an internal run.app hop, and the cocoindex engine’s LMDB state store lives on persistent local disk. The product application is untouched: it remains on Vercel + managed Supabase, decoupled from the pipeline except for the single inbound pipeline-runs/record webhook. The outcome is a cheaper, simpler, datapath-reachable single-host deployment that is overwhelmingly reversible back to Cloud Run.

Problem

The pipeline as deployed on Cloud Run cannot run its datapath. cocoindex and pullmd are both ingress: internal Services with zero VPC egress configured, so the cocoindex→pullmd HTTP call cannot complete — the reachability defect tracked at {62.1}. The premise “cocoindex→pullmd works today as deployed” was disproven in RESEARCH; the datapath has never run end-to-end. Worse, the cocoindex Service deploys green because its /health probe is a liveness no-op on a separate thread — green readiness never implied a working datapath. On top of the broken topology, Cloud Run imposes a structural cost floor (two warm minScale=maxScale=1 Services, plus ~$200/mo Filestore the moment LMDB persistence is wanted, since GCS-FUSE cannot mmap/lock the LMDB) for infrastructure that has never processed a single ingestion. B1 dissolves the reachability defect by co-location and eliminates the cost floor by using local disk.

Code-intelligence orientation (cited verbatim per the code-intel block):

• gitnexus_query({query: "cocoindex pipeline ingestion pullmd", repo: "knowledge-hub"}) surfaced no execution processes ("processes": []) but returned the pipeline definitions: ingest_file (scripts/cocoindex_pipeline/flow.py:1039-1489), app_main (scripts/cocoindex_pipeline/flow.py:1523-1867), _pullmd_to_markdown (scripts/cocoindex_pipeline/adapters.py:115-188), extract_source_provenance (scripts/cocoindex_pipeline/adapters.py:233-278), and the cocoindex server thread symbols _health_handler (scripts/cocoindex_pipeline/server.py:131-144) and start_cocoindex_thread (scripts/cocoindex_pipeline/server.py:168-225) — all in the Cocoindex_pipeline module.
• gitnexus_query({query: "pipeline runs webhook record CRON_SECRET verifyCronAuth", repo: "knowledge-hub"}) surfaced process proc_146_post (POST → SupabaseError, cross_community, 4 steps) wiring the inbound webhook: POST (app/api/internal/pipeline-runs/record/route.ts:95-174, step 1, module Freshness-transitions) → recordPipelineRun (lib/pipeline/record-run.ts:159-268, step 2). Auth helper: verifyCronAuth (lib/cron-auth.ts:15-25, module Freshness-transitions). The pipeline-side emitter is _emit_pipeline_run_webhook (scripts/cocoindex_pipeline/flow.py:551-644).
• gitnexus_context({name: "_emit_pipeline_run_webhook", repo: "knowledge-hub"}) — verdict scope LOW: 1 direct caller (app_main in the same file), zero outgoing edges indexed, participates in no cross-community process. The webhook emitter is a single-caller leaf; re-pointing its target URL has negligible blast radius.
• gitnexus_context({name: "_pullmd_to_markdown", repo: "knowledge-hub"}) — verdict scope LOW: 2 direct callers (convert_binary_to_markdown, extract_source_provenance, both scripts/cocoindex_pipeline/adapters.py), zero outgoing edges indexed, no process participation. The pullmd HTTP adapter is internal to the pipeline module; the cocoindex→pullmd target is env-only (PULLMD_SERVICE_URL, adapters.py:125, confirmed by OQ-66-2), so the localhost repoint touches no code.

External-library grounding: this is an infrastructure/ops artefact. It cites no third-party library symbols that require an import-and-call check — the surfaces named are KH-internal Python symbols, Cloud Run/Compose manifests, an inbound TS route, and hosted-platform capability sets (Coolify), none of which is an importable API. The one external pin touched in passing, cocoindex[postgres]==1.0.3, is already empirically pinned in the codebase (RESEARCH §6); no cocoindex symbol shape is asserted here, so the empirical-grounding axis is satisfied without a fresh import-and-call check.

Goals / Non-goals

Goals (B1, in scope):

• Co-locate cocoindex + pullmd (+ sidecars) on one host; make the datapath reachable.
• Persist the cocoindex LMDB on local disk; eliminate Filestore.
• Repoint the inbound webhook with zero app/Vercel change.
• Move secrets on-host; retire WIF + cloud-run-deploy.yml; back up the LMDB; monitor the datapath; keep everything reversible.

Non-goals (explicitly OUT of scope for B1):

• B2 — the multi-tenant client offering. The self-host-Supabase crux and the AGPL-pullmd conveyance decision (RESEARCH §2.4, §5 OQ-66-7) are open product/legal decisions, deliberately deferred. B1 does not force either. B1 keeps managed Supabase; the KB store of record does not move.
• App cutover. There is none. The Vercel app + managed Supabase are untouched.
• kb_pipeline retirement (RESEARCH §4-ii) — a separate Curator carry, not a B1 item.
• Cost-export hardening (OQ-66-5) — the directional cost conclusion stands; pulling the precise GCP billing export is out of B1’s behaviour scope.

Behavior

The “user” here is the operator running and maintaining the co-located pipeline, plus the downstream consumers of the resulting pipeline_runs rows and the Supabase KB. Each invariant below is an observable, testable property an operator (or a test/probe) can verify against the B1 deployment. UK English; dates DD/MM/YYYY.

Topology and reachability

1. Single-host co-location. cocoindex, pullmd, and pullmd’s Playwright (/render) and Trafilatura (/extract) sidecars all run on one host. An operator inspecting the host sees the full container set running together; there is no second standing host for the pipeline. The host is the IONOS VPS (8 vCPU / 16 GB / 480 GB NVMe) per OQ-66-1.

2. cocoindex reaches pullmd over the host-local network. The cocoindex→pullmd HTTP call resolves to http://localhost:3000 or a compose-network service alias (e.g. http://pullmd:3000) — never an internal *.run.app URL. An operator can issue the same request cocoindex makes ({PULLMD_SERVICE_URL}/api?url=… with the Bearer token) from the cocoindex container and receive a pullmd response. The cocoindex→pullmd target is set purely by the PULLMD_SERVICE_URL env var; no source change is involved (OQ-66-2).

3. The reachability defect is dissolved, not patched. There is no ingress: internal classification and no VPC/NAT/egress dependency anywhere in the cocoindex→pullmd path. The cross-service hop that could not complete on Cloud Run (both Services internal-ingress, zero VPC egress) does not exist on B1 — the two services share a host-local network. A test that previously failed because the hop could not route now succeeds because there is no hop to route.

4. pullmd’s internal sidecar wiring is preserved. pullmd continues to call its Playwright and Trafilatura sidecars over in-pod/compose localhost (PLAYWRIGHT_URL, TRAFILATURA_URL) exactly as it did on Cloud Run. B1 adds cocoindex into the same network; it does not re-wire pullmd’s existing sidecar topology.

LMDB persistence

5. LMDB lives on persistent local disk. The cocoindex engine’s LMDB state store (COCOINDEX_DB → e.g. /cocoindex-state/lmdb) is mounted on a persistent local-disk volume (a Docker named volume / Compose volumes: mount), not a tmpfs/emptyDir in-memory medium. An operator inspecting the mount sees a real disk-backed volume, not tmpfs.

6. LMDB survives container restart. After the cocoindex container is restarted (or the host reboots) without destroying the volume, the LMDB memo state is intact: the engine resumes from its prior incremental state rather than rebuilding from scratch. A test can write memo state, restart the container, and observe the state persisted.

7. The LMDB is cocoindex’s engine store, not a KH datastore. Persisting it is a KH infra choice; the database belongs to the cocoindex engine. Supabase remains the canonical KB store of record. Losing the LMDB volume forces a memo-state rebuild only — never KH data loss — because the canonical content lives in Supabase (independently backed up).

Inbound webhook — no app change

8. The webhook repoint is a single host env var; the app does not change. The cocoindex pipeline emits its run record by POSTing to the app’s /api/internal/pipeline-runs/record endpoint (Bearer CRON_SECRET). On B1 the target comes from a host-side env var (PIPELINE_RUN_WEBHOOK_URL) pointing at the same Vercel app URL, with the same CRON_SECRET. There are zero changes to app code, to proxy.ts, to the route handler, or to any Vercel configuration. The endpoint, its auth (verifyCronAuth), and its row-writing behaviour (recordPipelineRun → pipeline_runs) are byte-for-byte as they are today.

9. A run on the new host lands a pipeline_runs row. After the co-located pipeline completes (or fails) a run, a corresponding pipeline_runs row appears in Supabase with the same shape it would have had from Cloud Run (run status, stage counts, items created, error class / retry count on failure). The webhook is host-agnostic by construction: any host that can reach the Vercel app over HTTPS and holds CRON_SECRET produces the same row.

Secrets on-host

10. Secrets live in Coolify-native per-app encrypted env. The ~8–10 secret values the pipeline needs (e.g. COCOINDEX_DB_DSN, CRON_SECRET, PULLMD_API_TOKEN, pullmd admin creds, PIPELINE_RUN_WEBHOOK_URL, the Anthropic key, Supabase keys) are stored in Coolify’s per-application encrypted env management on the host, hidden-after-save. They are not read from GCP Secret Manager at runtime (OQ-66-3). Infisical is deferred to B2.

11. PULLMD_SERVICE_URL largely disappears as a secret. Under B1 the cocoindex→pullmd target is http://localhost:3000 (or a compose alias) — non-sensitive host-local configuration — so it need not be a managed secret value the way the Cloud Run run.app URL was.

12. The GCP deploy identity is retired, not migrated. The GCP deploy service account / WIF credentials are not copied onto the host — B1 eliminates gcloud run deploys entirely, so there is no deploy-SA on the host side. An operator auditing host secrets finds runtime secrets only, never a GCP deploy-SA JSON.

Deploy and rollback

13. Build happens off-host on GitHub runners. The heavy two-image build (cocoindex ~5.3 GB + pullmd Playwright ~3.7 GB, ~9 GB combined) runs on GitHub Actions runners and never on the 16 GB host (OQ-66-6). The host’s CPU/RAM is never contended by a build; a build in progress does not degrade a live ingestion.

14. Push-to-deploy pushes pinned images, then Coolify pulls. A push to the deploy branch triggers a thin GitHub Action that builds and pushes pinned (tagged, not :latest) images to the registry (ghcr.io), then signals Coolify (deploy webhook/API) to pull and deploy those images. The host pulls finished images; it does not build them.

15. Rollback is re-pointing an image tag — no rebuild. To roll back, an operator re-points the deployed image tag to a prior pinned tag and redeploys; this is near-instant and does not trigger a rebuild on the host. Rolling forward again is the same operation with a newer tag.

16. cloud-run-deploy.yml + WIF are retired by the cutover. Once B1 is live, the Cloud Run deploy workflow and its WIF authentication are no longer the deploy path. The manifests and the workflow remain in git history (supporting reversibility, invariant 24); they are simply not the active path.

Backup and restore

17. The LMDB volume is backed up on a schedule to S3. A scheduled job snapshots the cocoindex LMDB and stores it in S3-compatible storage (the same bucket Coolify uses for its DB backups). Coolify’s native S3 backups are database-only and do not cover a raw Docker named volume (OQ-66-4), so this LMDB backup is a dedicated job, not Coolify’s built-in DB backup.

18. The LMDB snapshot is taken safely while the store is live. The backup uses an LMDB-consistent hot-copy mechanism (mdb_copy, which copies through a read transaction using LMDB’s MVCC read-snapshot) so it produces a consistent snapshot without quiescing or stopping the engine. A naive cp/tar of the live data.mdb is never used (it can capture a torn page set). An operator can verify a snapshot taken during active writes restores to a consistent, openable LMDB.

19. A restore path is defined and exercisable. An operator can restore the LMDB from the most recent S3 snapshot onto the persistent volume and resume the engine. The restored state is a valid cocoindex memo state; if the snapshot is stale or unavailable, the fallback is a memo-state rebuild (invariant 7), not data loss.

20. Supabase remains the canonical store and is backed up independently. Nothing in the LMDB backup path touches Supabase; the KB store of record continues under its own backup regime, unchanged by B1.

Datapath monitoring

21. Monitoring watches pipeline_runs row arrival, not just /health. Because the cocoindex /health probe is a liveness no-op (green ≠ ingesting), the datapath health signal is pipeline_runs row arrival in Supabase, not container liveness. The monitoring surface treats “container up, /health green” as insufficient evidence that the datapath is working.

22. A datapath stall is observably distinct from container liveness. If the containers are up and /health is green but no pipeline_runs rows arrive within the expected window for an active ingestion, the monitoring surface flags a datapath stall — it does not report the pipeline as healthy on liveness alone. Conversely, a successful ingestion is confirmed by the row arrival, not by the probe.

Cost floor removed

23. The Cloud Run cost floor is gone. The B1 deployment runs with: no Filestore ($200/mo eliminated — local disk mmaps/locks the LMDB natively, which GCS-FUSE never could); one host instead of two warm minScale=maxScale=1 Services; and no per-deploy two-image Cloud Build (builds run on GH runners, invariant 13). An operator comparing the standing monthly cost sees the single-host VPS line (£21/mo all-in on IONOS per OQ-66-1) in place of the warm-×2 + Filestore + NAT/Artifact-Registry floor.

Reversibility

24. Every B1 item is re-pointable back to Cloud Run. The Cloud Run manifests and cloud-run-deploy.yml remain in git, so the topology can be re-platformed back: the cocoindex→pullmd URL reverts with one env var (invariant 2 / OQ-66-2), the webhook target reverts with one env var (invariant 8), secrets can move back to Secret Manager, and the deploy path can revert to WIF. No B1 step requires an irreversible code change.

25. The LMDB memo state is the single genuinely stateful artefact. Reversibility is bounded only by the LMDB: it is the one artefact whose loss costs a rebuild (invariant 7). Everything else (host, compose, secrets, deploy path, webhook target) is config that re-points cleanly. The KB itself is never at risk because Supabase is canonical and independently backed up.

In-scope-adjacent (for {66.4} PLAN to pick up, not a core invariant)

kpf dead-manifest cleanup (RESEARCH §4-iii). kpf is decommissioned yet still live in 6 cloudrun files plus the for TENANT in phew kpf deploy loop in cloud-run-deploy.yml, so the retired kpf tenant is actively redeployed on every push. The B1 cutover — which retires cloud-run-deploy.yml and its loop entirely (invariant 16) — is the natural moment to drop kpf from the deploy path and delete the two kpf Job manifests + the stale README kpf commands. PLAN should surface this as a B1-cutover side-task. It is not a core B1 behaviour invariant and must not bloat the invariant set; it is recorded here so decomposition can pick it up without re-deriving it. (Note: kb_pipeline retirement, RESEARCH §4-ii, is a separate Curator carry and is not a B1 item.)