Skip to content
Canonical

RESEARCH — On-prem deployment pivot: B1 co-locate + Coolify ops + B2 groundwork (ID-66.1)

RESEARCH — On-prem deployment pivot: B1 co-locate + Coolify ops + B2 groundwork (ID-66.1)

Section titled “RESEARCH — On-prem deployment pivot: B1 co-locate + Coolify ops + B2 groundwork (ID-66.1)”

Task: ID-66 — On-prem deployment pivot — B1 co-locate + Coolify ops + B2 groundwork. Subtask: {66.1} RESEARCH. Author date: 29/05/2026. Status: DECISION-SUPPORT, B1 ratified. Liam ratified Option B1 (co-locate cocoindex + pullmd) in S282. This document does not re-litigate the A-vs-B1 choice; it operationalises B1 — enumerating what we need to pivot now, evaluating Coolify as the ops substrate (for B1 and the longer-term B2 client offering), establishing a directional cost baseline, and recording precise corrections to the s281 framing doc for a follow-on Planner to apply.

This is the {66.1} artefact. {66.2} PRODUCT, {66.3} TECH, {66.4} PLAN are separate fresh-Planner instances (Q-PLANNER-2). Every load-bearing claim is tagged [VERIFIED] (grounded in a repo file / manifest read this session) or [ASSUMPTION] (needs a decision or an empirical probe). Open questions for Liam are collected in §5 as OQ-66-N.

TL;DR

Section titled “TL;DR”

B1 = the s281 §3 P4 “co-locate” topology: run cocoindex + pullmd on one host so cocoindex reaches pullmd over http://localhost:3000, and the cocoindex LMDB state store

  • corpus live on a persistent local disk. This dissolves the {62.1} reachability defect (no cross-service hop → no ingress: internal classification → no VPC/NAT), and kills the ~$200/mo Filestore (local disk mmaps and locks natively, unlike GCS-FUSE). The app is untouched: the only app↔pipeline coupling is the inbound pipeline-runs/record webhook (Bearer CRON_SECRET), which works from any host that can reach the Vercel app over HTTPS [VERIFIED]. The pivot’s real work is host + Compose + persistent volume + a push-to-deploy path to replace cloud-run-deploy.yml/WIF + secrets
  • monitoring + backup. Coolify is a strong fit as the managed-PaaS ops layer B1 gives up vs Cloud Run, and is the substrate the original ADS plan already named for B2 (the on-prem-for-clients offering) — with two caveats: the Feb-2026 Coolify evaluation was written for a now-stale stack (Logto/Turso/FastAPI, not Next.js/Supabase/cocoindex/pullmd), and B2 re-opens the AGPL-pullmd conveyance question. B1 is largely reversible; B2 is partly one-way and is not decided here.

1. B1 pivot checklist — “what we need right now to pivot”

Section titled “1. B1 pivot checklist — “what we need right now to pivot””

The components below are what B1 requires to finish wiring the pipeline on a co-located host. Each is tagged reversible (can be undone / re-platformed back to Cloud Run, since the manifests still exist and the app is Supabase-decoupled) or one-way (incurs a commitment that is costly to unwind). Note up front: none of these items touches the product app — the app↔pipeline seam is Supabase plus the single inbound webhook (item e).

(a) One host — UK VPS or GCE VM — running Docker Compose or Coolify

Section titled “(a) One host — UK VPS or GCE VM — running Docker Compose or Coolify”

A single standing host that runs the co-located container set. Two host families:

  • UK VPS (Fasthosts / IONOS / OVH have UK data centres; Hetzner is EU-only) — aligns with the eventual B2 data-residency story and is the cheapest standing-host option. [VERIFIED — provider/UK-DC table in coolify-evaluation.md §3 “Server Provider Considerations”; stack-context caveat: that table was written for the bid-manager project, but the provider/residency facts transfer.]
  • GCE VM in europe-west2 — keeps the workload inside the existing GCP project and WIF/secret tooling, smallest blast radius from today’s setup, but does not advance the UK-VPS B2 direction. [ASSUMPTION — not yet costed against a VPS; see §3.]

Coolify’s minimum host spec is 2 CPU / 2 GB RAM / 30 GB disk [VERIFIED — coolify-evaluation.md §2 “Server Requirements”], but the workload dominates: the cocoindex Service runs cpu=2/mem=4Gi and pullmd’s Playwright sidecar is ~3.7 GB, so the host must comfortably exceed Coolify’s bare minimum. cocoindex pre-warms ~1.8 GB Docling weights into the image (~5.3 GB image) [VERIFIED — services/README.md §“Docling pre-warm”

  • staging-pullmd.yaml:161-166]. Sizing target [ASSUMPTION]: ≥4 vCPU / ≥16 GB RAM / ≥80 GB disk to hold both containers warm plus the LMDB volume and image layers; confirm against the live image sizes before provisioning (see OQ-66-1).

Tag: reversible (a standing host can be torn down; the Cloud Run manifests remain).

(b) A Compose file co-locating cocoindex + pullmd over localhost

Section titled “(b) A Compose file co-locating cocoindex + pullmd over localhost”

A docker-compose.yml (or Coolify “Docker Compose” build pack) declaring the two services on one host so cocoindex calls http://localhost:3000 (pullmd) instead of an internal run.app URL. Much of the shape already exists: pullmd already runs as a multi-container pod (pullmd :3000 + Playwright sidecar :8002//render + Trafilatura sidecar :8001//extract) talking over in-pod localhost today, via PLAYWRIGHT_URL / TRAFILATURA_URL — this is explicitly “closest to the upstream docker-compose shape” [VERIFIED — staging-pullmd.yaml:16-19, 102-108]. So the pullmd half is already a localhost-coupled compose unit; B1 adds the cocoindex service into the same compose network and repoints the cocoindex→pullmd call from PULLMD_SERVICE_URL (a Secret Manager run.app URL today) to http://localhost:3000 (or a compose-network service alias).

The cocoindex→pullmd HTTP call is made by scripts/cocoindex_pipeline/_pullmd_to_markdown.py over PULLMD_SERVICE_URL [VERIFIED — services/README.md §“pullmd AGPL boundary (O-Q3)”]. The Bearer-token auth (PULLMD_AUTH_MODE=single-admin, Authorization: Bearer pmd_…) is retained on localhost — defence-in-depth is cheap to keep [VERIFIED — staging-pullmd.yaml:21-26, 92-93].

Tag: reversible (the compose file is additive; cocoindex’s HTTP-client adapter is URL-configurable via env, so re-pointing back to a run.app URL is a one-line env change — verify the adapter reads the URL from env and has no hardcoded host, OQ-66-2).

(c) cocoindex’s LMDB on a persistent local volume

Section titled “(c) cocoindex’s LMDB on a persistent local volume”

cocoindex’s engine keeps an LMDB state store (COCOINDEX_DB → /cocoindex-state/lmdb) that today sits on an in-memory emptyDir tmpfs (medium: Memory, sizeLimit: 512Mi) — ephemeral, wiped on every cold start [VERIFIED — staging-phew-cocoindex.yaml:60-103 + cocoindex-deploy.md §0/§1]. On a co-located host, this becomes a real local-disk volume (a Docker named volume / Compose volumes: mount). Local disk mmaps and locks natively, which LMDB requires; GCS-FUSE cannot mmap/lock, so it was never an option, and Filestore (~$200+/mo, 1 TiB min) was the only Cloud-Run-native persistent FS — co-locating on local disk eliminates that ~$200/mo cost entirely [VERIFIED — cocoindex-deploy.md §1 + staging-phew-cocoindex.yaml:93-98 + id-62 RESEARCH §3.3]. The same local volume can also host the corpus directory (COCOINDEX_SOURCE_PATH), which is currently "" (idle) with no corpus volume mounted at all [VERIFIED — id-62 RESEARCH §3.2].

Important framing (correction iv, see §4): the LMDB is cocoindex’s own persistent incremental store, not a KH datastore. The decision to persist it across restarts is a KH infra choice; the database itself belongs to the cocoindex engine [VERIFIED — cocoindex-deploy.md §0 table: “the cocoindex ENGINE’s internal LMDB state store … cocoindex’s internal store is LMDB-only”].

Tag: reversible (volume can be re-created; but the persisted memo state is the one stateful artefact — backing it up matters, see item f).

(d) Push-to-deploy path replacing cloud-run-deploy.yml + WIF

Section titled “(d) Push-to-deploy path replacing cloud-run-deploy.yml + WIF”

Today’s deploy is a GitHub Actions workflow (cloud-run-deploy.yml) that authenticates via WIF (Workload Identity Federation), runs Cloud Build, and applies manifests with bare gcloud run services replace / gcloud run jobs replace (no --vpc-egress/--network flag anywhere — this is the root of the {62.1} reachability defect) [VERIFIED — cloud-run-deploy.yml:140-158, 243-296 + id-62 RESEARCH §OQ-62-9]. B1 replaces this whole chain. Two replacement shapes:

  • Coolify webhook — Coolify’s native push-to-deploy: a Git webhook fires on push, Coolify pulls, builds (Dockerfile / Compose), deploys, runs health checks, routes traffic [VERIFIED — coolify-evaluation.md §4 “CI/CD Workflow”].
  • Thin GH Action → SSH/registry — a much smaller workflow than cloud-run-deploy.yml that pushes the image to a registry and SSH-triggers a docker compose pull && up -d (or calls Coolify’s deploy API). [ASSUMPTION — either shape works; Coolify-native is less bespoke code to own.]

The WIF + Cloud Build + per-tenant-manifest-replace machinery is retired by B1. The per-tenant NEXT_PUBLIC_CLIENT_ID bake-at-deploy guard [VERIFIED — cloud-run-deploy.yml:222-247] must be preserved in whatever replaces it if multi-tenant deploys continue (relevant to B2).

Tag: one-way-ish (retiring cloud-run-deploy.yml/WIF is reversible in principle since the file remains in git history, but it’s the most “infrastructure-rewrite” item — treat the cutover as a deliberate step, not a toggle).

(e) Repoint the inbound pipeline-runs/record webhook — NO app change

Section titled “(e) Repoint the inbound pipeline-runs/record webhook — NO app change”

The cocoindex sidecar POSTs to /api/internal/pipeline-runs/record on the Vercel app (Bearer CRON_SECRET, verifyCronAuth) to land pipeline_runs rows [VERIFIED — app/api/internal/pipeline-runs/record/route.ts:91-93 + s281 §1]. This is the only app↔pipeline coupling, and it is inbound (pipeline → app). It works from any host that can reach the Vercel app over HTTPS and holds CRON_SECRET — the sidecar reads its target from PIPELINE_RUN_WEBHOOK_URL (a Secret Manager mount today) [VERIFIED — cloud-run-deploy.yml:375 + route file]. On B1, this becomes a host env/secret pointing at the same Vercel URL. No app code changes; no Vercel changes. [VERIFIED — the webhook is host-agnostic by construction.]

Tag: reversible (a single env var on the new host; trivially re-pointable).

(f) Host secrets + monitoring + backup

Section titled “(f) Host secrets + monitoring + backup”

The ops surface B1 takes on from the managed platform:

  • Secrets — today 10 base secrets + cocoindex-specific (PIPELINE_RUN_WEBHOOK_URL, PULLMD_SERVICE_URL, PULLMD_API_TOKEN, COCOINDEX_DB_DSN) + pullmd admin creds, all in GCP Secret Manager and mounted via --set-secrets [VERIFIED — cloud-run-deploy.yml:347-409]. On B1 these become host-side env / a secrets store (Coolify’s per-application env management with hidden-after-save secrets [VERIFIED — coolify-evaluation.md §4 “Environment Management”]). COCOINDEX_DB_DSN (the Supabase pooler string) and CRON_SECRET are the load-bearing ones for the webhook + the app’s asyncpg pool. [Migration of secret VALUES out of Secret Manager is operator work — OQ-66-3.]
  • Monitoring / alerts — Cloud Run gave health-gated replace + revision Ready signals
    • Sentry. B1 needs equivalent host + container monitoring + alerting (Coolify supplies deployment/server/disk monitoring + Discord/Telegram/email notifications) [VERIFIED — coolify-evaluation.md §1 feature table]. Caveat: the cocoindex /health probe is a no-op liveness signal (see §4 correction i) — green health does not mean the datapath runs — so host monitoring must watch the datapath (e.g. pipeline_runs row arrival), not just container liveness. [ASSUMPTION — datapath-level monitoring design is TECH work.]
  • Backup — the LMDB volume (cocoindex memo state) is the one stateful artefact on the host (Supabase remains the KB store and is independently backed up). Needs a volume backup (Coolify offers automated backups to S3-compatible storage; primarily framed for its managed databases — confirm it covers arbitrary named volumes, OQ-66-4) [VERIFIED that Coolify does S3 DB backups; ASSUMPTION that it covers a raw LMDB volume].

Tag: reversible for secrets/monitoring; the LMDB backup is the genuinely new stateful-ops burden (but losing it only forces a memo-state rebuild, not data loss — Supabase is canonical).

§1 summary — the reversible/one-way split

Section titled “§1 summary — the reversible/one-way split”
ItemComponentReversible?
(a)One host (UK VPS or GCE VM) + Compose/CoolifyReversible
(b)Compose co-locating cocoindex + pullmd over localhostReversible (verify URL-from-env, OQ-66-2)
(c)LMDB on persistent local volume (kills ~$200/mo Filestore)Reversible (memo state is the stateful bit)
(d)Push-to-deploy replacing cloud-run-deploy.yml + WIFOne-way-ish (infra rewrite; git history remains)
(e)Repoint inbound pipeline-runs/record webhookReversible (one env var; NO app change)
(f)Host secrets + monitoring + backupReversible secrets/monitoring; new LMDB-backup burden

The single biggest finding: B1 is overwhelmingly reversible because the app is decoupled through Supabase + one inbound webhook. The pipeline re-platforms without touching the product. [VERIFIED]

2. Coolify fit — B1 (ops layer) and B2 (client substrate)

Section titled “2. Coolify fit — B1 (ops layer) and B2 (client substrate)”

2.1 B1 — does Coolify supply the managed-PaaS ops layer B1 gives up vs Cloud Run?

Section titled “2.1 B1 — does Coolify supply the managed-PaaS ops layer B1 gives up vs Cloud Run?”

B1’s stated cost (s281 §3) is that ops “shifts onto us” — losing managed patching, health-gated replace, WIF-native CI, and the runbooks. Coolify is precisely a self-hostable PaaS that re-supplies that layer. Mapping each ops requirement Cloud Run provided to a Coolify capability [all VERIFIED against coolify-evaluation.md §1/§4 unless noted]:

Ops requirement (lost with Cloud Run)Coolify capabilityVerdict
Push-to-deployGit push-to-deploy (GitHub/GitLab/Bitbucket/Gitea) + webhooks + APICovered
TLS/SSLAutomatic Let’s Encrypt issuance + renewalCovered
Multi-service deployDocker Compose build pack (multi-service, predefined networks)Covered — direct fit for the (b) compose file
Volume persistencePersistent storage / volume managementCovered (the (c) LMDB volume) — confirm raw-volume backup, OQ-66-4
BackupsAutomatic backups to S3-compatible storagePartial — DB backups are first-class; raw LMDB-volume backup unconfirmed (OQ-66-4)
Health checksHealth-check configuration + container restart on failureCovered — but note the cocoindex /health no-op caveat (§4-i): datapath monitoring is extra
Monitoring/alertsDeployment/server/disk monitoring + Discord/Telegram/emailCovered for infra; datapath-level monitoring is bespoke

B1 Coolify verdict: YES / strong fit, with one partial (LMDB-volume backup) and one caveat (the health probe is a no-op, so liveness ≠ datapath-runs — datapath monitoring must be added on top). Coolify materially re-supplies the managed-PaaS layer B1 otherwise gives up, and its Docker-Compose build pack maps directly onto the (b) co-location compose file. [VERIFIED for the capability set; the no-op-probe caveat is the one place “green” must not be trusted as “working”.]

2.2 B2 — is Coolify the substrate the original ADS plan named?

Section titled “2.2 B2 — is Coolify the substrate the original ADS plan named?”

The s281 doc records that the original ADS plan was on-prem-flavoured (“Coolify on UK VPS”, per-tenant DB) — so B2 is a return to a latent direction, not a new invention [VERIFIED — s281 §4 + state-of-the-product.md §0.1/§1 as cited there]. The Feb-2026 Coolify evaluation already worked the B2 multi-tenant model in depth:

  • Multi-server, one dashboard — Coolify (self-hosted or Cloud) manages many servers from one pane [VERIFIED — coolify-evaluation.md §5 Option B/C].
  • Per-client projects — one Coolify project per client; environments (production/staging) within each [VERIFIED — §5 “Coolify projects to organise by client”].
  • Docker-network isolation — Coolify destinations (separate Docker networks per client) give network-level isolation even on shared hosts [VERIFIED — §5 “destinations (Docker networks)”].
  • UK VPS data residency — the provider table (Fasthosts/IONOS/OVH UK DCs) supports the per-client residency choice [VERIFIED — §3 provider table].
  • The evaluation’s headline recommendation: Option C (Hybrid) — shared infra for cost-sensitive clients, dedicated VPS for regulated/premium [VERIFIED — §5 “Multi-Tenant Recommendation”].

B2 Coolify verdict: YES — Coolify is the substrate the ADS plan already named, and the multi-tenant model (projects + destinations + multi-server + UK VPS) is exactly the B2 shape. [VERIFIED]

2.3 Stack-refresh caveats (Feb-2026 evaluation is partly stale)

Section titled “2.3 Stack-refresh caveats (Feb-2026 evaluation is partly stale)”

The Coolify evaluation predates the KH stack pivot. Platform-fit claims transfer; stack specifics are STALE [VERIFIED — the doc header says “Bid Manager Project”, and its integration sections name Logto/Turso/FastAPI]:

  • Auth: the doc names Logto (one-click Coolify service). KH uses Supabase Auth (GoTrue), not Logto. The “deploy Logto via Coolify” step does not apply. [VERIFIED — coolify-evaluation.md §3 step 7 / §6 names Logto; KH is Supabase per CLAUDE.md.]
  • Database: the doc names Turso (LibSQL/SQLite) as external DB. KH uses Supabase (Postgres + pgvector), one project per client. Coolify can host PostgreSQL one-click, but whether KH self-hosts Supabase on Coolify or keeps managed Supabase cloud is the B2 crux (s281 §4) and is out of scope for B1 (B1 keeps managed Supabase). [VERIFIED — doc names Turso §4; KH/Supabase per CLAUDE.md; the self-host-Supabase question is the s281 B2 crux.]
  • Runtime: the doc’s Nixpacks/Dockerfile examples target FastAPI/Python. KH’s co-located workload is the cocoindex Rust+Python image + the pullmd AGPL Node/Python container set — these deploy via the Dockerfile / Docker-Compose build packs (not Nixpacks auto-detect). The pinned images (aeternalabshq/pullmd:2.0.0 + sidecars; the cocoindex Cloud Build image) carry over directly. [VERIFIED images — staging-pullmd.yaml
    • services/README.md; ASSUMPTION that Coolify Compose ingests them unchanged.]

2.4 The AGPL-pullmd conveyance question (B2 only)

Section titled “2.4 The AGPL-pullmd conveyance question (B2 only)”

For B1 (our infra), pullmd’s AGPL status is no new exposure — we already operate pullmd as a network service [VERIFIED — s281 §1 + services/README.md O-Q3]. For B2 (shipping into a client perimeter), AGPL’s network-use clause is joined by a conveyance question. The s281 doc records three mitigations in order of cleanliness [VERIFIED — s281 §4 “AGPL distribution”]:

  1. Don’t convey — the client pulls the upstream public pullmd image themselves; KH never conveys the binary. (Cleanest.)
  2. Host it — KH keeps hosting pullmd as a network service even for on-prem clients (only the KB store is on-prem) — but this re-opens the data-egress question, weakening the sovereignty claim.
  3. Drop pullmd for Docling — fall back to Docling (MIT, already in the cocoindex image) for HTML on-prem, accepting lower extraction quality.

This is a B2 product/legal decision, not a B1 item. B1 does not trigger it. [VERIFIED that B1 is unaffected; ASSUMPTION on which mitigation B2 takes — Liam’s call, deferred.]

3. Cost baseline (directional, not precise)

Section titled “3. Cost baseline (directional, not precise)”

Option A (stay on Cloud Run) real spend to date: approximately ~$70 total on Cloud Run since setup, with NO users and NO ingestions — i.e. the pipeline has never run the datapath, so this ~$70 is essentially idle warm-instance + build + registry cost, not workload cost. [ASSUMPTION — the ~$70 figure is operator-reported, not pulled from a billing export this session; flagged for confirmation, OQ-66-5.] On top of this, GitHub Actions minutes per deploy: cloud-run-deploy.yml runs Cloud Build for two images (the base pipeline image + the cocoindex image, the latter a +30–45 min first build for Docling pre-warm) plus the deploy/secret-mount/smoke steps, on ubuntu-latest, timeout-minutes: 45 [VERIFIED — cloud-run-deploy.yml:120, 195-211 + services/README.md pre-warm table]. So every pipeline-code or manifest push spends real CI minutes on a heavy two-image build.

Cloud Run’s structural cost floor (the thing B1 removes): 2× minScale=maxScale=1 warm Services @ cpu2/mem4Gi held warm permanently (for LMDB warmth + Playwright cold-start avoidance) + ~$200/mo Filestore once LMDB persistence is wanted + NAT/egress + Artifact Registry [VERIFIED — s281 §2 + cocoindex-deploy.md §1]. The warm-instance + Filestore floor is the price of the topology, and we are paying for a managed autoscaler we deliberately pin to 1.

B1 single-host economics: one standing host instead of two warm Services + Filestore + NAT. A UK VPS sized for the workload is roughly single tens of $/mo (the coolify-evaluation.md §3 table shows ~$5-20/mo for small VPS, but the cocoindex+Playwright workload needs a larger box — directionally still well under the Cloud Run warm-×2 + $200 Filestore floor) [VERIFIED that the Filestore is killed; ASSUMPTION on the precise VPS $ — directional only, OQ-66-1/OQ-66-5]. The decisive cost line is the Filestore kill (~$200/mo eliminated outright by local disk), plus collapsing two warm Services to one host, plus no per-deploy two-image Cloud Build. Directional conclusion: B1 is materially cheaper at single-tenant scale. [VERIFIED directionally; not priced precisely — a true comparison needs the actual GCP billing export and a chosen VPS plan.]

4. Cloud-run doc corrections (for a follow-on Planner to apply to the s281 doc)

Section titled “4. Cloud-run doc corrections (for a follow-on Planner to apply to the s281 doc)”

These four corrections are recorded here precisely; the s281 doc (docs/research/s281-cloud-run-vs-onprem-implications.md) is NOT edited by this Planner — a follow-on Planner applies them. Each correction names the target location in the s281 doc and the corrected claim.

(i) “Pipeline never ran” = setup INCOMPLETE, not a failure — the probe is a no-op

Section titled “(i) “Pipeline never ran” = setup INCOMPLETE, not a failure — the probe is a no-op”
  • Target: s281 TL;DR bullet 2 and §1 “reachability defect” — currently phrased “broken as deployed … never run end-to-end.”
  • Correction: the datapath has never run because setup is INCOMPLETE, not because a working thing broke. Critically, the cocoindex Service’s readiness/liveness probe is httpGet /health on a separate aiohttp thread, so the Service deploys green (revision Ready) without the datapath ever running — app_main returns to idle when COCOINDEX_SOURCE_PATH="" (the shipped value) and /health stays up regardless [VERIFIED — staging-phew-cocoindex.yaml:45-46, 80-92 (probe is /health, source path "") + id-62 RESEARCH §3.1-3.2 (idle-mode clean return; /health on its own thread) + cocoindex-deploy.md §0 (“the worker thread crashes at boot while /health stays green on its own thread”)]. The probe is a liveness no-op for the datapath: green ≠ ingesting. This reframes “never run” from a defect to an unfinished-wiring state, and warns that Cloud Run’s health-gated deploy gave false assurance.

(ii) kb_pipeline is NOT removed — scripts/kb_pipeline/*.py still present

Section titled “(ii) kb_pipeline is NOT removed — scripts/kb_pipeline/*.py still present”
  • Target: s281 §1 workload table — the kh-pipeline-{phew,kpf} row describes “the batch ingest CLI (scripts/kb_pipeline/)”. The s281 doc does not claim removal, but downstream framing has implied the legacy pipeline is gone; this correction pins the actual state.
  • Correction: scripts/kb_pipeline/ is still present — 21 .py files (e.g. chunk.py, classify.py, dedup.py, embed.py, extract.py, extract_answer.py, eval_holder_rule.py, …) [VERIFIED — ls scripts/kb_pipeline/*.py = 21 files this session]. The canonical replacement (scripts/cocoindex_pipeline/, 14 .py files) is the go-forward path, but kb_pipeline retirement is a tracked Curator carry, not done. Record kb_pipeline as “extant, retirement pending” rather than “removed”.

(iii) kpf retired but STILL live in 6 cloudrun manifests → dead-manifest cleanup candidate

Section titled “(iii) kpf retired but STILL live in 6 cloudrun manifests → dead-manifest cleanup candidate”
  • Target: s281 §1 workload table footnotes + appendix — “cocoindex + pullmd Services are phew-only; only the Jobs have kpf variants.”
  • Correction: kpf is decommissioned (the pullmd manifests say “phew-only; kpf decommissioned”) yet kpf is still live across 6 cloudrun files [VERIFIED — grep -rl kpf cloudrun/ = 6 files this session]:
    1. cloudrun/jobs/prod-kpf.yaml — dedicated kpf Job manifest (still deployed by the CI for TENANT in phew kpf loop).
    2. cloudrun/jobs/staging-kpf.yaml — dedicated kpf Job manifest (same loop).
    3. cloudrun/jobs/staging-phew.yaml — kpf reference in comments.
    4. cloudrun/services/prod-pullmd.yaml:3 — “kpf decommissioned” comment.
    5. cloudrun/services/staging-pullmd.yaml:3 — “kpf decommissioned” comment.
    6. cloudrun/services/README.md:47,65,76 — kpf SA-mapping + kpf smoke-verify commands for a kh-cocoindex-pipeline-kpf Service that has no manifest (the cocoindex/pullmd Services are phew-only). Furthermore, cloud-run-deploy.yml still loops for TENANT in phew kpf for Job deploy, secret-mount, and smoke-verify [VERIFIED — cloud-run-deploy.yml:225, 264, 310, 362, 389, 414], so the retired kpf tenant is actively redeployed every push. Flag as a dead-manifest cleanup candidate (a tracked Curator/follow-on carry — the B1 cutover is the natural moment to drop kpf from the deploy loop and delete the two kpf Job manifests + the stale README kpf commands).

(iv) The LMDB is COCOINDEX’s persistent incremental store, not a KH datastore

Section titled “(iv) The LMDB is COCOINDEX’s persistent incremental store, not a KH datastore”
  • Target: s281 §1 “Cost / licence baseline” Filestore bullet + §3 “Filestore” row — currently “give LMDB a persistent FS”, framed as a KH-LMDB-persistence decision.
  • Correction: keep the framing that persisting it across cold starts is a KH infra choice (correct), but add that the database itself belongs to the cocoindex engine — it is “the cocoindex ENGINE’s internal LMDB state store … cocoindex’s internal store is LMDB-only; there is no Postgres backend for it” [VERIFIED — cocoindex-deploy.md §0 table + staging-phew-cocoindex.yaml:60-71 comments distinguishing COCOINDEX_DB (engine LMDB path) from COCOINDEX_DB_DSN (KH’s asyncpg Supabase pool)]. The nuance: KH owns the persistence/topology decision; cocoindex owns the store. This matters for B1 because the local-disk volume hosts cocoindex’s memo state (backup/restore is about cocoindex’s incremental engine, not KH’s data — Supabase remains the KB store of record).

5. Open questions for Liam (OQ-66-N) — resolve before / during PRODUCT.md

Section titled “5. Open questions for Liam (OQ-66-N) — resolve before / during PRODUCT.md”
  • OQ-66-1 — Host family + sizing. UK VPS (Fasthosts/IONOS/OVH; advances B2 residency) vs GCE VM (smallest delta from today, keeps GCP tooling)? And confirm the box size against the live image sizes (cocoindex ~5.3 GB image + cpu2/mem4Gi; pullmd Playwright ~3.7 GB) — the ≥4 vCPU / ≥16 GB / ≥80 GB target is an estimate, not a measured floor. [ASSUMPTION]
  • OQ-66-2 — Is the cocoindex→pullmd URL fully env-configurable? B1’s localhost repoint (item b) and reversibility both assume _pullmd_to_markdown.py reads PULLMD_SERVICE_URL from env with no hardcoded host. Confirm before TECH commits (cheap empirical check — grep the adapter). [ASSUMPTION]
  • OQ-66-3 — Secret migration mechanics. Which secrets store on the host (Coolify env / a host secrets manager / SOPS-encrypted file), and the operator process to move the values out of GCP Secret Manager (COCOINDEX_DB_DSN, CRON_SECRET, pullmd creds, etc.)? [ASSUMPTION]
  • OQ-66-4 — Coolify backup of a raw LMDB volume. Coolify’s S3 backups are first-class for its managed databases; confirm it can back up an arbitrary named Docker volume (the cocoindex LMDB), or whether a separate volume-backup mechanism is needed. [ASSUMPTION]
  • OQ-66-5 — Cost confirmation. Pull the actual GCP billing export for the ~$70 figure and the GitHub Actions minutes, and price a concrete VPS plan, to turn §3’s directional conclusion into a real comparison. [ASSUMPTION]
  • OQ-66-6 — Push-to-deploy shape. Coolify-native webhook vs thin GH Action → SSH/Coolify API? (Coolify-native is less bespoke code to own; either works.) Carries the NEXT_PUBLIC_CLIENT_ID-bake guard preservation question for multi-tenant deploys. [ASSUMPTION]
  • OQ-66-7 — B2 is NOT decided here. The self-host-Supabase crux (s281 §4) and the AGPL-pullmd conveyance mitigation (§2.4) remain open product/legal decisions, deliberately deferred. B1 does not force either. [ASSUMPTION — explicitly out of B1 scope.]

6. Verification notes (external-API / infra facts)

Section titled “6. Verification notes (external-API / infra facts)”

This RESEARCH is an infrastructure / ops artefact. It cites no third-party library symbols that require an import-and-call check: the cited surfaces are KH-internal manifests (Cloud Run YAML), CI YAML, a TS route, a Python pipeline directory listing, and the Coolify feature set (a hosted-platform capability list, not an importable API). The one external pin touched in passing — cocoindex[postgres]==1.0.3 [VERIFIED — requirements.txt] — is already empirically pinned in the codebase (id-62 RESEARCH §9 records “Verified empirically against installed cocoindex 1.0.3”) and no cocoindex symbol is asserted here, so no re-verification is needed for this artefact.

Infra facts cited are file-grounded — every ingress setting, env value, volume shape, probe definition, manifest count, and deploy-loop claim carries a file:line or a grep/ls count run this session (the kpf-in-6-files and kb_pipeline-21-files counts were both produced live). The cost figures (§3) are the only directional [ASSUMPTION] claims and are flagged as such (OQ-66-5).

7. Files of record

Section titled “7. Files of record”
  • docs/research/s281-cloud-run-vs-onprem-implications.md — the framing doc; B1 = §3 P4 co-locate. The spine of this RESEARCH. (Corrections in §4 target this doc; NOT edited here.)
  • docs/research/coolify-evaluation.md — Feb-2026, written for the old bid-manager stack (Logto/Turso/FastAPI); platform-fit claims transfer, stack specifics are stale (§2.3).
  • docs/specs/id-62-fixture-staging-infra/RESEARCH.md §OQ-62-9 — reachability synthesis (P1 ruled out; P2 vs P4; B1 = P4).
  • cloudrun/services/{staging,prod}-phew-cocoindex.yaml — cocoindex Service: ingress: internal, minScale=maxScale=1, in-memory LMDB tmpfs, COCOINDEX_SOURCE_PATH="", no VPC, /health probe.
  • cloudrun/services/{staging,prod}-pullmd.yaml — pullmd AGPL Service + Playwright + Trafilatura sidecars over in-pod localhost (the existing compose-shaped unit); ingress: internal; “kpf decommissioned” comment.
  • cloudrun/jobs/{prod,staging}-{phew,kpf}.yaml — the per-tenant batch Jobs; the two *-kpf.yaml are dead-manifest cleanup candidates (§4-iii).
  • cloudrun/services/README.md — Service-vs-Job rationale, pullmd AGPL boundary (O-Q3), per-tenant SA mapping; carries stale kpf cocoindex-Service smoke commands (§4-iii).
  • .github/workflows/cloud-run-deploy.yml — WIF CI deploy; bare gcloud run … replace, no VPC flag; for TENANT in phew kpf loops (§4-iii); two-image Cloud Build (§3).
  • docs/runbooks/cocoindex-deploy.md §0/§1 — the two-DB distinction (engine LMDB vs KH asyncpg DSN), Filestore ~$200+/mo rationale, GCS-FUSE/LMDB incompatibility.
  • app/api/internal/pipeline-runs/record/route.ts — the only app↔pipeline coupling (inbound webhook, Bearer CRON_SECRET; host-agnostic — §1-e).
  • scripts/kb_pipeline/*.py (21 files) / scripts/cocoindex_pipeline/*.py (14 files) — the legacy-extant + canonical pipeline directories (§4-ii).
  • requirements.txt — cocoindex[postgres]==1.0.3 pin (§6).