OQ-CLOSURE — ID-66 on-prem B1 pivot: five gating Open Questions closed

Task: ID-66 — On-prem deployment pivot (B1 co-locate cocoindex + pullmd on one host). Purpose: Close the five Open Questions that gate {66.2} PRODUCT.md: OQ-66-1, OQ-66-2, OQ-66-3, OQ-66-4, OQ-66-6. (OQ-66-5 cost-export and OQ-66-7 B2 decisions are deliberately left open per RESEARCH §5.) Author date: 29/05/2026. Method: OQ-66-2 closed by repo grep (empirical, this session). OQ-66-3/4/6 closed by WebFetch/WebSearch against Coolify docs + upstream sources. OQ-66-1 closed by analysis over the priced VPS options + a light web check on GH Actions pricing. Every load-bearing claim is tagged [VERIFIED] (grounded in a file read or doc fetched this session) or [ASSUMPTION].

This file does not edit RESEARCH.md or any worktree artefact. It is the OQ-closure input the {66.2} Planner consumes.

OQ-66-2 (EMPIRICAL) — Is the cocoindex→pullmd target URL fully env-configurable?

VERDICT: FULLY ENV-CONFIGURABLE. B1’s localhost repoint is a one-env-var change; full reversibility holds. No code change required. [VERIFIED]

The cocoindex→pullmd HTTP call is made by _pullmd_to_markdown(url) in scripts/cocoindex_pipeline/adapters.py. The target host is read purely from the environment with a fail-fast and no hardcoded host and no default fallback:

scripts/cocoindex_pipeline/adapters.py:125 — pullmd_url = os.environ.get("PULLMD_SERVICE_URL")
scripts/cocoindex_pipeline/adapters.py:126-130 — if not pullmd_url: raise RuntimeError(...) (fail-fast; there is no or "http://…" default). [VERIFIED]
scripts/cocoindex_pipeline/adapters.py:143 — the request is built as f"{pullmd_url}/api" with params={"url": url} — the entire scheme+host+port comes from the env var; only the /api path and the ?url= query are literal. [VERIFIED]

Corroborating sweeps run this session:

A repo-wide grep for any hardcoded host (run.app / http:// / https:// / localhost / 127.0.0.1) inside scripts/cocoindex_pipeline/*.py returned zero non-comment hits. [VERIFIED — grep this session]
The cocoindex manifests (cloudrun/services/{staging,prod}-phew-cocoindex.yaml) carry no inline PULLMD_* env block at all — the only inline value: lines are CLIENT_ID=phew, COCOINDEX_SOURCE_PATH="", COCOINDEX_DB=/cocoindex-state/lmdb, and the LMDB size limit. [VERIFIED — grep of staging-phew-cocoindex.yaml]
PULLMD_SERVICE_URL is injected into the cocoindex Service only via the deploy workflow’s declarative --set-secrets string: .github/workflows/cloud-run-deploy.yml:387 — COCOINDEX_SECRETS="...,PULLMD_SERVICE_URL=PULLMD_SERVICE_URL:latest,PULLMD_API_TOKEN=PULLMD_API_TOKEN:latest,COCOINDEX_DB_DSN=...". [VERIFIED]
The only pullmd + run.app literals in the whole repo are (a) the pullmd Service’s own PUBLIC_URL self-reference (cloudrun/services/prod-pullmd.yaml:76, staging-pullmd.yaml:101) — that is pullmd telling itself its public URL, not the cocoindex→pullmd target — and (b) docs/runbook prose. No cocoindex-side hardcoded run.app target exists. [VERIFIED — repo-wide grep]

B1 mechanics: repoint by setting PULLMD_SERVICE_URL=http://localhost:3000 (or a compose service alias such as http://pullmd:3000) in the cocoindex container’s host env. Reverting to a run.app URL is the same one-line change. The Bearer-token auth path (PULLMD_API_TOKEN, also env-only at adapters.py:131-136) is unchanged and can be kept on localhost as defence-in-depth. The reversibility tag on RESEARCH §1 item (b) is confirmed correct. [VERIFIED]

OQ-66-3 (WEB) — Coolify + Infisical as the secret store

VERDICT: Infisical is self-hostable on Coolify TODAY via the one-click service template (sovereignty-aligned). But a NATIVE Infisical-as-secret-backend integration is NOT shipped — it is an open proposal (GitHub Discussion #7956, opened 14 Jan 2026). So for B1, the realistic options are (1) use Coolify’s own per-app encrypted env management as the B1 secret store, or (2) self-host Infisical on Coolify and inject via the Infisical CLI/agent at build/runtime. RECOMMENDATION: B1 should use Coolify-native env management; treat self-hosted Infisical as a B2 enhancement once the native integration lands. [VERIFIED for the two facts; recommendation is analysis]

Findings:

Infisical runs one-click self-hosted on Coolify. The Coolify services catalogue lists Infisical (https://coolify.io/docs/services/infisical); it deploys via Coolify’s service template onto your own server, not as a hosted/cloud dependency. This is sovereignty-aligned (the secret store lives on the same UK host family as the rest of B1). [VERIFIED — coolify.io/docs/services/infisical + Discussion #7956]
Native “Infisical as the exclusive secret backend” is NOT yet a feature. GitHub Discussion #7956 (coollabsio/coolify, opened 14 Jan 2026) proposes a native integration where enabling Infisical on a project disables Coolify’s local env management and Coolify pulls secrets fresh from Infisical at deploy time. As of this session it remains a proposal in the feedback phase — not merged, no version/date committed. [VERIFIED — Discussion #7956]
Today’s working pattern (until native lands): self-host Infisical via the Coolify template, then inject secrets through the Infisical CLI / token + project-ID at build and runtime (Dockerfile or entrypoint), OR simply manage the values in Coolify’s own per-application env UI (hidden-after-save secrets, per RESEARCH §1-f / coolify-evaluation §4). [VERIFIED — Discussion #7956 “current workarounds”]

Migrating KH’s secrets off GCP Secret Manager. The values to move are small and known (RESEARCH §1-f): CRON_SECRET, COCOINDEX_DB_DSN (Supabase pooler), PULLMD_SERVICE_URL (→ becomes http://localhost:3000 under B1, see OQ-66-2, so it largely disappears as a secret), PULLMD_API_TOKEN, PULLMD_ADMIN_EMAIL/PASSWORD, PIPELINE_RUN_WEBHOOK_URL, plus the Anthropic key and Supabase keys the pipeline uses. Migration is a one-time manual copy of ~8-10 values: gcloud secrets versions access latest --secret=<name> to read each, then paste into Coolify’s env UI (or infisical secrets set if Infisical is self-hosted). There is no service-account-creds problem on the host side — B1 eliminates WIF/GCP SA creds entirely (no gcloud run deploy, see OQ-66-6), so the GCP deploy-SA JSON does not migrate; it is simply retired. [VERIFIED for the secret inventory — RESEARCH §1-f + cloud-run-deploy.yml:387,409; ASSUMPTION that the manual-copy process is sufficient — it is operator work, not automated]

Implication for B1 reversibility: Coolify-native env management keeps the secret store on-host and re-pointable; nothing about it is one-way. Self-hosting Infisical adds an extra container + its own backing store (another stateful artefact to back up) — disproportionate for B1’s ~10 secrets, but the right shape for B2 multi-tenant secret isolation once the native backend integration ships. [ASSUMPTION — sizing judgement]

OQ-66-4 (WEB) — Can Coolify back up a RAW Docker named volume (the LMDB)?

VERDICT: NO native raw-volume backup. Coolify’s scheduled S3 backups are DATABASE-ONLY (pg_dump / mysqldump / mariadb-dump / mongodump). The cocoindex LMDB lives on a raw Docker named volume, which Coolify will NOT back up natively. Workaround: a scheduled sidecar/cron that runs mdb_copy (LMDB-consistent hot snapshot) → tar/restic → the same S3 bucket Coolify already uses. This is a small bespoke job, not a blocker. [VERIFIED for the Coolify limitation + the LMDB-safety mechanism]

Findings:

Coolify backups cover managed databases only. The backup docs enumerate PostgreSQL, MySQL, MariaDB, MongoDB (plus Coolify-itself), schedulable by cron, externalisable to S3-compatible storage. There is no generic/raw Docker-volume backup feature and no volume-level backup. [VERIFIED — coolify.io/docs/databases/backups]
The cocoindex LMDB is not a Coolify-managed database — it is the cocoindex engine’s own store on a raw named volume (RESEARCH §1-c, correction iv). So Coolify’s first-class S3 backup machinery does not apply to it. RESEARCH §2.1’s “Partial” verdict on the backup row is confirmed. [VERIFIED]

Workaround (recommended): a small scheduled job — either a Coolify “scheduled task” / host cron, or a tiny sidecar container sharing the LMDB volume — that:

Runs mdb_copy /cocoindex-state/lmdb /snapshot (optionally mdb_copy -c to compact and drop free pages), then tar/restic the snapshot to the S3 bucket Coolify already targets for DB backups.

LMDB-specific safety (this is the load-bearing detail): LMDB is an mmap’d, copy-on-write MVCC store. A read transaction takes a snapshot of the current valid meta page at its start and is immutable for that transaction’s lifetime regardless of concurrent writes, so mdb_copy produces a consistent snapshot while the store is open — no quiesce, no stop-the-world required. Do not naively cp/tar the live data.mdb file while writes are in flight (a plain byte copy can capture a torn page set); use mdb_copy, which copies through a read transaction and is the documented safe hot-backup path. Caveat: a long-running mdb_copy concurrent with heavy writes can transiently grow the on-disk file (the read txn pins old pages from reuse); for B1’s low write volume this is negligible. [VERIFIED — lmdb.tech / lmdb.readthedocs.io / mdb_copy(1) man page: “environment can be copied regardless of whether it is currently in use”; MVCC read-snapshot semantics]

Sizing note: the snapshot must land on the same host or stream to S3; the LMDB is small (the Cloud Run tmpfs sizeLimit was 256-512 MiB), so the backup job is cheap and fast. [VERIFIED — staging-phew-cocoindex.yaml LMDB size limit 268435456 = 256 MiB]

OQ-66-6 (WEB + analysis) — Push-to-deploy shape

VERDICT: Use a thin GitHub Action → registry push → Coolify pull (option ii), NOT Coolify-native on-host build (option i). Rationale: the heavy ~9 GB two-image build (cocoindex ~5.3 GB + Playwright ~3.7 GB) must NOT run on a 4-vCPU/16-GB host — Coolify’s own guidance is that resource-intensive builds can crash the server, and an on-host build would contend with the live pipeline for the same CPU/RAM. Build on GH runners, push pinned images to ghcr.io, then call Coolify’s deploy webhook/API to pull. The NEXT_PUBLIC_CLIENT_ID build-time bake is NOT a constraint on THIS decision — it is a Vercel/app + per-tenant cocoindex-manifest concern, not part of the cocoindex+pullmd host build — but the GH-Action shape preserves it cleanly if a baked image is ever needed. [VERIFIED for the build-host tradeoff + the bake locus; recommendation is analysis]

Comparison:

Axis	(i) Coolify-native git webhook (build ON host)	(ii) Thin GH Action → registry → Coolify pull
Heavy 2-image build runs on	Host CPU/RAM — competes with the live cocoindex+Playwright workload; Coolify docs warn resource-intensive builds crash servers	GH runners — isolated from the host; host only pulls finished images [VERIFIED — Coolify CI/CD docs]
GH Actions minutes	~0 (build is on-host)	Build minutes consumed, but KH is a private repo with 2,000 free min/mo, and GH cut runner prices ~40% on 1 Jan 2026; a periodic two-image build fits comfortably [VERIFIED — GH billing docs + Dec-2025 changelog]
Build caching	Coolify/host Docker layer cache (lost if host disk pressure prunes)	GH Actions cache + registry layer cache; more durable, shareable
Rollback	Re-deploy a prior commit (host rebuilds it again — slow for a 9 GB build)	Re-point the image tag via Coolify’s PATCH API then deploy webhook — instant, no rebuild [VERIFIED — Coolify API tag-edit for rollback]
Secret exposure	Build secrets would sit on the host build context	Build secrets stay in GH Actions; host only ever holds runtime secrets (smaller blast radius)
`NEXT_PUBLIC_CLIENT_ID` bake	n/a to the cocoindex/pullmd images	If a per-tenant baked image is ever needed, GH Actions bakes it as a build-arg per tenant — identical to today’s pattern

NEXT_PUBLIC_CLIENT_ID clarification (important). The repo has no Dockerfile for the Next.js app — the app builds on Vercel, and NEXT_PUBLIC_CLIENT_ID is baked there (per-tenant branding). On the Cloud Run / pipeline side, the bake-and-assert guard lives at .github/workflows/cloud-run-deploy.yml:222-247, where it validates the NEXT_PUBLIC_CLIENT_ID baked into the per-tenant Job manifests (phew/kpf) before gcloud run jobs replace. B1 co-locates cocoindex + pullmd, neither of which is the Next.js app, so the B1 host build does not touch the NEXT_PUBLIC_CLIENT_ID bake at all — the per-tenant branding bake remains a Vercel concern, untouched by this pivot. The guard at 222-247 is part of the retiring cloud-run-deploy.yml; if multi-tenant pipeline deploys continue under B2, the equivalent assertion is re-implemented in the thin GH Action (a build-arg + a slug-pattern check), which is trivially preserved in option (ii) and is the RESEARCH §1-d preservation requirement. [VERIFIED — no Dockerfile in repo; cloud-run-deploy.yml:222-247 is the pipeline-side bake guard; RESEARCH §1-d]

Why not Coolify-native build for the app too: moot — the app stays on Vercel under B1 (RESEARCH §1 “the app is untouched”). Option (ii) governs only the cocoindex+pullmd image delivery to the on-prem host.

OQ-66-1 (analysis + light web) — Host choice + migration effort

VERDICT: Pick IONOS (£15/mo, 8 vCPU / 16 GB / 480 GB NVMe) + £6/mo 100 GB backup ≈ £21/mo all-in. For ~£3/mo more than Fasthosts all-in it buys 2× vCPU, 2× disk, and 10× backup — and the 240 GB Fasthosts disk is the real risk: ~9 GB of images + LMDB + a GROWING corpus on local disk will erode 240 GB faster than 480 GB, and disk is the one dimension that is painful to grow after the fact. Migration effort GCE→VPS is a 1-2 day job (low-single-days band). On timing: PIVOT NOW — the migration effort is small, fixed, and one-off, while staying on GCE past the trial means paying the warm-×2 + (eventual) ~$200/mo Filestore floor in real money for a topology B1 deletes. [Analysis grounded in RESEARCH §1/§3 + the priced options; GH-pricing + Coolify facts VERIFIED above]

Workload fit against the two options:

	Fasthosts £16 + £2 backup = £18/mo	IONOS £15 + £6 backup = £21/mo
vCPU	4	8
RAM	16 GB	16 GB
Disk	240 GB NVMe	480 GB NVMe
Backup	10 GB	100 GB
Fit for ~4 vCPU / 16 GB workload	Meets the stated CPU floor exactly (no headroom)	Comfortable headroom (build never runs here — see OQ-66-6 — so 8 vCPU absorbs concurrent ingest + Playwright + monitoring)
Disk runway	~9 GB images + LMDB + corpus → 240 GB gets tight as corpus grows	480 GB gives real corpus runway
Backup capacity	10 GB cannot hold image+volume history	100 GB holds LMDB snapshots + headroom

The workload is RESEARCH §1-a: cocoindex Service ~cpu2/mem4Gi + pullmd’s Playwright sidecar ~3.7 GB, cocoindex image ~5.3 GB (Docling pre-warm ~1.8 GB), plus the LMDB volume and a growing corpus on local disk. Both boxes meet the 16 GB RAM line; IONOS wins on the two axes that bite later — disk runway and backup capacity — for ~£3/mo. The 16 GB RAM is the one dimension neither option exceeds; with builds off-host (OQ-66-6) this is adequate for B1 single-tenant, but it is the dimension to watch if the corpus + concurrent ingest grow (a RAM bump is the likely first scale step). [VERIFIED workload figures — RESEARCH §1-a; ASSUMPTION on the 16 GB adequacy under load — directional, matches the ≥16 GB sizing target]

Migration effort GCE→VPS (same Docker images): the images are unchanged (cocoindex Cloud Build image + aeternalabshq/pullmd:2.0.0 + sidecars), so the work is infra plumbing, not code:

Step	Effort
Stand up VPS + base hardening	~1-2 h
Install Coolify (one-script install)	~0.5-1 h
Author the co-location compose (cocoindex + pullmd + sidecars over localhost; OQ-66-2 makes the repoint a one-env-var change)	~2-4 h
Create + mount the LMDB persistent volume + the corpus volume	~0.5-1 h
Migrate ~8-10 secret values out of GCP Secret Manager into Coolify env (OQ-66-3)	~1-2 h
Thin GH Action → ghcr.io → Coolify deploy webhook (OQ-66-6) + retire `cloud-run-deploy.yml`/WIF	~3-5 h
DNS/SSL (Coolify auto-Let’s-Encrypt) for any host-facing endpoint; repoint the inbound `pipeline-runs/record` webhook env (no app change, RESEARCH §1-e)	~1-2 h
LMDB `mdb_copy`→S3 backup job (OQ-66-4)	~1-2 h
First end-to-end ingest smoke + datapath monitor (watch `pipeline_runs` row arrival, not just `/health` — RESEARCH §4-i)	~2-3 h

Total: roughly 1.5-2.5 working days for a single competent operator — a low-single-days band, not weeks. The reason it is small: B1 is overwhelmingly reversible and the app is decoupled (RESEARCH §1 headline), so there is no app cutover, no data migration (Supabase stays put), and no DNS change for users. [VERIFIED that the app is untouched + the webhook is host-agnostic — RESEARCH §1-e; ASSUMPTION on the hour estimates — operator-effort judgement]

Pivot-now vs stay-on-GCE-until-trial-expires:

Stay on GCE post-trial: you start paying the Cloud Run structural floor in real money — 2× minScale=maxScale=1 warm Services @ cpu2/mem4Gi held warm permanently, plus the ~$200/mo Filestore the moment LMDB persistence is actually wanted, plus NAT/egress + Artifact Registry (RESEARCH §3). That is a recurring monthly bleed for a topology B1 deletes — and the datapath has never even run, so it is pure overhead. [VERIFIED — RESEARCH §3]
Pivot now: spend the one-off ~1.5-2.5 days, then run at ~£21/mo all-in on IONOS with no Filestore, one host instead of two warm Services, and no per-deploy two-image Cloud Build. [VERIFIED directionally — RESEARCH §3 “B1 is materially cheaper at single-tenant scale”]

The trade-off math is lopsided: the migration is a small fixed one-off; staying is an open-ended monthly cost for infrastructure you are abandoning. With the GCP trial budget running out, the trial expiry is the natural forcing function — do the pivot before the trial lapses so you never pay the post-trial GCE floor. The only reason to delay would be if the ~1.5-2.5 days cannot be found before expiry; if so, the fallback is to scale the GCE Services to zero / tear down Filestore to stop the bleed while the pivot is scheduled (still cheaper than running the warm floor). [ASSUMPTION — depends on Liam’s calendar + exact trial end date, which this Planner does not have; OQ-66-5 cost-export would sharpen the £/$ figures but does not change the direction]

Closure summary

OQ	Verdict (one line)	Confidence
OQ-66-2	Fully env-configurable (`adapters.py:125`, no hardcoded host) — B1 repoint = one env var; reversibility holds	[VERIFIED]
OQ-66-3	Infisical self-hosts on Coolify, but native secret-backend is an unshipped proposal; use Coolify-native env for B1, Infisical for B2	[VERIFIED facts / analysis rec]
OQ-66-4	No native raw-volume backup (DB-only); workaround = scheduled `mdb_copy`→S3; LMDB MVCC makes hot snapshot safe	[VERIFIED]
OQ-66-6	Thin GH Action → registry → Coolify pull (build off-host); `NEXT_PUBLIC_CLIENT_ID` bake is Vercel/app-side, not a B1-host concern	[VERIFIED facts / analysis rec]
OQ-66-1	IONOS (8 vCPU/16 GB/480 GB, ~£21/mo all-in) over Fasthosts; migration ~1.5-2.5 days; pivot now before the GCP trial lapses	[analysis over VERIFIED inputs]

Still open (per RESEARCH §5, not in this closure’s scope): OQ-66-5 (pull the real GCP billing export + price the chosen VPS to harden §3), OQ-66-7 (B2 self-host-Supabase crux + AGPL-pullmd conveyance — product/legal, deferred).