ID-66 close-out specs — DRAFT (S299, LEDGER-FREE)

ID-66 close-out specs — DRAFT (S299, LEDGER-FREE)

STATUS: DRAFT — NOT a ledger. This is a clean spec draft for the four remaining ID-66 close-out subtasks ({66.12}, {66.13}, {66.14}, {66.15}). It is authored ledger-free: this session did not edit docs/reference/task-list.json, did not run scripts/ledger-cli.ts, and did not touch any ledger. A concurrent session owns the ledger. A later session lifts the details / testStrategy / tags below into the four subtask rows of task 66 in docs/reference/task-list.json.

Why this draft exists: the four subtasks already carry design-intent details (the TECH-change framing). This draft upgrades them to implementation-ready specificity — exact file paths, env-var names, the exact mechanism an executor runs, the concrete uuids/paths from the live host, and the gotchas (including a blocking schema/code mismatch found during research — see {66.15} §Gotcha-A and the Cross-cutting flag at the bottom).

Authored: 2026-06-02 (S299). Sources read: .github/workflows/onprem-deploy.yml, deploy/coolify/docker-compose.{production,staging}.yaml, deploy/coolify/.env.staging.example, docs/runbooks/{onprem-b1-deploy,staging-coolify-cocoindex,cocoindex-deploy}.md, docs/specs/ID-66-onprem-pivot/TECH.md §§4–7, docs/themes/canonical-pipeline/reference/cocoindex-write-model.md §6, app/api/internal/pipeline-runs/record/route.ts, lib/pipeline/record-run.ts, lib/cron-auth.ts, scripts/cocoindex_pipeline/flow.py (_emit_pipeline_run_webhook :602–697, flow-start emit :2031, terminal emit :2329), scripts/cocoindex_pipeline/server.py (_health_handler :131–148), the pipeline_runs schema migrations, and a live staging pg_constraint introspection of pipeline_runs_status_check.

---

0. Shared host / infra constants (load-bearing — copy verbatim into the executor brief)

These are the live values an executor / operator needs; they are scattered across onprem-b1-deploy.md and cocoindex-write-model.md §6, consolidated here.

Constant	Value	Source
Host (prod)	root@77.68.122.71, SSH key ~/.ssh/kh_ionos_ed25519 (ed25519, no passphrase)	onprem-b1-deploy.md “Access”
Coolify dashboard	http://77.68.122.71:8000 (FQDN over 443 pending {66.13} Option B)	onprem-b1-deploy.md {66.6}
Coolify app (prod, git compose)	name kh-onprem-pipeline-production, uuid ybiczck7f7e1xbdev3bk89cr	cocoindex-write-model.md §6; onprem-deploy.yml COOLIFY_APP_UUID
COCOINDEX_IMAGE_TAG env uuid	l92p743mjory7qaopxqjx58e	cocoindex-write-model.md §6
COCOINDEX_SOURCE_PATH env uuid	rqjlptckuuqvt4q8nz0wd98t	cocoindex-write-model.md §6
ghcr image	ghcr.io/ai-solution-hub/kh-cocoindex-pipeline:sha-<40-char gitsha> (public; never :latest)	onprem-deploy.yml IMAGE/TAG
LMDB engine store (in-container)	/cocoindex-state/lmdb (COCOINDEX_DB)	compose cocoindex.environment.COCOINDEX_DB
LMDB engine store (on-host)	/var/lib/docker/volumes/ybiczck7f7e1xbdev3bk89cr_cocoindex-state/_data/lmdb	cocoindex-write-model.md §6 (smoke sequence)
Prod named volume	cocoindex-state (compose project kh-onprem)	docker-compose.production.yaml
Staging named volume	cocoindex-state-staging (compose project kh-onprem-staging)	docker-compose.staging.yaml
Prod app webhook target	https://kh.phew.org.uk/api/internal/pipeline-runs/record	CLAUDE.md “Production URL”; route path
Staging app webhook target	https://knowledge-hub-git-staging-tw-group.vercel.app/api/internal/pipeline-runs/record	.env.staging.example PIPELINE_RUN_WEBHOOK_URL
Coolify deploy webhook	GET {COOLIFY_BASE_URL}/api/v1/deploy?uuid=<app-uuid>&force=false (Bearer COOLIFY_API_TOKEN)	onprem-deploy.yml deploy-cocoindex
Coolify env PATCH	PATCH {COOLIFY_BASE_URL}/api/v1/applications/<app-uuid>/envs (PATCH = update-in-place; POST duplicates the key — footgun)	onprem-deploy.yml; cocoindex-write-model.md §6
GH secrets (Production env)	COOLIFY_API_TOKEN (exists, S294), COOLIFY_BASE_URL=http://77.68.122.71:8000 (operator-created)	onprem-b1-deploy.md operator checklist

Directory-convention note for the executor: the four subtask details name deploy/onprem/backup/lmdb-backup.sh and deploy/onprem/monitor/datapath-watch.ts (from TECH.md §§5–6, written before the host dir was created). The repo’s actual on-host deploy dir is deploy/coolify/ (compose + .env.staging.example live there; deploy/onprem/ does not exist). Recommendation: create the new scripts under deploy/coolify/backup/ and deploy/coolify/monitor/ to match the existing convention, OR create deploy/onprem/ if the orchestrator prefers the spec-literal path. This draft uses deploy/coolify/{backup,monitor}/ and flags the deviation; the lifting session should pick one and update the subtask details to match. Either way this is a path decision, not a mechanism decision.

---

{66.12} — Rollback-by-tag mechanics + runbook

• Dispatch shape: verify/runbook (doc deliverable + a verified manual procedure). The mechanism — Coolify env PATCH + redeploy — is already wired by the deploy-cocoindex job in onprem-deploy.yml; nothing new is built. The deliverable is a runbook section plus a one-time live verification an operator runs.
• Depends on: {66.11} (the deploy webhook / tag-bump job — already wired in code) and at least two pinned sha- tags existing in ghcr.io (i.e. ≥2 successful runs of onprem-deploy.yml). No dependency on {66.13}/{66.14}/{66.15}.
• Code touched: none (NO KH application code; NO workflow change).

details (draft)

Document, in docs/runbooks/onprem-b1-deploy.md (new {66.12} Rollback by tag section, sibling to the existing {66.11} section), the rollback = re-point the image tag, no rebuild procedure. Rollback and roll-forward are the identical operation with a different tag value; the only stateful artefact (LMDB) is unaffected by an image swap.

The procedure (two equivalent paths):

1. Coolify UI path (operator): in app kh-onprem-pipeline-production (uuid ybiczck7f7e1xbdev3bk89cr) → Environment Variables → edit COCOINDEX_IMAGE_TAG (env uuid l92p743mjory7qaopxqjx58e) to the prior pinned sha-<commit> tag → Save → Deploy. Coolify re-clones the git compose and pulls the prior tag (near-instant; the host never builds).
2. API path (scriptable / mirrors the Action):

   # PATCH (update-in-place — POST would create a DUPLICATE key, the §6 footgun)
   curl --fail-with-body -sS -X PATCH \
     "${COOLIFY_BASE_URL}/api/v1/applications/ybiczck7f7e1xbdev3bk89cr/envs" \
     -H "Authorization: Bearer ${COOLIFY_API_TOKEN}" \
     -H "Content-Type: application/json" \
     -d '{"key":"COCOINDEX_IMAGE_TAG","value":"sha-<PRIOR_COMMIT>","is_buildtime":true,"is_runtime":true}'
   # then trigger the pull-deploy
   curl --fail-with-body -sS -X GET \
     "${COOLIFY_BASE_URL}/api/v1/deploy?uuid=ybiczck7f7e1xbdev3bk89cr&force=false" \
     -H "Authorization: Bearer ${COOLIFY_API_TOKEN}"

Burn-safe by construction (carry the {66.11} guard into this section): a rollback PATCHes only COCOINDEX_IMAGE_TAG. It must never touch COCOINDEX_SOURCE_PATH (env uuid rqjlptckuuqvt4q8nz0wd98t) — leaving it empty boots the rolled-back image idle (no corpus walk → zero Anthropic calls), so a rollback can never re-arm the burn footgun. Cross-reference the {66.11} “COCOINDEX_SOURCE_PATH must be confirmed empty” warning.

Tag-discovery aid for the runbook: list available rollback targets with gh api /orgs/ai-solution-hub/packages/container/kh-cocoindex-pipeline/versions (or the ghcr UI), or read the onprem-deploy.yml run history (each run’s TAG=sha-<github.sha>). The image is public (made public S288 — Coolify pulls anonymously), so no registry creds are needed for the pull.

Satisfies invariant 15 (rollback re-points a tag with no host rebuild). NO KH application code changes; NO workflow change — onprem-deploy.yml already supports both halves of the op.

testStrategy (draft)

With two pinned sha- tags in ghcr.io, an operator PATCHes COCOINDEX_IMAGE_TAG from the newer tag to the prior tag and redeploys; the container reports the prior image’s tag within seconds, no build is triggered on the host (confirm via docker logs/Coolify deploy log showing a pull, not a build), the LMDB volume is untouched, and the runbook documents the exact UI + API steps.

---

{66.13} — Repoint inbound webhook env + DNS/SSL (staging + prod)

• Dispatch shape: operator (the load-bearing work is host-env edits + a DNS A-record + Coolify auto-Let’s-Encrypt — none doable from an agent terminal). Pair with a small verify/runbook add (the DNS/SSL steps + a git diff proof that no code changed).
• Depends on: {66.8} (compose declares the PIPELINE_RUN_WEBHOOK_URL + CRON_SECRET env passthrough — done) and {66.10} (CRON_SECRET injected via Coolify env — operator). Independent of {66.12}/{66.14}/{66.15}.
• Code touched: NONE. The webhook repoint is two host env vars only. Do not edit flow.py, route.ts, proxy.ts, or Vercel config (see Gotcha-A — they are already correct).

details (draft)

Part 1 — webhook env repoint (env-only, zero code): set the cocoindex container’s Coolify env so the best-effort run webhook reaches the app:

• PIPELINE_RUN_WEBHOOK_URL
  • prod app kh-onprem-pipeline-production: https://kh.phew.org.uk/api/internal/pipeline-runs/record
  • staging app (kh-onprem-staging): https://knowledge-hub-git-staging-tw-group.vercel.app/api/internal/pipeline-runs/record (already the placeholder in deploy/coolify/.env.staging.example).
• CRON_SECRET — the same value the app verifies (verifyCronAuth in lib/cron-auth.ts compares Authorization: Bearer <CRON_SECRET> against process.env.CRON_SECRET). Staging must use the staging app’s CRON_SECRET; prod must use the prod app’s. A mismatch → the route returns 401 and no pipeline_runs row lands (the webhook is best-effort, so the pipeline keeps running — the failure is silent except for the Python-side pipeline-run webhook returned HTTP 401 log line).

_emit_pipeline_run_webhook (scripts/cocoindex_pipeline/flow.py:639–640) reads both vars from os.environ; if either is unset it logs PIPELINE_RUN_WEBHOOK_URL or CRON_SECRET not set — skipping and returns (no row, no crash). So the env repoint is the entire activation.

Gotcha-A (NO code change — proof obligation): the route handler (app/api/internal/pipeline-runs/record/route.ts), its verifyCronAuth gate, its recordPipelineRun → pipeline_runs write, and proxy.ts (the /api/* bypass already covers /api/internal/*) are byte-for-byte correct as-is. The {66.13} acceptance test is partly a negative: a git diff must show no change to flow.py or the route. (See the separate Cross-cutting flag at the bottom for an in_progress constraint defect that is NOT part of {66.13} — it must not be “fixed” here; flag it, don’t touch.)

Part 2 — DNS + SSL (operator infra): provision the HTTPS ingress surface the ID-62 reachability hand-off ({66.16}) will later use:

1. Point a DNS A record (e.g. coolify.kh.phew.org.uk, or a dedicated pipeline-ingress host such as pipeline.kh.phew.org.uk) → 77.68.122.71. The exact hostname is an operator/Liam choice; record it in the runbook.
2. Set it as the resource/instance FQDN in Coolify so auto-Let’s-Encrypt issues a cert and the host serves the endpoint over HTTPS on 443. This is the Option-B hardening already sketched in onprem-b1-deploy.md {66.6}; doing it here also lets 8000 be re-closed at the IONOS provider firewall later.
3. Open 443 in the IONOS Cloud Panel cloud firewall if not already (host ufw already allows 80, 443).

Note: the app webhook itself terminates on Vercel (kh.phew.org.uk / *.vercel.app), which is already HTTPS — so Part 1 needs no host TLS. The host DNS/SSL (Part 2) is for the inbound host-facing ingress surface ID-62 needs (/stage etc., per server.py:_stage_handler), bundled into this subtask per TECH.md change 7 + OQ-66-8.

Satisfies invariants 8, 9 (only-env-change; a completed/failed run lands a pipeline_runs row of the same shape as Cloud Run). NO KH application code changes.

testStrategy (draft)

git diff shows no change to flow.py or the route handler; with PIPELINE_RUN_WEBHOOK_URL + CRON_SECRET set on the host, a supervised smoke run lands a pipeline_runs row (HTTP 200 from the route); the chosen host FQDN serves HTTPS with a valid Let’s-Encrypt cert (curl -I https://<fqdn> → 2xx/3xx, no cert error); a deliberate wrong CRON_SECRET yields a 401 + the Python HTTP 401 log line (proving auth is enforced).

---

{66.14} — LMDB mdb_copy → S3 backup job

• Dispatch shape: code-executor for the script (deploy/coolify/backup/lmdb-backup.sh — a self-contained bash artefact a task-executor can write + commit) + operator for the schedule + S3 credentials (a Coolify scheduled task / host cron, and the S3 bucket + access keys, which an agent terminal cannot provision) + verify/runbook for the restore-procedure doc.
• Depends on: {66.9} (the LMDB named volume cocoindex-state + mount at /cocoindex-state — done). Independent of {66.12}/{66.13}/{66.15}.
• Code touched: new file only (deploy/coolify/backup/lmdb-backup.sh); NO KH application code; backup path touches Supabase not at all (inv 20).

details (draft)

Create deploy/coolify/backup/lmdb-backup.sh — a hot-snapshot → S3 job. It exists because Coolify native S3 backups are database-only and do NOT cover a raw named volume (OQ-66-4) — verified via the database_backups MCP surface (scoped to database_uuid, not application volumes).

Mechanism (the load-bearing safety claim — inv 18):

1. mdb_copy -c <src> <dst> — mdb_copy copies through an LMDB read transaction; LMDB’s MVCC read-snapshot is immutable for the txn lifetime regardless of concurrent writes, so it produces a consistent hot snapshot with no quiesce / no container stop. The -c flag compacts and drops free pages. A naive cp/tar of the live data.mdb is NEVER used (torn-page risk). mdb_copy is a CLI tool shipped with LMDB (liblmdb / lmdb-utils package), not an importable symbol (TECH.md §“mdb_copy is a CLI tool”).
   • Source-path gotcha: mdb_copy’s source arg is the directory containing data.mdb/lock.mdb (the env dir) when the env was opened as a dir, OR the data.mdb file with -n if opened MDB_NOSUBDIR. cocoindex uses COCOINDEX_DB=/cocoindex-state/lmdb as a directory env → source is /cocoindex-state/lmdb, dest is a fresh dir (e.g. /snapshot). Confirm the MDB_NOSUBDIR flag empirically on first run (ls /cocoindex-state/lmdb → expect data.mdb lock.mdb).
2. tar (or restic) the snapshot and ship it to the same S3-compatible bucket Coolify already targets for its DB backups (so backup retention lives in one place). Object key suggestion: s3://<bucket>/lmdb/cocoindex-state/<env>/lmdb-<UTC-ISO8601>.tar.zst.

Three viable execution surfaces (the script must work in any; pick at schedule time):

• (a) Coolify scheduled task on the cocoindex app (scheduled_tasks MCP: resource=application, uuid=ybiczck7f7e1xbdev3bk89cr, command=<script>, frequency=<cron>, container=cocoindex). Runs inside the cocoindex container, which already mounts the volume at /cocoindex-state — but the buildpack image ships no mdb_copy, tar-zstd, or aws/restic (it ships only the Python pipeline). So surface (a) requires baking those tools into the image (rejected — bloats the slim image / breaks the Inv-9 boundary) or is not viable as-is. Flag: prefer (b)/(c).
• (b) Host cron + a tiny throwaway tooling container that bind-mounts the on-host volume path /var/lib/docker/volumes/ybiczck7f7e1xbdev3bk89cr_cocoindex-state/_data and runs an image that does carry mdb_copy + awscli (e.g. docker run --rm -v <vol>/_data:/cocoindex-state:ro -v /snapshot:/snapshot <tooling-img> lmdb-backup.sh). Recommended — keeps the cocoindex image slim and the backup tooling independent.
• (c) Host cron directly on the IONOS host if lmdb-utils + awscli/restic are installed on the host (the host is the operator’s to provision). Simplest; the script reads the on-host _data path directly.

The script must be surface-agnostic: take LMDB_SRC (default /cocoindex-state/lmdb), SNAPSHOT_DIR (default mktemp -d), and S3 target + creds from env (BACKUP_S3_BUCKET, BACKUP_S3_PREFIX, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ENDPOINT_URL for non-AWS S3, AWS_REGION); fail loudly (set -euo pipefail), and clean up the temp snapshot in a trap.

Schedule (operator): B1 write volume is low and the LMDB is small (≤256–512 MiB historically) → a daily (or twice-daily) cron is ample. Retention is the operator’s choice within the 100 GB IONOS backup allowance (OQ-66-1). Record the chosen cron + retention in the runbook.

Restore procedure (document in onprem-b1-deploy.md, new {66.14} section): (1) stop the cocoindex container (Coolify Stop / docker stop); (2) fetch the latest S3 snapshot tar; (3) untar onto the persistent volume so the env dir lands at /cocoindex-state/lmdb (on-host: /var/lib/docker/volumes/ybiczck7f7e1xbdev3bk89cr_cocoindex-state/_data/lmdb); (4) restart the container. The restored store is a valid cocoindex memo state. Fallback: if the snapshot is stale/unavailable, the cost is a memo-state rebuild on the next ingest (inv 7) — slower, never KH data loss (Supabase is canonical, inv 20).

Satisfies invariants 17, 18, 19, 20. NO KH application code changes.

testStrategy (draft)

A scheduled run lands an LMDB snapshot tar in the S3 bucket, separate from Coolify’s DB backup (inv 17); a snapshot taken during active writes (run the script while a supervised smoke ingest is mid-flight) restores to a consistent, openable LMDB — verify by mdb_stat/opening the restored env, or by restoring it and confirming the cocoindex container boots and reports memo hits (inv 18); the documented restore path resumes the engine (inv 19); the backup path issues zero Supabase calls (inv 20).

---

{66.15} — Datapath monitor — pipeline_runs stall-watch

• Dispatch shape: code-executor for the monitor (deploy/coolify/monitor/datapath-watch.ts — a standalone read-only Supabase query a task-executor can write + commit to a clean test) + operator for the schedule (Coolify scheduled task / host cron) and the alert-channel wiring (Coolify Discord/Telegram/email webhook, or a Sentry DSN). Carries a blocking prerequisite — see Gotcha-A.
• Depends on: {66.8} (the pipeline emits pipeline_runs rows via the webhook — needs {66.13}‘s env repoint to actually land rows) and the Cross-cutting in_progress constraint fix (see bottom — without it, the stall-watch’s primary signal, an in_progress row that never goes terminal, can never exist, because the row insert is rejected at the DB). Independent of {66.12}/{66.14}.
• Code touched: new file only (deploy/coolify/monitor/datapath-watch.ts); NO KH pipeline/app code change (the monitor only reads Supabase). DO NOT modify the pipeline. The constraint fix is a separate migration, flagged below — not part of this monitor file.

details (draft)

Create deploy/coolify/monitor/datapath-watch.ts — a standalone, read-only stall-watch. Core premise (RESEARCH §4-i; server.py:_health_handler :131–148): “container up + /health green” is INSUFFICIENT evidence the datapath works — /health reflects only the cocoindex worker thread’s liveness (it returns 503 only if the worker thread crashed), not whether ingestion is producing rows. A successful ingestion is confirmed by pipeline_runs row arrival, never by the probe.

Monitor logic:

1. Connect to Supabase with the service-role key already present in the cocoindex Coolify env (SUPABASE_URL + SUPABASE_SERVICE_ROLE_KEY); the monitor is a service-role reader (bypasses RLS structurally, like the cron handlers). Reuse the pipeline_runs read shape from app/api/admin/pipeline-runs/recent/route.ts as the reference query (select pipeline_name, status, started_at, completed_at, op_id, error_message).
2. Stall conditions to flag (operator-tunable thresholds, recorded in the runbook):
   • (A) Stuck-in-flight: an op_id with an in_progress (flow-start) row but no terminal-status row (completed / completed_with_errors / failed) for the same op_id after STALL_THRESHOLD_MINUTES (default e.g. 30 min). This is the canonical “run stuck in_progress” signal. Requires the Cross-cutting in_progress constraint fix (Gotcha-A) — without it, no in_progress row ever lands, so condition (A) is dead.
   • (B) Terminal failure: a recent terminal row with status='failed' (or a run-rate of completed_with_errors above a threshold) within the window. recordPipelineRun already fires a Sentry error on failed and a warning on completed_with_errors, so (B) is partly covered by the existing alert path — the monitor adds a host-side cross-check / digest.
   • (C) Silence during an active-ingestion window: when ingestion is known to be active (operator sets a flag / a corpus is staged), no new pipeline_runs row (by created_at/started_at) within EXPECTED_RUN_WINDOW → flag a datapath stall (the “rows should be arriving but aren’t” case — distinct from idle mode, where no rows is correct).
   • Idle-mode guard: when COCOINDEX_SOURCE_PATH is empty (idle boot), no rows is the CORRECT state — the monitor must not alert on silence in idle mode (else it pages on every burn-safe idle deploy). Gate condition (C) on an explicit “ingestion expected” signal.
3. Alert routing: emit to a Coolify notification channel (Discord/Telegram/ email webhook URL via env, e.g. MONITOR_ALERT_WEBHOOK_URL) or Sentry.captureMessage. Keep it a single env-configurable sink so the operator picks the channel without a code change.

Config (env, recorded in runbook): STALL_THRESHOLD_MINUTES, EXPECTED_RUN_WINDOW_MINUTES, INGESTION_EXPECTED (bool gate for condition C), MONITOR_ALERT_WEBHOOK_URL (or SENTRY_DSN), plus the Supabase service creds.

Schedule (operator): a Coolify scheduled task or host cron every N minutes (default e.g. every 10–15 min). As with {66.14}, the buildpack cocoindex image ships no Node/tsx runtime, so run the monitor either on the host (node/bun/tsx) via cron, or as a tiny scheduled tooling container — not inside the cocoindex container. (If a TS runtime on the host is undesirable, an equivalent scheduled Supabase SQL — a pg_cron-style or external-cron psql query returning a boolean stall flag — is an acceptable alternative implementation per TECH.md §6; the TS file is the proposed default.)

Gotcha-A — BLOCKING schema/code mismatch (flag; do NOT fix inside this file): The live pipeline_runs_status_check constraint (confirmed by pg_constraint introspection on staging turayklvaunphgbgscat) is: CHECK (status = ANY (ARRAY['running','completed','completed_with_errors','failed'])) — it does NOT include in_progress. But the flow-start emit (scripts/cocoindex_pipeline/flow.py:2031–2038, status="in_progress") and the route’s PipelineStatusSchema (route.ts) do emit/accept in_progress, and recordPipelineRun does a plain INSERT (not an upsert). Result: every flow-start webhook insert currently violates the CHECK constraint and throws — caught by recordPipelineRun’s never-throws guard → a Sentry error fires, but no in_progress row ever lands. This silently breaks the monitor’s primary signal (condition A). Fix options (the lifting/owning session decides — this is a Cross-cutting item, see bottom): (i) a migration widening the constraint to include 'in_progress' (smallest, matches the code’s intent — note migrations auto-apply to prod on main push per cocoindex-write-model.md §6, so pair the constraint widen with a smoke); or (ii) change the flow-start emit to the already-permitted 'running' value and make the terminal emit an upsert on op_id (bigger, changes the row model). This draft recommends option (i) as the minimal correct fix. It is NOT part of the {66.15} monitor file — flag it for a separate migration subtask; do not edit flow.py/route.ts here.

Gotcha-B — not CocoInsight: CocoInsight is a dev-time LMDB inspector, not a prod stall-watch — do not conflate. A separate session scopes CocoInsight. This monitor reads pipeline_runs in Supabase, full stop.

Gotcha-C — started_at vs completed_at: pipeline_runs.started_at defaults to now() at insert; recordPipelineRun also stamps completed_at = now() on every insert (including the flow-start row), so completed_at is not a reliable “is this run still in flight” signal. Key the stall logic off op_id + status (terminal row present for the op_id?), not off completed_at being null.

Satisfies invariants 21, 22. NO KH application/pipeline code changes (monitor reads only; the constraint fix is a separate flagged item).

testStrategy (draft)

With containers up and /health green but no pipeline_runs rows in the expected active-ingestion window, the monitor flags a datapath stall (NOT “healthy”); an in_progress row with no terminal row past the threshold flags condition A (after the constraint fix lands); a real ingestion that lands a terminal row reports healthy on row arrival, not on the probe; in idle mode (empty COCOINDEX_SOURCE_PATH) the monitor stays quiet (no false stall). Unit-test the stall predicate against a mocked Supabase client (fixture rows: idle / in-flight / stalled / terminal-failure).

---

Cross-subtask dependency summary

{66.11} (done, wired) ──► {66.12} rollback-by-tag        [verify/runbook]
{66.8}+{66.10} ─────────► {66.13} webhook env + DNS/SSL  [operator (+runbook)]
{66.9} ─────────────────► {66.14} lmdb-backup.sh + S3    [code-executor + operator + runbook]
{66.8} (+{66.13} to land rows,
        +CONSTRAINT FIX) ► {66.15} datapath stall-watch  [code-executor + operator]

• {66.12}, {66.13}, {66.14} are mutually independent and can be done in parallel.
• {66.15} is gated on the in_progress constraint fix (Cross-cutting flag) for its primary signal, and benefits from {66.13} (so rows actually land), but the monitor file can be written + unit-tested independently of both.

Dispatch-shape table (which suits a code-executor subagent vs operator vs doc)

Subtask	Title	Dispatch shape	New artefact(s)	Who/what
{66.12}	Rollback-by-tag mechanics + runbook	verify/runbook	runbook section in onprem-b1-deploy.md	Operator runs a 1-time live tag re-point verify; doc the steps. No code, no workflow change.
{66.13}	Repoint inbound webhook env + DNS/SSL	operator (+ small runbook)	2 Coolify env values (staging+prod) + DNS A-record + Coolify auto-LE; runbook DNS/SSL note	Host/infra only. Code change = NONE (proof: git diff clean on flow.py/route).
{66.14}	LMDB mdb_copy → S3 backup	code-executor (script) + operator (schedule + S3 creds) + runbook (restore)	deploy/coolify/backup/lmdb-backup.sh; restore section in runbook	Executor writes/commits the bash artefact; operator schedules it (host cron / Coolify scheduled task) + provisions the S3 bucket/keys.
{66.15}	Datapath monitor — pipeline_runs stall-watch	code-executor (monitor) + operator (schedule + alert channel)	deploy/coolify/monitor/datapath-watch.ts (+ unit test); separate migration to widen the in_progress constraint	Executor writes/commits the read-only monitor + test; operator schedules it + wires the alert sink. Blocked-signal until the constraint fix lands.

Cross-cutting flag (NOT one of the four subtasks — surface to the orchestrator)

pipeline_runs.status CHECK constraint omits in_progress while the cocoindex flow-start emit + the /api/internal/pipeline-runs/record route both emit/accept it and recordPipelineRun does a plain INSERT. Confirmed live on staging (turayklvaunphgbgscat): CHECK (status = ANY (ARRAY['running','completed','completed_with_errors','failed'])). Today every flow-start (in_progress) insert is rejected at the DB → caught by the helper’s never-throws guard → a Sentry error fires but no flow-start row lands. This is a pre-existing defect (predates ID-66; surfaced here because {66.15}‘s stall-watch depends on the in_progress row existing). Recommended fix: a small migration widening the constraint to ARRAY['running','in_progress','completed','completed_with_errors','failed'] (matches the code’s intent and the route’s PipelineStatusSchema). Caveat: migrations auto-apply to prod on main push via the Supabase GitHub integration (cocoindex-write-model.md §6) — pair the widen with a pg_constraint re-introspection and a supervised smoke. This is flagged, NOT fixed by this session (ledger-free + the brief says flag existing-symbol changes, do not edit). Suggest the lifting session file it as a new ID-66 subtask (e.g. {66.x} “widen pipeline_runs status constraint for in_progress”) that {66.15} depends on, or fold the migration into {66.15}‘s brief as its first slice.