Skip to content
Canonical

V1 Sequencing — S308 carry / pick-up doc

V1 Sequencing — S308 carry / pick-up doc

Section titled “V1 Sequencing — S308 carry / pick-up doc”

⚠ SUPERSEDED (S309). All carried items landed — ID-76 done (migration #106 live on staging+prod), bl-222/bl-218 done, ID-66.14/66.15 done, bl-221 spec authored. The deltas are folded into v1-completion-sequence.md (the live forward map). This file is retained as a point-in-time S308 record only.

Purpose. S308 pushed the held canonical branch to main + aligned staging, closed ID-66.11/66.12, root-caused 66.13, and ratified ID-76. Several items carry to the next session. This doc is the executable pick-up list: once the carried items + the named verifications below are complete, then fold the deltas into v1-completion-sequence.md (deliberately NOT edited this session — it should reflect a settled state, and these items are mid-flight). Pair with the S309 continuation prompt.

No secrets in this file (repo is on the ID-68 path to public). Live secret values (CRON_SECRET, the Vercel bypass token, etc.) live only in the Coolify staging/prod env — read them with mcp__coolify__env_vars reveal=true when needed.

Authored: S308 (2026-06-04).

0. What S308 settled (DONE — no carry)

Section titled “0. What S308 settled (DONE — no carry)”
  • Branches aligned at 2f70f766. canonical-pipeline-setup → main (FF) → staging (FF) all pushed to the same tip. The held-branch hold is released. Staging was a pure ancestor (~1200 commits stale, S53-era) — the align is a clean FF, nothing lost.
  • Pandoc durability caveat RESOLVED. main now carries the pandoc layer (4c9f27f2) + pypandoc_binary in requirements.txt, so the onprem build off main is pandoc-capable. The main push triggered the onprem cocoindex rebuild (see §3 monitor).
  • Schema parity verified. Staging (turayklvaunphgbgscat) and prod (rovrymhhffssilaftdwd) DBs are both at all 105 migrations, byte-identical to the repo. No db push needed; the redeployed apps run cleanly against the live schema (the only app-coupled migration, pipeline_runs … in_progress, is already live on both).
  • ID-66.11 → done (deploy-automation prod-live). ID-66.12 → done (rollback-by-tag runbook section). See ledger journals.
  • 66.13 webhook-AUTH fixed — the Vercel-protection 401 is resolved by the bypass token (Liam re-added it to the staging Coolify PIPELINE_RUN_WEBHOOK_URL). Verified live: the synthetic POST went 401 → 404 (past protection).

1. Carried implementation items (executable next session)

Section titled “1. Carried implementation items (executable next session)”

A. ID-76 — make 'cancelled' a real pipeline_runs.status (RATIFIED: Option A)

Section titled “A. ID-76 — make 'cancelled' a real pipeline_runs.status (RATIFIED: Option A)”

Decision (Liam-ratified S308): cancellation → 'cancelled' always (pending AND mid-processing), superseding the D-8 completed_with_errors shortcut. Rationale: matches the “a value we expect to see” model; partial work is preserved in pipeline_runs.result regardless; stops spurious Sentry warnings on every user cancel; improves success-rate accuracy. Confirmed real bug: a pending cancel orphans the pipeline_runs row at 'running' forever (UI polls forever), table-wide across all 3 queue producers (markdown_batch, batch_reclassify, bid_draft_all).

The full plan (7 edits + tests) — verified blast-radius, single-caller writers:

  1. Migration — supabase migration new pipeline_runs_status_check_add_cancelled; mirror 20260602163234_…add_in_progress.sql (drop-if-exists + re-add CHECK) adding 'cancelled' to (running, in_progress, completed, completed_with_errors, failed). No search_path (no function). Verify on the LOCAL stack via supabase db reset (sandbox-off; TCP 127.0.0.1:54322 is sandbox-blocked). DO NOT db-push to staging/prod from the executor — orchestrator owns live DB application (and prod/staging are at 105; this would be #106).
  2. Pending-cancel orphan fix — app/api/jobs/[id]/cancel/route.ts: widen the job select to fetch payload (for pipeline_run_id); after the processing_queue UPDATE, when the pre-cancel status was 'pending', do a best-effort service-client UPDATE on pipeline_runs (status='cancelled', completed_at, error_message='cancelled before processing started') scoped by pipeline_run_id. Covers all 3 producers. Do NOT use recordPipelineRun (INSERT-only → would create a 2nd row).
  3. Orchestrator — lib/ingest/markdown-orchestrator.ts:~489: flip the cancelled branch 'completed_with_errors' → 'cancelled'; widen FinaliseRunParams.status (~733) + computeRunStatus unions + the finalProgress switch to admit 'cancelled'. Both single-caller (runImportPhase).
  4. batch_reclassify dispatch — lib/queue/dispatch.ts:~271-286: cancelled branch → 'cancelled'; widen the local union; change the inline Sentry guard (~328) so 'cancelled' does NOT alert (status === 'failed' || status === 'completed_with_errors').
  5. Producer type + Sentry guard — lib/pipeline/record-run.ts: add 'cancelled' to PipelineRunStatus (~41-45); guard (~252) → if (status === 'completed' || status === 'in_progress' || status === 'cancelled') return; (cancelled is silent).
  6. Route zod — app/api/internal/pipeline-runs/record/route.ts PipelineStatusSchema (~58-63): NO change (cocoindex never emits 'cancelled'; don’t imply a capability that doesn’t exist).
  7. List filter zod — lib/validation/schemas.ts:1900 PipelineRunsParamsSchema.status: ['running','completed','failed','cancelled'] → add the two missing real values → ['running','in_progress','completed','completed_with_errors','failed','cancelled']. Already correct (no edit): fetchers.ts:195-200 row type + schemas.ts:3141 PipelineRunRowSchema + upload-tab-content.tsx:277-303 UI all already handle 'cancelled'.

Tests: cancel-route tests (assert the pending-cancel pipeline_runs UPDATE); markdown-orchestrator test (cooperative-cancel status completed_with_errors → 'cancelled', .result still carries partial summary); batch_reclassify dispatch test (status + no Sentry on cancel); record-run test (status:'cancelled' emits no Sentry.captureMessage); pipeline-runs param test (?status=completed_with_errors + ?status=in_progress now valid).

Also: correct the ID-76 title — drop the stale (bl-218) cross-ref (bl-218 is the unrelated “cocoindex idle-boot test skipif guard”). ID-76 carries its own problem statement.

Execution note: the harness isolation:"worktree" was broken this session (phantom dir; bash → main repo, Write/Edit → split root). Implement in the main session (direct), OR via a manually-created git worktree add with the agent using absolute paths + git -C. Either way checker-gate the result (the user wants task-checker verification of code work — a checker would have caught the runbook app-name slip).

B. bl-222 — staging on-prem Coolify deploy arm in onprem-deploy.yml

Section titled “B. bl-222 — staging on-prem Coolify deploy arm in onprem-deploy.yml”

Unblocked: Liam added COOLIFY_API_TOKEN to the Staging GH environment (the only blocker). Build: add a staging deploy arm to .github/workflows/onprem-deploy.yml (current deploy-cocoindex is prod-only: push:[main], environment: Production, hardcoded uuid ybiczck7f7e1xbdev3bk89cr). Staging app uuid = ljurtiab99kb6dst38k76419, environment: Staging, COOLIFY_BASE_URL already on Staging GH. Checker-gate. Then bl-222 closes.

C. 66.13 staging row-landing — the re-test (a “necessary test”)

Section titled “C. 66.13 staging row-landing — the re-test (a “necessary test”)”

After the Vercel staging rebuild completes (poll GET …/api/internal/pipeline-runs/record?x-vercel-protection-bypass=<token-from-coolify-env> → 405 = route live; 404 = still old build), re-run the synthetic webhook POST (status completed, a labelled op_id) with Authorization: Bearer <CRON_SECRET-from-coolify-env> → expect HTTP 200 {ok:true} → then SELECT … FROM pipeline_runs WHERE pipeline_name LIKE 's308%' on staging confirms the row landed → 66.13 Part 1 fully closed. (Bypass token is already on the staging Coolify env; CRON_SECRET too.) Delete the test row after if desired. If the bypass param reverts again on a Coolify redeploy, that is a Coolify env-persistence quirk (NOT a git/push issue — the compose uses ${PIPELINE_RUN_WEBHOOK_URL} substitution with no git default/env_file); set it as a durable runtime env var.

D. 66.13 Part 2 — DNS/SSL (operator, NOT a cutover gate)

Section titled “D. 66.13 Part 2 — DNS/SSL (operator, NOT a cutover gate)”

For the ID-62 /stage inbound surface (NOT the outbound webhook, already HTTPS to Vercel; NOT the corpus-on-disk re-ingest). Both cocoindex apps have fqdn: null (live-confirmed). Steps: DNS A-record (e.g. pipeline.kh.phew.org.uk) → 77.68.122.71; set as the Coolify app FQDN → auto-Let’s-Encrypt; ensure 443 open on the IONOS cloud firewall (host ufw already allows 80/443). Confirm whether ID-62 actually needs it before the cutover, or defer.

2. Monitor / verify (deploys triggered by the S308 push)

Section titled “2. Monitor / verify (deploys triggered by the S308 push)”
  • onprem cocoindex prod rebuild (GH Actions onprem-deploy.yml, run on the main push): ~up to 75 min (Docling + Rust wheel). It rebuilds the prod image from main (= canonical content, pandoc-capable) and Coolify-deploys. Verify a healthy idle boot after (SOURCE_PATH="" on prod → burn-safe). Rollback = re-point Coolify COCOINDEX_IMAGE_TAG to a prior sha- tag (no rebuild). Image tag = sha-2f70f766….
  • Vercel prod app redeployed to current code (verified no hard dep on any unapplied migration). Vercel staging app rebuilding off the aligned staging branch (gains the cocoindex route — gates §1C).
  • docs-site deploy (main) FAILED on the push (gh run view 26945129201 --log-failed). Likely the known ID-9 docs-site CI fragility ({9.20}/{9.21} blocked). Investigate + characterise next session; not blocking the pipeline.
  • Dependabot: GitHub flags 4 default-branch vulns (1 critical, 2 moderate, 1 low) now that the branch is on main. Triage next session.

3. Before updating v1-completion-sequence.md

Section titled “3. Before updating v1-completion-sequence.md”

Update the V1 doc after the above settle, folding in: ID-66.11/12 done + the residual shape (66.13 observability-not-a-gate, 66.14/15 operator-gated builds carried); the branch-alignment + pandoc resolution; ID-76 ratified→done (once built); the verified headline that PullMD/ID-75 does NOT gate the re-ingest cutover (only 62.10 + new-URL ingest); and the 66.13 reclassification (run-record webhook is best-effort observability — the content re-ingest writes via COCOINDEX_DB_DSN regardless; prod observability proven via {66.16}).

4. Session learnings worth carrying

Section titled “4. Session learnings worth carrying”
  • Harness isolation:"worktree" is unreliable here — produced a phantom non-git dir; bash resolved to the main repo while Write/Edit split off. A dispatched executor correctly refused to edit the main repo. Prefer main-session direct-implement or a manual git worktree add + absolute-path discipline; read-only checker/investigation agents are unaffected.
  • The staging Vercel app was ~1200 commits stale (built from the staging git branch @ S53-55) — staging.vercel.app was serving ancient code. The staging-mirror sync had not run in a very long time. Aligned this session.
  • 66.13 was three layered problems, not one: Vercel protection (fixed by bypass) → stale staging app (no route → 404) → (DB was fine all along). The synthetic-POST probe (past-protection 401→404→405) is the fast way to localise which layer.
  • Schema parity was perfect (both DBs at 105) — de-risked the prod deploy entirely.
  • database.types.ts + lib/mcp/plugin-bundle.ts are sandbox read-denied (show as “Operation not permitted” in git status — harmless noise, not real changes).
  • Coolify env_vars reveal=true dumps ALL secrets to the MCP client — handle transiently, never echo into prose/commits/docs.