S299 staging smoke — findings + carries

S299 staging smoke — findings + carries

Live Path-A+B smoke on the Coolify staging stack (turayklvaunphgbgscat), S299. Ledger-free session — this doc carries the findings for the s299a ledger session / next session to fold into the ledgers. Companion: v1-completion-sequence.md, pullmd-cocoindex-intent-s299.md, /tmp/claude/s299-smoke/runbook.md.

Run 1 — op_id dd2306bc-461e-4fd7-b28c-8a7890a8c32e

Corpus on cocoindex-state-staging: path-a/short-clause.md (.md) + charnwood/itt-services.docx (.docx) + .kh-workspace-map.json (maps charnwood/ → workspace b0000000-…01). COCOINDEX_SOURCE_PATH=/cocoindex-state/corpus → rel_paths have corpus/ stripped (the runbook A.6 PKs were computed with corpus/ in the seed — wrong; actual rel_path = path-a/short-clause.md).

✅ Path-A content (passthrough .md) — GREEN on staging

short-clause.md → source_documents=1, content_items=1 (id cc28dd04-680d-59d2-9936-71897611e397), content_chunks=1, entity_mentions=2. Re-validates the S297 prod proof on staging; confirms Stage 2 passthrough + Stage 3/4/5 writes against the staging Supabase.

🔴 FINDING-1 — Docling conversion broken (bytes vs Path); blocks ALL docx/pdf/xlsx + Path-B forms

scripts/cocoindex_pipeline/adapters.py:101 _docling_to_markdown: converter.convert(content_bytes) passes raw bytes, but this Docling version’s DocumentConverter.convert() requires a Path or DocumentStream → pydantic ValidationError (input_value=b'PK\x03\x04…' = the .docx zip magic). The .docx never converted → 0 content rows AND 0 form rows for it (form_templates pipeline=0). This was the first-ever live exercise of the Docling path (S297 prod smoke was .md passthrough), so the first Path-B smoke did its job and surfaced it. Mocked tests missed it (Docling mocked).

• Impact: blocks Path-B (forms) + all office-doc content ingestion → load-bearing for re-ingest.
• Fix: wrap bytes in docling … DocumentStream(name=<filename>, stream=BytesIO(bytes)) (thread the filename into _docling_to_markdown), or write a temp file and pass its Path. Code change → image rebuild. Batch with the 42.10/.url work next session (one rebuild), then re-smoke Path-B.
• Dispatch: code-executor — gitnexus impact on adapters.py first; add a test that exercises REAL bytes (not a mocked Docling) so the regression can’t recur.

🔴 FINDING-2 — pipeline-run webhook HTTP 401 (run-tracking can’t land)

The flow POSTs run records to PIPELINE_RUN_WEBHOOK_URL at in_progress (start) and completed (end); both returned HTTP 401 with an “Authentication Required” HTML page → pipeline_runs rows never land (0 for this op). Separate from the in_progress constraint fix (which is correct + applied to staging). The webhook URL hits an auth-protected endpoint (likely Vercel deployment protection on the staging app, or a missing/wrong auth header/secret).

• Impact: run-tracking + the {66.15} stall-monitor are blind on staging until fixed; it also masks end-to-end validation of the in_progress constraint fix.
• Fix: {66.13} — set the correct staging PIPELINE_RUN_WEBHOOK_URL target + auth (Vercel protection-bypass token, or CRON_SECRET, whichever the /api/internal/pipeline-runs/record route expects). The runbook had flagged this URL as “leave empty until {66.13}” — it is set but auth-walled.

🟡 FINDING-3 — workspace_resolution logs ERROR for unmapped content paths (noisy, non-fatal)

The .md (under path-a/, not in the manifest) logged cocoindex.stage_error stage=workspace_resolution error_class=extraction_validation_failed “No manifest mapping prefixes rel_path=‘path-a/short-clause.md’ (Inv-5: unmapped path)” — yet the content rows landed (content is workspace-agnostic). So this is an ERROR-level log for every content file not under a form-mapped prefix. During re-ingest (mostly content, few forms) this fires per content file. Consider downgrading to a soft-warn when no form resolves. Minor observability nit.

in_progress constraint fix — DONE on staging; carry to prod

Migration supabase/migrations/20260602163234_pipeline_runs_status_check_add_in_progress.sql applied to staging only (constraint now ['running','in_progress','completed','completed_with_errors','failed']; in_progress INSERT verified). bl-166 also fixed (route.ts:151 downcast removed; tsc clean; 89 tests pass). Commit fb73cf1f on worktree-agent-a41193115d107d9ce.

• TODO: cherry-pick onto canonical-pipeline-setup + push to main (prod rollout) — batch with the next-session Docling+.url rebuild push (prod also emits in_progress, so prod needs this constraint).
• Ledger: bl-166 → resolved.

Latent gap (flagged, out of scope) — cancelled status

lib/query/fetchers.ts:195-200 + the markdown-orchestrator reference a cancelled pipeline_runs status NOT in the constraint (0 live rows). If a cancel path ever writes it directly, it’d be rejected. Backlog note.

42.10 / URL re-ingest — deferred to next session

• 42.10 cannot pass on current code (cocoindex localfs-only; adapters.py:69 hands PullMD a local path it can’t read). See pullmd-cocoindex-intent-s299.md + /tmp/claude/s299-smoke/run2-42.10-plan.md.
• LIAM STEER (load-bearing): URL/RSS feeds are remote (intelligence + future “research” workspaces) — do NOT default to a localfs .url pointer-file; evaluate a proper cocoindex remote/feed source. Separate batch-reingest (ID-45) from ongoing-feed-ingestion. Captured in pullmd-cocoindex-intent-s299.md §4.0.
• Next session: spec the URL re-ingest mechanism (Liam-owned design) + fix Docling (FINDING-1), batch into ONE cocoindex image rebuild + re-smoke Path-B + 42.10.

State at close

• Staging app STOPPED (burn-safe). Corpus remains on cocoindex-state-staging. COCOINDEX_SOURCE_PATH left set to /cocoindex-state/corpus (re-smoke = redeploy a Docling-fixed image). Prod idle (sha-c3ebf3c5).
• in_progress migration live on staging only (prod rollout pending the batched main-push).