Skip to content

PLAN — Fixture-staging live-verification infra — co-located /stage route + localhost verify driver (B1 on-prem) (ID-62.4)

PLAN — Fixture-staging live-verification infra — co-located /stage route + localhost verify driver (B1 on-prem) (ID-62.4)

Section titled “PLAN — Fixture-staging live-verification infra — co-located /stage route + localhost verify driver (B1 on-prem) (ID-62.4)”

Task: ID-62 — Fixture-staging live-verification infra. Subtask: {62.4} PLAN. Predecessors (read in full): {62.2} PRODUCT (docs/specs/id-62-fixture-staging-infra/PRODUCT.md, 29 invariants Inv-1..29, ratified S283 under B1) + {62.3} TECH (docs/specs/id-62-fixture-staging-infra/TECH.md, Slices A/B/C′/D/E′

  • {42.10}, 1:1 per-invariant verification map, ratified S283) + the ratified directive set docs/specs/id-62-fixture-staging-infra/P4-RECONCILIATION.md (§4 net shape) + the B1 host model docs/specs/ID-66-onprem-pivot/RESEARCH.md (§1-a..f). Author date: 29/05/2026 (S283).

SUPERSEDES the S279 P2 “in-VPC Cloud Run Job + operator entry-gate {62.0}” PLAN. This is a wholesale re-spec, not an amendment. The prior PLAN decomposed an in-VPC Cloud Run Job verify mechanism (old {62.7}), a kh-pipeline-image reuse + cnb-launcher entrypoint, a cocoindex-live-verify.yml doing WIF + gcloud run jobs execute, and an operator entry-gate {62.0} (curl internal /health from a throwaway Job) carried as a blocking note in every Subtask. The S282 ratification of Option B1 (co-locate cocoindex + pullmd on one on-prem host — ID-66 RESEARCH) dissolved that design’s organising spine. The whole Cloud-Run-Job / WIF / cnb-launcher / entry-gate block is deleted, not relabelled. Git history preserves the old version. Cross-reference: P4-RECONCILIATION §1 (WHOLESALE verdict), §4 (B1 net shape).

This is the {62.4} artefact authored by a fresh Planner instance (Q-PLANNER-2 — NOT the {62.1}/{62.2}/{62.3} author). It formalises TECH’s {62.4} slice outline into TM-shape Subtask records. It does NOT relitigate the ratified basis (B1 co-located host; localhost reachability; /stage-load-bearing default per OQ-62-P4-2; /health kept per OQ-62-P4-3; entry gate deleted; in-repo Python verify-driver module per OQ-62-TECH-11; co-located driver / host Vitest for the {42.10} Inv-9 + Tier-3 home per OQ-62-TECH-12). It does NOT author code and does NOT edit task-list.json — the Orchestrator writes the {62.5+} records from §6 below into docs/reference/task-list.json.


Six implementation Subtasks {62.5}{62.10}, sibling-only dependencies within Task 62. There is NO entry-gate Subtask and NO entry-gate blocking note — the OLD PLAN’s operator precondition ({62.0}, OQ-62-9: curl internal /health from a Cloud Run Job) is deleted because it is moot under B1 (no internal-ingress Service to probe, no Cloud Run Job — P4-RECONCILIATION §1, §4 item 7; TECH §“DELETED (not carried)”). It is replaced by the B1 dual implementation-gating precondition (§4 below), carried as a blocking note in every {62.5+} details field.

This honours the §3.3 sibling-only constraint: no {62.M} depends on any other Task’s Subtask. Two Task-level edges (NOT Subtask cross-deps) are recorded by the Orchestrator in task-list.json: (a) ID-62 depends on ID-66 (the B1 host must exist — PRODUCT §ID-66-precondition / OQ-62-P4-4); (b) ID-42 depends on ID-62 (ID-62’s primitive unblocks {42.10} at the Task level — TECH closing §).

Subtask → TECH slice → PRODUCT-invariant map

Section titled “Subtask → TECH slice → PRODUCT-invariant map”
SubtaskTECH slicePRODUCT invariantsSibling deps
{62.5} POST /stage routeSlice AInv-1, Inv-2, Inv-3, Inv-4, Inv-5, Inv-6, Inv-8none
{62.6} corpus activation + manifest seedSlice BInv-15, Inv-16, Inv-17, Inv-24{62.5}
{62.7} co-located verify driverSlice C′Inv-7, Inv-8, Inv-9, Inv-10, Inv-11, Inv-12, Inv-13, Inv-14{62.5}
{62.8} stageFixture JSON→multipartSlice DInv-2, Inv-19, Inv-20, Inv-22, Inv-25, Inv-26{62.5}
{62.9} B1 trigger + Vitest go-liveSlice E′Inv-18, Inv-20, Inv-28, Inv-29{62.6}, {62.7}, {62.8}
{62.10} {42.10} parameterised invocation{42.10}Inv-21, Inv-22, Inv-23, Inv-27{62.9}

Count: 6 of 25 Subtasks — well within the §3.4 soft ceiling. No Task-boundary split is signalled. TECH outlined Slice E′ + the {42.10} invocation as one combined slice; this PLAN splits them into {62.9} (the ID-62 trigger + host Vitest go-live) and {62.10} (the {42.10} HTML invocation) because they have distinct invariant sets (Inv-18/20/28/29 vs Inv-21/22/23/27), {62.10} adds the Inv-23 localhost pullmd round-trip on top of the driver, and {62.10} is the ID-42 Task-level payoff the Orchestrator may want to dispatch / verify independently. The split keeps each Subtask’s details load-bearing without one oversized brief.

AGPL constraint home: the AGPL image-isolation constraint (TECH-CONSTRAINT-AGPL, mapped to PRODUCT Inv-8) lands in two Subtasks: {62.5} (the /stage handler adds NO pullmd/Playwright dependency, so the cocoindex image stays pullmd/Playwright-free) and {62.7} (the verify driver asserts the AGPL boundary held + flags the B1-deploy-path image-inspect equivalent for ID-66 to wire). See §3 carry 4 and the two details fields.


2. PLAN-level decisions (carried from TECH by reference — NOT re-opened)

Section titled “2. PLAN-level decisions (carried from TECH by reference — NOT re-opened)”

All PLAN-level OQs were RESOLVED in {62.3} TECH §“Resolved TECH-level OQs”. This PLAN carries them by reference and bakes the consequences into the relevant Subtask details; it does not re-open them.

  • OQ-62-11 (verify-driver runtime) — RESOLVED in TECH §C′. A thin in-repo Python module scripts/cocoindex_pipeline/verify_driver.py, run on the B1 host (python3 -m scripts.cocoindex_pipeline.verify_driver). NOT a Cloud Run image, NO cnb-launcher, NO Job manifest. Reads fixtures from the host’s repo checkout (docs/testing/test-data/**). Carried into {62.7} details. (Recorded as OQ-62-TECH-11 in OQ-pending.md; default = in-repo module.)
  • OQ-62-12 ({42.10} Inv-9 + Tier-3 home) — RESOLVED in TECH §C′/§E′/§{42.10}. The co-located driver / B1 host runs them over localhost: the Inv-9 /s/<id> round-trip is one GET http://localhost:3000/s/<id> within the driver invocation; the Tier-3 topology files go live on the host’s Vitest step where COCOINDEX_STAGING_URL resolves to localhost. Carried into {62.9}/{62.10} details. (Recorded as OQ-62-TECH-12; per-file live-vs-defer is {62.9}’s call.)
  • OQ-62-P4-2 (/stage HTTP vs direct disk-drop) — RESOLVED in TECH. /stage HTTP over localhost is the load-bearing default; direct disk-drop is a non-default, explicit---disk-drop-flag-gated fallback the driver MAY omit. Carried into {62.7} details. (Recorded as OQ-62-P4-2 in OQ-pending.md.)
  • OQ-62-P4-3 (keep /health?) — RESOLVED in TECH. KEEP the cheap GET /health endpoint as an optional B1 container-liveness hook; PRODUCT Inv-6 is a /stage-must-not-break-/health regression note, not a probe requirement. Carried into {62.5} details.
  • OQ-62-6 (titlePrefix injection) — RESOLVED in TECH. Filename-derived title, NO in-byte injection; the caller embeds TEST_PREFIX in the dest filename; the form tier polls name/storage_path via matchStoragePath. Carried into {62.5}/{62.8} details.
  • OQ-62-7 (wire encoding) — RESOLVED in TECH. multipart/form-data (one file part + text parts destPath, titlePrefix), NOT base64-in-JSON; StageFixtureArgs keeps fixturePath unchanged (helper reads bytes runner-side). Carried into {62.5}/{62.8} details.

3. Load-bearing sequencing carries (from TECH + the brief, grounded at source)

Section titled “3. Load-bearing sequencing carries (from TECH + the brief, grounded at source)”
  1. /corpus/.kh-workspace-map.json MUST exist at process start BEFORE app_main arms the watch. flow.py:app_main (1523–1867) reads COCOINDEX_SOURCE_PATH (1536), loads source_path/".kh-workspace-map.json" (1582), and on a missing/unparseable/schema-invalid manifest emits a structured manifest_missing/manifest_invalid stage error (1586–1595) + a failed pipeline-run webhook (1596–1605) then raises (1606) — aborting the WHOLE flow. The watch arms once at app_main; today the value is "" → idle return (1537–1543). Consequence: corpus activation ({62.6}) is NOT merely “set COCOINDEX_SOURCE_PATH=/corpus” — server.main() must guarantee a schema-valid manifest exists at the corpus root before start_cocoindex_thread() spawns the worker, or EVERY fixture produces zero rows. Baked into {62.6} details as the FIRST step within the slice; {62.6} pins the exact minimal-valid shape from the load_workspace_manifest validator.

  2. The mkdir -p + manifest seed belongs in server.main() (Python), at the pre-thread-spawn step. TECH §B places mkdir -p ${COCOINDEX_SOURCE_PATH} + the manifest seed inside server.main() (234–255), BEFORE start_cocoindex_thread() (252). On B1 there is no Cloud Run manifest and no buildpack cnb-launcher PATH trap (those were Cloud-Run-era; the OLD PLAN’s launcher reasoning is DELETED), but the placement reasoning still holds: the seed lives in the Python the container already runs, not a shell entrypoint wrapper. {62.6} details mandates this placement.

  3. Tier-3 topology tests reach localhost on the B1 host — NO skip-on-GH-runner gymnastics. Under B1 the Vitest go-live step runs ON the host, so COCOINDEX_STAGING_URL resolves to localhost and the 8 Tier-3 files (health-probe, sidecar-cold-start, sidecar-mime-coverage, sidecar-version-metadata, stage-topology, latency-budget, transient-retry, audit-log-shipping) go live rather than skipping (TECH §E′ Tier-3 reframe — REDUCED under B1). {62.9} details mandates a per-file live-vs-defer decision (OQ-62-TECH-12), never a silently-green skip while claiming the topology tier is verified. The TECH §Testing “File → invariant tier map” is the per-file ledger.

  4. The AGPL boundary must survive the B1 deploy-mechanism change. The build-time cloudrun/cloudbuild-cocoindex.yaml:assert-inv9-no-pullmd-no-playwright (117–167) keeps pullmd/Playwright OUT of the cocoindex image. {62.5} keeps the image clean (the /stage handler imports only os/pathlib/uuid + already-present aiohttp.web). On B1, pullmd is a sibling container (ID-66 §1-b), NOT a cocoindex-image layer — the same container-isolation logic. {62.7} flags that the B1 push-to-deploy path (ID-66 §1-d) MUST wire an equivalent image-inspect check if it does not re-run the cloudbuild assertion, so the AGPL boundary is not silently lost when the deploy mechanism changes from Cloud Build to Compose/Coolify (this wiring is an ID-66 deploy-path concern; ID-62 asserts the constraint and flags it). The agpl-boundary.integration.test.ts (Tier 4, runs anywhere via image inspect) is the test-time backstop. Carried into {62.5} and {62.7} details.


4. IMPLEMENTATION-GATED precondition (replaces the deleted entry gate)

Section titled “4. IMPLEMENTATION-GATED precondition (replaces the deleted entry gate)”

The OLD PLAN’s operator entry-gate {62.0} (curl internal /health from a Cloud Run Job) is DELETED — moot under B1 (no internal-ingress Service to probe, no Cloud Run Job). It is REPLACED by the B1 dual implementation-gating precondition below, carried verbatim as a blocking note in every {62.5+} details field.

IMPLEMENTATION GATED — ID-62 implementation cannot begin until BOTH:

  • (a) this B1 re-spec is ratified by Liam (the {62.2} PRODUCT + {62.3} TECH + this {62.4} PLAN chain), AND
  • (b) ID-66 host stand-up is complete: the B1 host + Compose stack co-locating cocoindex + pullmd over localhost + a persistent local-disk volume hosting the corpus directory and the cocoindex LMDB all exist (ID-66 RESEARCH §1-a / §1-b / §1-c).

The verify driver and the corpus live ON the ID-66 host; without it there is nothing to run against. This is a Task-level dependency (ID-62 depends on ID-66 via Task.dependencies[], OQ-62-P4-4), consistent with the §3.3 sibling-only Subtask-dep constraint — it is NOT a Subtask cross-dependency. The Orchestrator records the edge in task-list.json; the Planner flags it.

What CAN proceed before (b): the Slice A /stage route ({62.5}) and the Slice D helper change ({62.8}) are host-agnostic — they pass python3 -m pytest scripts/tests/ and bun run test respectively without a live host. Only the live-tier go-live ({62.9}/{62.10} on-host run) and the corpus-activation verification ({62.6}’s live arming) wait on the ID-66 host. The Orchestrator MAY dispatch {62.5}/{62.8} ahead of the host once (a) holds; their details say so.


5. External-API verification (OQ-3 / Q-EX2)

Section titled “5. External-API verification (OQ-3 / Q-EX2)”

This PLAN introduces NO NEW third-party library symbols beyond those {62.3} TECH already empirically verified this session (29/05/2026, recorded in TECH §Verification):

  • aiohttp multipart symbols — aiohttp.web.Request.multipart, aiohttp.web.Request.post, aiohttp.MultipartReader, aiohttp.BodyPartReader.read, aiohttp.BodyPartReader.filename, aiohttp.web.json_response — all PRESENT against installed aiohttp 3.13.5 (pin aiohttp>=3.9.0,<4.0.0, requirements.txt:59).
  • cocoindex.connectors.localfs.walk_dirPRESENT against cocoindex[postgres]==1.0.3 (requirements.txt:44; call site flow.py:1683).

TS-side FormData/Blob/fetch (helper change {62.8}) are Web-standard Bun runtime built-ins — out of OQ-3 scope (framework/stdlib built-ins). StageFixtureArgs is internal KH (out of scope). The verify driver ({62.7}) uses stdlib + an already-pinned in-repo HTTP client (aiohttp/urllib/requests). No re-verification required at PLAN time; the {62.7}/{62.10} details carry the TECH verification block by reference so the Executor builds against the verified symbol set.

The OLD spec’s one infra import-and-call analogue — the internal→internal entry gate (old Inv-34 / OQ-62-9) — is DELETED, not carried (TECH §“DELETED”).

Code-intelligence orientation (PLAN-level, verbatim — per the planner-citation block)

Section titled “Code-intelligence orientation (PLAN-level, verbatim — per the planner-citation block)”
  • gitnexus_query({query: 'cocoindex server stage route corpus watcher verify driver fixture staging', repo: '…/knowledge-hub'}) — returned "processes": [], "process_symbols": [] (the aiohttp server surface is NOT modelled as an execution flow), but its definitions surfaced the load-bearing Python symbols this decomposition touches/extends, all confirming the PRODUCT/TECH grounding: Function:scripts/cocoindex_pipeline/flow.py:app_main (1523–1867), Function:scripts/cocoindex_pipeline/flow.py:ingest_file (1039–1489), Function:scripts/cocoindex_pipeline/flow.py:_emit_stage_error_log (363–396), and the Python test home Function:scripts/tests/test_cocoindex_server.py:aiohttp_app (144–154) + TestWorkerLiveness.test_health_503_when_worker_crashed (372–390). Out-of-index caveat: the TS staging helpers (stageFixture, pollContentItemsFor, dropFixture) are NOT in the GitNexus index (__tests__/integration/ is outside the indexed TS corpus) and did NOT appear — grounded by direct file read in PRODUCT/TECH, carried by reference here. …/knowledge-hub is a placeholder for the repo path (do not record the real machine path).
  • ast-dataflow / ts-morph — not applicable: server.py/flow.py/verify_driver.py are Python (ts-morph does not cover Python per .ast-dataflow/CLAUDE.md), and the one TS surface (fixture-staging.ts) is a test-corpus helper. The code-touching Subtasks’ details carry the discipline accordingly (gitnexus_impact/detect_changes for the indexed Python server symbols; direct read + grep for the Python verify-driver + the out-of-index TS helper).

6. Subtask records ({62.5}{62.10}) — TM-shape, for the Orchestrator to write

Section titled “6. Subtask records ({62.5}–{62.10}) — TM-shape, for the Orchestrator to write”

id is local to Task 62 (integer). dependencies are sibling-only (integers, other Subtasks of Task 62). description ≤250 chars and testStrategy ≤300 chars are HARD-GATED by the ledger CLI — every record below is authored within budget; all load-bearing narrative lives in the unbudgeted details field. The Orchestrator appends these to docs/reference/task-list.json under Task 62; this PLAN does NOT write the ledger.

  • title: Add co-resident POST /stage multipart byte-drop route to cocoindex server
  • description: Add _stage_handler + build_app() registration in server.py: read multipart body, loud-reject 400 on unset/missing COCOINDEX_SOURCE_PATH, write bytes under corpus root, reject path-escape, echo {destPath, requestId}.
  • details: IMPLEMENTATION GATED — ID-62 implementation cannot begin until BOTH (a) this B1 re-spec is ratified by Liam, AND (b) ID-66 host stand-up is complete (B1 host + Compose co-locating cocoindex+pullmd + persistent local-disk volume — ID-66 §1-a/b/c). NOTE: this Subtask is host-agnostic — it passes python3 -m pytest without a live host, so the Orchestrator MAY dispatch it once (a) holds, ahead of (b). The verify driver + corpus live on the ID-66 host. Edit scripts/cocoindex_pipeline/server.py: add async def _stage_handler(request) and register app.router.add_post("/stage", _stage_handler) in build_app() (153–161) alongside the existing add_get("/health", _health_handler) (160). Handler steps (TECH §Slice A): (1) read os.environ.get("COCOINDEX_SOURCE_PATH"); if unset/empty → web.json_response({"error": "COCOINDEX_SOURCE_PATH is unset"}, status=400); if set but not Path(p).exists() → 400 naming the path; 5xx reserved for an unambiguous server-side mount failure [Inv-5]; (2) reader = await request.multipart(); iterate parts; capture the file part bytes via await part.read(decode=False), the destPath + titlePrefix text parts; 400 if file or destPath absent (a path-only request with no bytes is rejected) [Inv-2]; (3) resolve target = corpus_root / destPath; reject path-escape (os.path.realpath(target) must be within realpath(corpus_root); reject ..//absolute) with 400 writing nothing; os.makedirs(target.parent, exist_ok=True); write bytes [Inv-3]; (4) web.json_response({"destPath": <written path>, "requestId": uuid.uuid4().hex}, status=200) [Inv-4]; (5) import only os/pathlib/uuid + already-present aiohttp.web — NO pullmd, NO Playwright, so the cloudbuild assert-inv9-no-pullmd-no-playwright (cloudrun/cloudbuild-cocoindex.yaml:117–167) stays green (TECH-CONSTRAINT-AGPL) [Inv-8]; do NOT touch _health_handler/worker_is_healthy/stage must NOT flip /health to 503 [Inv-6]. The route is registered on the SAME build_app() app the daemon thread runs inside (no second process/Service) [Inv-1]. OQ-62-6 carry: /stage performs NO in-byte title injection — the caller embeds TEST_PREFIX in the dest filename and ingest_file derives the title from the source path. OQ-62-7 carry: multipart/form-data, NOT base64-in-JSON. Tests in scripts/tests/test_cocoindex_server.py (existing aiohttp_app fixture 144–154 + aiohttp_app.router.resolve / make_mocked_request in-process pattern 158–176, no socket): assert the route table includes POST /stage; assert 400 on unset + on missing-dir COCOINDEX_SOURCE_PATH (monkeypatch env); assert a well-formed multipart write lands bytes at corpus_root/destPath (tmp_path corpus); assert path-escape 400; assert existing /health 200/503 tests still pass. Run python3 -m pytest scripts/tests/test_cocoindex_server.py. aiohttp multipart symbols verified PRESENT in TECH §Verification (29/05/2026, aiohttp 3.13.5) — build against that set. Code-intel: this is a Python symbol (ast-dataflow/ts-morph N/A — Python is not covered; use direct read + grep); build_app is indexed in GitNexus module Cocoindex_pipeline — run gitnexus_impact({target: "build_app", direction: "upstream"}) before editing and gitnexus_detect_changes() before commit.
  • testStrategy: python3 -m pytest scripts/tests/test_cocoindex_server.py green: route table includes POST /stage; 400 (named, not 5xx, not silent) on unset/missing COCOINDEX_SOURCE_PATH; multipart write lands bytes under the corpus root; path-escape rejected; /health 200/503 unchanged.
  • status: pending
  • dependencies: []

{62.6} — Corpus activation (COCOINDEX_SOURCE_PATH=/corpus + boot mkdir + manifest seed)

Section titled “{62.6} — Corpus activation (COCOINDEX_SOURCE_PATH=/corpus + boot mkdir + manifest seed)”
  • title: Activate cocoindex corpus watch on B1 persistent disk (/corpus + boot seed)
  • description: Set COCOINDEX_SOURCE_PATH=/corpus in the B1 Compose env; seed /corpus + a schema-valid .kh-workspace-map.json in server.main() before the worker spawns so app_main arms the watch instead of aborting; add a manifest-shape test.
  • details: IMPLEMENTATION GATED — ID-62 implementation cannot begin until BOTH (a) this B1 re-spec is ratified by Liam, AND (b) ID-66 host stand-up is complete (B1 host + Compose co-locating cocoindex+pullmd + persistent local-disk volume — ID-66 §1-a/b/c). The corpus lives on the ID-66 host’s persistent disk; live arming verification waits on (b). SEQUENCING (LOAD-BEARING, PLAN §3.1): app_main (flow.py:1523–1867) reads COCOINDEX_SOURCE_PATH (1536) then loads source_path/".kh-workspace-map.json" (1582) and emits a manifest_missing/manifest_invalid stage error (1586–1595) + a failed webhook (1596–1605) then raises (1606) on a missing/unparseable/schema-invalid manifest. The watch arms ONCE at app_main; today "" = idle return (1537–1543). So a schema-valid manifest MUST be present at the corpus root at process start, not merely before the first fixture, or EVERY fixture produces zero rows. Changes: (1) document COCOINDEX_SOURCE_PATH=/corpus in the B1 Compose env (ID-66’s Compose file — NOT a Cloud Run manifest), where /corpus is a directory on the host’s persistent local-disk volume hosting the cocoindex LMDB (ID-66 §1-c); the corpus survives container restarts — NO Filestore, NO ephemeral-tmpfs caveat, NO new volume design [Inv-15, Inv-24]. (2) scripts/cocoindex_pipeline/server.py main() (234–255) — BEFORE start_cocoindex_thread() (252): source = os.environ.get("COCOINDEX_SOURCE_PATH"); if set, os.makedirs(source, exist_ok=True) then, if Path(source)/".kh-workspace-map.json" is absent, write a minimal schema-valid manifest. Pin the EXACT minimal shape from the load_workspace_manifest validator (read the validator to confirm the minimal-valid fields; the degenerate-but-legal shape, e.g. a schema_version + empty mappings list, must parse without raising). The seed lives in the Python the container runs, NOT a shell entrypoint wrapper (PLAN §3.2) [Inv-17]. (3) scripts/tests/test_cocoindex_service_manifests.py (or the B1-Compose-shape test the slice chooses): assert COCOINDEX_SOURCE_PATH == "/corpus"; add a server.main() test (or extend test_cocoindex_server.py) that with COCOINDEX_SOURCE_PATH=<tmp> set, the pre-spawn block creates the dir + seeds a manifest that load_workspace_manifest parses without raising. Activation = container (re)start (the watch arms once at app_main); the failure mode is observable, never a false green — unset path → /stage loud-reject (Inv-5), invalid manifest → app_main abort → downstream Vitest poll times out diagnostically (Inv-16/Inv-18) [Inv-16]. Code-intel: main/app_main are Python symbols in GitNexus module Cocoindex_pipeline (ast-dataflow/ts-morph N/A — Python; use direct read + grep); run gitnexus_impact({target: "main", direction: "upstream"}) before editing + gitnexus_detect_changes() before commit.
  • testStrategy: python3 -m pytest scripts/tests/test_cocoindex_service_manifests.py scripts/tests/test_cocoindex_server.py green: COCOINDEX_SOURCE_PATH == /corpus, no corpus volume added; main() seeds /corpus + a .kh-workspace-map.json that load_workspace_manifest parses without raising.
  • status: pending
  • dependencies: [5]

{62.7} — Co-located verify driver (in-repo Python module, localhost /stage, stage-only)

Section titled “{62.7} — Co-located verify driver (in-repo Python module, localhost /stage, stage-only)”
  • title: Build co-located verify driver (verify_driver.py, localhost /stage, stage-only)
  • description: Add scripts/cocoindex_pipeline/verify_driver.py — a thin in-repo Python module, run on the B1 host, that POSTs fixture bytes multipart to localhost:<port>/stage, exiting 0 iff all 2xx. Stage-only; no Supabase/SQL/Vitest; no Job/WIF/secrets.
  • details: IMPLEMENTATION GATED — ID-62 implementation cannot begin until BOTH (a) this B1 re-spec is ratified by Liam, AND (b) ID-66 host stand-up is complete (B1 host + Compose co-locating cocoindex+pullmd + persistent local-disk volume — ID-66 §1-a/b/c). The driver runs ON the ID-66 host by construction. OQ-62-TECH-11 RESOLVED (TECH §C′): a thin in-repo Python module scripts/cocoindex_pipeline/verify_driver.py, invoked as python3 -m scripts.cocoindex_pipeline.verify_driver --fixtures <set> by the operator or a B1 push-to-deploy/operator hook (ID-66 §1-d / OQ-66-6). NO dedicated container image, NO cnb-launcher, NO Cloud Run Job manifest, NO WIF gcloud run jobs execute, NO Secret-Manager --set-secrets. Behaviour: (1) read the (fixture set) — a list of (fixturePath-in-repo-checkout, destPath, titlePrefix) tuples — from the host repo checkout (docs/testing/test-data/**; the host HAS the repo). For each, read the bytes and POST multipart (file part + destPath + titlePrefix text parts) to http://localhost:<port>/stage — the LOAD-BEARING DEFAULT per OQ-62-P4-2 (do NOT raw-write the corpus disk by default) [Inv-7, Inv-12]. Because it runs on the same host as the cocoindex container, the request reaches /stage over loopback / the host’s private container network — no ingress: internal to defeat, no 403 [Inv-7]. (2) Stages only — does NOT poll Supabase, does NOT run the invariant SQL probes, does NOT embed a copy of the Vitest assertions; no Supabase query originates from the driver process [Inv-8 stages-only half, Inv-9]. (3) Exit 0 iff every /stage POST returned 2xx; exit non-zero on any 4xx/5xx/timeout/connection-refused, surfacing the failing fixture + the /stage response status/body so the operator can diagnose without re-running [Inv-10]. Re-running re-stages (overwrite-on-name; the watcher re-processes on change); no clean-corpus precondition; cleanup (dropFixture) is the test layer’s responsibility, not the driver’s [Inv-11]. Secrets (the cocoindex localhost port; for {42.10}, the pullmd Bearer token + Supabase service-role creds the Vitest step needs) read from the B1 host secrets store (Coolify env / host secrets manager, ID-66 §1-f) — never echoed in logs; NO GCP Secret Manager [Inv-7 secrets half]. AGPL (TECH-CONSTRAINT-AGPL / PLAN §3.4): the driver does not bake pullmd into the cocoindex image; FLAG that the B1 push-to-deploy path (ID-66 §1-d) MUST wire an image-inspect equivalent of cloudbuild-cocoindex.yaml:117–167 (zero pullmd/Playwright in the cocoindex image) if it does not re-run the cloudbuild assertion, so the AGPL boundary is not silently lost when the deploy mechanism changes — this wiring is an ID-66 deploy-path concern; ID-62 asserts + flags it; agpl-boundary.integration.test.ts (Tier 4) is the test-time backstop [Inv-8 AGPL half]. Direct disk-drop (--disk-drop) is a non-default explicit-flag fallback the driver MAY omit (OQ-62-P4-2) [Inv-12]. Uses stdlib + an already-pinned in-repo HTTP client (aiohttp/urllib/requests) — no new external symbol (PLAN §5). Local check: python3 -m pytest scripts/tests/ — the stage-loop is unit-testable with a mocked HTTP client (assert exit-code semantics; assert NO Supabase/SQL symbol is imported). Code-intel: verify_driver.py is a NEW Python module (no upstream callers yet; ast-dataflow/ts-morph N/A — Python; use direct read + grep); run gitnexus_detect_changes() before commit to confirm only the expected new symbols land.
  • testStrategy: python3 -m pytest scripts/tests/ green for the verify-driver stage-loop: exit 0 iff all /stage 2xx; non-zero + failing-fixture log (status/body) on any failure; no Supabase/SQL import in the driver source; default codepath issues an HTTP POST to localhost /stage (not a raw fs write).
  • status: pending
  • dependencies: [5]

{62.8}stageFixture JSON→multipart wire-contract change

Section titled “{62.8} — stageFixture JSON→multipart wire-contract change”
  • title: Switch stageFixture from JSON body to multipart bytes (StageFixtureArgs unchanged)
  • description: In fixture-staging.ts, change stageFixture to read fixture bytes runner-side and POST a multipart file part (+ destPath/titlePrefix), dropping the JSON body. StageFixtureArgs, the throws, and the response read stay unchanged.
  • details: IMPLEMENTATION GATED — ID-62 implementation cannot begin until BOTH (a) this B1 re-spec is ratified by Liam, AND (b) ID-66 host stand-up is complete (ID-66 §1-a/b/c). NOTE: this Subtask is host-agnostic — it passes bun run test without a live host, so the Orchestrator MAY dispatch it once (a) holds, ahead of (b). OQ-62-7 RESOLVED in TECH: multipart/form-data (NOT base64-in-JSON) — binary .xlsx/.docx/.pdf fixtures. Edit fixture-staging.ts stageFixture (100–139): KEEP StageFixtureArgs (51–71) exactly as-is — callers compile unchanged (OQ-62-7) [Inv-26]. Replace the JSON body construction (115–119): read bytes at args.fixturePath runner-side (fs.readFile / Bun.file); build a FormData with a file part (Blob/File of the bytes, named basename(args.destPath)), a destPath text part, and a titlePrefix text part; fetch(endpoint, { method: 'POST', body: formData }) — DROP the explicit Content-Type: application/json header at 115 (fetch sets the multipart boundary). The behaviour invariant: bytes on the wire, not a path the writer can’t see [Inv-2, Inv-19]. KEEP verbatim: the unset-COCOINDEX_FIXTURE_STAGING_URL throw (105–108), the non-2xx throw (125–128), the { destPath?, requestId? } response read (131–136). Do NOT touch pollContentItemsFor (179–222), dropFixture (376+), or the form helpers (546+) — they run wherever the Vitest tier runs (on the B1 host under B1) against live Supabase (reachable from anywhere) [Inv-22]. OQ-62-6 carry (Inv-25): no /stage in-byte injection — the content tier matches ilike('title', '${titlePrefix}%') (199) on a filename-derived title; the form tier polls name/storage_path via matchStoragePath (559). CARRY (Inv-20): dropFixture’s entity_mentions cleanup stays best-effort (411–425, ID-49.5 deferred per S273 OQ-1) — confirm the entity_mentions table/FK shape when the live tier first produces rows so cleanup does not leak across runs (a note for the first live run, not a code change here). FormData/Blob/fetch are Bun built-ins (out of OQ-3 scope). Local check: bun run test (NOT bun test) — the typed contract must not break (StageFixtureArgs unchanged); add a helper-level unit asserting stageFixture sends FormData (no application/json header; a file part present) against a mocked fetch. Code-intel: TS test helper, NOT in the GitNexus index (__tests__/integration/ outside the corpus — TECH §Context); consumers are *.integration.test.ts (test corpus), so no gitnexus_impact/ast-dataflow callers run is required — the change is internal to the helper’s request construction; ground by direct read.
  • testStrategy: bun run test green: StageFixtureArgs unchanged (callers compile); stageFixture sends FormData with a file part and NO application/json header (asserted against a mocked fetch); env-gate + non-2xx throws + {destPath, requestId} response read preserved.
  • status: pending
  • dependencies: [5]

{62.9} — B1 trigger + host Vitest go-live + Inv-29 launch-flip doc item

Section titled “{62.9} — B1 trigger + host Vitest go-live + Inv-29 launch-flip doc item”
  • title: Add B1 trigger (operator/host hook) + host Vitest go-live + Inv-29 launch-flip doc
  • description: Add the B1 trigger artefact (operator/host hook replacing gcloud run jobs execute, with a commented launch-flip marker), run the cocoindex Vitest tier on the host with the COCOINDEX_* env block, and add the Inv-29 item to the sequencing doc.
  • details: IMPLEMENTATION GATED — ID-62 implementation cannot begin until BOTH (a) this B1 re-spec is ratified by Liam, AND (b) ID-66 host stand-up is complete (B1 host + Compose + persistent volume — ID-66 §1-a/b/c). The trigger + Vitest go-live run ON the ID-66 host; this Subtask’s live verification waits on (b). The B1 trigger REPLACES the OLD cocoindex-live-verify.yml’s WIF + gcloud run jobs execute — NO gcloud run jobs execute, NO WIF, NO Cloud Run Job in the trigger path [Inv-28]. KEEP the policy (topology-agnostic): on-demand, NOT inlined into the PR-blocking ci.yml integration job, NOT scheduled today; the existing PR-blocking integration job is unchanged [Inv-28]. Changes: (1) the B1 trigger artefact — a host runbook step (or a Coolify scheduled-task stub / thin workflow that SSHes the host, the slice chooses) that runs the verify driver (python3 -m scripts.cocoindex_pipeline.verify_driver) then the Vitest go-live step on the host; a non-zero driver exit fails BEFORE the Vitest step. Include a commented-out scheduled-cadence block with a # FLIP ON AT LAUNCH (ID-62 Inv-29) marker so the launch flip is a one-line uncomment [Inv-29]. (2) the Vitest go-live step — run the cocoindex Vitest tier (bun run test:integration or a cocoindex-scoped invocation, NOT bun test) ON the B1 host with the env block: COCOINDEX_STAGING_URL (= http://localhost:<port>), COCOINDEX_FIXTURE_STAGING_URL (= http://localhost:<port>), COCOINDEX_SOURCE_PATH (= /corpus), plus live-Supabase creds from the host secrets store — so the previously-skipped Tier-1/Tier-1b/Tier-2/Tier-4/Tier-5 tests EXECUTE and assert against live Supabase [Inv-18, Inv-20]. TIER-3 REFRAME (LOAD-BEARING, PLAN §3.3 / OQ-62-TECH-12): under B1 the Vitest step runs ON the host so COCOINDEX_STAGING_URL = localhost — the 8 Tier-3 files (health-probe, sidecar-cold-start, sidecar-mime-coverage, sidecar-version-metadata, stage-topology, latency-budget, transient-retry, audit-log-shipping) reach localhost and CAN go live (no skip-on-GH-runner gymnastics). Decide per file: EITHER go live on the host now, OR explicitly defer with a tracked note. Do NOT set the gate vars in a way that turns these green-by-skipping while claiming the topology tier is verified (TECH §Testing “File → invariant tier map” is the per-file ledger) [Inv-18 Tier-3 half]. Tier-1b (extract-memoisation, stage-5-failure-non-destructive, memo-hit-pipeline-run): confirm whether each needs a sibling Tier-1 stage to have run first (sequencing within the driver invocation). (3) Inv-29 action item: ADD “flip the live-tier verification to a scheduled/automatic cadence at launch” to the T13 “Pre-launch operational pre-decisions + observability” row of docs/themes/canonical-pipeline/reference/canonical-pipeline-sequencing.md (line 241) — Inv-29 is satisfied by the doc’s presence + the commented schedule: marker, NOT by wiring a live schedule now [Inv-29]. The staging happens via the driver ({62.7}); the host Vitest asserts produced rows only. Local check: any workflow/runbook YAML lints; bun run test still green (off-host the tier skips clean as designed — no localhost /stage/Supabase). Code-intel: no indexed-symbol edit (artefacts are YAML/runbook + a doc append + an env block); ground the test-tier file list by grep over __tests__/integration/cocoindex/*.integration.test.ts.
  • testStrategy: B1 trigger on-demand (operator/host hook, NO gcloud run jobs execute/WIF) with a commented launch-flip marker; ci.yml integration job unchanged; a host run runs the driver then the Vitest tier (Tier-1/1b/2/4/5 live; all 8 Tier-3 files decided, none silently-green); Inv-29 doc item.
  • status: pending
  • dependencies: [6, 7, 8]

{62.10} — Parameterised {42.10} HTML invocation (Inv-9 round-trip over localhost)

Section titled “{62.10} — Parameterised {42.10} HTML invocation (Inv-9 round-trip over localhost)”
  • title: Wire the {42.10} HTML/Inv-7-8-9 invocation through the same verify driver
  • description: Parameterise the verify driver with the {42.10} HTML fixture + Inv-7/8/9 set: it stages one HTML source via /stage; Inv-7/8 assert via host Vitest against live Supabase; the Inv-9 /s/<id> round-trip runs from the driver over localhost.
  • details: IMPLEMENTATION GATED — ID-62 implementation cannot begin until BOTH (a) this B1 re-spec is ratified by Liam, AND (b) ID-66 host stand-up is complete (B1 host + Compose co-locating cocoindex+pullmd over localhost + persistent volume — ID-66 §1-a/b/c); the Inv-9 round-trip needs pullmd as a localhost sibling (ID-66 §1-b). OQ-62-TECH-12 RESOLVED (TECH §{42.10}): the co-located driver runs the Inv-9 round-trip over localhost; the old in-VPC-Job split-home DISSOLVES (pullmd is a sibling container). The SAME verify driver + /stage route ({62.7}), invoked with the (HTML fixture, Inv-7/8/9 assertion set) parameterisation, produces the {42.10} proof — ONE primitive parameterised by (fixture set, assertion set); no HTML-specific staging branch [Inv-21]. Changes: (1) extend scripts/cocoindex_pipeline/verify_driver.py (or a sibling invocation mode) to accept the HTML fixture set and, for that mode only, after staging the HTML and letting the worker ingest, read the produced pullmd_share_id from the source_documents row (via the host-secrets-store Supabase creds) and GET http://localhost:3000/s/<share_id> (pullmd on localhost, ID-66 §1-b), asserting a 200 non-empty response and pullmd_share_id = X-Share-Id; surface pass/fail through the driver exit code [Inv-23]. The round-trip is one HTTP call WITHIN the driver invocation, not a separate external curl; not attempted off-host [Inv-23]. (2) the host Vitest assertions (the existing {42.10} test home, or a NEW __tests__/integration/cocoindex/inv-7-8-9-pullmd-e2e.integration.test.ts): poll content_items.content_text non-empty markdown (Inv-7) + source_documents.extraction_method = pullmd_* matching the live X-Source header (Inv-8) via the live service-role client (Supabase reachable from anywhere) — the HTML fixture is staged by the driver, not by beforeAll [Inv-22]. These are live-Supabase assertions, reachable from anywhere (Inv-18 half) [Inv-27]. The single assertion surface is the Vitest tier; the driver does NOT re-implement the row assertions (Inv-9 no-duplication carried from {62.7}). ID-42 {42.10} is unblocked at the TASK level (ID-62 in ID-42’s Task.dependencies[]), NOT via a Subtask cross-dep — this Subtask depends ONLY on its sibling {62.9}. Local check: python3 -m pytest scripts/tests/ for the extended driver’s HTML-mode probe (mocked DB read
    • mocked pullmd GET, exit-code semantics); bun run test for the new/extended Vitest file’s typed shape (skips clean off-host). Code-intel: verify_driver.py extension (Python; ast-dataflow/ts-morph N/A — use direct read + grep); run gitnexus_detect_changes() before commit.
  • testStrategy: A host run with the {42.10} HTML parameterisation: the driver stages the HTML + asserts the Inv-9 /s/<id> -> 200 round-trip via exit code; host Vitest asserts Inv-7 (content_text non-empty markdown) + Inv-8 (extraction_method = pullmd_*) against live Supabase; {42.10} proof green.
  • status: pending
  • dependencies: [9]

7. Sibling-only dependency verification (the DAG)

Section titled “7. Sibling-only dependency verification (the DAG)”

All {62.5}{62.10} dependencies reference Subtasks of Task 62 ONLY (5, 6, 7, 8, 9). No edge points outside Task 62. The dependency DAG:

{62.5} ──┬──> {62.6} ──┐
├──> {62.7} ──┼──> {62.9} ──> {62.10}
└──> {62.8} ──┘
  • {62.5} (Slice A /stage route) — no deps; host-agnostic, buildable ahead of the host.
  • {62.6}, {62.7}, {62.8} — each depends on {62.5} (corpus activation, the driver, and the helper all consume the /stage route’s multipart contract).
  • {62.9} — depends on {62.6} (activated corpus), {62.7} (the driver), {62.8} (the byte-posting helper).
  • {62.10} — depends on {62.9} (the activated corpus + driver + go-live step the {42.10} invocation reuses).

Sibling-only forcing function (§3.3) — HONOURED. No Subtask of Task 62 depends on a Subtask of any other Task; no escalation (Task split/merge) is required — the boundary is correct. The two cross-Task edges are at the Task level (Task.dependencies[]), recorded by the Orchestrator: ID-62ID-66 (the B1 host precondition); ID-42ID-62 ({42.10} unblocked by ID-62’s primitive). Neither is a Subtask cross-dep.

6 of 25 Subtasks — well within the §3.4 soft ceiling. No Task-boundary problem signalled.


  • 6 Subtasks {62.5}{62.10}, sibling-only deps, DAG in §7. The OLD entry gate ({62.0}, OQ-62-9) is DELETED (moot under B1); it is REPLACED by the B1 IMPLEMENTATION-GATED dual precondition (§4: re-spec ratified AND ID-66 host complete), carried as a blocking note in EVERY {62.5+} details.
  • PLAN-level OQs carried from TECH by reference, NOT re-opened (§2): OQ-62-11 (in-repo Python verify-driver module, no Job/image/cnb-launcher); OQ-62-12 (co-located driver / host Vitest for the Inv-9 round-trip + Tier-3, over localhost); OQ-62-P4-2 (/stage HTTP default, disk-drop a non-default fallback); OQ-62-P4-3 (/health kept); OQ-62-6 (filename-derived title); OQ-62-7 (multipart/form-data).
  • Load-bearing carries baked into details (§3): (1) the schema-valid /corpus/.kh-workspace-map.json MUST be seeded at process start in server.main() before the watch arms (else app_main raises → zero rows) — {62.6}; (2) the seed/mkdir lives in Python, not a shell entrypoint — {62.6}; (3) the 8 Tier-3 files get a per-file live-vs-defer decision, never silently-green — {62.9}; (4) the AGPL image-isolation constraint (pullmd a sibling container, NOT baked into the cocoindex image; the cloudbuild / B1-equivalent image-inspect check) lands in {62.5} (image stays clean) + {62.7} (flag the B1-deploy-path equivalent for ID-66).
  • Two Task-level edges for the Orchestrator to record in task-list.json: ID-62ID-66 (B1 host precondition); ID-42ID-62 ({42.10} unblock). Neither is a Subtask cross-dep.
  • Dispatch hint: {62.5} and {62.8} are host-agnostic — the Orchestrator MAY dispatch them once the re-spec is ratified (precondition (a)), ahead of the ID-66 host (precondition (b)). {62.6}/{62.7}/{62.9}/{62.10} need the live host for their on-host verification.
  • No new external-library symbols beyond TECH’s verified set (aiohttp multipart PRESENT, walk_dir PRESENT against the pins; TS FormData/Blob/fetch are Bun built-ins, out of OQ-3 scope). No ABSENT/SIGNATURE_DRIFT/BEHAVIOUR_DRIFT.
  • Ledger title carry (OQ-62-LEDGER-TITLE): Task.title in task-list.json still reads the superseded “in-VPC harness” phrasing — the Orchestrator updates it to the B1 phrasing (see OQ-pending.md). The Planner does not edit the ledger.