RESEARCH — Fixture-staging live-verification infra (ID-62.1)

RESEARCH — Fixture-staging live-verification infra (ID-62.1)

Task: ID-62 — Fixture-staging live-verification infra: /stage server + CI → internal-ingress reachability (shared in-VPC harness). Subtask: {62.1} RESEARCH. Status: spec-needed (RESEARCH-first). Gates ID-53 close-out (live-tier verification before relying on Stage-5). Promoted from: backlog-191 (S278). Author date: 29/05/2026.

This is the {62.1} artefact. {62.2} PRODUCT, {62.3} TECH, {62.4} PLAN are separate fresh-Planner instances (Q-PLANNER-2). This document records findings and options; it does NOT ratify a design. Open questions for Liam are numbered OQ-62-N and collected in §8.

---

1. Problem statement

The cocoindex integration tier — 39 tests (23 ID-49 + 16 {53.14} + {53.15} assertion) across ~35 *.integration.test.ts files under __tests__/integration/cocoindex/ — is authored-but-never-run-live. Every test is describe.skipIf(!ENABLED)-gated on a triplet of env vars (COCOINDEX_STAGING_URL + COCOINDEX_FIXTURE_STAGING_URL + COCOINDEX_SOURCE_PATH) plus live Supabase credentials. None of those vars is set in CI, so the tier skips clean and has never produced a single green live assertion.

The S278 investigation established the gap is NOT the original backlog-191 premise (merely “wire the env vars”). Two structural blockers sit underneath:

1. No /stage server exists. The client helper __tests__/integration/cocoindex/_helpers/fixture-staging.ts (stageFixture, lines 100–139) POSTs { fixturePath, destPath, titlePrefix } to ${COCOINDEX_FIXTURE_STAGING_URL}/stage (endpoint built at line 111), expecting a service to read the fixture bytes and write them into the cocoindex-watched corpus path. No Cloud Run Service serves /stage and no /stage handler exists in the Python. scripts/cocoindex_pipeline/server.py:160 registers exactly one route — app.router.add_get("/health", _health_handler) — and nothing else. ID-49.10 built only the CLIENT helper; the server was scoped to “ID-49.9” but never authored or deployed.

2. All staging cocoindex + pullmd Services are ingress: internal. Confirmed across manifests:

   • cloudrun/services/staging-phew-cocoindex.yaml:23 → run.googleapis.com/ingress: internal
   • cloudrun/services/prod-phew-cocoindex.yaml:23 → internal
   • cloudrun/services/staging-pullmd.yaml:60 → internal
   • cloudrun/services/prod-pullmd.yaml:35 → internal

   A GitHub-hosted CI runner is external to the GCP project network, so a direct fetch/curl against any of these returns HTTP 403 (network-layer block) even when the Service is healthy. This is documented authoritatively in two places:

   • .github/workflows/cloud-run-deploy.yml (ID-42.7 verify step, ~lines 433–440): “the Service is ingress: internal, so an external GitHub-hosted runner gets 403 (not 200) even when the Service is healthy — see docs/runbooks/pullmd-deploy.md §2.3.”
   • docs/runbooks/pullmd-deploy.md:193–199: “ingress: internal is a NETWORK-layer block — an identity token does NOT bypass it, so a direct external curl ${SERVICE_URL}/ returns 403/000 even when the Service is perfectly healthy … A from-network curl is only meaningful when run from the cocoindex Service or another in-project context.”

   Therefore even with all three COCOINDEX_* vars set in the CI integration env block, stageFixture’s fetch from a GitHub-hosted runner cannot reach an internal-ingress Service.

Real scope of ID-62 = (i) build a /stage fixture-drop server that writes into the cocoindex corpus path, (ii) solve CI → internal-ingress reachability via an in-VPC execution context, then (iii) wire the env. All three are prerequisites; wiring env alone is vacuous.

---

2. Shared root with ID-42 {42.10}

ID-42 {42.10} (“End-to-end HTML ingest proof against the deployed pullmd Service — Inv-7/8/9”) is currently status: blocked. Its journal (<info added on 2026-05-28T20:20:52.642Z>) records the identical blocker:

“{42.10} non-mocked Inv-7/8/9 E2E proof is NOT worktree-runnable: pullmd Service is ingress:internal (Inv-9 curl /s/ unreachable from a laptop) and the proof must drive the DEPLOYED cocoindex worker (Cloud Run live-fs-watch; PULLMD_SERVICE_URL / PULLMD_API_TOKEN / COCOINDEX_DB_DSN in Secret Manager only).”

Liam’s S277 decision deferred ID-42 retirement and recorded an operator runbook that explicitly names the candidate execution contexts:

“with gcloud+VPC access, stage one real HTML source into the STAGING cocoindex source binding, let the deployed worker ingest, then assert (Inv-7) content_items.content_text non-empty markdown; (Inv-8) source_documents.extraction_method = pullmd_* matching live X-Source; (Inv-9) pullmd_share_id = X-Share-Id AND curl {staging PULLMD_SERVICE_URL}/s/<share_id> -> 200 non-empty (from inside VPC / Cloud Run job / cloud shell).”

Both Tasks need the same primitive: an execution context that (a) sits inside the GCP project network so it can reach ingress: internal Services, (b) can write a real source file into the place the deployed staging cocoindex worker is watching, and (c) can read back from live Supabase to assert the produced rows. The only difference between the two is the fixture content and the assertion set:

• ID-62 stages arbitrary fixtures (PDF / docx / etc. per the cocoindex source-MIME coverage) and asserts the full Stage-1→6 invariant tier (39 tests) via pollContentItemsFor + the per-invariant SQL probes.
• ID-42 {42.10} stages one real HTML source and asserts Inv-7/8/9 (markdown body non-empty; extraction_method = pullmd_*; pullmd_share_id round-trips via GET {PULLMD_SERVICE_URL}/s/<id> — which itself must run from in-VPC).

Recommendation (carried to {62.2}/{62.3}): design ONE harness — an in-VPC execution context with a fixture-drop path into the staging cocoindex corpus — and parameterise it by (fixture set, assertion set). ID-42 {42.10} becomes one invocation of that harness with the HTML fixture + Inv-7/8/9 assertions; the ID-62 tier is the same harness with the cocoindex fixture corpus + the 39-test assertion set. The GET /s/<id> round-trip for Inv-9 runs inside the same in-VPC context, so it too clears the ingress:internal block.

---

3. The corpus path — WHERE does the staging cocoindex worker watch? (CRITICAL)

This is the load-bearing finding for the /stage server design. The /stage writer MUST target whatever the worker actually watches, and the answer constrains where the writer can run.

3.1 The watched path is an in-pod LOCAL filesystem path

scripts/cocoindex_pipeline/flow.py:app_main() (lines 1084–1223):

• Line 1096: source_path_str = os.environ.get("COCOINDEX_SOURCE_PATH", "").
• Lines 1097–1103: if unset → logs “running in idle mode” and returns cleanly (no raise). The daemon thread exits; aiohttp keeps /health alive.
• Line 1105: source_path = Path(source_path_str).
• Lines 1106–1113: if the path does not exist → idle mode, returns.
• Line 1186: source = localfs.walk_dir(source_path, live=True, recursive=True) — this is the cocoindex LocalFile source. It walks a local filesystem directory and (with live=True) arms cocoindex’s continuous fs-watch loop for the Service’s lifetime.

So the watched corpus is a POSIX local filesystem path inside the running pod, named by COCOINDEX_SOURCE_PATH. It is not a database, not an HTTP source, not a message queue — a directory the cocoindex Rust engine polls/watches via localfs.walk_dir.

3.2 The staging manifest ships idle (empty source path) and mounts NO corpus volume

cloudrun/services/staging-phew-cocoindex.yaml:

• Lines 45–46: COCOINDEX_SOURCE_PATH value: "" → idle mode (nothing watched today).
• Lines 77–79: the ONLY volumeMount is cocoindex-state → /cocoindex-state (the LMDB state store).
• Lines 99–103: the ONLY volumes entry is cocoindex-state, an in-memory (emptyDir { medium: Memory, sizeLimit: 512Mi }) tmpfs.

There is no volume mounted for a corpus directory. If COCOINDEX_SOURCE_PATH were set to, say, /corpus, that directory would be an ephemeral in-pod path on the container’s writable layer (or would need a new volume to be declared and mounted). The manifest-shape test scripts/tests/test_cocoindex_service_manifests.py (test_manifest_mounts_in_memory_volume_for_lmdb, lines 128–173) asserts the LMDB volume shape but says nothing about a corpus volume — there is none to assert.

3.3 GCS-FUSE is ruled out; Filestore is the only persistent shared-volume option

docs/runbooks/cocoindex-deploy.md:85–97 and the manifest comments (staging-phew-cocoindex.yaml:60–69, 93–98) establish:

• The LMDB store is in-memory tmpfs deliberately: “GCS FUSE does NOT support mmap/locking, so a Cloud Storage volume is LMDB-incompatible.”
• “the only … option that gives LMDB a real persistent FS is Filestore NFS (~$200+/mo).”

The same mmap/locking caveat does not necessarily apply to the corpus directory (localfs.walk_dir reads files; it does not mmap a database there). A GCS-FUSE mount might serve as a shared corpus drop-target between a separate writer and the watcher — but this is unverified and GCS-FUSE has known write-visibility / consistency characteristics that interact badly with an fs-watch loop. See OQ-62-2.

3.4 Architectural consequence: the /stage writer must share the pod or the volume

Because the watched path is in-pod local fs, the writer that drops fixture bytes there must be one of:

• (A) In the same pod / same process as the worker. The cocoindex worker already runs as a daemon thread inside the same aiohttp process that serves /health (server.py:start_cocoindex_thread, lines 169–226; main() spawns the thread at line 252 then web.run_app at line 255). So adding a POST /stage route to the same aiohttp app (server.py:build_app, lines 153–161) gives the handler direct filesystem access to the exact directory the daemon thread is watching — no volume sharing, no second process, no network hop to the corpus. The handler writes ${COCOINDEX_SOURCE_PATH}/<destPath>; walk_dir(live=True) picks it up on the next watch cycle. This is the architecturally minimal design and the recommendation for {62.3}.

• (B) A separate writer sharing a Filestore (or GCS-FUSE) volume mounted into both the writer and the cocoindex Service. Higher cost (Filestore ~$200+/mo) and more moving parts; only warranted if there is a reason /stage cannot live in the cocoindex Service process (OQ-62-3).

Constraint surfaced: if the corpus path stays in-pod/ephemeral (the cheapest option), the /stage writer cannot be a standalone Service — it must be co-resident with the watcher (option A). Any design that puts /stage in a separate Service without a shared persistent volume is incoherent: bytes written in the /stage pod are invisible to the cocoindex pod.

Note on idle-mode interaction: today COCOINDEX_SOURCE_PATH="", so app_main returns immediately and nothing is watched. To activate the live tier at all, COCOINDEX_SOURCE_PATH must be set to a real, existing directory and the Service restarted (the fs-watch arms once at app_main; server.py:195–198 confirms a clean idle return does not flag the worker crashed). The /stage server design must account for this: set COCOINDEX_SOURCE_PATH (e.g. /corpus), ensure the dir exists at boot, and redeploy. See OQ-62-1.

---

4. Reachability options (CI / operator → ingress:internal Services)

The harness needs an execution context inside the GCP project network. Four concrete options, with tradeoffs.

Option (a) — Cloud Run job executed in-VPC via gcloud run jobs execute ★ RECOMMENDED PRIMARY

A dedicated Cloud Run Job (analogous to the existing kh-pipeline-{phew,kpf} jobs) that runs the staging + assertion logic. Triggered from ci.yml (or by an operator) via gcloud run jobs execute <job> --region=europe-west2 --wait after WIF auth.

Why it fits:

• Internal reachability: a Cloud Run Job runs inside the GCP project, so it can reach ingress: internal Services (the cocoindex sidecar + pullmd). This is exactly how the deployed cocoindex Service already reaches the internal pullmd Service — over HTTP via PULLMD_SERVICE_URL (cloud-run-deploy.yml:277, 375, 387).
• Secret Manager access: the existing phew-pipeline-sa@kh-staging-494815 SA holds roles/secretmanager.secretAccessor (job manifest header, cloudrun/jobs/staging-phew.yaml:22–23), so the job can mount COCOINDEX_DB_DSN, PULLMD_SERVICE_URL, PULLMD_API_TOKEN, the Supabase service-role key, etc. — all of which “are in Secret Manager only” per the {42.10} journal.
• CI-native via existing WIF harness: cloud-run-deploy.yml:121–158 already shows the full pattern — permissions: { id-token: write }, google-github-actions/auth@v2 with workload_identity_provider: ${{ vars.GCP_WIF_PROVIDER }} + service_account: ${{ vars.GCP_DEPLOY_SA_EMAIL }}, then setup-gcloud@v2. A new ci.yml job (or step) reuses this verbatim to authenticate, then gcloud run jobs execute.
• Established operational prior art: gcloud run jobs execute kh-pipeline-phew appears in docs/runbooks/cloud-run-phase-1.md:543, 724, 732 and docs/reference/data-entry-points.md:111. docs/themes/canonical-pipeline/phase-0-investigation/0.2.5-build-not-wired.md:114 explicitly anticipates “any Cloud Scheduler / GitHub Action / manual gcloud run jobs execute invocation runs the container.” The pattern is blessed and exercised.

Tradeoffs:

• Setup cost: moderate. New Job manifest (cloudrun/jobs/{env}-cocoindex-verify.yaml or similar), a deploy step in cloud-run-deploy.yml (gcloud run jobs replace), the job’s container entrypoint (the staging + assertion driver), and the ci.yml execute step. All follow existing patterns; no novel infra primitive.
• Where do the assertions run? Two sub-shapes (OQ-62-5): (i) the job only stages the fixture (writes to the corpus), and the Vitest assertions run on the GitHub-hosted runner against live Supabase (Supabase REST is ingress: all — reachable from anywhere; pollContentItemsFor already uses the live service-role client); OR (ii) the job runs BOTH staging and assertions (e.g. a Python/TS harness inside the job) and CI just gates on the job’s exit code. Shape (i) keeps the 39 Vitest tests as the assertion surface (matches the authored consumers) but requires the corpus-write to be reachable — which loops back to §3.4 (the writer must be in-pod, so the job would POST to the cocoindex Service’s new /stage route from inside the VPC). Shape (ii) is self-contained but duplicates the assertion logic outside Vitest.
• Corpus reachability still needs §3.4 resolved. A job reaching the cocoindex Service’s /stage route is fine (internal→internal HTTP works). But the /stage route must exist and write in-pod (option A of §3.4). The job is the client of /stage, exactly as stageFixture is today — it just runs from inside the VPC instead of the GH runner.

Option (b) — Self-hosted GitHub runner inside the VPC

Register a self-hosted GH Actions runner on a GCE VM (or GKE pod) inside the project network. CI jobs labelled runs-on: [self-hosted, vpc] then execute with native internal reachability — stageFixture’s fetch and the Inv-9 curl /s/<id> “just work” with no code changes to the consumers.

Tradeoffs:

• Pro: the only option that requires zero change to the existing consumer code — COCOINDEX_FIXTURE_STAGING_URL points at the internal cocoindex URL and the runner reaches it directly. The 39 Vitest tests run unmodified on the self-hosted runner.
• Con — security: self-hosted runners on public-repo or fork-PR triggers are a well-known RCE surface (a malicious PR can run arbitrary code on the runner inside your VPC). KH is a private repo (github.com/ai-solution-hub/knowledge-hub), which reduces but does not eliminate the risk (Dependabot PRs, compromised dependencies). Requires ephemeral / hardened runner discipline.
• Con — maintenance: a long-lived VM (or autoscaling runner group) is standing infra to patch, monitor, and pay for. The Cloud Run job (option a) is ephemeral and pay-per-execution.
• Con — not the established pattern: KH has no self-hosted runners today; this introduces a new operational primitive vs. reusing the already-blessed gcloud run jobs execute + WIF path.

Option (c) — Flip the /stage Service ingress to all + IAM/OIDC token auth

Deploy /stage as a Service with run.googleapis.com/ingress: all (or internal-and-cloud-load-balancing) and require a Google-signed OIDC identity token (Authorization: Bearer $(gcloud auth print-identity-token)) so only the CI SA can call it. The GH runner then reaches it directly.

Tradeoffs:

• Pro: simplest CI wiring — a normal fetch from the GH runner with a bearer token; no job, no self-hosted runner.
• Con — security regression (weigh carefully): ingress: internal is a network-layer isolation. docs/runbooks/pullmd-deploy.md:193–195 is explicit: “an identity token does NOT bypass it.” Flipping to ingress: all removes that network isolation entirely and relies solely on application-layer IAM/OIDC. For a /stage endpoint that writes arbitrary bytes into the ingest corpus (which then flows through the full extraction pipeline into the live DB), exposing it to the public internet — even token-gated — is a materially larger attack surface than the current internal-only posture. A token leak or an IAM misconfiguration becomes a corpus-injection vector. The cocoindex + pullmd Services are deliberately internal-only by design (staging-pullmd.yaml:18–22); flipping one breaks that uniform posture.
• Con — pullmd cannot be flipped the same way for {42.10}: the Inv-9 GET /s/<id> round-trip hits the pullmd Service. Flipping pullmd to ingress: all is a much bigger decision (it is an AGPL third-party container with single-admin auth). So option (c) solves ID-62’s /stage reach but does NOT cleanly solve the shared {42.10} need — it would leave the shared-harness goal half-met.

Net: option (c) is the cheapest to wire but the worst on the security axis and the weakest on the shared-harness axis. Hold it as a fallback only if OQ-62-4 rules out the job harness.

Option (d) — Cloud Shell / manual operator run (interim, already documented)

The S277 {42.10} operator runbook (“from inside VPC / Cloud Run job / cloud shell”) already sanctions a manual operator run from Cloud Shell (which sits inside Google’s network and can be granted project access). This is not CI-native (no automated gate) but is the zero-new-infra path to get the first green live proof and unblock ID-53 close-out + the {42.10} Inv-10 gate, while the durable harness (option a) is built.

Tradeoffs: pro — available today, no build; con — manual, not repeatable in CI, not a regression gate. Best framed as the bridge that produces the first live evidence while option (a) lands.

§4 recommendation

Primary: option (a) — Cloud Run job via WIF, because it (1) reaches internal-ingress Services natively, (2) has Secret Manager access via the existing SA, (3) reuses the already-blessed WIF + gcloud run jobs execute pattern with minimal new infra, and (4) is the only option that cleanly serves BOTH ID-62 and {42.10} (the job runs the HTML proof

• the Inv-9 /s/<id> curl from inside the VPC too).

Bridge: option (d) — manual Cloud Shell / gcloud run jobs execute to get the first live evidence and unblock the gates while (a) is built.

Fallback: option (b) self-hosted runner if zero consumer-code-change is judged worth the standing-infra + security cost. Avoid option (c) unless OQ-62-4 forces it; the corpus-injection attack-surface and the half-met shared-harness goal make it the weakest.

---

5. /stage fixture-drop server design

Grounded in §3, the design that follows the existing architecture most cleanly:

• Where it runs: in the cocoindex sidecar Service process, as a new POST /stage route on the existing aiohttp app. server.py:build_app() (lines 153–161) currently adds only GET /health; add app.router.add_post("/stage", _stage_handler). The handler runs in the same process as the start_cocoindex_thread() daemon (lines 169–226), so it shares the pod filesystem with the live walk_dir(source_path, live=True) watcher.
• What it does: accept { fixturePath, destPath, titlePrefix } (the exact contract stageFixture already sends, fixture-staging.ts:116–120), read the fixture bytes, and write them to ${COCOINDEX_SOURCE_PATH}/<destPath>. Optionally inject titlePrefix into the file’s title metadata where the format supports it (the helper’s pollContentItemsFor matches content_items.title ILIKE '${titlePrefix}%', lines 196–199) — for formats where that is impractical, see OQ-62-6.
• Where the bytes come from: the fixturePath in StageFixtureArgs is “repository- relative or absolute” (helper doc, lines 52–57) — but the server is the reader, and the server runs in a Cloud Run pod that does NOT have the repo checked out. So either (i) the client (the in-VPC job) reads the fixture bytes and POSTs them in the body (changes the wire contract from a path to bytes — OQ-62-7), or (ii) the fixtures are baked into the cocoindex image at build time and fixturePath is resolved server-side against the image. The current contract sends a path, which implies the server can see it — true only if fixtures are baked into the image. This contract ambiguity must be resolved in {62.3} (OQ-62-7).
• COCOINDEX_SOURCE_PATH activation: set it to a real existing dir (e.g. /corpus), ensure the dir exists at container boot (a writable layer dir or a mounted volume per §3.4), and redeploy so app_main arms the watch (today it is "" → idle). The /stage handler should reject (clear 4xx) when COCOINDEX_SOURCE_PATH is unset, so a mis-wired deploy is loud rather than silently dropping bytes into a void.
• Response shape: echo { destPath, requestId? } (the helper reads these, fixture-staging.ts:130–138). destPath may be prefixed with the corpus root.
• AGPL boundary check: adding a route to server.py does NOT pull pullmd or Playwright into the cocoindex image, so cloudbuild-cocoindex.yaml’s Inv-9 assertion (lines 117–167, “zero pullmd entries, zero Playwright dependencies”) stays green. Confirm no new dependency trips it.

Open design fork: if OQ-62-3/OQ-62-7 conclude the server should be a separate Service (not co-resident), then §3.4 forces a shared persistent volume (Filestore) so the separate writer and the cocoindex watcher see the same bytes — significantly more infra. The co-resident POST /stage route is the strongly-preferred default.

---

6. CI wiring (once reachable)

Target: the integration job at .github/workflows/ci.yml (job integration, name “Integration tests”, begins ~line 734; env block ~line 803; runs-on: ubuntu-latest, environment: Staging). Today its env block sets Supabase + Anthropic + OpenAI + TEST_USER credentials but no COCOINDEX_* vars (confirmed: grep COCOINDEX .github/workflows/ci.yml returns nothing).

Two wiring shapes, depending on §4:

• If option (a) job harness: add a new step (or a new job) that (1) WIF-authenticates via the cloud-run-deploy.yml:140–158 pattern (id-token: write + google-github-actions/auth@v2 + setup-gcloud@v2), then (2) gcloud run jobs execute kh-cocoindex-verify-phew --region=europe-west2 --wait. The job itself carries the COCOINDEX_* + Secret-Manager-mounted vars (declared in its manifest / --set-secrets at deploy time, mirroring cloud-run-deploy.yml:387). If the Vitest assertions still run on the GH runner (shape (i) of §4a), the three COCOINDEX_* vars (COCOINDEX_STAGING_URL, COCOINDEX_FIXTURE_STAGING_URL, COCOINDEX_SOURCE_PATH) are added to the integration job’s env block from the Staging GitHub-environment vars/secrets — per the pattern in docs/runbooks/github-environments.md (vars vs secrets: non-sensitive URLs as vars.*, tokens as secrets.*). But note COCOINDEX_STAGING_URL/COCOINDEX_FIXTURE_STAGING_URL point at internal-ingress URLs the GH runner can’t reach — so shape (i) only works for the assertion half (pollContentItemsFor → live Supabase, which IS reachable); the staging half must happen inside the job. See OQ-62-5.
• Gating decision (CI-blocking vs operator-on-demand): whether this new live-tier proof is a PR-blocking CI gate or an on-demand / scheduled workflow is a product/infra call — OQ-62-8. The existing integration job is PR-blocking and staging-scoped; a live-fs-watch proof has higher latency and external-dependency variance (Anthropic tail latency, cold starts) that may argue for a separate non-blocking or scheduled workflow rather than inlining into the PR-blocking integration job.

GitHub-environment plumbing already present and reusable: vars.GCP_WIF_PROVIDER, vars.GCP_DEPLOY_SA_EMAIL (Staging + Production scopes), Supabase + Anthropic secrets, the Staging environment gate (environment: Staging, case-sensitive).

---

7. Consumers (the env-gated tests + helpers)

The artefacts that become live once the harness lands:

• Helpers — __tests__/integration/cocoindex/_helpers/fixture-staging.ts:
  • stageFixture(args) (lines 100–139) — POSTs { fixturePath, destPath, titlePrefix } to ${COCOINDEX_FIXTURE_STAGING_URL}/stage; throws if the var is unset (lines 104–108); throws on non-2xx (lines 123–128). This is the direct client of the /stage server ID-62 must build.
  • pollContentItemsFor(titlePrefix, opts?) (lines 179–223) — polls content_items (title ILIKE '${titlePrefix}%') via the live service-role client until a row lands or the 120s deadline. Gated on hasRealLiveDbCredentials(). Supabase REST is reachable from anywhere, so this half does not need in-VPC.
  • dropFixture(args) (lines 268–366) — best-effort cleanup across q_a_extractions, entity_mentions (deferred per S273 OQ-1), source_documents, then content_items, PK-scoped + titlePrefix-guarded.
  • hasFixtureStagingUrl() (lines 43–45) — the env gate.
• Env-gate triplet used by the test bodies: COCOINDEX_STAGING_URL, COCOINDEX_FIXTURE_STAGING_URL, COCOINDEX_SOURCE_PATH — typically combined into a local const ENABLED = HAS_STAGING_URL && HAS_FIXTURE_STAGING && HAS_SOURCE_PATH && hasRealLiveDbCredentials() and consumed via describe.skipIf(!ENABLED)(...). Confirmed pattern across files, e.g. admin-merge-coexistence.integration.test.ts:47–49, 88, classify-content-coexistence.integration.test.ts:50–52, 132, cross-document-dedup.integration.test.ts:40–42, 81, agpl-boundary.integration.test.ts:51, 59.
• Scope: 35 files under __tests__/integration/cocoindex/ reference the gate or the helpers (grep -rl count), including the shared test-helpers.ts and _helpers/fixture-staging.ts. The brief’s “~12 files / 39 tests” refers to the subset that actively stage fixtures and assert produced rows; the remaining files assert topology / health / version metadata and may gate on a subset of the triplet. {62.3}/{62.4} should enumerate which subset each invariant test needs (OQ-62-5 informs this).
• dropFixture table caveat: entity_mentions cleanup is best-effort because ID-49.5 is deferred (S273 OQ-1). When the live tier runs, confirm the table/FK shape so cleanup does not leak rows across runs.

---

8. Open questions (OQ-62-N)

• OQ-62-1 — Corpus path value + activation. What concrete value should COCOINDEX_SOURCE_PATH take in staging (e.g. /corpus), and is a Service redeploy acceptable as the activation step (it is required — app_main arms the watch once at boot; today the value is "" → idle)? Does activating the watch on the persistent staging Service have side effects on other consumers of that Service (none today since it is idle, but confirm)?

• OQ-62-2 — Is a shared corpus volume needed, and which kind? If /stage is co-resident with the worker (recommended), the corpus can be an ephemeral in-pod dir and no volume is needed. Confirm the corpus dir survives across the watch lifetime within a single revision (it does — same pod) and that ephemerality across cold starts is acceptable for a test fixture-drop (it should be — fixtures are re-staged per run). If a separate writer is chosen, decide Filestore (~$200+/mo, real NFS) vs GCS-FUSE (cheap but fs-watch/consistency-risky and unverified for localfs.walk_dir).

• OQ-62-3 — Co-resident /stage route vs separate Service. Confirm /stage should be a new POST route in the existing cocoindex server.py aiohttp app (recommended, §3.4 option A) rather than a standalone Service. A standalone Service forces a shared persistent volume (OQ-62-2) and is materially more infra.

• OQ-62-4 — Accept the job-harness build, or take the ingress-flip shortcut? Primary recommendation is the Cloud Run job (option a). If the build cost is judged too high for the value, the fallback is flipping /stage to ingress: all + OIDC (option c) — but this is a security regression (corpus-injection surface; “identity token does not bypass network isolation”, pullmd-deploy.md:193–195) AND does not cleanly serve {42.10}‘s pullmd /s/<id> round-trip. Liam’s call on the security-vs-effort tradeoff.

• OQ-62-5 — Where do the assertions run? (i) Job stages fixtures + GH-runner Vitest asserts against live Supabase (keeps the 39 authored tests as the surface, but the staging POST must happen inside the job since the cocoindex URL is internal); or (ii) the job runs both staging AND assertions and CI gates on exit code (self-contained but duplicates assertion logic outside Vitest). This determines whether the three COCOINDEX_* vars go in the ci.yml integration env block or only in the job manifest.

• OQ-62-6 — titlePrefix injection per file format. pollContentItemsFor matches content_items.title ILIKE '${titlePrefix}%'. For which fixture formats can /stage inject the prefix into title metadata, and what is the fallback when a format does not carry a title (e.g. derive the title from the filename / destPath)?

• OQ-62-7 — Wire contract: path vs bytes. stageFixture sends a fixturePath (repo-relative/absolute), implying the server can read it. But the Cloud Run pod has no repo checkout. Resolve: (a) bake fixtures into the cocoindex image and resolve the path server-side, or (b) change the contract so the client reads the bytes and POSTs them in the body (multipart/base64). Option (b) changes fixture-staging.ts:116–120 + the StageFixtureArgs shape; option (a) bloats the image and couples fixtures to deploys.

• OQ-62-8 — CI-blocking vs operator-on-demand vs scheduled. Should the live-tier proof be a PR-blocking gate (high latency + external-dependency variance inlined into the integration job), a separate non-blocking job, or a scheduled (nightly) workflow? ID-53 close-out needs a green live run, not necessarily a per-PR one. Recommend a separate workflow (or workflow_dispatch + schedule) over inlining into the PR-blocking integration job, but this is Liam’s call.

• OQ-62-9 — VPC egress / internal→internal reachability is assumed, not verified. The cocoindex Service reaches pullmd over PULLMD_SERVICE_URL today, implying same-project internal-ingress traffic works without an explicit VPC connector (no vpc-access/ vpc-connector/egress annotation exists in any manifest — confirmed by grep). A Cloud Run job’s egress to an internal-ingress Service should work the same way (same project), but this must be empirically verified before {62.3} commits to the job harness — run a throwaway gcloud run jobs execute that curls the internal cocoindex /health and confirm a 200 (not 403) from inside the job. (Per the OQ-3 empirical- verification discipline, this is the infra-side analogue of an import-and-call check.)

---

9. Verification notes (external-API / infra facts)

This RESEARCH cites no third-party library symbols that require an import-and-call check (the cited surfaces are KH-internal Python + CI YAML + Cloud Run manifests). The cocoindex symbol localfs.walk_dir(..., live=True, recursive=True) is the only external API referenced; it is already empirically pinned in the codebase (flow.py:1186 + the docstring at server.py:178–193 notes “Verified empirically against installed cocoindex 1.0.3”). No re-verification needed for this artefact.

Infra facts cited are file-grounded (not assumed): every ingress setting, env value, volume shape, route registration, and WIF-pattern claim carries a file:line citation in §1–§7. The one infra fact that is assumed and explicitly flagged for empirical verification is internal→internal job-to-Service reachability (OQ-62-9).

---

10. Summary for the Orchestrator

• Recommended reachability: Cloud Run job via WIF (gcloud run jobs execute, reusing the cloud-run-deploy.yml auth pattern + the phew-pipeline-sa SA) — the only option that natively reaches internal-ingress Services, has Secret Manager access, reuses blessed prior art, and serves both ID-62 and {42.10}. Bridge: manual Cloud Shell run for the first live evidence. Avoid the ingress-flip (security regression + half-met shared harness).
• Corpus-path finding: the watched corpus is an in-pod local filesystem path (flow.py:1186 localfs.walk_dir), COCOINDEX_SOURCE_PATH is "" today (idle), and no corpus volume is mounted (only the LMDB tmpfs). Consequence: the /stage writer must be co-resident with the worker — add a POST /stage route to the existing cocoindex server.py aiohttp app (shares the pod fs with the daemon-thread watcher).
• Shared harness: one in-VPC harness, parameterised by (fixture set, assertion set); {42.10} is one invocation (HTML fixture + Inv-7/8/9 incl. the /s/<id> curl from in-VPC), the ID-62 tier is the 39-test invocation.
• Load-bearing OQs for Liam: OQ-62-4 (build the job harness vs accept the ingress-flip security tradeoff), OQ-62-8 (CI-blocking vs operator-on-demand/scheduled), OQ-62-7 (wire contract: bake fixtures into the image vs POST bytes), OQ-62-5 (assertions on the GH runner vs inside the job). OQ-62-9 (internal→internal job reachability) needs an empirical curl-from-job check before {62.3} commits.

---

OQ-62-9 reachability synthesis (S281)

RESEARCH SYNTHESIS — ID-62 {62.1}: cocoindex→pullmd reachability + pivot option space (resolving OQ-62-9)

Executive finding (the premise inversion)

The deciding insight: the “cocoindex→pullmd works today, just replicate it” premise is UNVERIFIED and probably FALSE — and that changes which pivot is correct. Two facts collide:

1. VERIFIED (file-grounded): Both staging-phew-cocoindex.yaml and staging-pullmd.yaml are ingress: internal, and there is ZERO vpc-access / vpc-connector / run.googleapis.com/network / egress annotation in any manifest under cloudrun/, AND the deploy uses bare gcloud run services replace <manifest> with no --vpc-egress/--network/--subnet flags (cloud-run-deploy.yml:267, 293).
2. VERIFIED (Google Cloud docs): A Cloud Run service in the same project calling another ingress: internal service over its run.app URL is only classified internal “when traffic routes through a VPC network” — i.e. it requires Direct VPC egress or a VPC connector. Without it, egress goes to the internet first and the callee returns 404 (Cloud Run ingress docs, direct-VPC-egress codelab).

These two cannot both be true and have cocoindex reaching pullmd today. Per Google’s rule, the cocoindex Service as currently deployed (no VPC egress) should not be able to reach the internal pullmd Service. And there is no recorded evidence it ever has: ID-42 {42.10} (the live HTML-ingest proof) is status: blocked for want of exactly this, and the only repo mention of a successful end-to-end call (ledger-field-sweep-s269.md:414) is phrased as an acceptance criterion to prove, not an observation. Conclusion: cocoindex→pullmd is an untested assumption baked into the design, not a working datapath. (UNVERIFIED — gate everything on the live probe in §D.)

This reframes OQ-62-9. The S280 probe correctly found the bare Job fails (Job traffic classified EXTERNAL → GFE 404; Direct VPC egress on default subnet → 000 from a Cloud-NAT gap). But “invoke as a Service instead of a Job to collapse the problem” only works if a no-VPC same-project Service call to internal-ingress actually succeeds — and Google’s docs say it does not. So the real choice space is narrower than “Service vs Job”; it is “add a VPC datapath (some form) vs change ingress posture vs co-locate.” Jobs and Services are documented to behave the same for internal-ingress classification — there is no free Service-shaped escape hatch.

---

(a) cocoindex VPC / on-prem finding

VERIFIED — cocoindex (the OSS data-transform library, the one KH uses) imposes NO VPC or networking requirement of any kind. It is an Apache-2.0, pip-installable Python library (pip install -U cocoindex, Python 3.11–3.13, macOS/Linux/Windows) that runs on your own infrastructure with no cloud account, no private-networking prerequisite, and no license gate (install docs; github cocoindex-io/cocoindex, README “Incremental engine for long horizon agents”, current release v1.0.6 — KH pins 1.0.3, a few releases behind).

Liam’s belief (“cocoindex only provides VPC for enterprise customers”) is a category error — corrected. The “VPC / on-prem” language lives only on the commercial managed-service page (cocoindex.io/enterprise: “Deploy entirely inside your cloud. Data never leaves your perimeter,” SAML/OIDC SSO, audit lineage, branch-overlay, SLAs). That is a hosted-daemon upsell — cocoindex selling operating the daemon for you inside your perimeter. It is not a capability the OSS library lacks. Because KH self-hosts the cocoindex library inside its own Cloud Run pod, “VPC” is purely a KH deployment-topology choice (Cloud Run / GKE / on-prem), not a cocoindex purchase. (The earlier search hit “petabyte/on-prem/VPC” conflated the enterprise marketing page — and partly the separate cocoindex-code CLI product — with the core library; disregard.)

Corollary (VERIFIED-by-implication): since cocoindex is just a self-hosted library, the “$200/month persistent Filestore” cost noted earlier is not a cocoindex requirement either — it stems from KH’s own choice to give the LMDB state store a persistent FS (GCS-FUSE can’t mmap, per cocoindex-deploy.md:85–97). On-prem / a single co-located host sidesteps it entirely (the corpus and LMDB live on local disk). So “self-host on-prem ⇒ negates the $200/mo” is CORRECT, but note the $200/mo was always a KH-LMDB-persistence decision, never a cocoindex licensing item.

---

(b) pullmd VPC finding

VERIFIED — pullmd (github aeternalabshq/pullmd, v2.0.0) has NO VPC or private-networking requirement. It is a standard multi-container Docker topology: an Express main service on :3000 plus two Python FastAPI sidecars — Playwright/Chromium on :8002 (/render) and Trafilatura on :8001 (/extract) — communicating over an internal Docker bridge; only :3000 is exposed. Auth is Authorization: Bearer pmd_<token> (optional OAuth 2.1 via PULLMD_AUTH_MODE). No Cloud Run / serverless / VPC guidance exists in its docs (consistent with Liam seeing “no reference to VPC on the PullMD repo”). KH pins aeternalabshq/pullmd:2.0.0 + pullmd-playwright:2 + pullmd-trafilatura:2 (staging-pullmd.yaml:81,162,174).

Implication: pullmd’s ingress: internal posture is 100% a KH deployment decision, not pullmd’s. pullmd doesn’t care about networking; KH chose to network-isolate it (and PULLMD_AUTH_MODE=single-admin) because it’s an AGPL third-party container handling arbitrary URL fetches. So KH owns the entire reachability problem — neither upstream imposes it.

---

(c) Ranked pivot options (each tagged with the reachability mechanism it relies on)

Ranked by (collapses-the-problem × low-infra × serves-both-ID-62-and-{42.10} × security). All are gated on the §D probe — the probe result flips the ranking.

Branch point: does the §D probe show cocoindex→pullmd works TODAY (no VPC)?

If probe = WORKS (premise true after all — current datapath is somehow internal):

• P1 — Invoke the harness as a Service/route, not a Job (option (a) of the brief). [mechanism: same-project Service→internal-Service run.app, whatever makes the existing cocoindex→pullmd call work.] Add POST /stage to the cocoindex server.py aiohttp app (RESEARCH §3.4-A, co-resident with the walk_dir(live=True) watcher) — already the recommended /stage design. The in-VPC caller of /stage then just needs to be whatever class of caller the working cocoindex→pullmd call already is. This collapses the problem to zero new network infra and serves both Tasks. Strongly preferred if the probe is green.

If probe = FAILS (premise false — the likely outcome per Google’s docs):

The failure is not Job-specific; it’s that no VPC datapath exists at all, and the pipeline’s HTML extraction has never actually worked live. Reachability must be built:

• P2 — Direct VPC egress + Cloud-NAT config on the cocoindex Service. [mechanism: Direct VPC egress gives the pod an internal IP so same-project run.app traffic is classified internal.] Add --network/--subnet/--vpc-egress=private-ranges-only (or all-traffic if it also needs internet) to the cocoindex Service deploy, and fix the Cloud-NAT gap the S280 000 exposed. This is the canonical Google-documented fix (direct-VPC codelab), is not enterprise-gated, and fixes both the (never-working) cocoindex→pullmd production datapath and the harness reach. This is the highest-value pivot if the probe fails — because it also repairs the real pipeline, not just the test harness.
• P3 — Serverless VPC Access connector. [mechanism: VPC connector routes egress through the VPC → internal classification.] Older mechanism, same effect as P2, adds a standing connector resource (cost + maintenance). Choose only if Direct VPC egress is unavailable in europe-west2 for the project’s config. Functionally equivalent reachability outcome to P2; strictly more infra.
• P4 — Co-locate / collapse to one host (on-prem or single GKE pod / single Service). [mechanism: localhost — no cross-service network hop, no ingress classification at all.] Run cocoindex + pullmd in the same pod / same host so the adapter calls http://localhost:3000 (pullmd) and /stage writes to a local-disk corpus. This is closest to pullmd’s upstream docker-compose shape, eliminates the ingress problem entirely, and also kills the $200/mo Filestore (LMDB + corpus on local disk). Strong on-prem / GKE candidate; the cost is leaving the Cloud-Run-per-service topology. Best “topology pivot” answer; serves both Tasks.
• P5 — ingress=all + IAM/OIDC bearer (spec amendment, option (e)). [mechanism: drop network isolation; rely on Google-signed identity token at the app layer.] Cheapest CI wiring (a fetch + bearer from the GH runner), but a security regression for an endpoint that injects arbitrary bytes into the ingest corpus, and it does not cleanly solve the {42.10} pullmd /s/<id> round-trip (flipping the AGPL pullmd Service to public is a bigger call). Fallback only.
• P6 — internal DNS. [mechanism: private DNS name for the internal Service.] A naming convenience layered on P2/P3 — it does not by itself solve classification (you still need a VPC datapath). Not a standalone solution; fold into P2 if desired.
• P7 — Self-hosted GH runner in-VPC (RESEARCH §4-b). [mechanism: runner sits inside the project network.] Zero consumer-code change, but standing infra + fork-PR RCE surface. Orthogonal to the cocoindex→pullmd production datapath (only helps the test harness, not the real pipeline). Fallback.

Net recommendation: the probe outcome is binary-decisive. If green → P1 (collapse via Service/route, zero infra). If red (expected) → P2 (Direct VPC egress + Cloud-NAT) as the primary, because it fixes the actual broken production datapath, with P4 (co-locate / on-prem) as the strong topology-simplifying alternative that also deletes the $200/mo cost. Avoid P5.

---

(d) The precise next live-probe (the deciding test)

Run this BEFORE {62.3} commits to any pivot. It answers the one load-bearing unknown: does a same-project caller reach an ingress: internal Service without VPC egress, and is the cocoindex→pullmd datapath real? Run from Cloud Shell / operator with project access (kh-staging-494815), gcloud authed.

Probe 0 — establish the ground truth on the EXISTING (no-VPC) cocoindex Service (the deciding test):

1. Get the internal URLs: gcloud run services describe kh-cocoindex-pipeline-phew --region=europe-west2 --format='value(status.url)' and same for kh-pullmd-phew.
2. Confirm the manifests’ no-VPC claim against the LIVE revision (the deploy might differ from the manifest): gcloud run services describe kh-cocoindex-pipeline-phew --region=europe-west2 --format='yaml(spec.template.metadata.annotations)' — look for any run.googleapis.com/vpc-access-* / network-interfaces / vpc-access-egress. If present, the premise resolves WORKS via that VPC config (→ P1/P2 trivial). If absent, continue.
3. The decider: exec into the running cocoindex Service container (or trigger its existing pullmd adapter codepath) and curl the pullmd internal URL with the bearer token: curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $PULLMD_API_TOKEN" "$PULLMD_URL/health" (or the adapter’s actual endpoint). Cloud Run has no exec; achieve this by deploying a one-line debug route or reading the cocoindex Service’s own logs from a real ingest attempt.
   • 200 / 2xx ⇒ premise WORKS (same-project internal call succeeds with no VPC — contradicts Google’s general rule, so there’s hidden network config; harmless — adopt P1).
   • 404 (GFE) ⇒ premise FALSE, traffic classified EXTERNAL — the production datapath has never worked → adopt P2 (Direct VPC egress) or P4 (co-locate). This is the expected result given §Executive-finding.
   • 000 / connection refused ⇒ Cloud-NAT / egress gap (the S280 Job symptom reproduced at Service layer) → P2 + fix Cloud-NAT.

Probe 1 — only if Probe 0 = 200 (confirm a Job is the only thing that differs): gcloud run jobs execute kh-pipeline-phew … containing the same curl, to confirm Job-vs-Service is the real axis. If Service=200 and Job=404, the brief’s “invoke as Service not Job” collapse is validated → P1 definitively.

Stop condition / what it gates: Probe 0’s HTTP code alone selects the pivot family (WORKS→P1; FALSE→P2/P4). Do not author {62.3} TECH until Probe 0 returns a concrete code from the live staging revision.

---

Claim ledger (VERIFIED / UNVERIFIED)

• cocoindex OSS lib = Apache-2.0 self-hosted, no VPC/license/networking requirement — VERIFIED (install docs + GitHub README).
• “cocoindex VPC = enterprise-only” — VERIFIED FALSE (it’s a managed-daemon upsell on /enterprise; OSS self-host makes VPC a KH topology choice).
• pullmd v2.0.0 has no VPC requirement; ingress:internal is KH’s choice — VERIFIED (pullmd repo + KH manifests).
• $200/mo Filestore is a KH-LMDB-persistence decision, negated by on-prem/co-locate, not a cocoindex cost — VERIFIED (cocoindex-deploy.md + cocoindex licensing).
• No VPC annotation/flag anywhere in KH cloudrun manifests or deploy — VERIFIED (grep + cloud-run-deploy.yml).
• Same-project Service→internal-Service needs VPC egress per Google docs — VERIFIED (Cloud Run ingress docs).
• cocoindex→pullmd works today — UNVERIFIED, probably FALSE (no VPC datapath exists; {42.10} blocked; no live success on record). This is the gate — resolve with Probe 0.
• Jobs vs Services differ for internal-ingress classification — VERIFIED they do NOT differ per Google docs (so “Service collapses it” only holds if a no-VPC internal call works at all — which Probe 0 settles).

Files of record: docs/specs/id-62-fixture-staging-infra/RESEARCH.md (§4 options, §8 OQ-62-9), cloudrun/services/staging-phew-cocoindex.yaml, cloudrun/services/staging-pullmd.yaml, cloudrun/jobs/staging-phew.yaml, .github/workflows/cloud-run-deploy.yml:243-296, docs/runbooks/pullmd-deploy.md:193-199, docs/audits/ledger-field-sweep-s269.md:414.