RESEARCH — Cross-network content-ingest contract (Vercel ↔ IONOS): the canonical-store guarantee for agent/UI-created content

⚠️ DISPOSITION: NOT A PROCUREMENT DEPENDENCY (bannered S462). D1–D9 were never ratified and must not be ratified as written: the content_items grain was eliminated (id-131) and forms entry is manual-upload app-side (DR-014), mooting the procurement leg. If the cross-network UI-ingest question is revived for NON-form corpus content, re-baseline onto the source_documents/record_embeddings grain first.

RESEARCH — Cross-network content-ingest contract (Vercel ↔ IONOS)

Status: RESEARCH complete — adjudication PENDING Liam (§9 decision register). Authored 25/06/2026. RESEARCH-only — no Task opened, no ledger written, no implementation, no DDL. This document adjudicates a security-invariant-touching architecture decision (Inv-13); trade-offs are surfaced honestly rather than rubber-stamping any one option. Identity hygiene: public-track repo — no client identity tokens used below.

Trigger: the source-less / agent-created create_content_item ingest path is non-functional cross-network today — it errors at /stage. Surfaced while greening MCP eval FC-60 / FC-65, which have been decoupled to the synchronous reference_ingest branch as a stopgap (verified — see §3.6).
Question: what should the canonical ingest contract be for agent/UI-created content (the “propose-into-store / canonical-store guarantee” — bytes land durably so a future re-ingest re-derives the content_items row) given the Vercel ↔ IONOS network boundary?

1. Problem statement & why it matters

The platform spans two networks:

Vercel runs the Next.js UI + the MCP server (/api/mcp/mcp).
IONOS / Coolify (Server B) runs the CocoIndex ingestion worker behind Traefik.

The source-less create path stages bytes to the worker, then triggers a corpus walk that mints the DB rows. That staging endpoint (/stage) is unauthenticated and compose-internal by design (security invariant Inv-13). Because Vercel reaches the worker only over the public Traefik host — and Traefik routes only /walk, /health, /extract — a Vercel → worker POST /stage hits a host that has no router for /stage and is 404’d at the edge. The cross-network leg is therefore structurally broken.

Why it matters. Two distinct user-facing paths depend on this seam:

UI folder-drop (app/api/ingest/folder-drop/route.ts) — a human drops a file.
Agent/MCP source-less create_content_item (lib/mcp/tools/content.ts) — an agent creates a content item with no source_url.

Both call the same primitive, stageAndWalk (lib/upload/folder-drop.ts:185), and both are currently broken cross-network. The product intent behind this path is the canonical-store guarantee (PRODUCT §OQ-1 Option A, RATIFIED file-backed, content.ts:543-547): the bytes must land in a durable store so the pipeline is the single authority that materialises (and could re-derive) the content_items row — explicitly to avoid “net-new uncontrolled DB ingestion” (the WS-6 anti-goal, content.ts:546). The current breakage means agent-created content either cannot land at all, or lands only via the stopgap URL/RPC path that bypasses the canonical store.

2. Code-intelligence orientation

Per code-intel discipline, the corpus was navigated with gitnexus-class tooling plus direct reads. Domain vocabulary explored: cross-network ingest, stage, walk, corpus, source_documents, storage_path, canonical store. The execution flow under adjudication is the create_content_item → stageAndWalk → worker /stage + /walk → CocoIndex content branch → source_documents + content_items upsert chain. Every current-behaviour claim in §3 carries a file:line citation; no claim is asserted from prose.

Empirical-verification note (Q-EX2 scope). This RESEARCH cites in-repo code only (no external-library API surface is asserted here), so the import-and-call gate has no external symbols to verify at this stage. Forward-pointer: Options C and E below would require a Supabase Python Storage client (supabase-py storage.* / signed-URL download) and/or a CocoIndex object-storage source connector that do not exist in the worker today — those MUST be import-and-call verified against the pinned requirements.txt versions before any PRODUCT/TECH spec adopts them.

3. Verified current-state ground truth

3.1 The `stageAndWalk` client contract (`lib/upload/folder-drop.ts`)

Leg 1 — POST ${COCOINDEX_WORKER_URL}/stage (:212-216): multipart/form-data with file (raw bytes), destPath, titlePrefix. No Authorization header is sent (contrast Leg 2). Timeout 30 s (:95). Expects a JSON {destPath, requestId} 2xx (:243-252).
Leg 2 — POST ${COCOINDEX_WORKER_URL}/walk (:262-266): header Authorization: Bearer ${CRON_SECRET}. Timeout 10 s (:96). A 409 (walk already in flight) is treated as success — the staged file joins the running walk (:278-288).
Config (:131-169): requires both COCOINDEX_WORKER_URL and CRON_SECRET; a missing value is a loud FolderDropError('config', …), never a silent no-op.
destPath is consumed verbatim as the uuid5 PK seed (INV-1, :17-22, :99-129); only absolute/..-escaping paths are rejected.
Return shape (:61-74): {destPath, stageRequestId, sourceFile} — no content_items row id. sourceFile (basename of destPath) is the poll correlation key against content_items.source_file.

3.2 The worker HTTP app (`scripts/cocoindex_pipeline/server.py`)

Route	Auth	Edge-routed?	Behaviour
`GET /health` (`:249`)	none	yes	liveness; reflects worker-thread health (503 on crash).
`POST /stage` (`:265`)	NONE	no (Inv-13)	writes multipart bytes to the local-fs `COCOINDEX_SOURCE_PATH` corpus dir; named-400 on mis-wire; does NOT trigger ingestion.
`POST /walk` (`:450`)	bearer `CRON_SECRET`, fail-closed 503/401 (`:487-498`)	yes	one-shot `update_blocking(live=False)` corpus walk; single-flight 409; 202 accepted.
`POST /extract` (`:553`)	bearer `EXTRACT_API_TOKEN` (distinct blast radius, `:592-602`)	yes	pure HTML cleaner; 20 MB cap; 429 rate-limit.

/stage is genuinely unauthenticated — the handler does no bearer check; its only guard is COCOINDEX_SOURCE_PATH presence + path-containment realpath check (:299-368).
Corpus discovery is local-fs (:300, :362-372): /stage writes into the mounted volume at COCOINDEX_SOURCE_PATH (e.g. /cocoindex-state/corpus). There is no continuous watcher; ingestion fires only on an explicit /walk (:273-278).
DB connection is asyncpg over the Postgres pooler via COCOINDEX_DB_DSN (flow.py:1538-1562) — NOT the Supabase client SDK. The worker carries SUPABASE_* env (see §3.3 keyset) but does not use a Supabase Storage client anywhere in the Python pipeline (grep: no storage/bucket/download SDK usage in scripts/cocoindex_pipeline/).

3.3 Traefik rules + Inv-13 + the parity guard

All four composes route exactly /walk + /health + /extract, never /stage:
- docker-compose.platform.yaml:100 (Inv-13 comment :77-84)
- docker-compose.platform-staging.yaml:101 (:78-85)
- docker-compose.staging.yaml:109 (:87-94)
- docker-compose.production.yaml:86 (:64-71)
No ports:/expose: mapping on the cocoindex service (verified absent) — the service is reachable only on the internal Docker bridge. server.py:881-885 documents this as the primary Inv-13 enforcement (the bind is 0.0.0.0 only for bridge reachability).
The parity guard (scripts/ci/check-compose-parity.ts):
- Asserts the Traefik rule includes /walk + /health + /extract (REQUIRED_TRAEFIK_PATHS, :89) and excludes /stage (:232-238).
- POSTURE CORRECTION (honest, vs the dispatch brief). The dedicated workflow compose-parity.yml is WARN-ONLY / NON-REQUIRED — it runs on push to main, is NOT listed in ci.yml, and is not a merge gate (guard header :21-26). It exits non-zero on drift but does not block a PR by itself.
- However, a vitest test __tests__/deploy/compose-parity.test.ts exercises the same guard logic against the real compose, asserting drifts == [] (:49-51) and explicitly asserting that a widened scope routing /stage is flagged (:109-118). That test runs under bun run test, which is PR-blocking. So routing /stage would fail the blocking suite — via the unit test, not the warn-only workflow. Any option that edge-routes /stage (Option A) must update both the guard’s REQUIRED_TRAEFIK_PATHS //stage prohibition and this test, and revise the Inv-13 wording in all four composes.

3.4 `source_documents` schema (`squash_baseline.sql:7462`)

storage_path text NOT NULL, content_hash text NOT NULL, mime_type NOT NULL, file_size integer NOT NULL, source_url text (nullable), status CHECK ('uploaded','processing','processed','failed') default 'uploaded', plus workspace_id/uploaded_by/pipeline_run_id/op_id (nullable).

Critical finding — storage_path is polymorphic (overloaded by provenance):

Provenance	`storage_path` value	Citation
URL (reference_ingest)	the normalised source URL	`20260619130100_…sql:56`; `flow.py:2947`
Local-fs file (CocoIndex content branch)	the corpus-relative filesystem path (e.g. `agent-create/foo.md`) — also the uuid5 PK seed	`flow.py:2140`, `:1922`
Binary upload (UI)	a Supabase Storage object key in the `documents` bucket	`app/api/source-documents/[id]/binary-url/route.ts:47`, `:118`

So the same column means a URL, or a corpus fs path, or a Storage object key, depending on how the row was minted. This overloading is load-bearing for the Option C/E adjudication: there is no single, consistent “bytes live here” pointer today.

3.5 The CocoIndex content branch (how a `/stage`’d file becomes rows)

flow.py _ingest_content_branch (:2050) runs on a /walk: Stage 2 binary→markdown (:2075), Stage 3 LLM extraction (:2087-2098), Stage 6 deterministic uuid5 PKs seeded on rel_path (:2110-2111), then upserts source_documents (storage_path = rel_path, content_hash = content_fingerprint, source_url = None, :2137-2158) and content_items (:2172+). The uuid5 seed is rel_path, so re-walking the same path re-derives the same rows — this is the re-ingest/canonical-store mechanism, but it is keyed to the corpus-volume path, not object storage.

3.6 The MCP `create_content_item` branches + the FC stopgap

URL branch (content.ts:549-716): routes through the synchronous reference_ingest RPC (atomic source_documents + reference_items, server-side uuid5 PKs). Returns a real id synchronously. Does not touch the worker. B-25 hard invariant: the RPC signature must not change.
Source-less branch (content.ts:718-795): writes the markdown as a file via stageAndWalk into agent-create/ (:733-753); eventually-consistent, returns id: null + a source_file poll key (:782-788). This is the broken cross-network path.
FC stopgap (verified): functional-correctness.ts:1635 — FC-60 is “exercised via the SYNCHRONOUS reference_ingest”; :2095 — FC-65 creates its delete-test item “with no cocoindex-worker dependency (see FC-60)”. The evals were greened by avoiding the source-less /stage path, not by fixing it.

3.7 `COCOINDEX_WORKER_URL` is a single base URL

.env.example:44 (e.g. https://cocoindex-worker.example.com). The same base is used by /walk (folder-drop.ts:262, intelligence/pipeline.ts:508), /extract (clean-via-worker.ts:63), and /stage (folder-drop.ts:212). Because /walk and /extract must work, this base resolves to the public Traefik host — on which /stage is unrouted. There is no separate internal-only base URL configured for /stage. This is the mechanical root cause.

4. The `/stage` bearer-hardening (cross-cutting sub-decision)

/stage is unauthenticated today (server.py:265) — acceptable only under the Inv-13 premise that it is unreachable off the Docker bridge. It is worth bearer-gating as standalone defence-in-depth regardless of which option wins, because:

It is a file-write endpoint (open(target,'wb'), :371) — the highest-value primitive on the worker.
The eval harness and any future co-resident caller reach it over the bridge; a bearer adds a second layer behind the network boundary (the /walk + /extract precedent already exists — reuse the pattern, but with its own token, not CRON_SECRET, to keep blast radii separate per the /extract precedent :570-574).

Where it lands per option:

Option	`/stage` bearer status
A (public `/stage`)	PREREQUISITE — a public file-upload endpoint MUST be authed; bearer is non-negotiable, not optional.
B (Next.js API → worker seam)	defence-in-depth if B still forwards to `/stage`; required if that seam is edge-reachable.
C / E (storage-mediated)	`/stage` may be retired from the cross-network path (worker pulls instead) but remains for compose-internal/eval use → bearer-harden as hygiene.
Standalone	recommended now, independent of the larger decision (cheap, low-risk).

5. Options

Each option is assessed on: how it works · security posture · complexity/blast radius · latency + the synchronous-UUID question · re-ingest/canonical-store guarantee · cost to build · Inv-13 + parity-guard interaction.

Option A — Bearer-gated public `/stage`

How: add /stage to the Traefik rule in all four composes; bearer-gate _stage_handler (its own token); add the bearer to the stageAndWalk Leg-1 request; invert the parity guard’s /stage prohibition; revise Inv-13.
Security posture: weakest. Creates a public, edge-reachable file-upload endpoint on the ingestion worker — attack surface (upload abuse, path/zip bombs, storage exhaustion) exists even when authed. Directly contradicts the current Inv-13 stance and the deliberate client_max_size 50 MB “compose-internal-only” justification (server.py:688-693). Token leakage = remote write into the corpus.
Complexity/blast radius: 4 composes + guard inversion + guard test + folder-drop.ts Leg-1 + server.py handler + worker redeploy. Touches the security invariant itself.
Latency / synchronous-UUID: unchanged from today — eventually consistent, no synchronous id (PRODUCT §OQ-1 Option A). The agent still polls by source_file.
Canonical-store guarantee: preserved as-is (bytes land in the corpus volume; re-walk re-derives). Note the store is a single-host Docker volume, not object storage — durability/backup is weaker than Supabase Storage.
Cost: medium build, high governance cost (re-opening a ratified security invariant).
Inv-13: revises/weakens it. Both the guard and its blocking test must flip.

Option B — Next.js authenticated ingest API → worker seam

How: a Vercel-side app-authed route accepts content and forwards to the worker over an authenticated channel.
Security posture: better front door (app auth on Vercel), but the hard part is unchanged: it still needs a worker-reachable seam. If that seam is /stage, B collapses into “A with a nicer front door” (still edge-routes a write endpoint). If the seam is storage-mediated, B collapses into C. B is not independent — it is a front-end wrapper that must be paired with A’s or C’s back-end.
Complexity/blast radius: a new Vercel route + whichever back-end seam it wraps.
Latency / synchronous-UUID: depends on back-end; can return a synchronous stub id if paired with a DB-write, otherwise eventual.
Canonical-store guarantee: inherited from the paired back-end.
Cost: additive on top of A or C.
Inv-13: neutral by itself; inherits the paired back-end’s interaction.
Verdict: not a standalone answer — fold its app-auth idea into the chosen back-end.

Option C — Storage-mediated worker-pull (Inv-13 preserved)

How: Vercel writes the bytes to a Supabase Storage bucket (already reachable from Vercel — storage.from(...).upload/createSignedUrl is in use, §3.4). The worker pulls from that bucket (signed-URL download or service-key read) into its corpus dir (or reads it as a CocoIndex source), then walks. No new public endpoint; Inv-13 untouched.
Security posture: strongest. No edge-exposed write endpoint; the bucket is gated by Supabase RLS/service-key; the worker makes outbound HTTPS only (same posture as its existing Anthropic/OpenAI/webhook calls). Object storage gives a real durability/backup story.
Complexity/blast radius: net-new worker capability — the Python pipeline currently has no Supabase Storage client (§3.2). Requires adding supabase-py storage (or httpx signed-URL fetch) to the worker, plus a pull trigger. Creds already present (SUPABASE_URL/SUPABASE_SERVICE_ROLE_KEY in the worker keyset, check-compose-parity.ts:64-82) but unused for Storage today. storage_path overloading (§3.4) must be resolved — is the bucket key the new grain, and does the uuid5 PK seed change from rel_path to the object key (a migration/identity concern, flow.py:1922)?
Latency / synchronous-UUID: eventually consistent by default (upload → trigger → pull → walk). Can be paired with a synchronous DB stub (see §6) if the agent UX needs an id.
Canonical-store guarantee: strongest — bytes durably in object storage; a future re-ingest re-pulls + re-derives. This most literally satisfies “propose-into-store”.
Cost: highest engineering (worker Storage client + trigger + identity/grain decision), lowest governance (Inv-13 preserved).
Inv-13: preserved. Parity guard unchanged. NOTE from the brief confirmed: a storage_path grain already exists in the schema — but it is overloaded, so C must pick a clean grain rather than assume the column already models object storage for the file path.

Option D — Direct DB write via a `content_ingest` RPC (DISCOVERED; the “abandon the worker” option)

How: generalise the synchronous reference_ingest evidence-pair RPC (the FC stopgap already leans on it, §3.6) to source-less/file content — a content_ingest RPC that atomically mints source_documents + content_items server-side, exactly as the URL branch does today.
Security posture: good (owner-gated RPC; no new network seam).
Latency / synchronous-UUID: synchronous id immediately — the best agent UX; chains cleanly.
Canonical-store guarantee: NOT satisfied for bytes. The bytes never land in the corpus, so the CocoIndex pipeline can never re-derive the row from stored bytes — this is exactly the WS-6 “net-new uncontrolled DB ingestion” anti-goal flagged in content.ts:546. (The URL branch’s “guarantee” is re-fetch the URL, not re-derive from bytes — a weaker contract that only works because a URL is itself a durable pointer.)
Cost: low-medium (one RPC, mirrors an existing one).
Inv-13: untouched.
Verdict: viable only if Liam consciously drops the canonical-store-from-bytes guarantee for agent/UI creates and accepts the DB as the authority. Honest framing: this is “make the stopgap the contract”.

Option E — Storage-as-corpus / CocoIndex object-storage source (DISCOVERED; deeper variant of C)

How: instead of pull-into-local-fs, make the CocoIndex source backend read the Supabase Storage bucket directly (CocoIndex supports pluggable sources). The corpus becomes the bucket.
Security posture: same as C (no public endpoint).
Complexity/blast radius: deepest — changes the pipeline’s source connector and the identity grain (uuid5 seed moves off rel_path). Largest migration risk, but the cleanest end-state (one durable store, no local volume, no /stage at all for this path).
Canonical-store guarantee: strongest and most uniform.
Cost: highest; needs the empirical verification of a CocoIndex storage source connector (§2 forward-pointer) before commitment.
Inv-13: preserved (could eventually retire /stage entirely).
Verdict: the right long-term shape, likely too large for the immediate FC-60/65 unblock; record as the north star.

6. The synchronous-UUID tension (cross-option)

A genuine product tension the options expose:

Agent UX wants a synchronous content_items id to chain subsequent calls (get/update/assign). Only D (and a stubbed B) give that today.
The canonical-store guarantee wants bytes durably landed + pipeline-authoritative materialisation, which is inherently eventually consistent (A, C, E).

These can be reconciled by a synchronous stub + async enrichment: write a minimal content_items/source_documents row synchronously (returning an id) and land the bytes durably, with the walk later enriching the same uuid5-keyed row (idempotent upsert — flow.py:2104-2111 already upserts on a deterministic PK). This is a design question for PRODUCT, not a free lunch: it reintroduces a DB write on the create path (the WS-6 tension) and requires the synchronous id and the walk’s uuid5 seed to agree. Flagged as D2 in §9.

7. Comparison at a glance

Axis	A (public `/stage`)	C (storage-pull)	D (DB RPC)	E (storage-as-corpus)
Security posture	weakest (public write)	strongest	strong	strongest
Inv-13	revises	preserved	preserved	preserved
Parity guard + blocking test	must flip both	unchanged	unchanged	unchanged
Synchronous id	no	no (unless stubbed)	yes	no (unless stubbed)
Canonical store (re-derive from bytes)	yes (Docker vol)	yes (object store)	no	yes (object store)
Net-new worker code	small	medium (Storage client)	none	large (source connector)
Build cost	medium	high	low	highest
Governance cost	high	low	low	low
Unblocks FC-60/65	yes	yes	already (stopgap)	yes

(B omitted from the matrix — it is a front-end wrapper over A or C, not an independent back-end; §5 Option B.)

8. Recommendation

Primary recommendation: Option C (storage-mediated worker-pull), with the /stage bearer-hardening done now as standalone defence-in-depth, and Option E recorded as the long-term north star.

Rationale:

It preserves Inv-13 — the security invariant Liam ratified deliberately. No public write endpoint, no parity-guard inversion, no re-litigation of a settled security stance. Option A’s public file-upload endpoint is a real, permanent attack surface to buy back a path that C delivers without it.
It delivers the strongest canonical-store guarantee — bytes durably in object storage (with a backup story), re-pullable + re-derivable, which is the literal “propose-into-store” intent (PRODUCT §OQ-1 Option A) — and it does so without the WS-6 anti-goal that Option D accepts.
The Storage primitive already exists on the Vercel side (§3.4) — only the worker pull is net-new, and the worker already makes outbound HTTPS and already holds the Supabase creds; the missing piece is a Storage client, not network reachability.
It composes cleanly with the synchronous-stub reconciliation (§6) if the agent UX needs an id — without committing to the DB-authoritative model wholesale.

Caveats that gate the recommendation (must be resolved in PRODUCT/TECH):

The storage_path overloading (§3.4) must be resolved into a clean grain — do not assume the existing column already models the object-storage path for file provenance.
The uuid5 identity seed (currently rel_path, flow.py:1922) must be decided: keep the corpus-relative path as the seed (pull writes bytes to a path), or re-key to the object key (cleaner, but a migration/identity change).
The worker Storage client (supabase-py storage or httpx signed-URL) must be import-and-call verified against the pinned requirements.txt (§2 forward-pointer) before adoption.

If Liam prioritises immediate agent-UX (synchronous id) over the bytes-re-derive guarantee for agent/UI creates, Option D is the honest fast path (it makes the existing FC stopgap the contract) — but it explicitly abandons the canonical-store-from-bytes guarantee and accepts the WS-6 anti-goal. That is a product call, not a technical one — hence the decision register below.

9. Decision register (ratify before a PRODUCT spec)

#	Question	Options / default
D1	Which ingest contract? Storage-pull (C), public `/stage` (A), DB-RPC (D), storage-as-corpus (E).	Default: C (Inv-13-preserving, strongest store). Adjudicated §8.
D2	Synchronous id or eventual consistency for agent/UI creates? If synchronous, adopt the §6 stub-then-enrich pattern (id agrees with the walk’s uuid5 seed)?	Default: keep eventual (current PRODUCT §OQ-1 Option A) unless agent UX demands an id.
D3	Is the bytes-re-derive canonical-store guarantee a hard requirement for agent/UI creates, or is URL-style “re-fetch / DB-authoritative” acceptable (which would permit D)?	This single answer decides C/E vs D.
D4	Bearer-harden `/stage` now, as standalone defence-in-depth, independent of D1? With its own token (not `CRON_SECRET`), per the `/extract` blast-radius precedent?	Recommended: yes (cheap, low-risk; §4).
D5	Inv-13 disposition. Confirm it stays intact (C/D/E) — or, only if A is chosen, ratify the explicit weakening + the guard/test/4-compose edits (§3.3).	Default: keep Inv-13.
D6	`storage_path` grain & identity seed. Resolve the §3.4 overloading; decide whether the uuid5 PK seed stays `rel_path` or moves to the object key (`flow.py:1922`).	Required for C/E; migration-sensitive.
D7	Storage bucket & lifecycle. New dedicated bucket vs reuse `documents`; retention/cleanup of agent-create blobs; workspace scoping/RLS.	Required for C/E.
D8	Worker pull trigger. Does the existing `/walk` enumerate the bucket, or is there a per-object pull step before the walk? Does this change the `/walk` contract?	Required for C/E.
D9	Scope/tier. Is this a full spec-chain Task, or PRODUCT+PLAN? (Compound: schema/identity grain + worker code + product contract → likely full chain.)	Recommend full chain.

10. Honest caveats & corrections to the dispatch brief

Parity-guard posture. The brief described the guard as “an active CI guard … fails the build if a Traefik rule routes /stage”. Precisely: the dedicated workflow is warn-only / non-required, but a PR-blocking vitest test (__tests__/deploy/compose-parity.test.ts) asserts zero drift on the real compose and explicitly flags a /stage route — so routing /stage does fail the blocking suite, via the test, not the workflow (§3.3). The enforcement is real; the mechanism differs from the brief.
storage_path is not a ready-made Option-C grain. The brief’s note (“source_documents.storage_path already exists — investigate whether this is already the grain”) resolves to: it exists but is overloaded three ways (§3.4), so C cannot assume it already models object storage — D6 must settle the grain.
The worker can reach Supabase, but not via a Storage client today. Network feasibility for C/E is confirmed (outbound HTTPS + creds present), but the Python pipeline has no Storage SDK usage — that is net-new code requiring empirical verification (§2).
Option B is not independent — it is a Vercel front-door that must wrap A’s or C’s back-end (§5).