CocoInsight scoping — staging smokes + full re-ingest debugging aid

Provenance: S299 (2026-06-02). Read-only scoping. Authored ahead of the Path-A / Path-B staging smokes and the full re-ingest, to decide whether CocoIndex’s CocoInsight (“Insights”) inspector should be wired in as a per-target write / lineage debugging aid.

Bottom line (one-liner): defer-not-worth-it — CocoInsight’s server is not present in cocoindex 1.0.3 (the version we run), so there is nothing to enable; even if it were, it inspects the LMDB engine memo state, not the per-target Supabase row writes we actually need to debug. Use direct Supabase SQL (which we already have) for the smokes. See §6 for the full recommendation and the narrow conditions under which a later re-look (sidecar task) would make sense.

1. What CocoInsight IS

CocoInsight is CocoIndex’s dev-time flow inspector. It is a browser UI hosted at https://cocoindex.io/cocoinsight that connects to a local HTTP server which CocoIndex’s own CLI starts (cocoindex server …). The UI renders the engine’s view of a flow: declared flows / sources / targets, the rows the engine is tracking, and lineage between processing steps — sourced from the engine’s LMDB internal state (the same ops/memo store, not your destination DB).

Key architectural point (verified from the 0.x docs): the data stays local. CocoInsight is a static front-end served from cocoindex.io; it talks to your locally-bound server over CORS. The server is what reads LMDB and serialises flow/row/lineage info to the browser. So the page is remote, but the pipeline internals are served by a process you run — gated only by bind address + CORS (see §4 security).

What it shows is the engine’s incremental-processing bookkeeping (what the engine thinks each processing unit produced, fingerprints, lineage), not a SQL-level read of the rows that landed in content_items / source_documents / entity_mentions / q_a_extractions in Supabase. This distinction is the crux of the fit assessment (§5).

2. Can we enable/run it with cocoindex 1.0.3? — No.

This is the decisive finding. The version we run does not ship the server.

Verified against the installed package (pip show cocoindex → Version: 1.0.3, Location: ~/Library/Python/3.14/.../site-packages/cocoindex):

The 1.0.3 CLI has exactly five commands. Running the installed CLI:

1
$ python3 -m cocoindex.cli --help
2
Commands:
3
  drop    Drop an app and all its target states.
4
  init    Initialize a new CocoIndex project.
5
  ls      List all apps.
6
  show    Show the app's stable paths.
7
  update  Run an app in catch-up mode.

There is no server command:

1
$ python3 -m cocoindex.cli server --help
2
Error: No such command 'server'.

The console-script entry point is only the CLI group. cocoindex-1.0.3.dist-info/entry_points.txt: cocoindex = cocoindex.cli:cli. The Click group (cli.py) registers only ls / show / update / drop / init.
The public Python API has no serve/start-server function. dir(cocoindex) exposes start, start_blocking, stop, stop_blocking, update, update_blocking, show_progress, the LiveMap* / LiveComponent* types, etc. — nothing named server, serve, insight, or equivalent. App exposes only update / update_blocking / drop / drop_blocking.
The Rust core does not contain the server. strings core.abi3.so | grep -iE 'cocoinsight|49344|server_address|run_server|lineage|/api/' → no hits. The HTTP server that backed CocoInsight in the 0.x line is not compiled into this wheel.
ServerSettings is vestigial. cocoindex/setting.py (and _internal/setting.py) still define ServerSettings with address = "127.0.0.1:49344", cors_origins, and the env vars COCOINDEX_SERVER_ADDRESS / COCOINDEX_SERVER_CORS_ORIGINS. query_handler.py still references “tools like CocoInsight”. But ServerSettings is never instantiated or consumed anywhere in the package — it is dead config left over from 0.x. Setting COCOINDEX_SERVER_ADDRESS in our compose would do nothing.

Why: CocoIndex v1.0.0 (released 2026-04-22) did a “CLI restructuring — consolidated CLI with cocoindex update as the primary interface.” The server command (and the CocoInsight server path) is part of the 0.x surface; the official cocoindex server … -ci / cocoinsight_access docs that turn up in search describe the 0.x product (note the docs-v0/ doc paths). It is not in 1.0.x.

2a. For historical reference — how it worked in cocoindex 0.x

(Documented here only so the option is fully understood; none of this applies to 1.0.3.)

1
# 0.x only — DOES NOT EXIST in 1.0.3
2
cocoindex server path/to/app.py -a 0.0.0.0:49344 -ci -L

0.x flag	Meaning
`-a, --address IP:PORT`	Bind address; default `127.0.0.1:49344` (else `COCOINDEX_SERVER_ADDRESS`)
`-ci, --cors-cocoindex`	Allow `https://cocoindex.io` (the CocoInsight UI) CORS access
`-c, --cors-origin`	Comma-separated allowed origins
`-cl, --cors-local PORT`	Allow `http://localhost:<port>`
`-L, --live-update`	Watch sources + apply continuously while serving

Then open https://cocoindex.io/cocoinsight and point it at the server address.

2b. What 1.0.3 does give us for introspection

The replacement introspection surface in 1.0.x is static-structure only, via the CLI:

cocoindex ls — list apps + persisted state per environment.
cocoindex show <app> [--tree] — print the app’s stable paths (the processing-graph node tree). show --db <lmdb> --app-name <name> can read straight from a persisted LMDB without importing the module.

These print the shape of the flow (stable paths / components). They do not report per-target row counts, the rows a target wrote, or row-level lineage. There is no 1.0.3 CLI/API that surfaces “target X wrote N rows” from LMDB.

(The Python cocoindex.inspect API mirrors this: iter_stable_paths, list_stable_paths* — stable paths only, no row/lineage data.)

3. Our actual runtime (for grounding the fit assessment)

We run cocoindex 1.0.3 as the on-prem ingestion worker on Coolify (IONOS 77.68.122.71), in deploy/coolify/docker-compose.{production,staging}.yaml, alongside pullmd / playwright / trafilatura.
The worker does not use the CLI. scripts/cocoindex_pipeline/server.py runs KH_PIPELINE_APP.update_blocking(live=True) on a daemon thread, fronted by an aiohttp /health wrapper on port 8080 (the only exposed port; healthcheck probes it via bash /dev/tcp).
Engine state is LMDB at /cocoindex-state/lmdb (COCOINDEX_DB), on the cocoindex-state (prod) / cocoindex-state-staging (staging) named volume.
Writes to Supabase are user-managed row-upserts on a KH-owned asyncpg pool, each target on its own autocommit connection — see cocoindex-write-model.md. The destination rows live in Supabase, which LMDB/CocoInsight has no view of.

4. Fit with the Coolify stack (hypothetical — if a server existed)

Even setting aside §2 (it doesn’t exist in 1.0.3), wiring it as a sidecar would be awkward:

LMDB single-writer. LMDB allows one writer + many readers, but only within the same host process/OS lock scope. CocoIndex opens the env with a map and its own lock; pointing a second process (a CocoInsight server container) at the same cocoindex-state(-staging) volume while the worker holds it is not a supported concurrent-reader contract and risks lock contention / MDB_BAD_RSLOT-class issues. A safe read would likely require the worker stopped — i.e. you can’t watch a live smoke, which defeats the purpose.
Co-resident only. Because of the lock + the shared on-disk env, it would have to be a sidecar in the same compose mounting the same volume, not an independent service. That is more coupling, not less.
Exposing a UI port ties to {66.13}. The worker today exposes only :8080 (health, not even publicly routed). A CocoInsight server would add another listening port that the cocoindex.io front-end must reach over CORS — meaning a publicly reachable address (or a tunnel), which pulls in the DNS/TLS/reverse-proxy work tracked under {66.13} (staging FQDN + SSL “once DNS/SSL lands”). Net new exposure surface on the ingestion host.

5. Security assessment

CocoInsight is a pipeline-internals inspector — by design it surfaces flow structure and the engine’s row/lineage view, which for our flow includes LLM-extracted CLIENT content (extracted Q&A, entity mentions, chunk text held in the engine’s memo state). That is squarely the wrong thing to expose right now:

(a) ID-68 (repo private → public + IP separation). ID-68 is the active, irreversible de-ID / public-flip workstream (v1-completion-sequence.md §ID-68). Standing up an inspector that renders pipeline internals + client content — and whose UI is a third-party-hosted page at cocoindex.io reaching a port on our ingestion host — cuts directly against the IP-separation / minimise-exposure posture. The access model is bind-address + CORS only — there is no authentication on the cocoindex server; anyone who can reach the bound port can read the flow. On a public-routed host that is an open data-exposure hole.
(b) S299 pipeline_runs.error_message leak (already-known). S299 found the terminal failure webhook writes str(exc) un-redacted to pipeline_runs.error_message (flow.py:2335 → record-run.ts:212), echoing LLM-extracted client content verbatim (tracked in v1-ledger-mutation-plan-s299.md § “SECURITY-ADJACENT”, fix = bl-165 Option D ~5 lines). This proves we already have an un-redacted-client-content exposure path in the pipeline surface; adding a second, broader, unauthenticated exposure surface (CocoInsight) before ID-68 closes would compound it. Conversely the existing leak is the higher-priority, concrete hardening item — fix that, don’t add CocoInsight.

Verdict: even if available, CocoInsight would be unacceptable to expose during the ID-68 public-flip window without auth + private networking. That is a strong independent reason to defer beyond the “it doesn’t exist in 1.0.3” reason.

6. Concrete value for the smokes — and the recommendation

Does it help see which rows each target wrote / why a target produced 0 rows? No — and this is true independent of the version problem:

The smokes’ failure mode is per-target Supabase writes (e.g. the bl-158-style entity_resolution target silently writing 0 rows, or source_documents going 0→N as the cleanest oracle per cocoindex-write-model.md §6). Those rows live in Supabase.
CocoInsight reads LMDB engine state, not Supabase. It could show the engine attempted a processing unit, but the authoritative “did target X write N rows” signal is a SELECT count(*) against Supabase — which we already have, directly, with no new service, no new port, no exposure, and no LMDB-lock risk.
A target writing 0 rows is best diagnosed by exactly the existing tools: Supabase SQL row counts per target + the worker’s docker logs (declare_row path) + pipeline_runs. These see the real write path; CocoInsight does not.

RECOMMENDATION: defer-not-worth-it.

Not available in 1.0.3 — there is no server command to enable (§2). Adopting it would require either downgrading to 0.x (regressive — we depend on 1.0.x write-model behaviour documented in cocoindex-write-model.md) or waiting for/over the chance that a future 1.x reintroduces the server.
Wrong layer even if present — it inspects LMDB memo state, not Supabase target rows; the 0-rows debugging need is served better and already by direct Supabase SQL (§6).
Security-disqualifying during ID-68 — unauthenticated, exposes client content, pulls in {66.13} DNS/TLS, compounds the known S299 error_message leak (§5).

For the Path-A / Path-B / ID-42.10 staging smokes + the full re-ingest, use the existing toolchain: per-target SELECT count(*) in Supabase (staging branch), the worker docker logs, and the pipeline_runs rollup. source_documents 0→N remains the cleanest content-half oracle.

Narrow re-look condition (only then revisit as a sidecar task): if (i) a future cocoindex 1.x reintroduces a server / inspector command in our pinned version and (ii) ID-68 has closed with the ingestion host on a private network behind auth and (iii) a debugging need arises that genuinely requires engine-memo lineage (not Supabase rows) — then scope it as a co-resident, auth-gated, private-only sidecar under a dedicated task. Until all three hold, there is nothing to do.

7. Sources

Installed package (authoritative for our version): pip show cocoindex → 1.0.3; python3 -m cocoindex.cli --help (five commands, no server); python3 -m cocoindex.cli server --help → Error: No such command 'server'; dir(cocoindex) public API; cocoindex/setting.py ServerSettings (vestigial); cocoindex/query_handler.py (CocoInsight mention); _internal/core.abi3.so strings (no server symbols); cocoindex-1.0.3.dist-info/entry_points.txt.
CocoIndex 0.x docs (historical server behaviour only): https://cocoindex.io/docs-v0/core/cli (server flags), CocoInsight access docs (https://cocoindex.io/docs/cocoinsight_access per search; page 404s on direct fetch — flag/behaviour captured from the search excerpt + the v0 CLI page).
CocoIndex v1.0.0 release (CLI restructuring, 2026-04-22): https://github.com/cocoindex-io/cocoindex/releases.
KH internal: docs/themes/canonical-pipeline/reference/cocoindex-write-model.md (write model, 0-rows oracle), .../v1-ledger-mutation-plan-s299.md + .../v1-completion-sequence.md (S299 error_message leak, ID-68), deploy/coolify/docker-compose.{production,staging}.yaml (runtime topology), scripts/cocoindex_pipeline/server.py (update_blocking(live=True) daemon-thread boot), docs/runbooks/staging-coolify-cocoindex.md ({66.13} DNS/TLS).