Cocoindex Operational Capabilities — KH Mapping

Cocoindex Operational Capabilities — KH Mapping

Author: Phase 0.9 Prerequisite-2b investigation (Cocoindex operational sub-agent). Date: 13/05/2026. Status: RESEARCH — not a binding decision. Routes into WP4 02-data-flow.md, 03-tech-stack.md, and Liam ratification. Cocoindex version referenced: 1.0.3 (matches S1 / S2 / S14 probe installs).

---

§0 — Scope + sources

This doc maps cocoindex’s operational surface (CocoInsight, persistent control plane, freshness, scheduling, failure management) onto the KH tables / OQs / findings that defer to “after cocoindex investigation”.

Cocoindex sources reviewed (web, no source-code read this pass beyond what S1/S2/S14 already inspected):

• Front page https://cocoindex.io/ — names the eight always-on control-plane subsystems verbatim: live caching, pipeline catalog, version tracking, continuously learning, lineage, task scheduling, metrics collection, failure management.
• https://cocoindex.io/blogs/cocoinsight — comprehensive CocoInsight blog (what it is, lineage colouring, zero-retention).
• https://cocoindex.io/docs/getting_started/overview/ — declares the inspectable-everything claim + fault-tolerant runtime.
• https://cocoindex.io/docs/advanced_topics/internal_storage/ — LMDB key-value store, default 4 GiB map, 1024 named DBs, tunable.
• https://cocoindex.io/docs/advanced_topics/memoization_keys/ — fingerprint hierarchy (__coco_memo_key__() -> registered fn -> automatic canonicalisation -> serialised-object fallback); MemoStateOutcome two-phase validation (cheap state check, then expensive content hash).
• https://cocoindex.io/blogs/incremental-processing/ — three CDC strategies (push-based, metadata-scan, source-specific); lineage records for stale-row removal; behavior_version parameter for custom-function cache invalidation.
• https://medium.com/@cocoindex.io/data-consistency-in-indexing-pipelines-cocoindex-paradigm-d3229d29eb36 — three consistency strategies (key superset, ordinal bookkeeping, versioned-key soft delete); explicit ordinal-prevents-overwrite semantics.
• https://cocoindex.io/docs/cli — CLI surface (init/ls/show/update/drop); update flags: -L --live / --full-reprocess / -f / -q / --reset.
• https://news.ycombinator.com/item?id=43772582 (cocoindex Show HN) — author’s framing of data freshness as the design goal.

KH context reviewed (file paths absolute):

• /Users/liamj/Documents/development/knowledge-hub/CLAUDE.md — recordPipelineRun() gotcha (G6), pipeline_runs usage discipline.
• /Users/liamj/Documents/development/knowledge-hub/.claude/skills/cocoindex/SKILL.md — internal cocoindex skill; v1 API surface; @coco.fn(memo=True) is the standard memo decorator.
• /Users/liamj/Documents/development/knowledge-hub/docs/plans/phase-0-investigation/0.9-intended-architecture.md — Scenario A (S1-confirmed) baseline; audit_log planned new table; content_history kept; pipeline_runs retain-or-retire deferred.
• /Users/liamj/Documents/development/knowledge-hub/docs/plans/phase-0-investigation/0.9-spike-S1-cocoindex-schema-coupling.md — managed_by="user" lets KH keep all schema authority; cocoindex only writes rows. Crucial: cocoindex’s control plane runs alongside KH’s schema, not over it.
• /Users/liamj/Documents/development/knowledge-hub/docs/plans/phase-0-investigation/0.9-spike-S14-cocoindex-concurrency.md — v1 Cloud Run topology = single-orchestrator-instance + per-instance LMDB; concurrent writers are safe but waste work; crash recovery is clean.
• /Users/liamj/Documents/development/knowledge-hub/docs/plans/phase-0-investigation/10-feedback-investigation-findings/00-synthesis.md — N5 (upload silent-fail), N6 (pipeline_runs fate), N7 (op_id propagation).
• /Users/liamj/Documents/development/knowledge-hub/docs/plans/phase-0-investigation/10-feedback-investigation-findings/01-q19-q24-source-documents-audit-trail.md — Finding 01 §4.2 post-cocoindex audit-shape table.
• /Users/liamj/Documents/development/knowledge-hub/docs/plans/phase-0-investigation/feedback-findings-review.md — Liam’s routing of OQ-Q24-A / OQ-CX33-A / Theme C / Theme D to this sub-agent.

KH tables in scope:

• pipeline_runs (35 prod rows; KH-owned, populated via lib/pipeline/record-run.ts / start-run.ts / update-progress.ts).
• pipeline_failures — does not exist in current schema (grep returned 0 results across lib/, scripts/, supabase/migrations/). The brief’s framing of pipeline_failures is hypothetical; clarified at §2.4.
• content_history (1223 prod rows; trigger-driven via auto_version_content_items; five change_type values in use).
• audit_log (planned new table per Finding 01 §4.2 + Rec 7; not yet built).
• source_documents (629/0 prod fill; Option α/β decision pending — §4.6 of intended-arch).

Caveats up front (per brief’s constraint to flag uncertainty):

1. CocoIndex’s docs are product-marketing-shape, not internal-engineering-shape. The eight subsystems are named but rarely specced field-by-field. Some claims below (e.g. metrics catalogue, retention semantics on the ops-DB ledger) are inferred from blog posts + the Rust-engine string-search done in S14, not from a single canonical spec doc.
2. pipeline_failures is not a current KH table. The brief asked whether cocoindex replaces it; I read this as a forward-looking question (does KH need such a table once cocoindex is in?).
3. CocoInsight’s HTTP API surface is undocumented at the page level (the docs page cocoinsight_access returns 404 to WebFetch; the blog post + front page hint at the architecture). The CLI page lists no server command; the blog explicitly says cocoindex server -ci main is how it’s started. This is a doc-page-not-published gap, not a missing capability — confirmed in HN/Substack coverage.
4. Source-code follow-up is recommended before binding Recommendation 4 below (CocoInsight as audit-log substitute) — the on-prem CocoInsight server’s persistence model needs to be inspected in the cocoindex PyPI package + the cocoindex-io/cocoindex GitHub repo. This pass is web-evidence only.

---

§1 — CocoInsight (lineage + observability)

1.1 What it does

CocoInsight is “a platform for data lineage and data observability” (front-page tagline). It is a web UI that connects to a running cocoindex server via HTTP and renders:

1. Dataflow visualisation. Right panel shows the pipeline graph (@coco.fn nodes + connectors + targets); left panel shows step-by-step data preview (“Each field is tied to an input or output of a step in the dataflow transformation”).
2. Per-field lineage. Click any field or transformation step to inspect lineage:
   • Direct upstream dependencies (exact fields) — coloured blue.
   • Direct downstream outputs (exact fields) — coloured green.
   • “Every output field can be traced back to the exact set of input fields and operations that created it” (blog verbatim).
3. Per-record data preview at every node — before/after of the data at every transformation node, including intermediate values that aren’t persisted to any target.
4. Query debugging + statistics — listed as planned features in the blog; not yet GA.

1.2 Access mechanics (verbatim where possible)

Aspect	Detail
Launch	Run cocoindex server -ci main (per the CocoInsight blog). The -ci flag opts the local cocoindex server into CocoInsight access. Note: the server subcommand is NOT documented on the public CLI reference page (/docs/cli) which only lists init/ls/show/update/drop. The blog is the canonical reference. Action: confirm against cocoindex --help output in Phase B implementation.
Topology	UI runs on https://cocoindex.io/cocoinsight (managed-SaaS); points at YOUR cocoindex server (on-prem / Cloud Run / local).
Transport	HTTP. UI fetches flow schema + data from your server’s HTTP APIs.
Retention	”CocoInsight itself has zero data retention with your flows schema and data” (blog verbatim). The UI is a thin client; all state lives on your cocoindex server’s LMDB ops-DB.
Auth	Documented only obliquely (the -ci flag is the opt-in). Auth model details are not in the blog — TBD: investigate Phase B.

1.3 Data model — what CocoInsight reads

CocoInsight queries the cocoindex server’s HTTP API to retrieve:

• Flow definition. The @coco.fn graph (Python source structure, function names, target declarations).
• Per-row data state. Current value of every field at every node (read from LMDB cache + the materialised target rows in Postgres).
• Lineage edges. Source -> transformation -> target chains. These are derived from cocoindex’s internal source-to-target key-mapping store (see Medium article on data consistency, §1: “keys tracked in the internal storage is always a superset of those really exist in the target store”).

Because retention is zero on the UI side, the durability + query semantics of CocoInsight are entirely those of the underlying LMDB ops-DB:

• LMDB is per-instance, on the cocoindex server’s local filesystem.
• Default size 4 GiB virtual address space; 1024 named LMDB databases; tunable.
• No replication, no shared store across Cloud Run instances (S14 confirms LMDB-per-container is the v1 topology). This means CocoInsight on one instance shows only that instance’s view.

1.4 What CocoInsight captures (events)

Not exhaustive based on web docs alone, but inferable from the eight control-plane subsystems + the consistency model:

Event class	Captured?	Detail
Per-row source ingestion (path + content hash + ordinal)	Yes	Required by ordinal bookkeeping + lineage.
Per-@coco.fn invocation (fingerprint + memo hit/miss + duration)	Yes	Required by memo cache validation. Surfaced as “throughput / cost” metrics.
Per-row target write (table + PK + values + write timestamp)	Yes	Required by 3-phase commit consistency.
Per-row delete (soft-delete flag + GC schedule)	Yes	Required by versioned-key-soft-delete strategy.
Schema/code-change-triggered cache invalidation (fingerprint diff)	Yes	Required by behavior_version semantics.
Per-flow-run summary (rows scanned / rows changed / rows skipped)	Implied	Throughput metric exposed via UI.
Cost (LLM tokens, embedding calls, etc.)	Yes	Front-page lists “cost” alongside throughput + freshness. Granularity unclear — likely per-flow-run, possibly per-function.
User actions in the UI (clicks, etc.)	No retention	Zero-retention promise.
Compliance / business-event log (who-did-what-when human-readable)	NO	CocoInsight is a developer-pipeline-observability surface, not a compliance audit log. This is the load-bearing gap for KH §5 below.

1.5 KH need mapping

KH need	Covered by CocoInsight?	Notes
Per-row data lineage (which input bytes produced which output row)	YES — fully replaces ad-hoc tracing. Eliminates need for KH-side lineage queries on content_history.parent_id, source-doc-diff UI, etc.	Maps to: edit-flow §6 UC1 traceability, Q&A markdown sidecar provenance, Knowledge Map source-cluster view.
Per-flow-run throughput + cost dashboards	YES — out of the box. Front-page calls this out explicitly.	Maps to: pipeline_runs.progress JSONB telemetry; pipeline-health-tab UI (recordPipelineRun() consumers).
Per-@coco.fn cache hit/miss + memoisation effectiveness	YES.	Maps to: cost-tracking pattern Q4.12 (skill-seekers-style cost dashboards) — supersedes that pattern entirely.
Compliance audit log (human-readable change events, who/when/why per record)	NO.	KH audit_log (planned, Finding 01 Rec 7) is NOT replaced by CocoInsight. CocoInsight tracks transformation-level explainability, not business-event narrative. The two are orthogonal: cocoindex says “row R was derived from bytes X using fn F at time T”; audit_log says “Liam archived workspace W at time T because reason R”.
Per-content_item change log (KH content_history, 1223 rows)	Partial. CocoInsight + cocoindex’s source-to-target-key store covers the lineage portion (source bytes -> content_items row -> which fn produced it). The semantic change_type discriminator (create/owner_change/archive/edit/metadata_change) is NOT a cocoindex concept.	Trigger-driven auto_version_content_items should KEEP firing post-cocoindex; the rows still serve user-facing audit UX. CocoInsight COMPLEMENTS — it doesn’t replace.
Op_id rollback (UC3 sweep / UC8 dedup / UC9 taxonomy)	NO.	Op_id is a KH-business-domain concept; cocoindex has no equivalent. Op_id-stamped writes still need to land in content_history.op_id (or audit_log.op_id) per Finding 01 Rec 6 + Rec 7.
Multi-instance (multi-Cloud-Run) consolidated view	NO at v1.	LMDB-per-instance + zero-retention UI = each instance has its own view. v1 topology is single-orchestrator-instance per S14, so this is non-blocking. Plan for v2 if scaling beyond single-instance ingest.

1.6 Recommendation

Adopt CocoInsight for developer-/operator-facing pipeline observability + lineage debugging. Do NOT use it to replace KH-business-domain audit trails.

Specifically:

1. Retain content_history (1223 rows, 5 change_types). Trigger continues to fire on every materialised content_items change. The lineage that cocoindex tracks at the ingestion-pipeline layer COMPLEMENTS the change-log layer — they answer different questions.
2. Build the planned audit_log table per Finding 01 Rec 7 (id, op_id, op_type, actor_id, workspace_id, occurred_at, op_summary, affected_record_count, plus a new cocoindex_run_ref text column for cross-reference). This is a human-readable compliance surface; CocoInsight is a developer surface. The cocoindex_run_ref lets a compliance row link out to the CocoInsight UI for the underlying technical lineage.
3. Retire the ad-hoc per-component cost tracking previously planned for Q4.12 (skill-seekers-pattern) — CocoInsight already provides throughput + cost metrics.
4. Retire the source-doc-diff UI fate question (Finding 01 Rec 3, app/documents/[id]/diff/page.tsx) IF the diff visualisation moves to CocoInsight. Note: CocoInsight shows pipeline-stage data, not raw before/after-text-diff — so the diff UI may stay as a content-comparison surface independent of the lineage surface. Decision: TBD post-empirical CocoInsight comparison; not a hard retire.
5. Plan for v2 multi-instance consolidation. If KH ever scales beyond single-instance cocoindex ingest, a shared ops-DB or aggregator layer is needed (cocoindex doesn’t provide this).

---

§2 — Persistent Data Pipeline (the “control plane”)

2.1 What it does — the eight subsystems

The front page lists eight always-on subsystems. Mapped to what each does (web-evidence; some inferences flagged):

#	Subsystem	What it does	Evidence
1	Live caching	Per-@coco.fn memoisation; cache hit ⇒ skip invocation.	@coco.fn(memo=True) decorator (KH skill SKILL.md L99-100; SDK references).
2	Pipeline catalog	Registry of all Apps + their @coco.fn graphs + connector configurations. Surfaces via cocoindex ls.	CLI command ls lists registered apps. Catalog stored in LMDB.
3	Version tracking	Per-source SourceLogicFingerprint + per-field FieldDefFingerprint. Detects logic-vs-data-vs-context changes independently. behavior_version parameter on custom fns.	Memoization-keys doc; incremental-processing blog.
4	Continuously learning	Vague. Marketing-phrase for adaptive memo-cache behaviour; NOT documented as a separate technical subsystem. Inferred meaning: the memo-state two-phase validation (cheap state check ⇒ expensive content hash) “learns” which functions need re-validation based on source-state shifts. TBD: confirm via source-code read.
5	Lineage	Source-to-target key tracking; lineage records driving stale-row removal. Renders via CocoInsight.	Data-consistency Medium post; incremental-processing blog.
6	Task scheduling	NOT a cron scheduler — cocoindex does NOT replace KH’s cron infrastructure. This is the engine’s internal task queue (which @coco.fns to run, in what order, on which threads/processes). Live mode keeps the app running; catch-up mode runs the queue once. External scheduling (when to invoke cocoindex update) is the user’s responsibility.	CLI doc; live mode + catch-up mode descriptions. Cocoindex is INVOKED by an external scheduler; it doesn’t BE one.
7	Metrics collection	”Throughput, freshness, cost — all observable” (front page verbatim). Surfaces in CocoInsight. Granularity per-flow-run + per-@coco.fn likely; per-row possibly.	Front page; CocoInsight blog.
8	Failure management	”Retries, exponential back-off, dead-letter queues, and no-data-loss guarantees” (README verbatim). Embedded in the Rust core; not user-configurable from the outside per the docs reviewed.	README; getting-started overview (“Fault-tolerant runtime that gracefully retries transient failures”).

2.2 Data model — what the persistent control plane stores

Three layers (per the data-consistency Medium post + LMDB internal-storage doc):

Layer A — Catalog (LMDB):

• Registered Apps (name + main fn).
• Connector configs (e.g. localfs.walk_dir(...), postgres.mount_table_target(...)).
• @coco.fn graph topology.

Layer B — Per-row source-target key mapping (LMDB):

• Source key (e.g. file path or Postgres source row PK).
• Source ordinal (timestamp or version number for strict ordering).
• Target key(s) (the materialised row’s PK in target table).
• Fingerprints: SourceLogicFingerprint (operations the row depends on) + FieldDefFingerprint (per-field operations).
• Soft-delete flag for versioned-key soft delete (GC’d offline).

Layer C — Per-fn memo cache (LMDB):

• Function path (stable component subpath).
• Input fingerprint (canonical form of args).
• Cached output (encoded via custom encoder OR fallback serialised-object representation).
• MemoStateOutcome last-validated timestamp.

Layer D — Ops history / “ledger” (per the brief’s framing):

• Per-flow-run summary records: inferred to exist (required for throughput/cost metrics + CocoInsight “every step every record” claim) but NOT explicitly documented as a separate first-class API in the pages reviewed.
• TBD: source-code read to confirm whether per-flow-run history is queryable from Python via App.history() or similar, and what retention policy applies. Without this, KH cannot rely on cocoindex’s ledger as a persistent audit substrate.

2.3 KH need mapping

KH need	Covered by cocoindex control plane?	Notes
Per-flow-run roll-up (status / started_at / completed_at / progress / pipeline_name / items_created) — KH pipeline_runs	PARTIAL. Cocoindex captures all of these conceptually, but the API to query them from outside (in JS/TS app code) is not documented. KH pipeline_runs provides a STABLE TS-facing API that survives cocoindex schema changes.	Recommendation: KEEP pipeline_runs as KH-side rollup (Finding 01 Rec 8 option b). Insulates UI from cocoindex internals. recordPipelineRun() API stable. Closes OQ-Q24-A.
Pipeline failures (retries / DLQ / per-step errors) — KH pipeline_failures (does not exist)	YES. Cocoindex’s failure management is built-in. No need to build pipeline_failures in KH.	Surface failed-flow-runs via CocoInsight UI + summary metrics on KH pipeline_runs.status='failed'. No new KH table required.
Pipeline catalog (what flows are registered, what they do)	YES, fully. cocoindex ls + CocoInsight UI.	No KH-side need.
Version tracking (code changes invalidate cache; data changes re-run delta)	YES, fully. This is core to cocoindex.	KH inherits this for free under Scenario A.
Continuous learning (improve memo behaviour over time)	VAGUE. Marketing phrase; likely refers to memo-state-validation. No KH-specific need that depends on this.	N/A.
Throughput metrics	YES — out of the box via CocoInsight.	Plug pipeline-health-tab into CocoInsight or replicate the summary into pipeline_runs.progress JSONB.
Cost metrics (LLM tokens / embedding calls / etc.)	YES — front-page claim, granularity per-flow-run minimum.	Supersedes the Q4.12 skill-seekers-pattern need.
Task scheduling (cron-like / event-driven trigger of pipelines)	NO — DOES NOT REPLACE KH cron. Cocoindex is invoked by an external scheduler. KH continues to need cron / Cloud Run scheduled jobs / webhooks to invoke cocoindex update (or live mode to keep it running).	KH cron infrastructure stays. Cron jobs invoke cocoindex update <app.py> or post webhook events that trigger live-mode reaction. pipeline_runs row creation moves into the cron entry-point (recordPipelineRun() wraps the cocoindex invocation).
Failure / retry / DLQ semantics	YES. Cocoindex’s Rust core handles retries + back-off + dead-letter routing.	KH does not need to build pipeline_failures table. Use cocoindex’s failure surface + flag failed pipeline_runs.status='failed'.
Per-row crash recovery (partial-batch failure resumption)	YES. S14 confirmed: kill -9 mid-run, re-run resumes cleanly.	KH inherits this for free.
Multi-Cloud-Run-instance consolidated ledger	NO at v1. LMDB-per-instance.	Non-blocking at v1 (S14 single-orchestrator topology). Plan for v2.

2.4 Recommendation

Layer KH’s coarse rollup tables alongside cocoindex’s persistent control plane — don’t try to replace either:

1. pipeline_runs (KEEP). Acts as the KH-TS-facing API for pipeline-health-tab. Wraps cocoindex invocations. Stable shape per recordPipelineRun(). Closes OQ-Q24-A: RETAIN.
2. pipeline_failures (DO NOT BUILD). Cocoindex’s failure management absorbs this need. Surface failure summary via pipeline_runs.status='failed' + CocoInsight UI deep-link.
3. content_history (KEEP). As argued in §1.5: cocoindex covers ingest-lineage, content_history covers user-facing change-event semantics. Both are needed.
4. audit_log (BUILD per Finding 01 Rec 7). Compliance UX surface. Not covered by cocoindex.
5. Cron infrastructure (KEEP). Cocoindex doesn’t schedule; KH continues to invoke. Wrap cocoindex invocations with recordPipelineRun() so the KH-TS-facing rollup row exists.
6. Q4.12 cost-tracking pattern (RETIRE). Replaced by CocoInsight metrics + per-@coco.fn cache stats.

Implementation pattern (forward-looking for Phase B):

// scripts/cron/canonical-ingest.ts (pseudocode)
const runId = await startRun({ pipelineName: 'cocoindex_canonical', ... });

try {
  // Invoke cocoindex (subprocess or live-mode webhook)
  await spawnCocoindex(['update', 'flow.py']);
  await recordPipelineRun({ runId, status: 'completed', ... });
} catch (err) {
  // Cocoindex's internal failure surface has already captured the row-level error
  // We just record the overall run status
  await recordPipelineRun({ runId, status: 'failed', error: err.message, ... });
}

Closes OQ-Q24-A (RETAIN). Routes pipeline_runs into WP4 02-data-flow.md.

---

§3 — Freshness

3.1 What it is (cocoindex’s own definition)

From the cocoindex Show HN post + incremental-processing blog + front page:

• Front page slogan: “Built for data freshness.”
• Definition (incremental-processing blog): “If T is your most acceptable staleness, and you don’t want to recompute the whole thing repeatedly every cycle of T, you will need incremental processing more or less.”
• Operational meaning: Freshness = time between a source-byte change and the materialised target reflecting that change. Lower-bound: source-CDC-event latency (push-based sources) + cocoindex’s process-and-commit latency (typically sub-second per row in live mode).
• Measurement: Per-record source_ordinal (timestamp or version number from the source) + per-record target_committed_at (when cocoindex committed the materialised row). Freshness = now() - source_ordinal for the freshest unprocessed change, or target_committed_at - source_ordinal for processed rows.
• Surfaced via: CocoInsight metrics (“freshness” listed alongside throughput + cost).
• Trigger-driven re-processing: Three CDC strategies (push-based for Google Drive et al; metadata-scan for filesystems; source-specific listing APIs). Source-binding determines which.

3.2 What it is NOT

• NOT a TTL on materialised rows. Cocoindex doesn’t “expire” content; it tracks delta and re-processes the changed delta.
• NOT KH’s freshness enum (fresh / stale / expired / archived) which is a content-governance concept (does this knowledge still reflect reality?). Cocoindex’s freshness is an ingest-latency concept (is the materialised row up to date with the source bytes?).
• NOT user-facing. Cocoindex’s freshness is a developer/operator metric; it doesn’t drive UI.

3.3 KH need mapping

KH need	Covered by cocoindex freshness?	Notes
Coverage cadence (Q3.10) — when does coverage recompute?	PARTIAL. Cocoindex’s incremental delta tells you which content_items rows changed. Coverage compute is a downstream KH @coco.fn (or post-cocoindex post-target write) that listens to those changes. Cocoindex tells you WHEN to recompute (Δ events). Cocoindex does NOT define what coverage IS (that’s a KH-semantic concern).	Coverage trigger = cocoindex Δ event on the relevant tables (content_items, q_a_pairs, bid_questions, bid_question_matches). Coverage scope = KH-business-domain decision (template-only today; extend per Finding 04 Rec 4).
Content-governance freshness enum (KH content_items.freshness: fresh / stale / expired / archived)	NO. Different concept.	KH freshness enum is a knowledge-validity concept (driven by captured_date + ageing rules + user reviews). Cocoindex’s freshness is an ingest-latency concept. KH freshness STAYS.
Change-reports (formerly “digest”) trigger	YES — cocoindex Δ events ARE the trigger.	A change-report listens to cocoindex Δ events on the relevant subset (e.g. “feed_articles in workspace W”). Cocoindex emits the events; KH composes the report.
Re-classification trigger on user-initiated edit-back (Theme C state-machine; OQ-CX33-A)	YES — natural fit.	Under Candidate A (in-platform editor -> filesystem write -> cocoindex source-binding observes file change), cocoindex’s source-binding triggers re-extraction automatically on every edit. The state-machine question (Theme C) is WHICH edits trigger which re-classification — that’s a KH-business-domain layer ABOVE cocoindex’s freshness substrate. Cocoindex doesn’t gate edit-trigger semantics; it provides the substrate.
Re-extract behaviour on edit-back (OQ-CX33-A: does cocoindex re-extract?)	YES.	Edit-back writes the source file; cocoindex’s source-binding detects the change (file mtime or content hash change); the @coco.fn graph re-runs the delta. Cost: only the changed-file’s flow runs (incremental Δ). Closes OQ-CX33-A.
TTL / time-based re-extract	NO. Cocoindex doesn’t have a TTL concept; freshness is event-driven, not interval-driven.	KH content-governance freshness (fresh→stale→expired) is a separate KH-side ageing system. Drives KH-side review queue, NOT cocoindex re-processing.

3.4 Recommendation

Adopt cocoindex’s freshness substrate for ingest-latency observability; KEEP KH’s content-governance freshness enum entirely separate:

1. Cocoindex freshness = ingest-side substrate. Drives change-reports trigger (Q3.10), drives coverage recompute trigger (Q3.10), drives Theme C re-classification trigger. CocoInsight surfaces the metric for operator visibility.
2. KH content_items.freshness enum = governance-side concept. Driven by captured_date + ageing rules + user review. Unchanged by cocoindex adoption.
3. Change-reports rename (formerly “digest”). Change-reports listen to cocoindex Δ events on the relevant subset of tables and compose a report. This is a downstream @coco.fn or a post-cocoindex job that subscribes to cocoindex’s Δ-event surface (or polls content_items.updated_at).
4. Coverage scope (Q3.10). Still a KH-business-domain decision. Cocoindex tells you WHICH rows changed; the coverage compute decides what to do with that.
5. Theme C state-machine. Sits ABOVE cocoindex; cocoindex’s source-binding is the substrate that triggers re-extraction, but the decision “do we propagate this edit to re-classify the q_a_extraction, or is it a cosmetic edit that doesn’t?” is KH-business-domain. Recommendation: add an edit_intent enum (per Theme C in feedback-findings-review.md §1: cosmetic / data / structural) that gates which downstream @coco.fns re-run. Cosmetic edits skip re-classification; data + structural trigger full flow. This needs a small wrapper @coco.fn that reads edit_intent from a sidecar metadata file or DB column and conditionally invokes the heavier extraction fns. Closes Theme C.

---

§4 — Scheduling + failure management

4.1 Scheduling

Cocoindex does NOT replace KH’s cron infrastructure. Critical point: cocoindex’s “task scheduling” subsystem refers to its INTERNAL task queue (which fn to run, in what order, on which thread). External invocation — the equivalent of “run the canonical pipeline every 15 minutes” — is the user’s responsibility.

Mode	When you’d use it	KH integration
Catch-up mode (default)	Run on demand: cocoindex update flow.py. Scans sources, processes delta, exits.	Cron triggers cocoindex update. Wrap with recordPipelineRun().
Live mode (-L)	Long-running daemon; live sources stream changes continuously.	Cloud Run service (single instance per S14); container keeps running. recordPipelineRun() rollup tracks per-batch progress via mid-flight progress updates.
Hybrid (KH-pattern)	Live mode for files/Q&A pipeline + scheduled cron for cross-corpus jobs (coverage compute, dedup audit, etc.).	Both patterns coexist.

4.2 Failure management

From front page + README: “Retries, exponential back-off, dead-letter queues, and no-data-loss guarantees.”

Failure class	Cocoindex behaviour	KH need
Transient @coco.fn failure (e.g. LLM API 429)	Built-in retry + exponential back-off.	None — inherit.
Persistent @coco.fn failure (e.g. malformed input)	Routed to dead-letter queue. The DLQ surface is not documented at the page level; presumed accessible via CocoInsight + Python API.	TBD: how does KH access DLQ rows from TS? Likely: TS reads from cocoindex’s HTTP API or from a TS-facing pipeline_runs.failed_rows JSONB column we populate in the wrapper.
Per-row CHECK violation on target table (asyncpg PostgresError)	Per S1 §1.x: bubbles up as asyncpg.PostgresError from the engine’s upsert. Per-row failure isolation: S1-Q2 PENDING (likely the engine fails the whole batch unless wrapped).	Phase B first-step: verify per-row isolation. Per S1 Q2.
Process-level crash (SIGKILL)	Per S14 §1.x: recovery is clean; restart resumes from the LMDB cache.	None — inherit.
LMDB corruption	Not documented as a recovery path. TBD: investigate Phase B.	Disaster recovery plan: re-ingest from sources (cocoindex’s --full-reprocess flag re-runs everything against fresh LMDB).

4.3 KH need mapping

KH need	Covered?	Notes
Schedule canonical pipeline to run every 15 minutes	NO — KH owns.	Cron entry-point invokes cocoindex update. recordPipelineRun() wraps.
Schedule coverage recompute on Δ events	PARTIAL. Cocoindex Δ events trigger; KH schedules the recompute job.	Live-mode cocoindex with a downstream @coco.fn that reacts to Δ events. Or cron-poll Δ events into KH’s job queue.
Schedule change-reports daily 08:00	NO — KH owns.	Cron job composes change-reports by querying recent Δ events.
Retry transient LLM failures	YES.	Inherit.
Dead-letter rows that consistently fail extraction	YES, but surfacing is TBD.	Phase B: define how KH UI surfaces DLQ rows. Likely a new pipeline_runs.dlq_count rollup + CocoInsight deep-link for detail.
Replay failed rows after fix	YES — cocoindex update --full-reprocess or fingerprint-invalidate.	KH UI: “retry failed extractions” button. Implementation: bump behavior_version on the failed fn, or invalidate cache for affected rows.

4.4 Recommendation

1. KEEP KH cron infrastructure. Cocoindex doesn’t replace it.
2. DO NOT BUILD pipeline_failures table. Cocoindex’s DLQ + per-row failure isolation absorb this.
3. EXTEND pipeline_runs with a dlq_count rollup column (or surface via existing progress JSONB). Phase B detail.
4. Verify per-row failure isolation in Phase B first-step (S1-Q2 pending) — without per-row isolation, a single bad row could fail a whole batch and would require KH to build batch-splitting logic.

---

§5 — Summary table

#	Cocoindex capability	KH need	KH table/module affected	Recommendation
1	CocoInsight lineage UI	Per-row data lineage; pipeline debugging	(new dev surface)	ADOPT as developer-facing observability. Deep-link from KH admin UIs.
2	CocoInsight throughput / cost metrics	Pipeline-health-tab visibility; cost dashboards (Q4.12)	pipeline_runs.progress JSONB; Q4.12 cost-tracking pattern	ADOPT. Q4.12 skill-seekers pattern RETIRE.
3	CocoInsight zero-retention UI	Compliance audit log	(planned audit_log)	DOES NOT REPLACE. Build audit_log per Finding 01 Rec 7. Cross-link to CocoInsight via cocoindex_run_ref.
4	Pipeline catalog (cocoindex ls)	Registry of pipelines	(none today)	ADOPT. No KH-side need.
5	Version tracking (fingerprints + behavior_version)	Code-change-triggered cache invalidation	(none today)	ADOPT by Scenario A default.
6	Source-to-target lineage records (LMDB Layer B)	Stale-row removal; provenance queries	source_documents (Option α/β); content_history	REPLACES source_documents.version + parent_id + source_document_diffs (per Finding 01 §3.3). DOES NOT REPLACE content_history.
7	Per-@coco.fn memo cache (LMDB Layer C)	Skip expensive ops on unchanged inputs	(none today; manual caching in scripts)	ADOPT — replaces ad-hoc caching in scripts/kb_pipeline/.
8	Throughput / cost / freshness metrics	Pipeline-health-tab	pipeline_runs.progress	ADOPT. KEEP pipeline_runs as TS-facing rollup.
9	Task scheduling (internal queue)	KH cron / Cloud Run scheduled jobs	KH cron infrastructure	KEEP KH cron. Cocoindex doesn’t replace external scheduling.
10	Failure management (retry / back-off / DLQ)	Failed-extraction surface; retry UI	(pipeline_failures — not built)	REPLACES. Do not build pipeline_failures. Surface via pipeline_runs.status='failed' + DLQ rollup.
11	Freshness (ingest-latency metric)	Change-reports trigger; coverage cadence (Q3.10); Theme C state-machine	change-reports; coverage_runs; edit-flow §6	ADOPT as substrate. KH content_items.freshness enum (governance) stays SEPARATE.
12	Re-extract on source change	Edit-back re-classification (OQ-CX33-A); UC1 round-trip	edit-flow §6 UC1 / UC4 / UC6	NATURAL FIT. Closes OQ-CX33-A. Add KH-side edit_intent to gate which downstream fns re-run (Theme C).
13	Crash recovery (kill -9 safe per S14)	Pipeline robustness	(none)	ADOPT by default.
14	Multi-instance consolidated view	Cross-Cloud-Run-instance observability	(none)	NOT AVAILABLE. v1 single-instance per S14. Plan for v2 if needed.

---

§6 — Open decisions this resolves

For each open decision in feedback-findings-review.md that was routed to the cocoindex prereq, state the resolution status here:

OQ / Finding-item	Status	Resolution + reasoning
OQ-Q24-A — pipeline_runs retain-vs-retire	RESOLVED-CONFIRMED (Finding 01 Rec 8 option b — RETAIN).	Cocoindex’s per-flow-run ledger lives in LMDB; KH-TS-facing API to query it is not documented. Keeping pipeline_runs as KH-side rollup (populated via recordPipelineRun() wrapper around cocoindex invocations) insulates the UI from cocoindex schema and provides a stable contract. Pipeline-health-tab continues to read from pipeline_runs. CocoInsight deep-link provides per-run lineage detail.
OQ-Q24-C — op_id propagation (trigger vs app-stamped)	STILL-OPEN-because-not-cocoindex-decision.	Cocoindex doesn’t have an op_id concept. Op_id is a KH-business-domain rollback construct. The cocoindex prereq doesn’t move this OQ. Falls back to Liam’s working default in Finding 01: “app-stamped where rollback-op cohesion matters; trigger-driven elsewhere.” Routes to WP4 02-data-flow.md.
OQ-CX33-A — cocoindex re-extract on user-initiated edit-back	RESOLVED-CONFIRMED. Under Candidate A (in-platform edit -> filesystem write -> cocoindex source-binding observes file change), cocoindex automatically re-runs the affected flow on every source change. Incremental Δ ensures only the changed file’s flow runs. Cost: low (single-file delta). Trigger latency: sub-second for filesystem connector with metadata-scan.	Theme C state-machine (which edits trigger which re-classifications) sits ABOVE cocoindex — recommended via new edit_intent enum.
N5 (Finding 01) — upload-route silent-fail fix	STILL-OPEN-because-coupled-to-Option-α/β.	Cocoindex doesn’t directly resolve this. Under Option α, the silent-fail still needs fixing (the slim source_documents row must be written to maintain the binary-source manifest). Under Option β, the upload route’s source_documents insert becomes moot. Resolution gates on B2 (α/β decision).
N6 (Finding 01) — pipeline_runs fate	RESOLVED-CONFIRMED (RETAIN). Same as OQ-Q24-A above.
N7 (Finding 01) — op_id propagation	STILL-OPEN-because-not-cocoindex-decision. Same as OQ-Q24-C above.
Theme C — Edit / re-classification state-machine	PARTIALLY RESOLVED. Cocoindex provides the substrate (source-binding re-extracts on file change). The state-machine sits ABOVE: which edits trigger which downstream re-classifications. Recommendation: edit_intent enum (cosmetic / data / structural) gates downstream @coco.fn invocation. RBAC + concurrent-edit handling are KH-business-domain layers.	Routes to WP4 02-data-flow.md (substrate) + 05-qa-flow.md (edit_intent + state machine).
Theme D — Cocoindex freshness integration	RESOLVED-NEW-DIRECTION. Cocoindex freshness = ingest-latency substrate (metric + trigger). KH content_items.freshness enum = content-governance concept (orthogonal). They COEXIST. Coverage (Q3.10) trigger = cocoindex Δ events; coverage scope = KH-business-domain. Change-reports = downstream @coco.fn listening to cocoindex Δ events. Re-classification trigger (Theme C) = cocoindex source-binding + edit_intent gate.	Routes to WP4 02-data-flow.md + 08-new-features.md (change-reports).
Finding 01 Rec 3 — diff UI fate	PARTIALLY RESOLVED. CocoInsight provides pipeline-stage data lineage but NOT raw before/after-text-diff. The KH diff UI may stay as a content-diff surface independent of CocoInsight. Decision: TBD post-empirical CocoInsight comparison; not a hard retire.	Routes to Phase B post-CocoInsight evaluation.
Finding 01 Rec 8 — audit_log cocoindex-covered?	RESOLVED-CONFIRMED (NOT covered; BUILD audit_log). CocoInsight is a developer surface (pipeline-step lineage), not a compliance surface (human-readable business-event narrative). audit_log (per Rec 7) is genuinely needed. Add cocoindex_run_ref cross-link column.	Routes to WP4 02-data-flow.md.
Q3.10 — Coverage scope	PARTIALLY RESOLVED. Cocoindex Δ events provide the trigger. Scope is KH-business-domain. Aligned with Finding 04 Rec 4 (extend coverage to 4th gap-source).	Routes to WP4 08-new-features.md.
Q4.12 — Embedding pipeline cache + cost tracking	RESOLVED-NEW-DIRECTION. CocoInsight metrics supersede the skill-seekers-style cost dashboards. Cache is built-in via @coco.fn(memo=True).	Routes to WP4 03-tech-stack.md (CocoInsight adoption).
OQ-Q29-A (S9 equal-hash idempotency)	STILL-OPEN-because-empirical. Cocoindex’s content-hash idempotency is well-documented in principle; per S229 / S2 / S14 prior probes, the engine respects content-hash equality on re-writes. But S9 spike is the formal validation, status unclear in 0.9-decision-graph.md §5.	Recommend running S9 as Phase B first-step (cheap; ~1 hour).
pipeline_failures table (hypothetical)	RESOLVED-CONFIRMED (DO NOT BUILD). Cocoindex’s failure management (retries / back-off / DLQ) absorbs the need. Surface failures via pipeline_runs.status='failed' + per-run DLQ rollup count.	Routes to WP4 02-data-flow.md.

---

§7 — Phase B follow-ups (recommended)

These didn’t fit cleanly into §6 but should be checked in Phase B implementation:

1. Source-code read of cocoindex’s persistent ledger API. Confirm whether per-flow-run history is queryable via Python (App.history() or similar) and what retention applies. Without this, KH cannot rely on cocoindex’s ledger as an audit substrate — would force a heavier KH-side pipeline_runs shape.
2. CocoInsight on-prem deployment + auth model. The blog references the managed-SaaS UI pointed at on-prem server; production deployment (e.g. Cloud Run + Vercel) needs to verify auth + network topology. Source-code or community-forum search.
3. DLQ surfacing pattern. How does KH-TS UI access cocoindex’s DLQ rows? Three candidates: (a) cocoindex HTTP API; (b) post-cocoindex @coco.fn that mirrors DLQ to a KH table; (c) deep-link to CocoInsight UI per-row.
4. Per-row failure isolation (S1-Q2). Verify cocoindex doesn’t fail whole batches on a single CHECK violation. Critical for UC8 + UC4.
5. LMDB corruption recovery. Document disaster-recovery procedure (cocoindex --full-reprocess against fresh LMDB).
6. CocoInsight CLI verification. Confirm cocoindex server -ci main works in 1.0.3 — the CLI docs page (/docs/cli) doesn’t list server. Check cocoindex --help output in venv from S1.
7. Edit_intent enum binding. Where does it live? Three options: (a) content_history.edit_intent column; (b) sidecar metadata file alongside the markdown content; (c) wrapped in the cocoindex flow as a per-fn arg. Theme C decision.

---

§8 — Routing to WP4 sub-docs

For the architecture-implementation phase:

WP4 sub-doc	What this prereq contributes
02-data-flow.md	§1.6 + §2.4 + §3.4 + §4.4 recommendations on pipeline_runs, audit_log, content_history, cron infrastructure, freshness substrate. All Theme D bindings.
03-tech-stack.md	CocoInsight adoption decision (§1.6). Q4.12 retirement.
05-qa-flow.md	edit_intent enum proposal (§3.4 + §6 Theme C).
08-new-features.md	Change-reports as downstream @coco.fn (§3.3). Coverage scope decision gates on Q3.10 (KH-business).

---

End of prerequisite 2b — cocoindex operational mapping. Closes the OQs listed in §6; routes residuals to WP4 + Phase B first-step.