cocoindex evaluation — re-use vs integrate vs reinvent for KH ingest pipeline

cocoindex evaluation — re-use vs integrate vs reinvent for KH ingest pipeline

Date: 2026-05-08 Branch: content-items-investigation (worktree) Author: Claude (Opus 4.7, 1M context) Subject: cocoindex-io/cocoindex — incremental data engine for AI agents (Apache 2.0; PyPI cocoindex; ~75% Python / ~25% Rust) Status: Evaluation only — verdict-led but no production integration changes proposed without explicit decision. Lens: “what should we be doing” (canonical-pipeline collapse, empty DB pre-re-ingest) — NOT “lowest-disruption to current code”. Re-use battle-tested infrastructure where it pays.

---

1. What cocoindex is (single-paragraph)

cocoindex is a declarative incremental data-engineering engine that compiles a Python flow definition (sources → transforms → targets) into a Rust-backed runtime that maintains the target store in sync with the sources by recomputing only the delta when source data or transformation code changes. Per-row provenance + content-hash + code-hash memoisation drives the cache; unchanged rows skip every downstream stage. Out-of-the-box connectors include local FS, Amazon S3, Google Drive, OCI Object Storage, Postgres source, and 8+ targets (Postgres, pgvector, LanceDB, Qdrant, Turbopuffer, Neo4j, FalkorDB, SurrealDB, Kafka). Built-in transformations cover RecursiveSplitter chunking (with optional tree-sitter), SentenceTransformerEmbedder, LiteLLMEmbedder (any LiteLLM-supported model), entity resolution (faiss + Pydantic AI), and a “language detect from filename” helper. Custom transformations are user-defined Python async def functions decorated @coco.fn(memo=True). Pitch: “any folder of files (or stream) into a continuously fresh AI-consumable index, recomputing only what changed.” Apache 2.0.

---

2. Grounded codebase audit (read-only)

I could not run cocoindex hands-on in this session — the sandbox blocks PyPI SSL (the network proxy fails certificate verification, so pip install cocoindex errors). The evaluation below is grounded in:

1. The cocoindex GitHub README + python/cocoindex/connectors/ directory listing + python/cocoindex/ops/{text,litellm,sentence_transformers,entity_resolution} source
2. The examples/pdf_embedding/main.py reference flow
3. The examples/conversation_to_knowledge design doc (YouTube → SurrealDB knowledge graph)
4. KH source code already on disk (Phase 0.1-0.7 audits, lib/extraction/ 1,000 LOC, lib/bid-library-ingest/ 638 LOC, scripts/kb_pipeline/ 7,404 LOC)
5. Phase 0.7 synthesis + 0.7.4 source_documents architecture + the user’s 07-synthesis-feedback ratifications

2.1 What I verified hands-off

The cocoindex install story is genuine. PyPI listing exists. README quickstart shows: pip install -U cocoindex docling, write a main.py decorating two functions with @coco.fn, run cocoindex update main.py. No daemon required (a local SQLite “operations DB” tracks the cache). For production, the target store is whatever connector you bind — Postgres in our case.

The user code is small. The PDF→embedding example is roughly 30-40 LOC end-to-end:

@coco.fn(memo=True)
async def index_file(file, table):
    md = await docling.convert(file)               # PDF → markdown
    for chunk in RecursiveSplitter(2000, 500).split(md):
        emb = await SentenceTransformerEmbedder().embed(chunk.text)
        table.declare_row(text=chunk.text, embedding=emb)

@coco.fn
async def main(src):
    table = await postgres.mount_table_target(PG, "docs")
    table.declare_vector_index("embedding")
    await coco.mount_each(index_file, localfs.walk_dir(src).items(), table)

The Rust engine sees localfs.walk_dir → index_file → postgres table as a typed flow; on every cocoindex update, it diffs the source-file content-hashes against last run, only re-runs index_file for files whose content changed, and only re-writes target rows whose embedding output changed. That’s the “React-for-data” semantic.

Connector list (verified): localfs, amazon_s3, google_drive, oci_object_storage, kafka, postgres (as both source and target), sqlite, doris, lancedb, qdrant, turbopuffer, neo4j, falkordb, surrealdb. No URL fetcher. No RSS connector. No DOCX connector either — the pdf_embedding example uses docling (a separate IBM-research Python package) for PDF→markdown conversion. docling does support DOCX (and PDF, HTML, CSV, image OCR), so the integration path for DOCX is “use docling inside a @coco.fn”, same shape as the PDF example.

Built-in transforms (verified):

• RecursiveSplitter (text + optional tree-sitter syntax-aware chunking)
• SeparatorSplitter (regex-based)
• SentenceTransformerEmbedder (HF sentence-transformers models, e.g. all-MiniLM-L6-v2)
• LiteLLMEmbedder (any LiteLLM-supported model — covers OpenAI, Cohere, Vertex AI, Anthropic via litellm proxy)
• LiteLLMTranscriber (Whisper, ElevenLabs)
• Entity resolution module (faiss-based dedup; uses Pydantic AI for validation)
• language_detect_from_filename

Critical absences for KH:

• No URL extractor (we’d need to build it inside a @coco.fn using our existing Readability/Jina/Firecrawl chain)
• No RSS connector (no equivalent of P9’s feed-source registry / 4-tier extraction cascade)
• No Anthropic-native LLM connector (we’d go via LiteLLM, which adds a hop and a config layer; KH already uses Anthropic SDK directly with prompt-caching, which LiteLLM may or may not surface cleanly)
• No Q&A docx Pattern A/B table extractor (would need to be a custom @coco.fn, but our existing lib/bid-library-ingest/extract-qa-pairs.ts is 497 LOC of domain logic — porting to Python is non-trivial)

2.2 What I did NOT verify hands-on (and why)

• Code-hash memoisation behaviour when a transformation function changes. The README claims “memoization (caching by input and code hash)” — directionally credible (Rust-style hash of the bytecode + closure env), but I have not validated whether trivial whitespace changes invalidate the cache or whether they correctly do.
• Postgres target schema flexibility — does mount_table_target(PG, "content_items") accept a pre-existing schema with our 70+ columns and FKs, or does it want to own/create the table? This is the single biggest uncertainty for KH integration. I’ll come back to this in §4.
• Failure-isolation guarantees — when one row fails extraction, does cocoindex skip and continue, or halt the flow? README claims “failure isolation” (Rust core), but error-routing to KH’s ingestion_quality_log would need explicit wiring.
• Streaming / push semantics — cocoindex is “incremental engine for long-horizon agents”. Is it pull-based (poll source, diff, recompute) or push-based (source notifies engine)? The PDF example is pull. For RSS/webhooks, the model needs verification.
• Concurrency model — cocoindex update is presumably one process. KH would want N workers in parallel for cron + queue. Unverified whether the Rust engine partitions per-source-row across workers natively or if we’d need to shard manually.

I’d budget ~2-3 days of hands-on time to fully validate these unknowns before committing. Confidence on this evaluation: 78%. I drop below 80% specifically because I could not run the Postgres-target-onto-existing-schema test.

---

3. Architecture summary

┌─────────────────────────────────────────────────────────────────┐
│ User-defined Python flow (decorated @coco.fn)                   │
│   • Sources: localfs / S3 / GDrive / Postgres / Kafka / ...     │
│   • Transforms: RecursiveSplitter / Embedder / custom @coco.fn   │
│   • Targets: Postgres / pgvector / Qdrant / Neo4j / SurrealDB    │
└──────────────┬──────────────────────────────────────────────────┘
               │ (compiled into typed flow graph)
               ▼
┌─────────────────────────────────────────────────────────────────┐
│ Rust engine (production-grade per README claim)                 │
│   1. Per-row content-hash extraction from source                │
│   2. Per-transform code-hash + input-hash → memoised cache      │
│   3. Diff against last run's hashes → recompute only Δ          │
│   4. Per-row failure isolation                                  │
│   5. Parallel chunk processing                                  │
│   6. Target sync (CDC-style upsert + tombstone for deletes)     │
└──────────────┬──────────────────────────────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────────────────────────────┐
│ Local operations DB (SQLite by default)                         │
│   • Per-row provenance ledger                                   │
│   • Content-hash + code-hash cache                              │
│   • Lineage graph (source → transforms → target rows)           │
└─────────────────────────────────────────────────────────────────┘

3.1 Notable design properties

Property	Implication for KH
Declarative flow (DAG of typed sources/transforms/targets)	Maps cleanly to the Phase 0.7 4-layer canonical pipeline plan: Layer 1 transports, Layer 3 shape adapters, Layer 4 core stages all become @coco.fn decorated nodes.
Per-row content-hash memoisation	Solves the “is this a re-ingest?” problem at the engine level — same content → cache hit → no re-processing → no duplicate content_items rows. This is exactly the architectural problem 0.7.4 §5.4 flags.
Code-hash invalidation	When we change classifyContent prompt → all rows re-classify on next run. When we change inferLayer → only re-infer layers. When we add a stage → only that stage runs for prior rows. This is incremental re-ingest semantically — replaces our planned re-ingest cycle.
Pull-based by default (cocoindex update walks sources)	RSS would need a wrapper that syncs RSS items → localfs/Postgres source the engine reads. Or we keep our existing P9 RSS pipeline as upstream-of-cocoindex.
Apache 2.0 licence	Compatible with KH’s deployment. No GPL contagion.
Rust core, Python interface	Performance-critical paths are Rust; user-extensibility is Python. KH backend is TS-first; cocoindex would join the existing Python pipeline tier (scripts/kb_pipeline/) rather than the TS tier.
Local SQLite ops-DB	Adds a state file that needs to be persisted across runs (in our case, per Cloud Run job) — operationally similar to the cache.py graphify uses.
No native Anthropic	Goes via LiteLLM. Loses prompt-cache control (we’d need to verify LiteLLM surfaces cache_control headers). KH’s classification already uses prompt-caching aggressively per lib/ai/classify.ts.

3.2 Per-stage mapping to KH’s existing pipeline

KH stage (Phase 0.7 Layer 4)	cocoindex equivalent
Dedup gate (content-hash + suspected-duplicate)	Built-in — content-hash memoisation IS the dedup. KH’s content_text_hash GENERATED-ALWAYS column becomes the engine-managed primary key.
Side-channel writes (source_documents)	Custom @coco.fn — write source_documents row from inside the binary-shape adapter. Maps to a second target binding alongside content_items.
content_items INSERT	postgres.mount_table_target(PG, "content_items") — assuming schema-flexibility verification passes.
classifyContent	Custom @coco.fn wrapping our existing Anthropic SDK classify call OR LiteLLMEmbedder-style wrapper. Code-hash memoisation kicks in if prompt unchanged.
Embedding (text-embedding-3-large)	Replace with LiteLLMEmbedder("openai/text-embedding-3-large") OR keep our existing lib/ai/embed.ts inside a @coco.fn.
regenerateChunks	Replace with RecursiveSplitter OR wrap our existing lib/chunking/.
inferLayer / quality_score / extractDates	Custom @coco.fn chain — each becomes a memoised stage.
pipeline_runs observability	NOT built-in — we’d write a custom emitter @coco.fn that observes flow state and writes pipeline_runs. cocoindex has its own internal lineage but it doesn’t surface it as a Postgres row by default.
ingestion_quality_log	NOT built-in — same custom emitter pattern.

3.3 Per-shape mapping (5 KH shape adapters → cocoindex flow)

KH shape	cocoindex implementation
URL (P4)	Custom @coco.fn URL extractor (port of lib/extraction/url.ts to Python OR call our TS via subprocess/HTTP). cocoindex doesn’t have a URL connector. Path: keep our extractor, wrap as @coco.fn.
document-binary (P7)	localfs.walk_dir source + docling.convert(file) for PDF/DOCX → markdown. Replaces unpdf + mammoth + turndown chain.
document-text (P8 markdown batch)	localfs.walk_dir source + frontmatter-aware passthrough @coco.fn. Direct replacement for lib/ingest/markdown-orchestrator.ts.
qa-docx (P3 — 71% of prod)	Custom @coco.fn wrapping our lib/bid-library-ingest/extract-qa-pairs.ts Pattern A/B logic OR port to Python. Big port if we go full cocoindex.
rss-discovery (P9)	No native. Either (a) keep KH’s RSS pipeline as upstream feeder writing to a Postgres staging table, then bind postgres_source connector to that staging table, OR (b) port the 4-tier extraction cascade to Python @coco.fns. Higher integration cost.

---

4. Perspective (i) — Dev workflow value

cocoindex is not a dev-workflow tool in the sense graphify is (graphify is a structural code-graph indexer that Claude Code can query mid-session). cocoindex is a runtime data engine — it lives in production / cron, not in the agent’s dev loop.

The only dev-workflow angle is: does cocoindex change how Claude Code agents reason about the KH pipeline?

4.1 Modest dev-workflow benefits

1. Declarative flow as agent-readable artefact. A flow.py of 200-400 LOC describing “URL → extract → classify → embed → store” is much more navigable for an agent than the current 7,404 LOC of scripts/kb_pipeline/ + lib/extraction/ + lib/ai/ spread across two languages. Onboarding a new agent (or human) to “where does ingestion happen?” collapses from a Phase 0.1 audit to “read flow.py”.
2. Code-hash invalidation as regression signal. When an agent edits classifyContent, cocoindex’s next run will re-classify everything. This is observable — the agent can diff outputs and verify the change had the intended effect. It’s a stronger feedback loop than the current “ship to staging, manually trigger re-ingest, sample-check 5 rows” pattern.
3. Operations DB as ground-truth lineage. When debugging “why is row X classified as Y?”, the cocoindex ops-DB shows the exact code-hash + input-hash that produced Y. KH today has ai_call_log (deferred per NEW12) which is meant to fill this gap. cocoindex would absorb that requirement.

4.2 What it does NOT give us (dev-workflow side)

• No structural code graph. Graphify, knip, the planning audits — those tools all do structural analysis of the codebase. cocoindex does runtime data lineage. They’re orthogonal — adopting cocoindex doesn’t replace graphify; it complements it.
• No agent-mountable MCP server for querying the flow state. (We could build one — cocoindex exposes the ops-DB directly — but it’s not built-in.)
• No code-review value. cocoindex doesn’t lint, test, or verify code quality. Vitest, ESLint, knip, graphify still do their jobs.

4.3 Net for (i)

Marginal dev-workflow benefit, dominated by platform value. The flow-as-readable-artefact is real but small; the operations-DB lineage is real but mostly redundant with ai_call_log (deferred). The case for cocoindex stands or falls on platform value (Perspective ii).

---

5. Perspective (ii) — KH platform value

This is where the evaluation gets interesting. Apply the corrected lens: empty DB pre-re-ingest, no time pressure, “what should we be doing”, biased toward re-use of battle-tested infrastructure.

5.1 Conceptual overlap with Phase 0.7 canonical pipeline

The Phase 0.7 plan calls for:

1. Layer 1: 5 transports normalising into a single IngestRequest envelope
2. Layer 2: shape router
3. Layer 3: 5 shape adapters producing typed extracted content
4. Layer 4: canonical pipeline core (dedup → side-channels → store → classify → embed → chunk → post-stages → observability)

cocoindex’s flow model maps almost 1:1. Compare:

Phase 0.7 layer	cocoindex equivalent	Verdict
Layer 1 transports	Each transport becomes a different “source” binding (localfs for cron, postgres source for queue, custom HTTP source for API/MCP)	Direct map.
Layer 2 shape router	Discriminated-union dispatch INSIDE the source/transform — cocoindex does typed flow graphs natively	Direct map.
Layer 3 shape adapters	Each adapter is a @coco.fn decorated function	Direct map.
Layer 4 canonical core	Sequence of @coco.fn decorated stages writing to typed table targets	Direct map.
ingestion_quality_log	Custom emitter @coco.fn observing flow state	Custom but straightforward.
pipeline_runs	Custom emitter	Custom but straightforward.
Re-ingest cycle (planned weeks 3-4 of Stream 1)	Replaced entirely by cocoindex’s incremental Δ. When source data lands in fresh DB, cocoindex update runs the full flow once; future runs only recompute changed rows.	Major collapse.

This is the most striking finding: cocoindex’s incremental engine semantically IS the canonical pipeline core. The Phase 0.7 plan was going to spend 2 weeks building a “canonical foundation” that handles dedup gate + side-channels + observability + re-runnable stages. cocoindex provides that foundation as battle-tested infrastructure.

5.2 The “live, continuously fresh context” claim — does it map to KH freshness/governance?

The cocoindex pitch: “live, continuously fresh context for AI agents and LLM apps to reason over effectively — with minimal incremental processing.”

KH has two analogous concerns:

1. content_items.freshness_checked_at — governance cron checks every N days whether a URL’s content has changed. If yes, re-ingest.
2. Re-upload detection (P7 detect_reupload RPC) — when a binary is re-uploaded, decide if it’s identical / new_version / fresh.

cocoindex addresses (1) directly. Bind a URL source to the engine; it pulls + content-hashes on every cocoindex update; if hash changed, downstream stages re-run; cache otherwise. This eliminates the freshness governance code path for URL ingests — replaced by the engine’s standard incremental model.

cocoindex addresses (2) cleanly. Binary uploads are detected as “new content-hash for same key” and the engine re-runs the binary’s flow (re-extract, re-classify, re-embed, re-write target rows). This collapses the re-upload-creates-new-content_items friction point from 0.7.4 §5.4: cocoindex would UPDATE the existing target row rather than INSERT a new one (assuming the target binding uses the source-document-key as upsert key).

The user’s question on 07-synthesis-feedback line ~88 — “if we have a source document which then gets extracted as a content item(s) and then I edit that content item in the platform, and then upload a new version of the document (including a different filename), what’s happening in the platform from a data flow perspective, and data provenance?”

Today: re-upload with new filename → new source_documents row with parent_id = NULL (filename mismatch — detect_reupload keys on filename) → new content_items row, prior content_items row orphaned → user sees 2 KB items, mental model broken.

Under cocoindex: the source-document key would NOT be filename — it’d be a logical-document-id (FK from source_documents). User uploads policy-v2.docx (different filename, same logical doc). The flow’s source connector emits a row with the logical-doc-id key — engine sees same key + new content-hash → recomputes downstream → UPDATEs the same content_items target row → new content_history v2 entry via DB trigger. Re-upload friction architecturally resolved.

But: this requires the user to indicate “this re-upload is the new version of doc X” at upload time — typically via a UI affordance (“update existing document?”). cocoindex doesn’t infer logical identity from content alone; that’s a product decision (which prior P0-1 silent-fail debate also surfaced). The engine enables the right behaviour; doesn’t automate it.

5.3 Multi-source coverage — does it cover all 5 KH input shapes?

Shape	cocoindex coverage	Effort to wire
URL	No native connector. Custom @coco.fn wrapping our existing extractor.	~2-3 days port of TS extractFromUrl to Python (or call out via HTTP/subprocess from Python).
document-binary (PDF/DOCX)	Native via docling (separate package). PDF + DOCX + HTML + image OCR.	~1 day to wire docling + pgvector target + source_documents side-channel write.
document-text (markdown)	Native via localfs.walk_dir + custom frontmatter parser @coco.fn.	~1 day to wire.
qa-docx (Pattern A/B)	No native. Custom @coco.fn — port of extract-qa-pairs.ts 497 LOC OR wrap via subprocess.	~1 week port if we go Python-native; ~2 days subprocess-wrap if we keep TS.
rss-discovery	No native. Custom upstream feeder (KH’s existing P9) writes to staging Postgres table; cocoindex postgres_source connector binds.	~3 days integration design + wiring.

Coverage verdict: 3 of 5 shapes well-covered, 2 of 5 require porting effort. docling specifically (PDF + DOCX) is a strong fit. The qa-docx Pattern A/B parser is the biggest port cost — but it’s also the path serving 71% of prod data, so it’s worth the investment.

Code count comparison:

	KH today	cocoindex equivalent
Python pipeline	7,404 LOC across 18 files	~500-800 LOC of flow.py + adapter @coco.fn’s (rough estimate; depends on whether we port qa-docx or wrap)
TS extraction surface	~1,000 LOC across lib/extraction/ + lib/bid-library-ingest/	Folded into Python @coco.fn’s (or wrapped via subprocess)
Orchestration code	Significant (recordPipelineRun, ingestion_quality_log helpers, dedup gate, etc.)	Mostly absorbed by engine; thin custom emitters

Net code delta: ~6,000-7,000 LOC retired, replaced by ~500-800 LOC of declarative flow + ~50-150 LOC of custom emitters + cocoindex package as dependency. This is the dominant code-collapse opportunity in the Phase 0.7 plan.

5.4 Schema coupling — KH’s entity_mentions + entity_relationships Supabase tables vs cocoindex output

Per §2.2, the single biggest uncertainty is whether cocoindex’s postgres.mount_table_target(PG, "content_items") accepts a pre-existing 70-column schema with FKs and triggers, or whether it wants to own/create the table.

Two scenarios:

Scenario A: cocoindex accepts existing schema. We bind mount_table_target to existing content_items table; the engine performs upserts keyed by some primary key we declare (probably content_text_hash or a logical-doc-id). All existing FKs (created_by, source_bid, source_document_id, etc.), triggers (content_history_auto_version, ensure_v1_history_at_commit), and RLS policies continue to function. Engine respects the schema; we tell it which columns to write. This is the desired path.

Scenario B: cocoindex wants schema ownership. Engine generates target tables based on the flow’s declared schema; pre-existing content_items is incompatible. We’d need to either (a) drop and let cocoindex re-create (loses 70 columns of KH-specific data), or (b) introduce a “shadow target” table that cocoindex owns + a sync trigger to KH’s content_items. This is the bad path — would force significant schema rework.

I could not verify which scenario applies without hands-on. The cocoindex examples I read all create new tables for new pipelines (pdf_embedding creates a docs table from scratch). I have not found an example binding to a pre-existing complex schema. This needs validation in a 1-day spike before any commitment.

If Scenario A holds, schema integration cost is trivial (~half day to wire each target binding).

If Scenario B holds, schema integration cost is significant (~2-3 weeks to design + build a shadow-table + sync layer, OR to migrate content_items to cocoindex-owned shape — the latter conflicts with the existing KH FK/RLS posture).

5.5 entity_mentions + entity_relationships specifically

Even if Scenario A holds for content_items, the entity tables raise a separate question: cocoindex’s “entity_resolution” module (faiss + Pydantic AI) does dedup-by-embedding-similarity, but it does NOT use KH’s existing entity_aliases registry, entity-type taxonomy, or the Pass-1/Pass-2 classification flow.

Two paths:

1. Keep KH’s classify.ts + Pass 2 entities. Wrap as a @coco.fn and emit entity_mentions rows as a target. cocoindex’s role is incremental re-runnability + caching; classification semantics stay KH’s.

2. Replace with cocoindex’s entity_resolution module. Loses 1,399 LOC of scripts/kb_pipeline/classify.py + the 7-domain keyword classifier + Pass 1/Pass 2 distinction + the KH entity-type taxonomy. Major rework. Probably wrong shape for KH given the bid-management domain specificity.

Recommendation: path (1). cocoindex is the orchestration substrate; KH’s classification stays as-is, just becomes a @coco.fn.

5.6 NEW5/NEW7 — do we need binary storage at all?

Per the user’s 07-synthesis-feedback line 61: “Do we need to store binaries? If all content is extracted to markdown anyway, and we have the file provenance (for audit purposes), could all content be saved as markdown in a DB column/table?”

cocoindex doesn’t change this answer directly, but it reframes it:

• If binaries are NOT stored, the source for cocoindex’s binary shape is a transient upload → extract → write markdown to DB. Source-key: the logical-doc-id assigned at upload. Re-upload of same logical-doc-id → engine sees new content-hash → re-extracts → updates target row. Cleaner than today’s source_documents bucket-write story because the engine doesn’t need persistent binary storage to detect change — it stores the content-hash in its ops-DB.
• The audit-trail concern (legal/compliance — “show me the original PDF the user uploaded”) would be handled by uploading to a dedicated audit bucket (or SharePoint integration in v2 per user’s note) — separate from cocoindex’s flow. cocoindex tracks WHAT was extracted; the bucket tracks WHAT WAS UPLOADED. They’re decoupled.

Net: cocoindex makes “no binary storage in main flow” cleaner. The user’s hypothesis (line 61) — that binaries don’t need to be a primary storage concern — aligns well with cocoindex’s source-key + content-hash model.

5.7 Q&A pipeline collapse — could cocoindex absorb the 440 prod rows from .docx with track changes?

The user’s 07-synthesis-feedback lines 37-47 raises significant concerns about Q&A:

• 440 rows in prod from .docx files (some with track changes — handled via python-docx’s open_document_safe() per CLAUDE.md gotchas)
• Markdown version exists (provided by client after they ran a training course; client now provides .md as preferred format)
• The user notes: “Q&A pairs would have ‘arrived at the route already formed via paste from spreadsheet UI’, which isn’t the case” — the synthesis was wrong. Actual ingest happened via scripts/import_bid_library.py (P3 Python pipeline).
• “we can’t guarantee the format that we could receive Q&A pairs in from a client unless we standardise the format within which we ingest them or extract them”

cocoindex’s role here: the engine doesn’t solve format-standardisation — that’s still our extractor’s responsibility. But it does provide two things:

1. A clean place to write the format-detection + extraction logic. A qa_docx_adapter @coco.fn takes file → returns ExtractedContent[] (one per Q&A pair). Pattern A/B logic stays in Python (or wraps existing TS via subprocess).
2. Incremental re-runnability when format-handling code changes. When we improve the Pattern B parser, code-hash invalidates, all Q&A docx rows re-extract automatically. Today, that’s a manual re-ingest cycle.

The 440-row migration scenario: if we adopt cocoindex pre-re-ingest, the .docx binaries would be the source for cocoindex; on first run, all 440 rows extract via the canonical adapter; future changes to the adapter (e.g. adding Pattern C) re-run only on next cocoindex update. Far cleaner than today’s “manual re-ingest with —batch-tag” pattern.

But: this requires .docx binaries to be available (the user noted in 0.7.4-Q4 that they’re not — 387 of 440 Q&A pairs have no source binary). Open question: do we have the original 440 .docx files preserved somewhere? If yes, cocoindex absorbs the migration cleanly. If no, we backfill from existing source_file text (cheap option per 0.7.4-Q4) — same as the synthesis recommended, but the new architecture makes the backfill easier (it’s a one-time localfs source binding to a folder of synthetic .md files representing the existing 387 rows).

5.8 Comparison to graphify — overlap, complement, or compete?

They are orthogonal.

	graphify	cocoindex
Scope	Code + docs (a folder) → knowledge graph	Any source → any target, incrementally
Output	graph.json + graph.html + GRAPH_REPORT.md (analysis artefacts)	Live target stores (Postgres / Neo4j / vector DBs)
Mental model	Static analysis snapshot	Continuously synced data flow
Role for KH	Dev workflow tool + post-launch concept-map feature	Production ingestion pipeline (replaces ~80% of KH’s ingest code)
Lifecycle	Run on demand	Run continuously / on cron
Schema impact	None (writes external files)	Significant (binds to Postgres targets directly)
Built with	Python + tree-sitter + NetworkX	Python + Rust (incremental engine)

They complement each other cleanly. Graphify analyses code structure; cocoindex orchestrates data flow. Both adopted: graphify is the dev-time map (“how is the codebase structured?”); cocoindex is the runtime engine (“how does data flow through?”). The dual-adoption story is internally consistent.

The only conflict: both want to be the “knowledge graph” answer post-launch. Graphify’s Option A (docs/plans/phase-0-investigation/graphify-evaluation.md §5.2) was “borrow analysis primitives, build a workspace concept map from entity_relationships”. cocoindex includes Neo4j / FalkorDB / SurrealDB targets natively — we could bind entity_mentions + entity_relationships to a Neo4j target and get the graph for free. This duplicates Option A’s effort.

Resolution: adopt cocoindex first (it has graph-DB targets built-in), then evaluate whether graphify’s specific analysis primitives (Leiden cluster + god-nodes + surprising-connections) add value above what cocoindex’s graph + Neo4j community-detection adds. They might, but the incremental case for graphify Option A weakens significantly if cocoindex is in.

5.9 Options ranked by ROI (KH platform)

Option A — Full canonical pipeline replacement with cocoindex

Concept: Phase 0.7 Stream 2 (Phase A foundation + Phases B/C/D/E/F/G) replaces the canonical-core build with cocoindex flow. All 5 shape adapters become @coco.fn decorated functions. Layer 4 stages are sequenced as cocoindex transforms. Postgres targets bound to existing schemas.

Why high-ROI under the corrected lens:

• Replaces 9-12 weeks of canonical-pipeline construction work with battle-tested infrastructure.
• Collapses ~6,000-7,000 LOC of KH ingestion + orchestration code.
• Re-ingest cycle (planned weeks 3-4) is replaced by cocoindex update running once on the empty DB — the engine’s incremental model handles all subsequent re-runs.
• Re-upload UPDATE behaviour (A2 ratified yes) becomes architecturally enforced by the engine’s source-key + content-hash model.
• ingestion_quality_log (NEW8 ratified yes) becomes a custom emitter — ~half a day to wire (vs ~3 days for TS-side helper port).
• Code-hash memoisation gives us incremental re-classification for free when prompts change — eliminates the need for custom batch-reclassify scripts.
• The user explicitly asks (07-synthesis-feedback line 52): “interested to understand how the following third-party tools may be beneficial here, keeping in mind the feedback I provided in the graphify evaluation feedback doc, around ‘re-use’ and ‘integration’ versus re-inventing the wheel” — this is the re-use answer for ingestion architecture.

Risks:

• Schema-coupling unknown (Scenario A vs B per §5.4) — must validate with 1-day spike.
• No native Anthropic SDK — go via LiteLLM, may lose prompt-cache control. ~2 days to prototype + verify.
• Q&A docx port — 497 LOC of TS extract logic; either Python rewrite (~1 week) or subprocess-wrap (~2 days). Subprocess-wrap is fine.
• RSS pipeline integration — P9 stays as upstream feeder; ~3 days to design the staging-table handoff.
• Operations DB persistence — local SQLite needs to be persisted across Cloud Run jobs. Probably mounted from a GCS volume; ~1 day operational setup.
• Adoption time — Liam is the product owner; agentic dev would need to build proficiency with cocoindex declarative model. Estimated 1-2 weeks of learning curve.

Effort estimate (under corrected “no time pressure” lens):

Phase	Work	Effort
Spike	Schema-coupling validation; LiteLLM Anthropic prompt-cache test; ops-DB persistence design	3-5 days
Foundation	cocoindex install + flow.py skeleton + Postgres targets bound + first shape (markdown text) end-to-end	1 week
Shape adapters	URL + document-binary (docling) + qa-docx subprocess-wrap	1.5-2 weeks
Custom emitters	pipeline_runs + ingestion_quality_log + post-stage adapters (inferLayer, summary, expiry, quality_score)	1 week
RSS integration	Staging-table feeder + cocoindex postgres_source binding	3-5 days
Migration off existing pipeline	Retire scripts/kb_pipeline/, lib/extraction/ (or absorb), lib/bid-library-ingest/ (subprocess-wrapped)	1 week
Cutover + re-ingest	Run on empty staging DB; verify outputs match expectations; cut over prod	1 week

Total: ~6-8 weeks, vs the Phase 0.7 Stream 2 estimate of 9-12 weeks. But the work is fundamentally different — instead of building canonical orchestration, we’re integrating cocoindex’s. The savings come from re-use, not optimisation.

Confidence: 75%. The schema-coupling unknown is the dominant risk. If Scenario B applies, effort balloons to 12-16 weeks and the case weakens.

---

Option B — Hybrid: cocoindex for new shapes, KH legacy for existing

Concept: Use cocoindex for shapes where it’s a clean fit (markdown text, document-binary via docling) and keep KH’s existing pipeline for URL + qa-docx + RSS where it’s already mature.

Why moderate-ROI:

• Lower risk than Option A (less surface area touched).
• Preserves KH’s existing Anthropic prompt-caching for classification.
• Avoids the qa-docx port question entirely.

Why ultimately weaker than Option A:

• We end up with two pipelines side-by-side — cocoindex’s incremental model for some shapes, KH’s manual orchestration for others. Compounds the existing 10-path complexity rather than collapsing it. Defeats the canonical-pipeline purpose.
• The Phase 0.7 plan’s whole point is collapse-to-one-core. Splitting between two engines is anti-collapse.

Effort: ~3-4 weeks, but result is architecturally worse than either Option A or pure KH Phase 0.7.

Confidence: 80%. Lower-risk, but I’d recommend against because it doesn’t deliver on canonical-pipeline goals.

---

Option C — Cocoindex as “outer layer” only (Phase 0 verifier + diff engine)

Concept: Don’t replace the pipeline. Use cocoindex narrowly for two things:

1. Re-runnable diff engine — bind cocoindex to KH’s existing content_items table as source, run a flow that re-classifies + re-embeds when prompts change, write back to the same table.
2. Lineage observability — operations DB becomes the queryable lineage record (replaces deferred ai_call_log work).

Why low-leverage:

• KH’s existing pipeline keeps doing all the work; cocoindex only handles re-runs.
• Most of cocoindex’s value (incremental sources, target sync, code-hash memoisation across the full flow) is unused.
• Doesn’t help with canonical-pipeline collapse.

Effort: ~2 weeks. But ROI is low; mostly duplicates ai_call_log work.

Verdict: don’t pursue. Either go full Option A or skip cocoindex entirely.

---

Option D — Skip cocoindex entirely

Concept: Build Phase 0.7 Stream 2 as planned (~9-12 weeks of canonical-pipeline work in TS), keep KH’s existing Python tier for what it does well.

Why this is a defensible choice:

• TS-first orchestration matches KH’s main codebase posture (Vercel, Next.js, MCP).
• Python pipeline (~7,404 LOC) is real, working, tested. It serves 71% of prod (P3 Q&A path) and the URL/markdown paths.
• Re-use of existing code is also “re-use” — Phase 0.7 already plans to refactor, not rewrite.
• No new dependency surface. No Rust runtime in production.

Why ultimately weaker than Option A under corrected lens:

• The user explicitly raised cocoindex (and pullmd, skill-seekers) in 07-synthesis-feedback line 52 with the framing “re-use vs re-invent”.
• The corrected lens is “what should we be doing”, not “what’s lowest-disruption”.
• Phase 0.7 Stream 2 is mostly building a poor-man’s cocoindex (canonical core with side-channels, re-runnable stages, observability). cocoindex is the productionised version.

Confidence: 65%. The case for skipping cocoindex weakens significantly under the empty-DB-pre-re-ingest framing. But it’s coherent if the unknowns from §2.2 surface as blockers.

---

5.10 Where cocoindex specifically helps with NEW5/NEW7 (per the prompt)

NEW5 — D2 markdown storage_path: In Option A, the question dissolves. cocoindex’s source key (logical-doc-id from source_documents) replaces storage_path as the durable identifier. The bucket-vs-column debate becomes moot for the main flow; the bucket (if kept at all) is a separate audit-trail concern.

NEW7 — Re-upload detection in D2 v1: Engine handles it natively. New content-hash for same source-key → recompute → UPDATE existing target row → content_history v2 written via the auto-version trigger. No detect_reupload RPC needed for the markdown path; the existing P7 RPC retires for the binary path too.

Re-upload UPDATEs existing content_items (A2 — user ratified yes): Engine semantics deliver this for free when the target binding is keyed correctly. The biggest single UX unlock from Phase 0.7 §3.1 is architecturally enforced rather than custom-built.

5.11 Where cocoindex DOES NOT help

• Q&A docx Pattern A/B parser quality. That’s domain logic — no engine substitutes for it. We still need our extract-qa-pairs.ts, it just becomes a @coco.fn.
• Classification prompt quality. We still tune our Anthropic prompts; cocoindex just orchestrates running them.
• RSS relevance scoring. P9’s 4-tier extraction + Claude relevance gate is domain-specific; cocoindex doesn’t replace it.
• Content-quality scoring (quality_score). Custom KH logic, becomes a @coco.fn.
• Semantic search runtime. cocoindex builds the index; queries against it are still our search code.
• MCP server. P10’s create_content_item tool is OAuth-gated and goes through KH’s API; cocoindex is downstream of that. No change.
• The user’s question about MCP tools (line 55: “P10 MCP create — this is a priority area because it’s where the client is already utilising Claude Desktop and Claude CoWork to create content”). cocoindex doesn’t change MCP at all. The MCP tool’s job is to take a content payload and call the canonical pipeline; whether the pipeline is KH-built or cocoindex-built doesn’t affect the tool’s interface.

---

6. Recommendations (corrected framing applied)

6.1 Adoption verdict

Recommendation: Re-use, with hybrid integration. Specifically: Adopt cocoindex as the ingestion-pipeline orchestration substrate (Option A), conditional on the schema-coupling spike confirming Scenario A.

This is NOT “defer to post-launch”. The user has explicitly said:

• Now is the right time for architectural changes (“we should look at not what we’re doing right now but what we should be doing”)
• Empty DB pre-re-ingestion makes this the cheapest possible time
• Re-use battle-tested infrastructure where it pays
• No time pressure
• 6-8 week effort under Option A is competitive with 9-12 weeks of the existing Phase 0.7 Stream 2 plan

The effort comparison (cocoindex vs build-our-own canonical core) tilts cocoindex’s way:

Dimension	Build-our-own (Phase 0.7 Stream 2)	Adopt cocoindex (Option A)
Effort	9-12 weeks	6-8 weeks (post-spike)
Code retained	All of lib/extraction/, scripts/kb_pipeline/, much of lib/ingest/	Most extractors retire; replaced by ~500-800 LOC declarative flow
Re-ingest cycle	Manual, weeks 3-4 of Stream 1	Replaced by engine’s incremental Δ
Re-upload UPDATE behaviour	Custom build	Engine-native
ingestion_quality_log	~3 days helper port	~half day custom emitter
Code-hash invalidated re-classify	Custom batch script	Engine-native
Long-term maintenance	KH owns 7,400+ LOC	KH owns ~500-800 LOC + cocoindex dependency
Risk	Known (we’ve planned each step)	Schema-coupling unknown (resolvable by spike)

The spike — 3-5 days hands-on verification — is the gate. After the spike, the decision is informed.

6.2 Sequencing within Phase 0.7

Stream 1 (re-ingest readiness gates) work that the user already approved to proceed on a separate top-level worktree (07-synthesis-feedback line 80) is unaffected by this recommendation. Items 1, 3, 5, 6, 7, 11 — all schema/migration/wiring fixes. Those continue.

Stream 1 items on hold per user (items 2, 4, 8, 9, 10, 12, 13) are partially absorbed by cocoindex Option A if adopted:

• Item 2 (D2 source_documents parity) — cocoindex’s source-key model dissolves the question.
• Item 4 (P7 silent-fail) — fixed differently; cocoindex’s failure-isolation replaces try/catch handling.
• Item 8 (OVERSIGHT helpers) — folded into cocoindex flow stages.
• Items 9 + 10 (RSS + batch chunk regen wiring) — chunks regenerate automatically when chunking logic changes.
• Item 12 (TS-side ingestion_quality_log) — replaced by custom emitter.
• Item 13 (Path 7 swallow-catch UI) — still needed (it’s UX, not pipeline), but the underlying pipeline failure-isolation comes free from the engine.

So post-cocoindex-adoption, Stream 1 collapses from 13 items to ~6 items (items 1, 3, 5, 6, 7, 11 — all schema-side). The “hold” items become unnecessary.

6.3 Recommended sequence

1. Week 1: cocoindex spike. 3-5 days hands-on. Validate Schema Scenario A. Test Postgres-target binding to pre-existing content_items table. Verify LiteLLM-via-Anthropic prompt-caching. Test code-hash memoisation behaviour. Build a toy flow that reads markdown, classifies via Claude, writes to a staging Supabase branch.
2. Week 1 in parallel: Stream 1 gates approved by user (items 1, 3, 5, 6, 7, 11). ~6 hours of work, no blocking dependency on cocoindex decision.
3. Decision gate: post-spike, evaluate. Go / no-go on Option A.
4. If go: Weeks 2-8: Option A implementation as scoped above.
5. If no-go: Weeks 2-12: Phase 0.7 Stream 2 as originally planned.
6. Week 9 (or 13): Re-ingest (Option A: it happens as part of cocoindex update; original plan: explicit re-ingest cycle).

6.4 Specific decisions for Liam

ID	Decision	Recommendation	Confidence
CC1	Run cocoindex schema-coupling spike (3-5 days) before committing?	Yes — this is the most important uncertainty; cheap to resolve; gates the rest	95%
CC2	If spike confirms Schema Scenario A: adopt cocoindex Option A as canonical pipeline substrate?	Yes, conditional — re-use of battle-tested infrastructure is exactly the corrected-lens answer	75%
CC3	If spike confirms Scenario B: fall back to Phase 0.7 Stream 2 as planned?	Yes — schema rework cost is too high otherwise	90%
CC4	Run cocoindex spike on the same worktree as graphify install, in parallel with user-approved Stream 1 items 1/3/5/6/7/11?	Yes — orthogonal work, independent merge windows	90%
CC5	Port qa-docx Pattern A/B logic to Python or wrap TS via subprocess?	Subprocess-wrap initially (~2 days vs ~1 week port); revisit post-launch	75%
CC6	Keep KH’s Anthropic SDK direct for classification, route via LiteLLM, or rewrite classifier prompts?	Spike must resolve this — depends on whether LiteLLM surfaces prompt-cache headers cleanly. Default: keep direct, wrap as @coco.fn	70%
CC7	If Option A adopted: do P9 RSS pipeline stays as upstream feeder, OR port to cocoindex?	Stay as upstream feeder v1 — RSS-discovery is a pre-content-items concern; can port post-launch if value emerges	80%
CC8	If Option A adopted: do MCP create_content_item tool stays as KH API → cocoindex flow’s external trigger?	Stay as KH API — MCP is the user-facing interface, cocoindex is the runtime; they layer cleanly	95%
CC9	If Option A adopted: graphify Option A (workspace concept map) revisit — pursue, defer, or drop in favour of cocoindex Neo4j target?	Defer graphify Option A — cocoindex Neo4j target gives us the same primitive natively. Re-evaluate post-launch	70%
CC10	Operations DB persistence: GCS volume on Cloud Run, or persistent Postgres table?	GCS volume — simpler, matches cocoindex defaults, deterministic. Persistent Postgres is overkill	80%

---

7. Open questions

1. Schema-coupling Scenario A vs B. Resolves via 1-day spike. Highest priority unknown.

2. LiteLLM Anthropic prompt-cache surface. Does LiteLLM pass through cache_control headers transparently? If not, KH would lose ~10-30% cost savings on classification (depending on prompt-cache hit rate). Resolves via 1-day prototype.

3. docling vs KH’s unpdf + mammoth + turndown chain. docling is a fully different PDF/DOCX library. Output may differ subtly from current extraction (different tables handling, different image OCR, different markdown output). Need a side-by-side comparison on ~50 prod-representative files. Resolves via 2-day comparison study.

4. Operations DB at scale. When KH has 100k+ content_items, the SQLite ops-DB grows substantially. Performance characteristics unknown. May need to swap to a Postgres-backed ops-DB. Resolves via load-test in spike.

5. Concurrency on Cloud Run. cocoindex update is one process. Cloud Run jobs run on N workers. Does cocoindex partition across workers natively, or do we need to shard input ranges manually? Resolves via documentation review + spike.

6. Code-hash semantics under TypeScript subprocess wrapping. If we wrap our existing TS extract-qa-pairs.ts via subprocess (recommendation CC5), what does cocoindex use as the “code hash”? The Python wrapper bytecode? The TS file content? Both? Resolves via documentation + 1-day test.

7. RSS staging-table handoff design. P9 writes to staging Postgres table; cocoindex postgres_source reads. Deletion semantics, retry-on-transient-failure, dead-letter handling — all need explicit design. ~3 days.

8. The user’s “pullmd” and “skill-seekers” tools also surfaced in feedback — those evaluations are separate. Recommendation: complete this cocoindex eval, then pullmd, then skill-seekers, before final architecture decision. They may add or change the picture.

9. Backout plan. If cocoindex underperforms in production (reliability, performance, support burden), how do we revert? pip uninstall is trivial; restoring KH’s existing pipeline depends on whether we’ve deleted code. Recommendation: keep scripts/kb_pipeline/ and lib/extraction/ archived in git after migration; allow ~3 months of dual-readability before deletion.

10. Anthropic-SDK direct vs LiteLLM cost overhead. LiteLLM proxy adds ~50-200ms latency per call. For a multi-stage flow (extract → classify → embed), this compounds. Resolves via spike timing measurement.

---

8. Confidence assessment

Section	Confidence	Reason
§1 What it is	92%	Sourced from README + repo structure + connector listing; verified via examples
§2 Grounded audit	78%	Could not run hands-on (sandbox SSL block); structural understanding solid but operational details unverified
§3 Architecture summary	85%	Sourced from official docs + connectors + ops directory; some operational properties (concurrency, ops-DB scaling) unverified
§4 Dev workflow value	88%	Conclusion (mostly orthogonal to dev workflow) is well-founded
§5 KH platform value	75%	Schema-coupling unknown is the dominant uncertainty; Options A-D well-reasoned; effort estimates ±25%
§6 Recommendations	78%	Conditional structure (spike-then-decide) is robust; the unconditional pieces (e.g. “don’t pursue Option C”) are higher confidence
§7 Open questions	90%	Per-item unknowns explicitly enumerated; spike resolves the top 4

Overall evaluation confidence: 80%.

Below 90% specifically because:

1. Hands-on validation impossible in sandbox. All unknowns from §2.2 are real and unresolved.
2. Schema-coupling Scenario A vs B. This single dimension changes the recommendation materially. Without verifying via spike, the Option A recommendation carries the largest single point of risk.
3. LiteLLM Anthropic surface. KH leans heavily on prompt-caching (lib/ai/classify.ts); if LiteLLM doesn’t surface cache_control headers, cost savings degrade.
4. Effort estimates are ±25-30% under “we haven’t done this before” framing. If schema-coupling needs a shadow-table pattern (Scenario B fallback), effort jumps from 6-8 weeks to 12-16 weeks.

I am, however, confident in the framing — that under the corrected lens (re-use vs re-invent, empty DB, no time pressure), cocoindex is a serious candidate that deserves a spike-and-decide treatment. NOT a “defer to post-launch” recommendation.

---

9. Appendix

9.1 The user’s question on 07-synthesis-feedback line 87 — full data-flow trace under cocoindex Option A

“If we have a source document which then gets extracted as a content item(s) and then I edit that content item in the platform, and then upload a new version of the document (including a different filename), what’s happening in the platform from a data flow perspective, and data provenance?”

Today (P7 + 0.7.4 §5.4):

1. Upload policy-v1.docx → P7 INSERTs source_documents row 1 (filename=policy-v1.docx, parent_id=NULL, version=1) → INSERTs content_items row A (source_document_id=row1.id) → DB trigger writes content_history v1.
2. User edits content_items row A in platform → DB trigger writes content_history v2 (change_type=edit).
3. User uploads policy-v2.docx (different filename) → detect_reupload fails to match (filename differs) → INSERTs source_documents row 2 (filename=policy-v2.docx, parent_id=NULL, version=1 — fresh chain) → INSERTs content_items row B (source_document_id=row2.id) → DB trigger writes content_history v1 for row B.
4. Result: two content_items rows (A with user edits, B with new doc); two unconnected source_documents chains; user has to manually decide what to do with the orphaned A.

Under cocoindex Option A:

1. Upload policy-v1.docx via UI → POST to /api/upload → KH API generates a logical document key K (e.g. derived from user-confirmed “this is a new document” or from explicit document-id input), writes binary to bucket if needed, INSERTs source_documents row 1 (filename=policy-v1.docx, document_key=K, version=1, parent_id=NULL) → cocoindex source binding sees new row in source — runs the binary-shape adapter → extracts to markdown → flows through canonical core stages → UPSERTs content_items row A (keyed by K) → DB trigger writes content_history v1.
2. User edits content_items row A in platform → DB trigger writes content_history v2. cocoindex’s source state for K is unchanged (the binary input didn’t change); engine ignores. The platform-edit is independent of the engine’s incremental flow.
3. User uploads policy-v2.docx AND tells the platform it’s “the new version of K” → KH API generates source_documents row 2 (filename=policy-v2.docx, document_key=K, version=2, parent_id=row1.id — chained) → cocoindex source binding sees existing key K with new content-hash → re-runs binary-shape adapter → extracts new markdown → flows through canonical core stages → UPSERTs content_items row A (same key K, NOT a new row) with new content → DB trigger writes content_history v3 (change_type=source_document_accepted or similar).
4. Result: ONE content_items row A, full content_history showing v1 (initial extract) → v2 (user edit) → v3 (new doc version). source_documents chain shows row1 → row2 (parent_id chain). Re-upload UPDATE behaviour architecturally enforced.

The hard part is step 3a — “user tells the platform it’s the new version of K”. That’s a UX decision (UI affordance: dropdown “this is a new version of: [existing document]”). cocoindex doesn’t infer logical identity from content alone; it relies on the source emitting the right key. The engine enables the UPDATE-not-INSERT behaviour; the UI drives it.

This matches A2 ratification (“re-upload UPDATEs existing content_items, creating content_history v2 — yes, v1 target”) cleanly.

Provenance under cocoindex:

• source_documents.parent_id chain: identical to today’s model (KH-owned).
• content_history: identical to today’s model (KH-owned, DB-trigger-driven).
• NEW: cocoindex operations DB ledger — per-row “what input + what code hash → what output” for every flow run. This is the stronger provenance answer that complements (doesn’t replace) content_history. When debugging “why did extract produce X”, the ops-DB has the answer.
• NEW: if cocoindex’s Neo4j target is bound (see CC9), the entity_relationships graph is continuously synced — gives us the post-launch concept-map primitive with no extra work.

9.2 Comparison matrix vs build-our-own canonical pipeline

Capability	KH Phase 0.7 Stream 2 (build)	cocoindex Option A (re-use)	Winner
Canonical pipeline core	Custom build (~2 weeks Phase A)	Native engine	cocoindex
Per-shape adapters	Custom build (~6 weeks Phases B+E+F)	Custom @coco.fns on top of engine (3-4 weeks)	cocoindex
Re-ingest cycle	Manual (~1 week, weeks 3-4)	Engine-native	cocoindex
Re-upload UPDATE behaviour	Custom build	Engine-native	cocoindex
ingestion_quality_log	TS helper port (~3 days)	Custom emitter (~half day)	cocoindex
Code-hash invalidated re-classify	Custom batch script	Engine-native	cocoindex
Anthropic prompt-caching	Direct SDK (works today)	Via LiteLLM (verify in spike)	KH Phase 0.7 (today)
RSS pipeline	Stays in KH — no change	Stays in KH as upstream feeder	Tie
MCP create_content_item	Stays in KH	Stays in KH; calls API which feeds cocoindex	Tie
Schema flexibility	KH owns schema entirely	Schema-coupling unknown (Scenario A vs B)	KH Phase 0.7 (known)
TypeScript-first orchestration	KH posture matches	Adds Python tier (already present in scripts/kb_pipeline/)	KH Phase 0.7
Long-term LOC maintenance	~7,400+ LOC	~500-800 LOC + dependency	cocoindex
Battle-testing	We’re building it	Apache 2.0 OSS, used by community	cocoindex
Backout cost	Trivial (we own it)	Code archived for ~3 months, then removable	KH Phase 0.7 (slight)

Tally: cocoindex wins 7, KH-build wins 4, ties 2. Weighted by impact, the cocoindex wins are larger (canonical pipeline core, re-ingest cycle, re-upload behaviour are major unblocks).

9.3 References

cocoindex sources read:

• https://github.com/cocoindex-io/cocoindex (README, root)
• https://cocoindex.io (homepage)
• https://cocoindex.io/docs/getting_started/quickstart
• https://github.com/cocoindex-io/cocoindex/tree/main/python/cocoindex (package structure)
• https://github.com/cocoindex-io/cocoindex/tree/main/python/cocoindex/connectors (14 connectors)
• https://github.com/cocoindex-io/cocoindex/tree/main/python/cocoindex/ops (transformations)
• https://github.com/cocoindex-io/cocoindex/blob/main/python/cocoindex/ops/text.py
• https://github.com/cocoindex-io/cocoindex/blob/main/python/cocoindex/ops/litellm.py
• https://github.com/cocoindex-io/cocoindex/blob/main/examples/pdf_embedding/main.py
• https://github.com/cocoindex-io/cocoindex/tree/main/examples/conversation_to_knowledge

KH sources cross-referenced:

• docs/plans/phase-0-investigation/0.7-synthesis.md (canonical pipeline plan)
• docs/plans/phase-0-investigation/07-synthesis-feedback.md (user ratifications + framing corrections)
• docs/plans/phase-0-investigation/0.7.1-path-normalisation-feasibility.md (4-layer architecture)
• docs/plans/phase-0-investigation/0.7.4-source-documents-history-relationship.md (source_documents architecture, 1:N relationship, re-upload friction)
• docs/plans/phase-0-investigation/graphify-evaluation.md + graphify-evaluation-feedback.md (re-use lens template)
• docs/plans/phase-0-investigation/trpc-evaluation-feedback.md (framing corrections)
• KH code: lib/extraction/ (1,000 LOC), lib/bid-library-ingest/ (638 LOC), scripts/kb_pipeline/ (7,404 LOC), lib/ai/classify.ts, lib/ai/embed.ts, app/api/upload/route.ts

9.4 Files NOT investigated hands-on (acknowledged gaps)

• cocoindex Rust engine source (only Python interface read)
• cocoindex examples/conversation_to_knowledge full source code (only design.md + spec.md summary)
• LiteLLM Anthropic adapter behaviour with prompt-caching
• docling DOCX-with-track-changes handling vs python-docx open_document_safe() (the KH gotcha pattern)
• cocoindex production deployments at scale (no third-party benchmark surveyed)

These all resolve via the recommended 3-5 day spike (CC1).

---

End of cocoindex evaluation.

Top three findings:

1. cocoindex’s incremental engine semantically IS the canonical pipeline core that Phase 0.7 Stream 2 was planning to build. Adopting it (Option A, conditional on schema-coupling spike) replaces 9-12 weeks of custom orchestration construction with battle-tested infrastructure, collapses ~6,000-7,000 LOC of KH ingestion code, and architecturally enforces the A2-ratified re-upload-UPDATE behaviour for free.

2. Re-use over re-invent applies cleanly here. Per the corrected framing (07-synthesis-feedback) and graphify-feedback’s “re-use vs integrate vs reinvent” lens, cocoindex is the right candidate to evaluate seriously. The user explicitly raised it. The empty-DB pre-re-ingest moment is the cheapest possible time. No time pressure.

3. The single dominant unknown is schema-coupling. Whether postgres.mount_table_target accepts pre-existing 70-column schemas with FKs and triggers (Scenario A) or wants schema ownership (Scenario B). Resolvable via 1-day spike. Unresolved, the recommendation is conditional. Resolved as Scenario A, the recommendation is firm Option A. Resolved as Scenario B, the fallback is Phase 0.7 Stream 2 as originally planned.

Top three uncertainties:

1. Schema-coupling Scenario A vs B (highest priority — gates Option A).
2. LiteLLM Anthropic prompt-cache passthrough (cost-affecting — affects total ROI).
3. Q&A docx Pattern A/B port (route-of-71%-of-prod — subprocess-wrap mitigates initially).

Recommendation: 3-5 day spike, then decide. Run the spike in parallel with user-approved Stream 1 items 1/3/5/6/7/11 on a separate worktree. Decision at end of week 1. If go: 6-8 weeks Option A. If no-go: 9-12 weeks Phase 0.7 Stream 2.