Phase B Prerequisite 2a — Cocoindex Examples + Capabilities Deep-Dive

Phase B Prerequisite 2a — Cocoindex Examples + Capabilities Deep-Dive

⚠️ S256 ERRATA — DO NOT TREAT AS CANONICAL FOR ExtractByLlm / LlmSpec / LlmApiType SHAPES. This survey was performed 2026-05-13 against the cocoindex 0.3.x examples-repo HEAD (not pinned to the 1.0.3 PyPI baseline cited in the metadata block below). The cocoindex 1.0.0 restructure REMOVED the ExtractByLlm / LlmSpec / LlmApiType symbols cited throughout this doc — they are ABSENT in the installed cocoindex==1.0.3 pin. The canonical record for cocoindex 1.0.3 LLM-extraction integration is docs/research/cocoindex-1.0.3-extractbyllm-spec-reality-investigation.md (S255 deliverable, ratified S256). The current canonical pattern in 1.x is Path A — @coco.fn(memo=True)-decorated function calling anthropic SDK directly with Pydantic validation; live examples-repo files patient_intake_extraction_baml / patient_intake_extraction_dspy / paper_metadata confirm this shape. Treat the §2 “Structured Extraction” + “patient_intake_extraction” + “paper_metadata” subsections below as intent record for the layered-fn / Pydantic-typed-output pattern, NOT as a literal implementation guide. The §2 KH need-mapping rationales survive intact; only the cocoindex-side API symbol names changed.

Audit date: 2026-05-13 Branch: content-items-investigation Author: Research agent (Opus 4.7, 1M context) Scope: Read-only research. Survey the cocoindex 1.0.3 examples repo plus Structured Extraction + Entity Resolution capabilities, and map each affordance to a KH need (citing feedback-findings-review.md §1 themes + Finding docs + spike outcomes). Surface gaps. No code edits. Cocoindex version baseline: 1.0.3 (PyPI, confirmed by S2 + S14 + S1 spike installs) — BUT the examples-repo HEAD at audit time was 0.3.x for the citation lines below; see S256 errata block above. Examples list source: https://github.com/cocoindex-io/cocoindex/tree/main/examples (27 examples enumerated 2026-05-13).

Reading order:

• §1 — Capability summary table (one-row-per-example)
• §2 — Per-example deep dive with KH mapping (postgres_source → text_embedding → files_transform → meeting_notes_graph_neo4j → pdf_embedding → pdf_to_markdown → entity_resolution + Structured Extraction + remaining examples)
• §3 — Summary mapping table (capability ↔ KH use case ↔ planned/gap)
• §4 — Top-3 recommendations
• §5 — Open questions surfaced
• §6 — Sources

---

§1 — Capability summary

Example / capability	One-line	Maps to KH need?
postgres_source	Reads from existing PG table, writes derived rows + embeddings to another PG table	Yes — feedback loop / WP3 ledger query
text_embedding	Markdown → chunks → embeddings → pgvector	Yes — primary content_chunks.embedding flow
files_transform	LocalFS markdown → HTML transform (templated, memoised)	Yes — generic transform shape (sidecar generator)
meeting_notes_graph_neo4j (+ _falkordb variant)	LLM-extract Persons/Tasks/Meetings → graph DB; entity resolution dedup	Yes — Knowledge Map (CX.32) + entity_relationships materialise
pdf_embedding	PDF → docling → markdown → chunks → embeddings → pgvector	Yes — binary-shape adapter for Phew PDFs
pdf_to_markdown	Pure PDF → markdown via docling; one .md per .pdf	Yes — Theme B form pipeline step 3 (markdown sidecar)
patient_intake_extraction_* (markitdown / BAML / DSPy)	PDF intake form → typed dataclass via LLM	Yes — directly maps to Theme B form pipeline step 4 (classify-form-data) + Theme A form-agnostic Q&A
paper_metadata	PDF → extract title/authors/abstract + chunks/embeddings	Yes — form metadata sidecar pattern
conversation_to_knowledge	YouTube → audio → transcript → LLM-extract Persons/Tech/Orgs/Statements + entity-resolution → SurrealDB graph	Yes — pattern reference for Phew “named clients/competitors/employees” dedup
entity_resolution (cocoindex.ops.entity_resolution)	FAISS-shortlist + LLM-pair-resolver → canonical name groupings	Yes — Q&A pair dedup; person/org dedup; NOT file-level near-duplicate (per S2)
code_embedding (+ _lancedb)	Python source → tree-sitter chunk → embeddings	No direct KH use; reference only
image_search (+ _colpali)	Images → CLIP embedding → Qdrant	No KH use (no image-search product surface yet)
audio_to_text	Audio file → LiteLLM Whisper → PG text rows	No v1 KH use (Phew has no audio corpus)
csv_to_kafka + kafka_to_lancedb	Streaming source ↔ streaming target	No v1 KH use; reference for live mode
entire_session_search	AI session transcripts → embeddings	No v1 KH use (overlaps mempalace)
gdrive_text_embedding	Google Drive markdown → embeddings	Deferred — v2 connector per 0.9-spike-S2
amazon_s3_embedding	S3 markdown → embeddings	Deferred — staging bucket pattern only
oci_object_storage_embedding	OCI Object Storage + OCI Streams → embeddings	No KH use
multi_codebase_summarization	Walks N repos → instructor + Gemini summary per file → markdown output	No KH use (we have graphify)
hn_trending_topics	HN scrape → analysis	No KH use

Net coverage: ~10 of 27 examples map directly to a KH need; ~8 are reference-only patterns; ~9 are out-of-scope for v1.

---

§2 — Per-example deep dive

Example: postgres_source

What it demonstrates:

• A flow whose source is an existing PG table (source_products), not a filesystem walk. Reads structured rows, computes derived fields, generates embeddings, writes results to another PG table.
• Uses cocoindex.connectors.postgres.PgTableSource(use_context(PG_DB), table_name=..., row_type=DataClass).
• fetch_rows() returns an async iterable; pairs cleanly with mount_each(..., source.fetch_rows().items(key=lambda r: r.id), table).
• Source PG and target PG may be the same database or distinct (env-toggled via SOURCE_DATABASE_URL).

Direct mapping to KH need:

• KH need: WP3 ledger query pattern (Finding 02 §3.3, citation lookups for bid-drafting + Q&A search) — covered Y. Cocoindex’s postgres_source makes the “read existing KH table, derive views, write back” pattern first-class. The same shape applies to the q_a_extractions → curated q_a_pairs promotion flow (0.9-intended-architecture.md §6.2 step 5).
• KH need: feedback loop (Finding 05 — bid_response outcomes feeding KG) — covered Y. The completed bid_responses table can be a cocoindex source; transforms emit entity_relationships deltas + write outcomes to a bid_feedback_signals derived table.
• KH need: per-finding 01 Recommendation 4 (upload-route handles binary-only) — covered Y, indirectly. The upload route writes a source_documents row + drops the binary into localfs (Theme F). postgres_source on source_documents then kicks off the binary-shape adapter chain — i.e. the upload route becomes a thin envelope and cocoindex’s PG-source binding does the orchestration.

Whether KH planned arch already accounts for this:

• Reference: 0.9-intended-architecture.md §5.2.(i) Path 1 (external-folder edit → cocoindex ingest) and §6.2 (Q&A retrieval) implicitly use this pattern but don’t name PgTableSource explicitly. The spike 0.9-spike-S1 confirms target-side; source-side coupling is the converse and was NOT exercised.
• Verdict: partial. The intended architecture assumes localfs as the primary source (UC10 watch). PG-source as a complementary intake is understood but not engineered — needs an explicit decision: do we wrap the upload route as “write source_documents row + drop binary on localfs” and let localfs.walk_dir pick it up, OR do we wire PgTableSource(source_documents) to drive flows? The first is simpler and consistent with S2; the second gives row-level provenance immediately.

Gap / opportunity:

• The Q&A-pair promotion path (q_a_extractions → q_a_pairs curated) is a perfect PgTableSource use case — the source IS a PG table. The intended architecture doc draws this as a manual promote step but doesn’t say it’s a cocoindex flow. Recommendation: model the promotion as a PgTableSource(q_a_extractions, filter=approved) → q_a_pairs target flow. This gives us incremental promote (only changed extractions re-promote) for free.
• For bid_responses feedback (Finding 05): PgTableSource(bid_responses, filter=approved) → derive bid_feedback_signals → write KG edges.

---

Example: text_embedding

What it demonstrates:

• The canonical pipeline: localfs markdown → RecursiveSplitter chunks → SentenceTransformerEmbedder → pgvector PG target.
• Memoised per-file: @coco.fn(memo=True) async def process_file(...). Only changed files re-chunk + re-embed.
• target_table.declare_vector_index(column="embedding") — engine can manage the vector index DDL in SYSTEM mode only. In KH’s managed_by="user" mode (per S1) the vector index is owned by our migrations.
• Example does not create a vector index by default; queries sequential-scan. For prod scale this MUST be configured separately.

Direct mapping to KH need:

• KH need: content_chunks with embedding vector(1024) populated incrementally on content_items writes — covered Y. KH’s text-embedding-3-large (1024d, per CLAUDE.md) is not a SentenceTransformerEmbedder default model, but cocoindex’s LiteLLMEmbedder("openai/text-embedding-3-large") covers it. Alternative path: wrap KH’s existing lib/ai/embed.ts as a Python @coco.fn subprocess call.
• KH need: RecursiveSplitter for chunking, replacing lib/chunking/ — covered Y. Per the cocoindex skill, RecursiveSplitter(chunk_size, chunk_overlap) + optional language="markdown" is drop-in. Tree-sitter aware variants exist for code.
• KH need: match_content_items RPC + RLS — partial Y. Cocoindex writes via the asyncpg pool’s role; RLS enforces for that role per S1 §1 row 7. Recommendation in S1 is service-role-equivalent for the engine writer. The RLS-aware match RPC reads happen client-side (TS/Next) — cocoindex doesn’t read.

Whether KH planned arch already accounts for this:

• Reference: 0.9-intended-architecture.md §6.2 “INGEST LAYER (cocoindex Δ via folder watch)” diagram; 0.8.2-cocoindex-evaluation.md §3.2 “Embedding (text-embedding-3-large) → Replace with LiteLLMEmbedder("openai/text-embedding-3-large") OR keep our existing lib/ai/embed.ts inside a @coco.fn.”
• Verdict: aligned. Q4.4 in §7.1 explicitly says “KEEP KH current (text-embedding-3-large 1024d) — verify cocoindex compatibility in spike.” S1 confirms the schema side (vector(1024) writes work via _vector_encoder pgvector text encoding).

Gap / opportunity:

• LiteLLM-via-Anthropic prompt-caching is unresolved (eval §7 open question #2). For text-embedding-3-large, there is no prompt-cache (it’s an embedding model, not a chat model). So this gap does not apply to embedding. It DOES apply to the classification path (classifyContent + inferLayer) which uses Anthropic chat completions with cache_control.
• Vector index ownership — KH owns the IVFFlat/HNSW index in its migrations (managed_by=“user”). The cocoindex declare_vector_index call must be omitted, OR set such that the engine acknowledges existing index. Verify in S1 follow-up (S1 §5 open question S1-Q4).
• Per-method scoring (OQ-Q112-A, embedding vs fulltext separate columns): cocoindex’s flow can produce both an embedding and a fulltext-search column in the same row, via additional @coco.fns. Not a single primitive — handled by additive flow stages.

---

Example: files_transform

What it demonstrates:

• Generic “in-folder → transform → out-folder” pattern. Source: **/*.md. Transform: MarkdownIt("gfm-like").render(text). Target: localfs.declare_file(outdir / outname, html, create_parent_dirs=True).
• The transform function is purely pluggable. Same shape works for: markdown → HTML; PDF → markdown; DOCX → markdown; XLSX → markdown.
• Auto-cleanup: deleting a source file removes the target file. This is the load-bearing semantic for the markdown-sidecar lifecycle.

Direct mapping to KH need:

• KH need: Theme B form-extraction pipeline step 3 (Markdown conversion via reusable cocoindex pipeline converter — feedback-findings-review.md §1 Theme B) — covered Y. files_transform IS the reusable converter base; per-MIME variants plug in their conversion function (docling.convert for PDF, mammoth + turndown for DOCX, custom XLSX→md for spreadsheets).
• KH need: Markdown sidecar v1 baseline (Finding 02 §3.3 promote to v1) — covered Y. Binary → sidecar markdown is exactly files_transform’s shape, with the input source being binary/* and the output target being markdown/* co-located by file-stem.
• KH need: UC10 external-folder edit ingest (0.9-intended-architecture.md §5.2.(i)) — covered Y. live=True on walk_dir enables fs-watch (FSEvents/inotify per S2).
• KH need: Finding 06 source-doc explorer per-MIME viewer composition (DOCX/PDF/XLSX flows) — covered Y, indirectly. The viewer doesn’t consume the transform output; it consumes the markdown sidecar produced by the transform. Tiptap mounts the sidecar.

Whether KH planned arch already accounts for this:

• Reference: 0.9-intended-architecture.md §5.2 (whole) — sidecar pattern described abstractly; 0.9-spike-S2-cocoindex-folder-binding.md §1 confirms localfs source binding works with live=True. feedback-findings-review.md §1 Theme B step 3 + Finding 06 (per-MIME flow rewrites under markdown sidecar v1).
• Verdict: aligned. This is the most-cited cocoindex pattern across the KH planning corpus.

Gap / opportunity:

• Binary → sidecar fidelity per MIME type is not modelled. Theme B step 1 (“evaluate-form skill, one per format type”) and step 3 (“Structured Outputs skill OR reusable cocoindex converter”) imply per-MIME @coco.fns but the architecture doc treats this abstractly. Recommendation: explicit binary_to_md_* @coco.fn per MIME with shared output target (markdown/{stem}.md).
• DRAFT-vs-final near-duplicate detection (S2 §1 row 6) is NOT served by files_transform — exact-bytes content fingerprint. Must layer entity_resolution-style embedding similarity OR mempalace KG dedup on top (S10 spike substrate). Already flagged in the planning corpus.

---

Example: meeting_notes_graph_neo4j (+ falkordb variant)

What it demonstrates:

• LLM-extract structured Meeting records from markdown notes — Pydantic/dataclass output via instructor+LiteLLM.
• Entity resolution applied to Person names: FAISS-shortlist (sentence-transformer embeddings, cosine >0.7) → LLM pair-resolver confirms duplicates → builds canonical-name dict — collapsing “Alice”, “Alice Chen”, “alice c.” into one Person node.
• Two graph DB targets: neo4j and falkordb — drop-in via cocoindex’s connector registry.
• Node types: Meeting, Person, Task. Edges: ATTENDED, DECIDED, ASSIGNED_TO.
• Per-file @coco.fn(memo=True) extraction + global entity-resolution pass (the resolver is stateful across files, mounted at app scope).

Direct mapping to KH need:

• KH need: CX.32 Knowledge Map surface (0.9-intended-architecture.md §5.2.A.(i) — OPS-G-1 PROVISIONAL “Knowledge Map” user-facing primitive) — covered Y, but with substrate question. Cocoindex’s Neo4j/FalkorDB targets give the visualisation primitive natively. KH does NOT currently use Neo4j (per CLAUDE.md, schema is PG); the per-finding entity_relationships table is the PG-resident KG surface.
• KH need: Theme A form-agnostic Q&A — covered partial. The Meeting/Task/Person extraction pattern is structurally identical to Question/Answer/Response extraction. Substitute the schema: Form (extraction unit), Question, Response, Person/Org (citation provenance) — and the rest of the flow (chunked LLM extract + entity-resolution + graph) carries over.
• KH need: Phew named clients/competitors/employees dedup (Theme A + ontology prereq Finding 03 + 05) — covered Y. Identical pattern to Person-name dedup in this example. This is the clearest direct precedent for KH’s organisation/competitor/employee dedup need.

Whether KH planned arch already accounts for this:

• Reference: 0.9-intended-architecture.md §5.2.A (Knowledge Map provisional, gated on S7 client-docs comparison-data), §6.2 (Q&A flow architecture: 3 parallel transforms include (3) entity_relationships (KG facts)), §7.2 (mempalace adoption shape).
• Verdict: partial. The KG-as-output is named, but the substrate is ambiguous — 0.9-intended-architecture.md defers between (a) PG entity_relationships table + cocoindex flow, (b) Neo4j/FalkorDB target via cocoindex, (c) mempalace KG. Theme F (mempalace MCP direct vs wrapped) is the parallel decision in feedback-findings-review.md.

Gap / opportunity:

• The intended architecture has not committed to a graph-DB target. The entity_relationships PG table is the v1 substrate per §7; Knowledge Map is provisional. If we DO want the Neo4j/FalkorDB target option, this example is the template; otherwise the same pattern works with postgres.mount_table_target(entity_relationships).
• The Pydantic output_type schema in the example is short and inline. KH’s Q&A extraction schema is more complex (citation provenance, evaluation criteria, evidence requirements per Theme B step 4). The example doesn’t directly cover nested schemas — patient_intake_extraction_* does (see below).

---

Example: pdf_embedding

What it demonstrates:

• PDF binary → docling (DocumentConverter) → markdown → RecursiveSplitter chunks → SentenceTransformerEmbedder → pgvector PG target.
• Same shape as text_embedding but with docling-driven PDF → markdown pre-stage.

Direct mapping to KH need:

• KH need: binary-shape adapter for Phew PDFs (Telehouse PDF + bid library PDFs per docs/client-documentation-base/binary/ per S2 §2.2) — covered Y.
• KH need: replace unpdf + mammoth + turndown chain with docling (0.8.2-cocoindex-evaluation.md §3.3 — docling supports DOCX/PDF/HTML/image OCR) — covered Y, but fidelity unverified. Open question #3 in 0.8.2 §7: “docling vs KH’s unpdf + mammoth + turndown chain. Output may differ subtly (different tables handling, different image OCR, different markdown output). Need side-by-side comparison on ~50 prod-representative files.”

Whether KH planned arch already accounts for this:

• Reference: 0.8.2-cocoindex-evaluation.md §5.3 row 2 (document-binary PDF/DOCX) — “Native via docling (separate package). PDF + DOCX + HTML + image OCR. ~1 day to wire docling + pgvector target + source_documents side-channel write.”
• Verdict: aligned, with fidelity-gap open. The implementation path is named; the comparison study is deferred.

Gap / opportunity:

• DOCX track-changes: docling DOCX handling vs python-docx’s open_document_safe() (per CLAUDE.md gotcha). The PDF example doesn’t probe DOCX; the path-of-71%-of-prod (P3 Q&A docx) is uncovered by docling-PDF specifically. The pdf_to_markdown example shape extends, but DOCX track-changes vs docling is unverified.
• Side-channel write to source_documents: not shown in the example. Need a sibling @coco.fn in the same component that writes the source_documents row alongside the markdown extraction.

---

Example: pdf_to_markdown

What it demonstrates:

• Stripped-down PDF → markdown ONLY (no chunking/embedding). One .md per .pdf, written via localfs.declare_file. docling does the conversion. Memoised.
• This is the bare bones of Theme B step 3 — markdown sidecar generation.

Direct mapping to KH need:

• KH need: Theme B step 3 reusable cocoindex pipeline converter (feedback-findings-review.md §1 Theme B) — covered Y, directly named in Liam’s verbatim notes (§4.6: “PDF - To: See example at https://cocoindex.io/docs/examples/pdf-to-markdown/”).
• KH need: Markdown sidecar v1 baseline (Finding 02 §3.3) — covered Y. The sidecar lifecycle (one .md per binary, auto-delete on source delete) IS this example’s shape.
• KH need: Finding 06 PDF flow rewrite (PDF → markdown via cocoindex example) — covered Y, named directly.

Whether KH planned arch already accounts for this:

• Reference: feedback-findings-review.md §4.6 (Liam verbatim), 0.9-intended-architecture.md §5.2.A (sidecar pattern).
• Verdict: aligned + named — this is THE example Liam cited for PDF flow rewrite under markdown sidecar v1.

Gap / opportunity:

• None new — already in the plan. But: the example uses docling only. Theme B step 1 (evaluate-form skill) sits before docling. The KH-specific addition is: pre-conversion form-evaluation skill that picks the right converter (docling baseline vs custom XLSX-as-form-template converter, etc.) — that’s a wrapper around pdf_to_markdown, not a replacement.

---

Example: patient_intake_extraction_* (markitdown, BAML, DSPy variants)

What it demonstrates:

Three sibling examples that all extract structured patient data from PDF intake forms into typed Python objects (Pydantic/dataclass):

1. patient_form_extraction (markitdown + ExtractByLlm): PDF → MarkItDown markdown → ExtractByLlm(output_type=Patient, instruction=...) → PG table. Nested dataclasses (Patient contains Address, Contact, Insurance, list[Condition], list[Medication], list[Allergy], etc.). LLM: OpenAI gpt-4o by default.
2. patient_intake_extraction_baml (BAML): BAML provides type-safe structured-data extraction with native PDF support. Schema in baml_src/patient.baml. Generated Python client. Output: JSON files in output_patients/. Reads PDFs directly without separate markdown stage.
3. patient_intake_extraction_dspy (DSPy + Gemini 2.5 Flash vision): Converts PDFs to images → DSPy ChainOfThought with native image support → Pydantic models → JSON output. No markdown stage at all — vision-direct.

The canonical pattern across all three:

doc["patient_info"] = doc["markdown"].transform(
    cocoindex.functions.ExtractByLlm(
        llm_spec=cocoindex.LlmSpec(
            api_type=cocoindex.LlmApiType.OPENAI, model="gpt-4o"),
        output_type=Patient,
        instruction="Please extract patient information from the intake form."))

Direct mapping to KH need:

• KH need: Theme B step 4 (classify-form-data — verify approach from template-driven-completeness-spec.md, focus on data once form is markdown — feedback-findings-review.md §1 Theme B) — covered Y, directly. ExtractByLlm with a FormQuestionAnswerPair (or BidQuestion, RFPQuestion) typed output IS the classify-form-data primitive.
• KH need: Theme A form-agnostic Q&A — covered Y. The Q&A structure becomes a Pydantic schema:

  class QAPair(BaseModel):
      question: str
      answer: str
      question_kind: Literal["mandatory", "optional"]
      evaluation_criteria: str | None
      evidence_requirements: list[str]
  class Form(BaseModel):
      form_type: Literal["bid", "rfp", "pqq", "checklist"]
      form_format: Literal["docx", "xlsx", "pdf", "html"]
      qa_pairs: list[QAPair]

  ExtractByLlm(output_type=Form, ...) extracts the whole form in one go.
• KH need: form-question-extraction strategy (OQ-Q11-A, Finding 04) — covered Y. The strategy IS: ExtractByLlm(output_type=Form, ...) on markdown sidecar.
• KH need: extract-tender-questions / analyse-template replacement (feedback-findings-review.md §1 Theme B “Evaluate scripts/extract_tender_questions.py and scripts/analyse_template.py”) — covered Y. These Python scripts collapse into a @coco.fn wrapping ExtractByLlm.

Whether KH planned arch already accounts for this:

• Reference: 0.9-intended-architecture.md §6.2 (Q&A flow — 3 parallel transforms include “(2) q_a_extractions (mempalace miner pattern A/B)”); 0.8.2-cocoindex-evaluation.md §3.3 row 4 (qa-docx — “Custom @coco.fn — port of extract-qa-pairs.ts 497 LOC OR wrap via subprocess”).
• Verdict: PARTIAL — the planning corpus assumes pattern A/B parser + mempalace miner; it does NOT yet route through ExtractByLlm. The intended architecture’s Q&A approach defers to S8 spike + Q3.9 D14 TBD. ExtractByLlm is the missing primitive that the form-extraction pipeline (Theme B step 4) needs.

Gap / opportunity:

• MAJOR ALIGNMENT. ExtractByLlm + typed Python schema directly answers Theme B step 4 + Theme A form-agnostic Q&A + OQ-Q11-A. The intended architecture’s extract-qa-pairs.ts Pattern A/B parser repurpose (§6.3) becomes optional — ExtractByLlm may subsume it entirely if the LLM extraction at gpt-4o-or-better quality outperforms the Pattern A/B regex chain on noisy DRAFT docs.
• BAML + DSPy variants show two alternative paths: BAML for type-safe declarative extraction (lower API surface, more schema-as-code); DSPy for vision-direct extraction (skips markdown stage). For XLSX form-templates where table structure is load-bearing, BAML’s PDF-native support might preserve structure better than markitdown. For PDF forms with photo/scan content, DSPy vision-direct might outperform OCR-via-markdown. Worth a comparison spike in Phase 2.
• patient_form_extraction is the closest precedent for KH’s needs. Phew’s actual form-extraction problem (bid PQQ with 50-500 Q&A pairs, XLSX checklist forms, scanned PDF tender forms) IS the patient-intake-extraction problem at higher volume — and the example shows the entire shape end-to-end in production.

---

Example: paper_metadata

What it demonstrates:

• PDF → docling → markdown → ExtractByLlm(output_type=PaperMetadata) extracting title, authors, abstract → PG target (alongside chunks + embeddings). One row per paper.

Direct mapping to KH need:

• KH need: form metadata sidecar (feedback-findings-review.md §1 Theme B implied — form_templates + form_template_fields with form_type, versions, evaluation criteria, evidence requirements) — covered Y. PaperMetadata becomes FormMetadata with KH-specific fields.
• KH need: per-MIME metadata extraction alongside content extraction — covered Y. The pattern is “extract content + extract metadata in parallel, both write to typed PG tables.”

Whether KH planned arch already accounts for this:

• Reference: 0.9-intended-architecture.md §6.5 (sales proposal templates, polymorphic citations); Finding 03 Q3.11 templates rename + form_templates + form_template_fields.
• Verdict: partial. The metadata table shape is named (Theme A); the extraction primitive is not yet routed through ExtractByLlm.

Gap / opportunity:

• Theme A form_templates + form_template_fields schema needs a per-form-type metadata extraction. paper_metadata is the precedent — one @coco.fn produces the form-level metadata row; another produces the QA-pair rows (per patient_intake_extraction). Both write to KH-owned PG tables in managed_by="user" mode.

---

Example: conversation_to_knowledge

What it demonstrates:

• YouTube audio → AssemblyAI transcribe + diarisation → LLM-extract SessionMetadata (speaker mapping) → second LLM call extracts RawStatement with mentioned_person, mentioned_tech, mentioned_org lists → entity-resolution on each entity-name list across all sessions (FAISS embeddings + LLM-pair confirmation) → SurrealDB graph: nodes session/statement/person/tech/org; edges person_session/session_statement/person_statement/statement_mentions.
• Memoised at every stage (fetch_transcript, extract_metadata, extract_statements, process_session, compute_entity_embedding, resolve_entity_pair all memo=True).
• Bubble-sort entity resolution: incremental, applies across sessions (new session’s entities resolve against the existing FAISS index).

Direct mapping to KH need:

• KH need: Phew named clients/competitors/employees dedup (Theme A overlap, Finding 03/05) — covered Y, strongest precedent in the entire example set. “Statement” maps to “QA pair”; “Person/Tech/Org” maps to “Client/Competitor/Employee” (KH’s named entities).
• KH need: temporal graph (Q4.1 mempalace Shape A — temporal cols) — covered Y by analogue. The session_statement edge carries date; KH’s entity_relationships carries valid_from/valid_to.
• KH need: structured citation lookup (Finding 02 Q1.11 citations rename + extend) — covered Y. statement_mentions polymorphic edge → KH citations polymorphic (citing_entity_kind enum: bid_response / sales_proposal / training_brief / competitor_research per OQ-Q111-A).

Whether KH planned arch already accounts for this:

• Reference: 0.8.2-cocoindex-evaluation.md §5.5 (entity_mentions + entity_relationships); 0.9-intended-architecture.md §4.4 (extended entity_relationships); §7 (mempalace KG model).
• Verdict: aligned, but mempalace-vs-cocoindex graph-substrate question unresolved (Theme F). This example uses SurrealDB as a graph target. KH planning oscillates between (a) PG entity_relationships, (b) mempalace KG, (c) Neo4j/FalkorDB target. Theme F + Q4.5 are the Liam-decision items.

Gap / opportunity:

• The bubble-sort entity-resolution-across-sessions pattern is non-obvious from the cocoindex.ops.entity_resolution docs alone. This example shows how to apply resolve_entities incrementally across runs — each new session’s entities resolve against the global FAISS index built up to that point. KH’s Q&A pair dedup + person/org dedup needs exactly this pattern (incremental, not per-batch). Worth promoting to a named reference pattern in KH’s planning.

---

Capability: cocoindex.ops.entity_resolution

What it provides:

from cocoindex.ops.entity_resolution import resolve_entities
result = await resolve_entities(
    entities=raw_entities,          # list[str] of raw entity names
    embedder=embedder,              # SentenceTransformerEmbedder or similar
    resolve_pair=resolver,          # LlmPairResolver or custom callback
)
result.canonical_of("Microsoft Corp.")  # → "Microsoft"
result.canonicals()                      # → set of canonical names
result.groups()                          # → groupings by canonical
result.to_dict()                         # → full mapping

• Hybrid embedding-similarity-shortlist (FAISS, cosine ≥0.7 default) + LLM-pair-resolver confirmation.
• top_n=5 candidates per entity by default.
• Policies: PINNED (existing canonical wins absolutely) or PREFERRED (existing canonical preferred but LLM can override).
• Custom resolve_pair callback accepts any deterministic resolver (regex normaliser, lookup against KH’s entity_aliases registry, hybrid).

Direct mapping to KH need:

• KH need: Q&A pair dedup (Theme A — duplicate Q&A pairs across q_a_pairs curated and q_a_extractions derived) — covered Y, but with caveat: resolve_entities is for names (short strings). Q&A pair dedup is over (question + answer) tuples — longer, semantic. The shape works but the embedder + threshold need tuning. Alternative: KH’s own match_content_items RPC + a similarity threshold over question_embedding.
• KH need: content dedup (S10 DRAFT-vs-final per S2 §1 row 6) — covered Y at the near-duplicate name/title level, NOT covered for file-level (S2 confirms content_fingerprint is exact-bytes). For file-level near-duplicate, must apply this primitive at the chunk-embedding or document-summary level. S10 spike (0.9-spike-S10-dedup-substrate.md) addresses this.
• KH need: person/org dedup (Phew named clients/competitors/employees — Theme A) — covered Y, direct match. Identical to the meeting_notes_graph + conversation_to_knowledge precedents.
• KH need: bid_question_matches → question_matches discriminator (OQ-Q113-C) — covered partial. The dedup-of-questions across past bids is exactly this primitive’s use case.

Whether KH planned arch already accounts for this:

• Reference: 0.8.2-cocoindex-evaluation.md §5.5 path (1) — “Keep KH’s classify.ts + Pass 2 entities. Wrap as a @coco.fn and emit entity_mentions rows as a target. cocoindex’s role is incremental re-runnability + caching; classification semantics stay KH’s.” Path (2) was “Replace with cocoindex’s entity_resolution module… Probably wrong shape for KH given the bid-management domain specificity.”
• Verdict: partial — the eval chose path (1) but the planning corpus DOESN’T account for using resolve_entities selectively for the cases where it IS the right shape (Q&A pair dedup, person/org dedup). Path (1) keeps KH classify; path (2) replaces KH classify. The correct nuance is path 1 for classification, path 2 for named-entity dedup specifically.

Gap / opportunity:

• The cocoindex eval § 5.5 binary “keep all KH or replace all” framing was overly coarse. Mid-grained adoption (use resolve_entities for the named-dedup cases ONLY, keep KH classification) is the correct shape per the example set. Recommend updating 0.9-intended-architecture.md to reflect this.

---

Capability: cocoindex.functions.ExtractByLlm (Structured Extraction)

What it provides:

doc["extracted"] = doc["markdown"].transform(
    cocoindex.functions.ExtractByLlm(
        llm_spec=cocoindex.LlmSpec(
            api_type=cocoindex.LlmApiType.OPENAI, model="gpt-4o"),
            # api_type can be OLLAMA for on-premise, OPENAI, ANTHROPIC, GEMINI, etc.
        output_type=Patient,           # any Python dataclass / Pydantic model
        instruction="Please extract patient information from the intake form."))

• LLM-backed, schema-driven (output_type is a typed Python class — dataclass, Pydantic, NamedTuple).
• Supports nested schemas with list[T] / T | None / nested dataclass references.
• Multi-provider: OpenAI, Ollama (local), Gemini, Anthropic — via cocoindex’s LlmApiType enum.
• Alternative path via BAML (baml_src/*.baml) or DSPy (dspy.ChainOfThought) for cases where schema-as-code or vision-direct is preferred. Both are demonstrated in the patient-intake example set.
• Memoised: output_type + instruction + LLM model + input markdown all participate in the code-hash. Changing the schema or instruction → all rows re-extract.

Direct mapping to KH need:

KH use case for converting XLSX/PDF forms to structured Q&A data per Theme B:

# Pseudo-code mapping
@dataclass
class QAPair:
    question: str
    answer: str | None              # nullable for blank forms
    question_kind: Literal["mandatory", "optional", "info-only"]
    evaluation_criteria: str | None
    evidence_requirements: list[str]
    scope_tags: list[str]           # bid/RFP/PQQ-specific
    citations_to_extract: list[str] # in-answer citations for chain-of-evidence

@dataclass
class FormMetadata:
    form_type: Literal["bid", "rfp", "pqq", "checklist", "tender"]
    form_format: Literal["docx", "xlsx", "pdf", "html"]
    form_title: str
    deadline: date | None
    issuing_organisation: str | None
    evaluation_methodology: str | None

@dataclass
class Form:
    metadata: FormMetadata
    qa_pairs: list[QAPair]

# In flow
doc["form"] = doc["markdown_sidecar"].transform(
    cocoindex.functions.ExtractByLlm(
        llm_spec=cocoindex.LlmSpec(
            api_type=cocoindex.LlmApiType.ANTHROPIC, model="claude-opus-4-7"),
        output_type=Form,
        instruction=THEME_B_STEP_4_PROMPT,
    ))
# Two parallel target writes
form_templates_table.declare_row(row=FormTemplate(...form.metadata...))
for pair in doc["form"]["qa_pairs"]:
    qa_extractions_table.declare_row(row=QAExtraction(...pair...))

Whether KH planned arch already accounts for this:

• Reference: 0.9-intended-architecture.md §6.2 step (2) “q_a_extractions (mempalace miner pattern A/B)”; feedback-findings-review.md §1 Theme B step 4 (classify-form-data).
• Verdict: ExtractByLlm is NOT named in the planning corpus. The corpus assumes mempalace-miner-pattern + pattern A/B parser. ExtractByLlm is a simpler, more general primitive that subsumes both for the classify-form-data step. The planning corpus pre-dates the granular reading of patient-intake-extraction examples.

Gap / opportunity:

• MAJOR — this is the single largest “missing primitive” in KH’s planning corpus. Recommendation: update 0.9-intended-architecture.md §6 (Q&A flow) to use ExtractByLlm for Theme B step 4 (classify-form-data), with output_type=Form (or per-form-type variants). Pattern A/B parser repurpose (§6.3) becomes the fallback for cases where deterministic regex outperforms LLM (e.g. structured bid library docs with rigid Q1: / A1: shape). Mempalace miner pattern stays for the temporal-KG side.
• Anthropic-direct via cocoindex’s LlmApiType.ANTHROPIC is supported (per the docs evidence). This dissolves the LiteLLM-vs-Anthropic-prompt-caching open question from 0.8.2 §7 #2 for the structured-extraction path specifically — though the prompt-caching question on the existing KH classifyContent chain still stands.
• BAML and DSPy alternatives: BAML is type-safe declarative (lower code surface for schema definitions); DSPy is vision-direct (skips markdown intermediate stage entirely for image-heavy PDFs). For Phew’s XLSX form-templates (where table structure is load-bearing — Cloud Security Principles Checklist V5_3 noted in 0.9-intended-architecture.md §5.2.A.(ii)), a structured-output-with-table-aware-prompt approach may need a custom pre-stage that linearises tables before ExtractByLlm — or a vision-direct DSPy variant.

---

Other examples (one-line each, KH relevance noted)

Example	One-line	KH relevance
code_embedding / code_embedding_lancedb	Python source → tree-sitter chunk → embeddings	None for v1 — KH uses graphify for code understanding
image_search / image_search_colpali	Images → CLIP/ColPali → Qdrant	None for v1 — no image-search product surface
audio_to_text	Audio → LiteLLM Whisper → PG	None for v1 — Phew has no audio corpus
csv_to_kafka	CSV → Kafka topic	None for v1 — no Kafka in stack
kafka_to_lancedb	Kafka → conditional-routing → LanceDB	None for v1 — reference pattern only
entire_session_search	AI session transcripts → embeddings	None for v1 — overlaps mempalace KG; defer
gdrive_text_embedding	Google Drive → embeddings	Deferred — v2 connector (S2 confirms v1 connector list = localfs only)
amazon_s3_embedding	S3 → embeddings	Deferred — no v1 KH S3 source; staging-only
oci_object_storage_embedding	OCI Object Storage + OCI Streams → embeddings	None — no Oracle Cloud in KH stack
multi_codebase_summarization	Walks repos → instructor + Gemini summary per file → markdown	None for v1 — graphify covers code understanding
hn_trending_topics	HN scrape → analysis	None for v1 — reference only
meeting_notes_graph_falkordb	Same as neo4j variant, different graph target	Reference — alternative graph-DB target option
text_embedding_lancedb / text_embedding_qdrant / text_embedding_turbopuffer	text_embedding with different vector targets	Reference — KH commits to pgvector v1

---

§3 — Summary mapping table

Capability	KH use case	Already planned	Gap
localfs.walk_dir(live=True) source	UC10 external-folder edit; markdown sidecar lifecycle (Theme B step 3, Finding 06)	Aligned (§5.2.(i); S2 confirmed)	None
postgres.PgTableSource	Q&A extraction → curated promotion (§6.2); bid feedback loop (Finding 05)	Partial — source-side coupling implicit, not engineered	Recommend explicit PgTableSource(q_a_extractions) flow in §6.2
postgres.mount_table_target(managed_by="user")	All KH-owned PG tables (content_items, chunks, q_a_pairs, entity_relationships)	Aligned (S1 verdict)	None
RecursiveSplitter + LiteLLMEmbedder("openai/text-embedding-3-large")	Replace lib/chunking/ + lib/ai/embed.ts	Aligned (Q4.4)	Verify Anthropic prompt-cache survives via LiteLLM (chat path only; embeddings unaffected)
docling (PDF/DOCX/HTML/XLSX → markdown)	Theme B step 3; binary-shape adapter; Finding 06 PDF flow	Aligned (§5.3 row 2; named in Liam’s notes §4.6)	Side-by-side fidelity comparison vs KH’s existing unpdf + mammoth + turndown chain still deferred (eval §7 #3)
files_transform shape (@coco.fn(memo=True) localfs → localfs)	Markdown sidecar v1 baseline (Finding 02 §3.3); per-MIME generator	Aligned (most-cited pattern across planning corpus)	Per-MIME @coco.fn decomposition not yet drawn
ExtractByLlm with typed Python output_type (Structured Extraction)	Theme B step 4 (classify-form-data); Theme A form-agnostic Q&A; OQ-Q11-A; form-template metadata extraction	MISSING — major gap. Planning assumes pattern A/B parser + mempalace miner	Top recommendation: update §6.2 + §6.3 to use ExtractByLlm(output_type=Form, ...). Pattern A/B becomes fallback
cocoindex.ops.entity_resolution.resolve_entities (FAISS shortlist + LLM-pair-resolver)	Person/org/competitor dedup (Theme A); Q&A pair dedup; question_matches discriminator (OQ-Q113-C)	Partial — eval §5.5 took binary “keep all KH or replace all”; mid-grained adoption not modelled	Recommend selective adoption: resolve_entities for named-entity dedup, keep KH classify for content classification
LLM provider via LlmApiType.ANTHROPIC (native)	Anthropic-direct LLM calls in @coco.fn	Unverified in eval §7 #2 (eval assumed LiteLLM-only)	Resolves the prompt-cache via LiteLLM concern for the structured-extraction path
mount_each + memoisation across runs	Incremental re-classify on prompt change; freshness (Theme D)	Aligned (eval §5.2 — engine-native incremental)	None
Neo4j / FalkorDB / SurrealDB graph targets	Knowledge Map surface (CX.32 / OPS-G-1); entity_relationships materialised graph	Partial — provisional (§5.2.A) gated on S7 client-docs comparison	Decision deferred; cocoindex makes either path (PG entity_relationships OR Neo4j target) low-cost
Bubble-sort incremental entity-resolution across runs	Phew named entity dedup across all bids; clients/competitors corpus	Not modelled	Reference pattern from conversation_to_knowledge — promote to KH planning
live=True source watching + update_blocking(live=True)	Sub-second freshness (Theme D); SaaS dev iteration loop	Partial — S2 + S14 confirm; freshness integration with KH coverage/governance not yet drawn	Theme D — operational sub-agent for coverage + change-reports model integration
Operations DB (LMDB) + lineage	pipeline_runs retain-vs-retire (OQ-Q24-A); audit trail; ai_call_log deferred	Partial — S14 confirms LMDB single-writer manageable; coverage/governance integration unmapped	Theme D operational deep-dive needed
cocoindex.fn(version=N) explicit version bump	Force re-extract on schema/prompt change	Not named in planning corpus	Add to architecture-impl playbook
coco.use_context(CONTEXT_KEY) for shared resources	Anthropic SDK client; Supabase pool	Aligned (skill ref pattern 6)	None
Live mode (LiveMapFeed Kafka source)	None for v1	Out of scope	None
cocoindex init project scaffolding	Phase 2 first-step (S1 §8)	Aligned	None

---

§4 — Top-3 recommendations

Recommendation 1 — Update §6 (Q&A flow architecture) to centre ExtractByLlm

Why: cocoindex.functions.ExtractByLlm + typed Python output_type is the cleanest implementation of Theme B step 4 (classify-form-data) and Theme A (form-agnostic Q&A). It directly subsumes:

• extract-qa-pairs.ts Pattern A/B parser repurpose (§6.3 working recommendation) — becomes optional fallback for cases where deterministic regex outperforms LLM on rigid corpora.
• scripts/extract_tender_questions.py + scripts/analyse_template.py — these collapse into one @coco.fn wrapping ExtractByLlm(output_type=Form).
• The mempalace-miner-pattern-A/B path (§6.2 step 2) for the classify-form-data sub-step (mempalace miner stays for the temporal-KG edge generation, which is a separate concern).

KH-need trace:

• feedback-findings-review.md §1 Theme A (form-type generalisation) — ExtractByLlm with per-form-type output_type IS the form-agnostic shape.
• feedback-findings-review.md §1 Theme B step 4 (classify-form-data) — ExtractByLlm(output_type=Form, instruction=THEME_B_PROMPT) IS the implementation.
• feedback-findings-review.md §2.4 OQ-Q11-A (form-question-extraction strategy) — same primitive.
• 0.9-intended-architecture.md §6.2 step (2) (“q_a_extractions”) and §6.3 (Pattern A/B parser repurpose) — both need updating to centre ExtractByLlm.

Concrete actions:

1. Add Form / FormMetadata / QAPair Pydantic schema sketches to 0.9-intended-architecture.md §6.2 (alongside the existing flow diagram).
2. Update §6.3 working recommendation: extract-qa-pairs.ts retires for the Theme B step 4 path; relocates to optional fallback for rigid-corpus extraction (low-LLM-cost path).
3. Add the paper_metadata precedent for the form_templates + form_template_fields schema (Theme A + Finding 03 Q3.11).

Recommendation 2 — Adopt cocoindex.ops.entity_resolution selectively for named-entity dedup; keep KH classify

Why: The 0.8.2-cocoindex-evaluation.md §5.5 framing was binary (“keep all KH OR replace all KH”). The example set shows the correct nuance is mid-grained:

• Keep KH classify (Pass 1 keyword-domain + Pass 2 entity-type via Anthropic) for content classification — domain logic, prompt-tuned, leverages KH’s entity_aliases registry.
• Adopt resolve_entities for named-entity dedup specifically — Phew’s clients/competitors/employees (Theme A), Q&A pair dedup over (question + answer) tuples, bid_question_matches → question_matches discriminator (OQ-Q113-C). These are name-string-dedup problems, which resolve_entities is purpose-built for.
• The conversation_to_knowledge bubble-sort incremental dedup pattern is the direct precedent for Phew’s incremental client/competitor dedup across all bids over time. Promote this to a named reference pattern in the planning corpus.

KH-need trace:

• feedback-findings-review.md §1 Theme A (form-type generalisation, named entities).
• feedback-findings-review.md §2.3 OQ-Q113-C (bid_question_matches → question_matches discriminator) — DECIDED in principle; implementation now points at resolve_entities.
• feedback-findings-review.md §2.4 Finding 04 OQ-Q35-C (form requirement_type=‘checklist’ enum) — same dedup primitive applies to enum-value normalisation.

Concrete actions:

1. Update 0.8.2-cocoindex-evaluation.md §5.5 to reflect mid-grained adoption (Recommendation 2 from path 1 stands; recommend ADD path 2 selectively for named-entity dedup only).
2. Add named-entity-resolution flow to 0.9-intended-architecture.md §7 (mempalace KG/memory model) — resolve_entities runs as a stage in the cocoindex flow that materialises entity_relationships, with the canonical-name dict fed into the KG node ID generation.
3. Decide PINNED vs PREFERRED policy for KH’s entity_aliases registry as the existing-canonical source.

Recommendation 3 — Promote pdf_to_markdown + patient_form_extraction as the canonical example references for Theme B; commission a Phase-2-first-step verification of docling fidelity

Why: Liam already named pdf_to_markdown as the PDF flow under markdown sidecar v1 (feedback-findings-review.md §4.6). patient_form_extraction is the closest end-to-end precedent for Phew’s actual form-extraction problem at production scale. Two outstanding fidelity questions remain (0.8.2-cocoindex-evaluation.md §7 #3):

• docling vs unpdf + mammoth + turndown on KH’s representative corpus.
• docling DOCX with track-changes vs python-docx’s open_document_safe() (CLAUDE.md gotcha).

These were deferred at eval time and are now blocking Theme B step 3 commitment.

KH-need trace:

• feedback-findings-review.md §4.6 (Liam verbatim — pdf_to_markdown named for PDF flow rewrite).
• feedback-findings-review.md §1 Theme B step 3 (Markdown conversion via reusable cocoindex pipeline converter).
• 0.8.2-cocoindex-evaluation.md §7 #3 (docling fidelity comparison — deferred).
• CLAUDE.md (docling DOCX track-changes vs open_document_safe() gotcha not yet verified).

Concrete actions:

1. Add pdf_to_markdown and patient_form_extraction to 0.9-intended-architecture.md §5.4 (folder adapter abstraction) as the canonical references.
2. Phase 2 first-step (per S1 §8 checklist): run docling against the canonical baseline corpus (docs/client-documentation-base/binary/ 8 files + forms/ 2 files) and compare markdown output to the existing KH extractor chain. Document deltas. ~2 days.
3. Pre-commitment to Theme B step 3 (Markdown conversion approach) depends on this comparison outcome.

---

§5 — Open questions surfaced

ID	Question	Source	Lean
Q-EX1	Does ExtractByLlm with LlmApiType.ANTHROPIC pass through cache_control headers transparently?	Dissolves eval §7 #2 for structured extraction path	Likely yes; verify in Phase 2
Q-EX2	Can ExtractByLlm output_type be a Pydantic v2 model with discriminated unions (e.g. `Form = BidForm	RFPForm	PQQForm`)?
Q-EX3	Does ExtractByLlm cache hit on schema-equivalent-but-rewrite changes (e.g. field rename, comment change)?	Code-hash semantics — eval §7 #1	Likely re-extracts on any output_type bytecode change; deterministic
Q-ER1	For resolve_entities, what’s the cost of LLM-pair-resolver calls at Phew’s volume (~50-200 named entities across all bids)?	Cost projection for Theme A	Bubble-sort means ~O(n²) pairs worst-case; pre-filter via FAISS keeps it tractable
Q-ER2	How does resolve_entities interact with KH’s existing entity_aliases registry (PINNED vs PREFERRED)?	Phew has curated canonical names already	Use PINNED for registry entries; PREFERRED for new ones
Q-FT1	docling DOCX handling of python-docx track-changes pattern (CLAUDE.md gotcha) — does docling honour open_document_safe()-equivalent semantics?	Theme B step 3 fidelity	Likely needs an @coco.fn pre-stage that strips track-changes before docling sees it
Q-XL1	XLSX form-template conversion: docling reads XLSX but may flatten tables; for forms-as-tables (Cloud Security Principles Checklist), structure-preservation is load-bearing	Theme B + Finding 06 OQ-CX33-D (interim CTA on XLSX)	May need custom XLSX → structured-markdown converter inside @coco.fn; BAML PDF-native might serve PDF-form variants better
Q-GS1	Knowledge Map substrate decision (Neo4j/FalkorDB target via cocoindex vs PG entity_relationships + cocoindex flow vs mempalace KG)	0.9-intended-architecture.md §5.2.A provisional; Theme F	Both PG and Neo4j paths low-cost via cocoindex; defer until S7 client-docs comparison + OPS-G-1 productivity decision
Q-PS1	PgTableSource row-level row_type schema: does the engine require exact match to PG table, or does it accept a subset of columns?	Q-EX3-like — what columns participate in cache fingerprint	Likely subset acceptable for read; verify
Q-FR1	live=True watchfiles vs Cloud Run topology (S14 found single-instance recommended) — does live mode trigger orchestrator-instance reconciliation cleanly?	Theme D freshness operational integration	Likely yes per S14 single-orchestrator pattern; verify with Cloud Run job design

---

§6 — Sources

Cocoindex (primary):

• Cocoindex examples directory listing (27 examples)
• postgres_source example
• text_embedding example
• files_transform example
• meeting_notes_graph_neo4j example
• meeting_notes_graph_falkordb example
• pdf_embedding example
• pdf_to_markdown example (docs)
• patient_form_extraction (markitdown + ExtractByLlm)
• patient_intake_extraction_baml
• patient_intake_extraction_dspy
• paper_metadata example
• conversation_to_knowledge design
• audio_to_text example
• csv_to_kafka + kafka_to_lancedb examples
• code_embedding + code_embedding_lancedb examples
• image_search + image_search_colpali examples
• amazon_s3_embedding example
• oci_object_storage_embedding example
• multi_codebase_summarization example
• entire_session_search example
• Entity Resolution operation docs
• Structured Extraction blog (Ollama PDF)
• Patient Intake Form Extraction blog
• Manual Extraction example
• CocoIndex incremental processing blog
• CocoIndex homepage (freshness claim)

KH planning corpus cross-referenced:

• CLAUDE.md (KH conventions, gotchas, schema constants)
• .claude/skills/cocoindex/SKILL.md + references/connectors.md + references/patterns.md + references/api_reference.md
• docs/plans/phase-0-investigation/0.8.2-cocoindex-evaluation.md (eval baseline)
• docs/plans/phase-0-investigation/0.9-intended-architecture.md (cocoindex search hits — §§ 4.4, 5.2.(i), 5.2.A, 6.2, 6.3, 6.4, 6.5, 7.2; lines 48, 55, 63, 76, 115-118, 168-169, 184, 190, 226-227, 237, 265, 292, 314, 343, 351, 374, 439-440, 462, 464, 542, 612, 614, 619, 626, 638, 708, 723, 765, 819, 821, 835, 839, 845, 852, 880, 893, 905, 909-911, 966, 1014)
• docs/plans/phase-0-investigation/0.9-spike-S1-cocoindex-schema-coupling.md (Scenario A confirmed; managed_by="user")
• docs/plans/phase-0-investigation/0.9-spike-S2-cocoindex-folder-binding.md (localfs source binding; UC4.4 change classification; v1 connector list = localfs only)
• docs/plans/phase-0-investigation/0.9-spike-S14-cocoindex-concurrency.md (LMDB single-writer manageable; v1 = single-orchestrator-instance)
• docs/plans/phase-0-investigation/feedback-findings-review.md (§1 Themes A/B/C/D/E/F/G; §3.2 cocoindex input matrix; §4 Liam verbatim)
• docs/plans/phase-0-investigation/10-feedback-investigation-findings/00-synthesis.md (S233 directional outcomes)

Files NOT investigated hands-on:

• Cocoindex Rust engine source (only Python interface + docs read)
• Live ExtractByLlm + resolve_entities runs against KH staging (deferred to Phase 2 first-step)
• BAML / DSPy variant runs (reference only)
• docling DOCX-track-changes vs open_document_safe() empirical comparison (eval §7 #3 deferred)

---

End of cocoindex examples deep-dive.