TECH — Cocoindex ledger API

TECH — Cocoindex ledger API

Date: 18/05/2026 (S242) Status: [DEFERRED-v1.1] per RATIFIED-S243 — v1 ships only pipeline_runs rollup. This draft retained as v1.1 substrate. Spec type: TECH only (no PRODUCT.md — TS-facing API surface, not user-facing behaviour, per S241 ratification + canonical-pipeline-implementation-plan PLAN.md §5 row 2) Drafting task: T1.3 per docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md §4.1 Source ratifications:

• S241 Liam ratification per docs/specs/core-docs-pathway-assessment/S239-still-open-consolidation.md S241 addendum row 8 (CATEGORISED-S241 — tech spec needed).
• docs/plans/phase-0-investigation/architecture/02-data-flow.md §7.2 N6 RESOLVED (pipeline_runs retained as KH-side rollup; per-stage metrics scope STILL-OPEN — this spec covers that gap).
• docs/plans/phase-0-investigation/phase-b-prerequisite-2-cocoindex-deep-dive.md §3.4 + §3.5 (per-flow-run ledger gap + LMDB-per-container limitation).
• docs/specs/id-31-0.9-canonical-pipeline/TECH.md §11 bullet “TS-facing API for cocoindex per-flow-run ledger” (historical-substrate framing per S241 Liam direction; the gap closure happens in this spec, not by editing canonical-pipeline TECH inline).

---

§1 — Context

§1.1 What this spec covers

A TypeScript API surface for KH application code to query per-stage metrics about a single cocoindex pipeline run — beyond what the KH-side pipeline_runs rollup already exposes. The API allows the UI and MCP layers to answer questions like “which of the six cocoindex stages succeeded for this source document?” and “did the LLM extraction stage produce a typed output, or did the run finish in a partial-completed state before reaching it?” without forcing callers to query cocoindex’s internal LMDB ledger directly.

Per docs/plans/phase-0-investigation/architecture/02-data-flow.md §7.2 N6 RESOLVED, the cocoindex per-flow-run ledger is not exposed at the TS-facing API level today. KH’s pipeline_runs table (supabase/types/database.types.ts:1768-1842) is a rollup row that records pipeline-level status (completed / completed_with_errors / failed per lib/pipeline/record-run.ts:38-41) but does not capture per-stage metrics. This spec closes that gap by introducing a thin, read-only API over the cocoindex flow-run ledger, while keeping pipeline_runs as the durable rollup.

§1.2 Boundary statement — pipeline_runs vs cocoindex internal LMDB ledger

Two ledgers exist in the v1 architecture; this spec is the public TS surface over the second one, layered above the first.

Ledger	Owner	Persistence	Lifetime	Surfaced via
pipeline_runs (KH-side rollup)	KH application	Postgres table public.pipeline_runs (supabase/types/database.types.ts:1768)	Durable; retained per KH data-retention policy.	recordPipelineRun() from @/lib/pipeline/record-run (cron) + startPipelineRun() from @/lib/pipeline/start-run (request-response Pattern E). One row per cocoindex flow run at terminal status, plus mid-flight progress updates via lib/pipeline/update-progress.ts.
Cocoindex internal ledger (in-process LMDB)	Cocoindex Rust engine	LMDB-per-container, ephemeral per Cloud Run sidecar instance per phase-b-prerequisite-2-cocoindex-deep-dive.md §3.5.	In-process for the sidecar; cleared on restart unless explicitly persisted.	This spec’s API (read-only). Never exposed raw — see §1.4 below.

Boundary contract: pipeline_runs is the durable, queryable rollup that downstream UI and MCP code already depends on. This spec’s API is the per-stage detail lens that sits above the cocoindex ledger for the lifetime of a flow run. Where pipeline_runs carries terminal status (completed / completed_with_errors / failed) and aggregate counts, the ledger API carries per-stage detail (which of the six cocoindex stages per 02-data-flow.md §3.1 reached terminal state, with which op_id, and at what timestamp).

§1.3 What “partial resolution” means here

Per 02-data-flow.md §7.2 + docs/plans/phase-0-investigation/10-feedback-investigation-findings/00-synthesis-v2.md §5.2 row 5 (TS-facing-ledger STILL-OPEN entry — this spec closes it), a cocoindex flow run can finish in three terminal states:

1. Completed — all six stages (source walk → binary conversion → LLM extraction → embedding → entity resolution → Postgres UPSERT per 02-data-flow.md §3.1) executed to terminal success for the input. pipeline_runs.status = 'completed'.
2. Partial-completed — one or more downstream stages skipped or short-circuited because an upstream stage produced no output or a memoised result, but no stage errored. Example: binary conversion produced an empty content_text for a malformed PDF, so LLM extraction was skipped — the run is not a failed, but the Postgres UPSERT row has empty extraction fields. pipeline_runs.status is currently 'completed_with_errors' per lib/pipeline/record-run.ts:34-37 semantics, but the rollup cannot say which stage short-circuited. This spec exposes that detail.
3. Failed — a stage threw or the cocoindex engine rejected the run. pipeline_runs.status = 'failed'. Cocoindex’s retry / back-off / DLQ subsumes the failure-handling layer per 02-data-flow.md §7.3 (COCO.7 DO-NOT-BUILD); this spec only reports the per-stage outcome, it does not orchestrate retries.

The partial-completed case is the central motivator for this spec — without it, the UI has no way to surface “the file was ingested but extraction did not run” to the user other than by inspecting downstream content_items rows directly.

§1.4 Cocoindex internal LMDB exposure boundary (load-bearing)

This API never exposes the raw LMDB ledger to TypeScript callers. Two reasons, both rooted in phase-b-prerequisite-2-cocoindex-deep-dive.md:

1. §3.5 — LMDB-per-container. Multi-instance Cloud Run deployments (v1.1+) would yield inconsistent reads if KH code queried LMDB directly across sidecar instances. The API layer normalises to a single logical view per flow_run_id.
2. §3.4 — the deep-dive’s prescription (paraphrased): the per-flow-run ledger’s query semantics and retention are not documented at the TS-facing API level, and pipeline_runs is the KH-facing surface that insulates KH code from cocoindex internals (the deep-dive recommends documenting the cocoindex→pipeline_runs mapping rather than exposing the ledger directly). This spec extends that prescription by adding a per-stage detail lens that owns its own KH-side type surface, so cocoindex version upgrades update only the sidecar-client implementation and do not cascade into KH application code.

The API exposes a typed, stable, KH-owned view over the ledger. Raw LMDB inspection remains a sidecar-side debugging surface only (CocoInsight per 02-data-flow.md §5.4) and is never proxied through this API.

§1.5 Relevant code

• lib/pipeline/record-run.ts:128-212 — KH-side recordPipelineRun() (cron terminal-status writer; this API is the layer above this).
• lib/pipeline/start-run.ts:100-179 — startPipelineRun() Pattern E at-start UPSERT helper (returns pipeline_runs.id; this API consumes that id where available). JSDoc at lines 78-99.
• lib/pipeline/update-progress.ts — mid-flight progress UPDATE helper.
• supabase/types/database.types.ts:1768-1842 — pipeline_runs schema reference.
• scripts/ontology-sync/parse-flow.py:1-142 — current cocoindex flow stub (S241 state — full flow lands at T8 per PLAN.md §4.8).
• lib/supabase/safe.ts:104-194 — canonical Result<T, E> envelope + sb() / tryQuery() / isOk() helpers (this spec’s Result<T, LedgerApiError> re-uses the same discriminant shape — see §2.3).

---

§2 — Proposed changes

§2.1 Module landing

A new module lib/pipeline/ledger-api.ts houses the TS API surface. Companion types live in types/pipeline-ledger.ts so MCP tooling and UI code can import the typed return shape without pulling in the implementation. Test fixtures live at __tests__/lib/pipeline/ledger-api.test.ts.

Path	Purpose	Pattern
lib/pipeline/ledger-api.ts (NEW)	Implementation — the four exported functions per §2.2.	Service-module pattern (no React deps). Wraps cocoindex sidecar HTTP/gRPC client + falls back to pipeline_runs rollup.
types/pipeline-ledger.ts (NEW)	TS types: FlowRunMetrics, StageMetric, FlowRunStatus, LedgerApiError, LedgerResult<T>.	Pure type exports; no runtime.
lib/pipeline/ledger-client.ts (NEW, INTERNAL)	Cocoindex sidecar client. Not exported from any barrel — direct file import only per CLAUDE.md “No barrel re-exports” gotcha.	Stateless module; reads sidecar URL from process.env.COCOINDEX_SIDECAR_URL.
__tests__/lib/pipeline/ledger-api.test.ts (NEW)	Vitest tests per §3.	Vitest with vi.mock('@/lib/pipeline/ledger-client', ...); integration variant under __tests__/integration/.

The module does not export from any index file (per CLAUDE.md “No barrel re-exports”). Callers import directly: import { getFlowRunMetrics } from '@/lib/pipeline/ledger-api'.

§2.2 TS API surface — four functions

All functions are read-only; this API never writes to cocoindex’s ledger and never writes to pipeline_runs (use recordPipelineRun() / startPipelineRun() for that — see §2.5). All four functions take an optional options.workspace_id for service-role callers per §2.7.

§2.2.1 getFlowRunMetrics

export async function getFlowRunMetrics(
  flow_run_id: string,
  options?: { workspace_id?: string },
): Promise<LedgerResult<FlowRunMetrics>>;

Primary entry point. Returns per-stage metrics for a single cocoindex flow run keyed by flow_run_id (cocoindex’s stable op_id per 02-data-flow.md §3.2 + §5.1).

Lands at: lib/pipeline/ledger-api.ts:export async function getFlowRunMetrics.

Return shape (types/pipeline-ledger.ts):

export interface FlowRunMetrics {
  flow_run_id: string;              // cocoindex op_id
  pipeline_run_id: string | null;   // KH pipeline_runs.id when correlated; null for orphan runs
  status: FlowRunStatus;            // 'completed' | 'partial_completed' | 'failed' | 'in_progress'
  started_at: string;               // ISO-8601 UTC
  completed_at: string | null;      // ISO-8601 UTC; null for in-progress
  stages: StageMetric[];            // ordered per cocoindex 6-stage topology
  source_key: string | null;        // <connector>://<path-or-url> per 02-data-flow.md §2.2; null for non-source-bound runs
}

export type FlowRunStatus =
  | 'completed'
  | 'partial_completed'
  | 'failed'
  | 'in_progress';

export interface StageMetric {
  stage_name: CocoindexStageName;   // 6-value union per §2.4
  status: StageStatus;              // per-stage terminal state
  started_at: string;
  completed_at: string | null;
  duration_ms: number | null;
  short_circuit_reason: ShortCircuitReason | null;
  error_message: string | null;
}

export type CocoindexStageName =
  | 'source_walk'
  | 'binary_conversion'
  | 'llm_extraction'
  | 'embedding'
  | 'entity_resolution'
  | 'postgres_upsert';

export type StageStatus =
  | 'completed'
  | 'short_circuited'   // memo hit per @coco.fn(memo=True)
  | 'skipped'           // upstream produced no work
  | 'failed'
  | 'in_progress';

export type ShortCircuitReason =
  | 'memo_hit'          // content-hash matched stored hash
  | 'no_upstream_output'// e.g. binary conversion produced empty content_text
  | 'edit_intent_cosmetic'; // per 02-data-flow.md §8.2 cosmetic edits skip re-extraction

§2.2.2 listFlowRunsForSource

export async function listFlowRunsForSource(
  source_key: string,
  limit?: number,
  options?: { workspace_id?: string },
): Promise<LedgerResult<FlowRunMetrics[]>>;

Lists recent flow runs for a single source key (e.g. all runs that processed localfs:///bid-library/foo.docx). Defaults to limit = 20; pipeline_runs.workspace_id filtering applies per RLS so callers only see runs scoped to their workspace.

Lands at: lib/pipeline/ledger-api.ts:export async function listFlowRunsForSource.

§2.2.3 getPartialResolveDetail

export async function getPartialResolveDetail(
  flow_run_id: string,
  options?: { workspace_id?: string },
): Promise<LedgerResult<PartialResolveDetail | null>>;

When a flow run’s status === 'partial_completed', returns the structured reason on the success branch. When the status is completed / failed / in_progress, the success branch carries null (no partial-resolve detail applies). The discriminant pattern is therefore:

• { ok: true, data: PartialResolveDetail } — partial-completed run, structured detail present.
• { ok: true, data: null } — non-partial status; this is success, not an error.
• { ok: false, error: LedgerApiError } — sidecar / RLS / lookup failure.

Return shape:

export interface PartialResolveDetail {
  flow_run_id: string;
  terminating_stage: CocoindexStageName; // last stage that ran to terminal state
  skipped_stages: CocoindexStageName[];  // stages downstream of terminating stage that were skipped
  reason: ShortCircuitReason;
  reason_detail: string | null;          // human-readable; for UI display
}

Lands at: lib/pipeline/ledger-api.ts:export async function getPartialResolveDetail.

§2.2.4 correlateToPipelineRun

export async function correlateToPipelineRun(
  flow_run_id: string,
  options?: { workspace_id?: string },
): Promise<LedgerResult<string | null>>;

Resolves a cocoindex flow_run_id to the corresponding pipeline_runs.id (KH-side rollup) when correlation is recorded. The success branch carries null (not an error) when no rollup row exists for the flow run — this is a valid state for in-flight runs that have not yet emitted a rollup, and for any cocoindex-internal flow run that never produced a KH-side row.

Discriminant pattern (matches §2.2.3):

• { ok: true, data: string } — pipeline_runs.id correlated.
• { ok: true, data: null } — no rollup row exists; this is success, not an error.
• { ok: false, error: LedgerApiError } — Supabase / RLS failure.

Lands at: lib/pipeline/ledger-api.ts:export async function correlateToPipelineRun.

§2.3 Result envelope and error type

All four functions return LedgerResult<T>, which is the canonical KH Result<T, E> shape — the same discriminated union exported by lib/supabase/safe.ts:108-110:

types/pipeline-ledger.ts

import type { Result } from '@/lib/supabase/safe';

export type LedgerResult<T> = Result<T, LedgerApiError>;

Expanded, this resolves to:

type LedgerResult<T> =
  | { ok: true; data: T }
  | { ok: false; error: LedgerApiError };

The discriminant is ok (boolean), not error — matching lib/supabase/safe.ts:108-110 exactly. The isOk() type guard from @/lib/supabase/safe (defined at lib/supabase/safe.ts:190-194) narrows LedgerResult<T> correctly. This is load-bearing: the spec deliberately re-uses the canonical envelope so callers can interleave ledger-API results with any other tryQuery()-style result without translation.

LedgerApiError mirrors the KH convention for typed errors:

export interface LedgerApiError {
  code: LedgerApiErrorCode;
  message: string;
  cause?: unknown;            // original error if applicable
}

export type LedgerApiErrorCode =
  | 'sidecar_unreachable'     // cocoindex sidecar HTTP/gRPC failed
  | 'flow_run_not_found'      // ledger has no record of this flow_run_id
  | 'rls_denied'              // workspace scoping rejected the read
  | 'malformed_ledger_entry'  // ledger row failed schema validation
  | 'sidecar_timeout';

Caller pattern — branch on ok, never on error:

const result = await getFlowRunMetrics(flowRunId);
if (!result.ok) {
  // result.error is LedgerApiError here (narrowed by ok=false)
  // log, fall back to pipeline_runs rollup, etc.
  return null;
}
// result.data is FlowRunMetrics here (narrowed by ok=true)
return renderFlowRun(result.data);

The isOk() helper from @/lib/supabase/safe is the canonical narrowing aid; use it in conditionals where a positive-branch read is more ergonomic:

import { isOk } from '@/lib/supabase/safe';

if (isOk(result)) {
  // result.data is FlowRunMetrics
} else {
  // result.error is LedgerApiError
}

Functions never throw. The Pattern E precedent (lib/pipeline/start-run.ts:100-179) throws on fail-fast at-start writes; this API does not — it is a read API used by UI and MCP layers where a thrown exception would degrade UX. Following the tryQuery() precedent (lib/supabase/safe.ts:160-181) is the correct shape: every failure surfaces through { ok: false, error }.

§2.4 Cocoindex stage name mapping

The six stages map 1:1 to the topology in 02-data-flow.md §3.1:

KH CocoindexStageName	Cocoindex primitive	KH 02-data-flow.md §3.1 stage
source_walk	localfs.walk_dir(live=True)	Source walk
binary_conversion	files_transform + per-MIME @coco.fn	Binary conversion
llm_extraction	ExtractByLlm with typed output_type	LLM extraction
embedding	LiteLLMEmbedder("openai/text-embedding-3-large")	Embedding
entity_resolution	entity_resolution	Entity resolution
postgres_upsert	postgres.mount_table_target(managed_by="user")	Postgres UPSERT

The implementation maps cocoindex’s internal stage identifiers to this union at the sidecar-client boundary so the public API stays stable across cocoindex versions (per §1.4 — the KH-owned view that insulates application code from cocoindex internals).

§2.5 Relationship to pipeline_runs writers

This spec does not modify recordPipelineRun() or startPipelineRun(). Their write contracts stay exactly as documented. Where this API is useful to those writers is at the moment they decide which pipeline_runs.status to set: a cocoindex run that finished in partial_completed per §2.2.1 maps to completed_with_errors in pipeline_runs.status (per lib/pipeline/record-run.ts:34-37), with error_message populated from PartialResolveDetail.reason_detail. This mapping is not in this spec’s scope — it lives in the cocoindex flow scaffolding (T8 per PLAN.md §4.8). The spec is read-only.

§2.6 Sidecar client integration

lib/pipeline/ledger-client.ts (internal) is the sole component that speaks to the cocoindex sidecar. Two transport options:

1. HTTP REST — cocoindex sidecar exposes a small HTTP API for ledger queries. Preferred at v1 because it matches the Cloud Run deployment shape (sidecar already serves HTTP per 02-data-flow.md §4.1). Configured via process.env.COCOINDEX_SIDECAR_URL.
2. gRPC — cocoindex’s native protocol. DEFERRED-v1.1 unless HTTP throughput proves insufficient.

The client uses fetch (Next.js / Node native) with a 5s default timeout. Timeouts return LedgerApiError.code = 'sidecar_timeout'. Network errors map to sidecar_unreachable.

Fallback contract: when the sidecar is unreachable (e.g. degraded Cloud Run instance), the API does not silently fall back to pipeline_runs. It returns { ok: false, error: { code: 'sidecar_unreachable', ... } } so callers can decide whether to fall back. This avoids silent-failure regressions per CLAUDE.md “Silent failures in Supabase calls” gotcha (the same discipline applies here: fail-loud, let the caller route).

§2.7 Authorisation + RLS

Per 02-data-flow.md §6 + docs/specs/rls-pattern/{PRODUCT,TECH}.md, the API enforces workspace scoping at two layers:

1. pipeline_runs reads (used by correlateToPipelineRun and as fallback in listFlowRunsForSource) go through getAuthorisedClient() per CLAUDE.md “discriminated union” gotcha; RLS predicates on pipeline_runs.workspace_id filter the result set automatically. Failures route via authFailureResponse(auth) when called from an API route.
2. Cocoindex sidecar reads are workspace-scoped by passing the caller’s workspace_id as a query parameter; the sidecar applies the equivalent filter before returning ledger entries. Missing or mismatched workspace returns rls_denied.

The options.workspace_id parameter in §2.2 signatures is mandatory for callers that do not have an authorised Supabase client in scope (service-role contexts: MCP tools called from cron, internal audit jobs, integration tests). The implementation contract:

• Authorised-client callers (route handlers): omit options.workspace_id. The API extracts the workspace from getAuthorisedClient().
• Service-role callers: pass options.workspace_id explicitly. The API does not infer workspace from a service-role client (which has no per-user context) and returns rls_denied if both are absent.
• Mismatch: if a caller passes options.workspace_id that disagrees with the authorised user’s workspace, the API returns rls_denied rather than silently overriding (the spec’s “fail-loud” discipline per §2.6).

This is encoded in the §2.2 signatures (every function accepts options?: { workspace_id?: string }); the absence-AND-no-auth case is asserted in tests (§3.1 row correlateToPipelineRun row d).

§2.8 No new migrations

This spec introduces no schema changes. pipeline_runs schema is unchanged. The cocoindex sidecar already manages its own LMDB. No KH-owned table is added, removed, or altered.

§2.9 No new MCP tools at v1

The API is internal-TS only at v1. An MCP wrapper (e.g. mcp__kh__get_flow_run_metrics) is a v1.1 candidate once the UI surfaces stabilise — DEFERRED-v1.1, tracked in 08-new-features.md-style backlog rather than this spec.

---

§3 — Testing and validation

Acceptance tests align to the four functions in §2.2. Each function has a unit test (mocked sidecar) and at least one integration smoke test under __tests__/integration/.

§3.1 Unit tests — __tests__/lib/pipeline/ledger-api.test.ts

Vitest with vi.mock('@/lib/pipeline/ledger-client', ...) to stub the sidecar transport. Use vi.hoisted() per CLAUDE.md “vi.mock() hoisting” gotcha. Coverage targets per function:

Function	Test cases (minimum)
getFlowRunMetrics	(a) completed flow returns { ok: true, data } with all six stages status: 'completed'; (b) partial-completed flow returns { ok: true, data } with terminating stage + skipped downstream + correct short_circuit_reason; (c) failed flow returns { ok: true, data } with failed stage + error_message; (d) in-progress flow returns { ok: true, data } with status: 'in_progress' + null completed_at; (e) unknown flow_run_id returns { ok: false, error: { code: 'flow_run_not_found' } }; (f) sidecar timeout returns { ok: false, error: { code: 'sidecar_timeout' } }.
listFlowRunsForSource	(a) returns runs in started_at DESC order; (b) respects limit; (c) empty array returned as { ok: true, data: [] } when no runs; (d) workspace scoping rejects cross-workspace reads with { ok: false, error: { code: 'rls_denied' } }.
getPartialResolveDetail	(a) partial_completed status returns { ok: true, data: PartialResolveDetail } with terminating_stage matching ledger; (b) completed status returns { ok: true, data: null } (not an error); (c) failed status returns { ok: true, data: null }; (d) unknown flow_run_id returns { ok: false, error: { code: 'flow_run_not_found' } }.
correlateToPipelineRun	(a) correlated run returns { ok: true, data: pipeline_runs.id }; (b) orphan run returns { ok: true, data: null } (not an error); (c) Supabase read failure routes through tryQuery() and surfaces as { ok: false, error: LedgerApiError }; (d) service-role call with neither authorised client nor options.workspace_id returns { ok: false, error: { code: 'rls_denied' } } (per §2.7).

§3.2 Integration test — __tests__/integration/ledger-api.integration.test.ts

Real cocoindex sidecar required. Marks the file as *.integration.test.ts so it runs under bun run test:integration (real Anthropic + Supabase + cocoindex). One end-to-end happy-path:

• Trigger a small cocoindex flow against a fixture file (single PDF, ≤100KB).
• Poll getFlowRunMetrics(flow_run_id) until result.ok && result.data.status !== 'in_progress'.
• Assert result.data.status === 'completed', all six result.data.stages[].status === 'completed', correlateToPipelineRun(flow_run_id) returns { ok: true, data: <non-null pipeline_runs.id> }.

Skipped on CI by default; runs on demand per docs/runbooks/ci.md integration policy.

§3.3 Partial-resolve regression — fixture-driven

A synthetic partial-completed scenario lives in __tests__/lib/pipeline/fixtures/partial-resolve-malformed-pdf.json (mocked sidecar response). Test asserts (with result from getFlowRunMetrics):

• result.ok === true
• result.data.status === 'partial_completed'
• result.data.stages contains six entries
• result.data.stages.find(s => s.stage_name === 'binary_conversion').status === 'short_circuited' with short_circuit_reason === 'no_upstream_output'
• All stages downstream of binary_conversion have status === 'skipped'
• getPartialResolveDetail returns { ok: true, data } with data.terminating_stage === 'binary_conversion', data.skipped_stages === ['llm_extraction', 'embedding', 'entity_resolution', 'postgres_upsert'].

This regression locks the partial-resolve contract from 02-data-flow.md §7.2 into the test suite.

§3.4 Sidecar mock harness

Sidecar HTTP responses are stubbed via direct vi.mock('@/lib/pipeline/ledger-client', ...) at the module boundary — the same pattern named in §3.1. No new test-time dependency is introduced. Three canonical fixtures shipped with the spec:

• completed-flow.json — six stages, all completed
• partial-completed-flow.json — binary_conversion short-circuit, downstream skipped
• failed-flow.json — llm_extraction failed, embedding + entity_resolution + postgres_upsert never ran

Fixture path: __tests__/lib/pipeline/fixtures/ledger-api/.

If at T1.3 build time the team decides an HTTP-level mock harness is warranted (rather than module-level mocking), msw is the candidate library — see [GAP-LEDGER-004] for the dependency-add gate.

§3.5 Type-level validation

A type-only test (__tests__/lib/pipeline/ledger-api.types.test-d.ts) asserts the public types compile against expected usage patterns — for example, that LedgerResult<FlowRunMetrics> is correctly discriminated by the ok property, that isOk(result) narrows result.data to FlowRunMetrics, and that the negative branch narrows result.error to LedgerApiError. Uses expectType<T>() from tsd or equivalent.

§3.6 No E2E coverage at v1

No Playwright tests at v1. The API is consumed by internal TS callers (MCP tools and admin UI surfaces that have not yet been built). E2E coverage lands with the consuming UI in v1.1, not in this spec.

---

§4 — Risks and mitigations

§4.1 Cocoindex schema-volatility risk

Cocoindex’s internal ledger schema is not part of its public TS API. A cocoindex version upgrade may change the on-disk LMDB shape.

Mitigation: the sidecar-client boundary at lib/pipeline/ledger-client.ts is the only code that knows the cocoindex schema. The public types in types/pipeline-ledger.ts are KH-owned and stable. Cocoindex upgrades update ledger-client.ts only; the public API and consumer code stay unchanged. Track cocoindex upgrades in docs/runbooks/ (separate runbook out of scope for this spec).

§4.2 LMDB-per-container in multi-instance deployments

Per phase-b-prerequisite-2-cocoindex-deep-dive.md §3.5, cocoindex at v1.0.3 keeps the ledger in LMDB per container. Multi-instance Cloud Run deployments cannot give a consolidated view from a single LMDB read.

Mitigation v1: KH currently deploys cocoindex single-instance (one Cloud Run sidecar service) — so the gap does not surface. The API documents this v1 constraint in JSDoc on getFlowRunMetrics. Multi-instance support is DEFERRED-v1.1 — at that point the sidecar grows a thin coordinator that proxies queries across instances, but the public TS API does not change.

§4.3 Sidecar unreachable when cocoindex flow has completed

If the sidecar process restarts after a flow completes but before the KH UI reads the ledger, the LMDB entry may be gone (ephemeral per 02-data-flow.md §5.4). The API returns { ok: false, error: { code: 'flow_run_not_found' } } in that case.

Mitigation: callers gracefully fall back to pipeline_runs rollup when flow_run_not_found is returned for a flow_run_id known to have completed. The fallback policy lives in caller code (UI / MCP), not in this API — keeping this spec read-only and predictable per §2.6. Document the fallback recipe in JSDoc.

§4.4 No retroactive ledger for pre-launch runs

Cocoindex’s ledger only carries entries for runs the engine has actually executed. Any pre-launch pipeline_runs row (e.g. from existing KH cron handlers) has no corresponding cocoindex ledger entry. correlateToPipelineRun covers this — but the inverse query (getFlowRunMetrics(pipeline_runs.id)) is not in scope. Callers that hold a pipeline_runs.id and want per-stage detail must first locate the cocoindex flow_run_id (recorded in the pipeline_runs.result JSON when the run is cocoindex-originated).

Mitigation: the cocoindex flow scaffolding (T8 per PLAN.md §4.8) is responsible for populating pipeline_runs.result.flow_run_id when it writes the rollup row. Verify in T8 acceptance criteria.

§4.5 Bun fetch + HTTP 204 sandbox interaction

Per CLAUDE.md “Bun fetch hangs on HTTP 204 through sandbox proxy” gotcha. The intended cocoindex sidecar HTTP transport returns 200 with a JSON body (possibly an empty stages: [] array for new flow runs) — but this contract is itself STILL-OPEN per [GAP-LEDGER-003] below, so the assumption needs verification at the spike that closes that gap.

Mitigation: document in the spec that the sidecar contract is “200 with body” pending [GAP-LEDGER-003] resolution, and add a Vitest assertion that the client handles a 204 response by mapping to { ok: false, error: { code: 'malformed_ledger_entry' } } (defensive — the sidecar should never emit 204, but the client treats it as a contract violation rather than hanging). If [GAP-LEDGER-003] resolves with a 204-on-empty contract, revisit this section.

§4.6 Workspace scoping bypass risk

If the API is called from a context without an authorised Supabase client (e.g. a background job using the service-role client), the workspace scoping does not auto-apply. The implementation must require an explicit workspace_id for non-authorised callers (encoded in the §2.2 signatures as options?: { workspace_id?: string }).

Mitigation: the §2.2 signatures expose the options.workspace_id parameter explicitly. Default mode (“scope-from-auth”) extracts workspace_id from getAuthorisedClient(). Service-role callers pass options.workspace_id. Vitest test row §3.1 correlateToPipelineRun (d) covers the absence-AND-no-auth case (returns { ok: false, error: { code: 'rls_denied' } }).

---

§5 — Follow-ups

§5.1 v1.1 candidate work

1. MCP wrapper tools — mcp__kh__get_flow_run_metrics, mcp__kh__list_flow_runs_for_source per 06-mcp-tooling.md MCP-tool patterns. DEFERRED-v1.1 until UI surfaces stabilise.
2. gRPC transport — replace HTTP REST if throughput proves a constraint.
3. Multi-instance coordinator — required only if KH deploys multi-instance cocoindex sidecar (per phase-b-prerequisite-2-cocoindex-deep-dive.md §3.5).
4. Pipeline observability spec — docs/specs/pipeline-observability/TECH.md per PLAN.md §5 row 9 (CONDITIONAL — only if cocoindex per-stage metrics surface need exceeds pipeline_runs rollup at the UI level beyond what this API provides).

§5.2 0.9-collapse-candidates

This spec creates no entries in any 0.9-collapse-candidates.md. It is purely additive — no existing module retires as a result of this work.

§5.3 Gap flags

[GAP-LEDGER-001] Category: tech-spec needed — partial. Cocoindex’s exact JSON shape over HTTP is not documented in the KH planning corpus. Sources checked: phase-b-prerequisite-2-cocoindex-deep-dive.md §1.1 / §3.4 / §3.5 (covers the in-process LMDB ledger and the gap, but not the HTTP API shape if one is exposed); 02-data-flow.md §3 / §4 / §5; scripts/ontology-sync/parse-flow.py (Python flow stub, not the sidecar HTTP surface); the cocoindex package README is not part of the planning corpus. Prerequisite work: a 1-2h spike during T1.3 implementation to confirm the sidecar HTTP route surface and JSON shape, then refine §2.4 (stage name mapping) + §2.6 (transport) accordingly. Until the spike runs, the field names in StageMetric (e.g. short_circuit_reason values) are KH-proposed shapes that the implementation will reconcile against the sidecar contract. The public TS API types in types/pipeline-ledger.ts are KH-owned regardless of the reconciliation; only the mapping inside ledger-client.ts shifts.

[GAP-LEDGER-002] Category: investigation needed. The mapping from cocoindex’s stage-status vocabulary to KH’s StageStatus union (specifically the short_circuited vs skipped distinction) requires verification against cocoindex’s actual emitted values. Sources checked: 02-data-flow.md §3.2 (“Memoisation per-component-path scope”); phase-b-prerequisite-2-cocoindex-deep-dive.md §1.1 + §3.6 (S9 spike per 0.9-spike-S9-cocoindex-idempotency.md referenced but spike not yet run per PLAN.md §4.8 subtask 1). Prerequisite work: Spike #1 closure per PLAN.md §4.8 + the spike noted in [GAP-LEDGER-001]. Until both spikes run, the ShortCircuitReason union is provisional and may need an additional variant or a value rename.

[GAP-LEDGER-003] Category: investigation needed. Sidecar HTTP route surface (URL paths, request/response shapes) is undocumented in the KH corpus. Sources checked: 02-data-flow.md §4 (Cloud Run sidecar topology — names the components but not the HTTP routes); 03-tech-stack.md (no sidecar HTTP surface documented); phase-b-prerequisite-2-cocoindex-deep-dive.md (covers cocoindex internals, not the sidecar HTTP gateway). Prerequisite work: the same spike noted in [GAP-LEDGER-001]; this gap and [GAP-LEDGER-001] are likely resolved together. The §4.5 Bun-204 mitigation depends on this spike’s outcome — if the resolved contract turns out to be 204-on-empty, §4.5 needs revising. If the cocoindex sidecar does not expose an HTTP ledger surface at all, the implementation falls back to: (a) shipping a thin sidecar-side HTTP shim that reads the LMDB and serialises, or (b) deferring the API to v1.1 with a documented v1 shim that returns “ledger surface not available — use pipeline_runs rollup” for every call. Option (a) is preferred and tracked back to T8 cocoindex flow scaffolding (docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md §4.8 subtask 11 CocoInsight on-prem deployment posture — partially-related but not identical to this gap).

[GAP-LEDGER-004] Category: dependency-add gate. The test-design in §3 uses vi.mock('@/lib/pipeline/ledger-client', ...) directly — not msw — because msw is not currently a project dependency (verified against package.json at S242 W3 spec authoring time: no msw entry in dependencies or devDependencies). If at T1.3 build time the team decides an HTTP-transport-level mock harness is warranted (rather than module-level mocking), adding msw to devDependencies becomes a prerequisite. Sources checked: package.json (verified absent at spec authoring); CLAUDE.md “Mock pattern” gotcha (createMockSupabaseClient is the established shared-mock pattern; no precedent for msw in the codebase). Prerequisite work: T1.3 implementation phase decides between (i) module-level vi.mock() of ledger-client.ts (preferred — no dep-add, matches §3.1 pattern), or (ii) installing msw and writing HTTP-level handlers (deferred decision; would require updating package.json + adding a setup file). Until that decision lands, §3.4 specifies module-level mocking as the v1 default.

§5.4 Cross-spec coordination

• docs/specs/id-36-cocoindex-extraction-contract/TECH.md (T1.1 + T1.2 per PLAN.md §4.1) — the ExtractByLlm contract spec defines the LLM-extraction stage output shape; this API’s StageMetric.stage_name === 'llm_extraction' carries no payload of the extracted Q&A, only stage-status metadata. The two specs are orthogonal; this spec depends on T8 cocoindex flow landing but not on T1.1/T1.2 directly.
• docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md §4.8 (T8) — the cocoindex flow scaffolding consumes this spec to emit the flow_run_id into pipeline_runs.result (per §4.4 mitigation).
• docs/specs/id-31-0.9-canonical-pipeline/TECH.md §11 — historical-substrate per S241 Liam direction; the STILL-OPEN entry there is closed by this spec but the canonical-pipeline TECH is not edited inline per S241 critical-rule 5.
• docs/specs/silent-failure-prevention-spec.md — this API follows the silent-failure-prevention discipline (fail-loud LedgerResult<T> envelope re-using the canonical Result<T, E> shape from @/lib/supabase/safe); no spec amendment needed.

---

§6 — Changelog

• 18/05/2026 (S242 main, T1.3 per PLAN.md §4.1): scaffold drafted. Three gap-flags raised ([GAP-LEDGER-001] cocoindex HTTP route surface; [GAP-LEDGER-002] stage-status vocabulary mapping; [GAP-LEDGER-003] sidecar HTTP gateway shape). All API signatures + return shapes + error envelope defined; boundary with pipeline_runs rollup made explicit per §1.2; cocoindex internal LMDB exposure prohibition documented per §1.4. Source-attribution markers on every claim. UK English throughout.
• 18/05/2026 (S242 W3 fix-pass per docs/specs/id-36-cocoindex-ledger-api/verifier-reports/cocoindex-ledger-api-verifier.md):
  • B1 fix: rewrote §2.2.1 / §2.2.3 / §2.2.4 / §2.3 to use canonical { ok: true, data: T } | { ok: false, error: E } discriminant from lib/supabase/safe.ts:108-110. Introduced LedgerResult<T> = Result<T, LedgerApiError> alias; caller example branches on if (!result.ok); documented isOk() narrowing pattern. The data: null overloads in getPartialResolveDetail / correlateToPipelineRun are now expressed as { ok: true, data: null } per the canonical shape.
  • N1 fix: corrected lib/pipeline/start-run.ts citation from :1-60 to :100-179 (function body) with :78-99 noted for JSDoc.
  • N2 fix: §1.4 no longer attributes “schema-volatility insulation” as a literal phrase from phase-b-prerequisite-2-cocoindex-deep-dive.md §3.4. The source recommends keeping pipeline_runs as the KH-facing surface and documenting the cocoindex→pipeline_runs mapping; the spec now paraphrases that recommendation and frames the per-stage detail lens as an extension of the deep-dive’s prescription.
  • N3 fix: all 00-synthesis-v2.md citations now carry the full path prefix docs/plans/phase-0-investigation/10-feedback-investigation-findings/00-synthesis-v2.md.
  • S1 fix: msw removed from the §3.4 test design; module-level vi.mock('@/lib/pipeline/ledger-client', ...) adopted as v1 default (matches §3.1 pattern already named). msw’s absence from package.json is now tracked as [GAP-LEDGER-004] so any future move to HTTP-transport-level mocking goes through an explicit dep-add gate.
  • S2 fix: §2.7 overload requirement is now explicit in the §2.2 function signatures (every signature carries options?: { workspace_id?: string }); the §2.7 narrative restates the contract (service-role callers MUST pass options.workspace_id; absence + no auth → rls_denied); §3.1 row correlateToPipelineRun (d) covers the case.
  • S3 fix: §4.5 explicitly notes the contract claim depends on [GAP-LEDGER-003] resolution; if the resolved sidecar contract is 204-on-empty, §4.5 needs revising.