T8 PRODUCT — Cocoindex flow scaffolding + Cloud Run sidecar deploy

T8 PRODUCT — Cocoindex flow scaffolding + Cloud Run sidecar deploy

Spec slug: cocoindex-flow-scaffolding (single spec — RATIFIED per RESEARCH.md §5.3, confirmed by Liam @ S252 Wave 1 close) Subtask: ID-28.2 (PRODUCT, precedes {28.3 TECH} → {28.4 PLAN} per Q-PLANNER-2 fresh-per-Subtask discipline) Parent Task: ID-28 — T8 cocoindex flow scaffolding + Cloud Run sidecar deploy (docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md §4.8) Critical-path position: Wave 4. Gates: T1 (Q-EX2 contract) + T2 (Q-OQR1-16 combined-PR migration) + T6 (q_a_extractions schema) + T3 (RLS-pattern apply). Gates T7 (Phew Q&A first-ingest) per RATIFIED-S243. Author: task-planner (opus-4-7, isolation=worktree) on worktree-agent-a83a2c1a7a744a1b9 from base commit d20aa7cc. FRESH INSTANCE — NOT the {28.1} RESEARCH author (Q-PLANNER-2).

---

§1. Context

T8 stands up the cocoindex 1.0.3 pipeline scaffolding for the canonical document → embeddings flow, and the Cloud Run sidecar that hosts the heavy extractor binaries (Docling 1.8 GB + pullmd 3.7 GB Playwright). This PRODUCT.md captures the externally-observable behaviour the system MUST guarantee — the contract the rest of Knowledge Hub depends on for ingest, audit, retrieval, and failure-mode handling.

Consumers of the T8 surface (the “users” this spec speaks for):

1. Pipeline operators — engineers running the cocoindex Service, reading pipeline_runs for run history, reading Cloud Run logs for diagnostics.
2. Corpus consumers — MCP tools (create_content_item, q_a_search), app routes, and downstream cron jobs reading content_items / q_a_extractions / source_documents rows produced by the pipeline.
3. Audit-forensics consumers — governance + compliance roles reading audit_log to answer “who changed what row, when, via which path”, and engineers reading pipeline_runs to answer “which rows did pipeline run X write”.
4. Downstream dependents — T7 (Phew Q&A first-ingest) ships files into the cocoindex source-binding location once T8 is stable; T14 (cocoindex absorption cleanup) retires scripts/kb_pipeline/* after T8 stabilises.

Source-of-truth documents (read in full to ratify these invariants):

• docs/specs/id-28-cocoindex-flow-scaffolding/RESEARCH.md — full ratified RESEARCH (S252 Wave 1, 8 OQs at default).
• docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md §4.8 — T8 acceptance criteria + subtasks + scope in/out.
• docs/plans/phase-0-investigation/architecture/02-data-flow.md §3 (6-stage topology) + §4 (Cloud Run sidecar) + §5 (op_id + audit_log hybrid) + §7.2 (recordPipelineRun rollup).
• docs/plans/phase-0-investigation/0.9-decision-graph.md §11.4.1 (N7 op_id hybrid ratification).
• docs/specs/id-36-cocoindex-extraction-contract/PRODUCT.md — Q-EX2 ExtractByLlm contract (T1 dependency, fully ratified S241).

S252 Wave 1 ratifications baked into these invariants (per Liam, 8/8 OQ defaults accepted):

• O-Q1: source_documents.op_id ALTER is a T8 follow-up DDL slice.
• O-Q2: Cloud Run Service shape (not Job) per S14 single-orchestrator-instance topology.
• O-Q3: pullmd kept separate from Docling per AGPL network-service boundary.
• O-Q4: Container image ~5.3 GB budget; pre-warm Docling model layer.
• O-Q5: LiteLLM dependency-pin + WIF auth wiring at default.
• O-Q6: LMDB pre-warm via ephemeral re-fingerprint per S14.
• O-Q7: CocoInsight DEFERRED-v1.1 note included.
• O-Q8: T7 Phew first-ingest cutover — T8 ships empty source-binding; T7 stages files post-T8 stable.

---

§2. Behaviour invariants

Each numbered invariant is testable in isolation. The Checker compares against acceptance criteria per-invariant during ratification. Behaviour-only — implementation choices land in {28.3} TECH.md. UK English throughout. Cross-references use [file.md §X] or [file.md L<n>] format. Q-EX2-derived invariants cross-link directly to docs/specs/id-36-cocoindex-extraction-contract/PRODUCT.md rather than restating that contract.

Area A — Cocoindex flow observability

Inv-1. File-change detection. When a file is created, modified, or deleted under a tracked cocoindex source-binding location, the pipeline observes the change and emits exactly one pipeline run scoped to that change within the configured polling window. Verifiable: drop a file into the watched folder; within the polling-cadence window a corresponding pipeline_runs row appears with status in_progress (then transitioning to succeeded or failed per Inv-21). Cross-ref: 02-data-flow.md §2.1 + 02-data-flow.md §3.1 (source-walk stage); RESEARCH.md §1.2 (localfs.walk_dir(recursive=True) for nested corpora).

Inv-2. End-to-end latency. For a successfully-processed source file under the supported MIME set (PDF / DOCX / XLSX via Docling; HTML via pullmd; markdown direct), the resulting content_items row is observable via a primary-key SELECT within the latency budget defined per pipeline_runs SLA at v1 (acceptance-test budget: ≤ 120 s end-to-end on the 35-file canonical corpus per RESEARCH.md §4.2 cold-cache benchmark). Verifiable: write file → poll content_items by expected key → assert row present within 120 s. Cross-ref: PLAN.md §4.8 acceptance criterion “LocalFS source binding observes file change → pipeline run → content_items row written within polling cadence (integration test)”; RESEARCH.md §4.2 (memo-hit benchmarks confirming pipeline behaviour on 35-file corpus).

Inv-3. Six-stage topology observable per document. Each pipeline run executes the six canonical stages — source walk → binary conversion → LLM extraction → embedding → entity resolution → Postgres UPSERT — in order, and an external observer can determine for any given document which stage it last reached. Verifiable: a run that fails at the embedding stage MUST report the failure stage explicitly (per Inv-22 structured-log shape); a run that completes MUST have written embeddings to content_items.embedding. Cross-ref: 02-data-flow.md §3.1 (stage table); RESEARCH.md §1 (cocoindex constraints).

Entity resolution IS in scope at v1 (RATIFIED — OQ-C OVERRIDDEN by Liam, S265). Stage 5 (entity resolution) is a first-class stage of every run, not a deferred quality pass: the re-ingested corpus ships with resolved/canonicalised entities at v1. A completed run MUST have run the entity-resolution stage over the run’s entity mentions (dedup/canonicalisation across mentions), so a consumer reading the entity tables for a successfully-processed document observes canonicalised — not raw-only — entities. The implementation substrate (the faiss-backed resolve_entities op + its pin) is the TECH concern (§P-2, subtask 28.29); RESEARCH.md §R5 records the OQ-C override.

Inv-4. Idempotency on content-hash match. Re-running the pipeline over a file whose byte-contents have not changed since the last successful run does NOT produce new derivation rows (content_items update with no-op diff, q_a_extractions no duplicates, entity_mentions no duplicates). The pipeline short-circuits at the content-hash check via the memoisation contract. Verifiable: ingest a file twice without modification; assert q_a_extractions row count for that source is identical before and after the second run, and that content_items.updated_at either does not change or changes only its updated_at audit-trail field with a no-op diff per Inv-15. Cross-ref: 02-data-flow.md §2.2 (content-hash idempotency via @coco.fn(memo=True)); RESEARCH.md §4.2 (S2 Run B warm-cache = 0 invocations confirmed).

Inv-5. Nested-corpus coverage. When the tracked source-binding location contains nested subdirectories, files at any depth within the tree trigger the pipeline (the source-binding adapter does not silently skip nested files). Verifiable: place a file at <source>/a/b/c/file.md (3 levels deep) and confirm a content_items row is produced for it. Cross-ref: RESEARCH.md §1.2 (localfs.walk_dir(recursive=True) explicit default per CLAUDE.md cocoindex gotcha).

Area B — Sidecar deploy contract

Inv-6. Sidecar Service availability. The cocoindex pipeline executes inside a Cloud Run Service (not a Vercel function, not a Cloud Run Job at v1 per O-Q2), and that Service is reachable at a stable Service URL from the orchestrator-side Vercel routes. Verifiable: an HTTP health probe to the sidecar’s /health (or equivalent) returns 200 OK from the Vercel host within the Cloud Run Service warm-start window. Cross-ref: RESEARCH.md §2.1 (Vercel 250 MB bundle limit makes sidecar mandatory); §2.3 (existing Cloud Run baseline 80% ready); O-Q2 ratification (Service not Job).

Inv-7. Extractor binary availability at the sidecar boundary. Every extractor invocation the pipeline makes — Docling for PDF/DOCX/XLSX; pullmd for HTML — succeeds against the sidecar’s runtime environment for the supported MIME set, even when the orchestrator Vercel host has no local extractor binary. The orchestrator MUST NOT attempt to invoke an extractor binary in-process at the Vercel layer. Verifiable: ingest one file of each MIME (PDF, DOCX, XLSX, HTML, markdown) end-to-end via the canonical pipeline; each lands a content_items row with non-empty content text. Cross-ref: 02-data-flow.md §4.1 (Docling 1.8 GB + pullmd 3.7 GB exceed Vercel limit); RESEARCH.md §2.1.

Inv-8. Sidecar version identification in observability metadata. Every pipeline_runs row produced by a sidecar invocation carries metadata identifying the sidecar image (e.g. an image-SHA, build-tag, or equivalent stable identifier) sufficient for forensic correlation between a corpus row and the extractor build that produced it. Verifiable: query pipeline_runs.metadata (or equivalent column) for any successful run; the result MUST contain an extractor-identification field that can be cross-referenced against the Cloud Run image-deploy log. Cross-ref: RESEARCH.md §2.2 (gap inventory — extractor versioning is a net-new requirement); 02-data-flow.md §7.2 (recordPipelineRun() rollup surface).

Inv-9. pullmd licence boundary preservation. pullmd remains a separately-deployed network service that the cocoindex sidecar invokes via HTTP — NOT co-located in the cocoindex sidecar image. The AGPL “network-service” clause boundary is preserved. Verifiable: container-image inspection (e.g. docker image inspect) of the cocoindex sidecar image returns zero pullmd binaries / Playwright dependencies; pullmd traffic is observable as outbound HTTP from the cocoindex sidecar to the pullmd Service. Cross-ref: RESEARCH.md §2.2 (pullmd-co-location decision); O-Q3 ratification (kept separate per AGPL boundary); 02-data-flow.md §4.3 (pullmd retention rationale).

Inv-10. Docling cold-start mitigation. The first extractor invocation after a sidecar cold-start completes within an acceptable latency budget (cold-start tolerance: ≤ 60 s for the first Docling call per RESEARCH.md §2.2 + O-Q4 pre-warm ratification). The Docling model layer is pre-warmed in the container image such that no run-time model download blocks the first extraction. Verifiable: after a deliberate Cloud Run Service scale-to-zero + cold-start cycle, the first PDF ingest completes within 60 s end-to-end. Cross-ref: RESEARCH.md §2.3 (~44.75 s baseline Docling cold-start per phase-b-prerequisite-2d-docling-bakeoff.md §6); O-Q4 ratification.

Area C — op_id propagation contract

Inv-11. Per-row op_id stamping on pipeline-produced corpus rows. Every content_items row produced or updated by a cocoindex pipeline run carries the op_id of that run in its content_items.op_id column. The same invariant holds for q_a_extractions.op_id and source_documents.op_id (the latter via the T8 follow-up ALTER per O-Q1). Verifiable: ingest a file via a single pipeline run; query the row(s) the run produced; the op_id values match each other AND match the pipeline_runs.op_id for that run. Cross-ref: 02-data-flow.md §5.1 (N7 hybrid pattern — per-flow op_id); RESEARCH.md §3.1 + §3.3 + §R4; O-Q1 ratification (T8 absorbs source_documents.op_id ALTER).

Memo-respecting op_id semantic (RATIFIED — OQ-A, S265). “Produced or updated” is exact: op_id records the run that last materially produced or changed this row, NOT the most recent run that merely scanned it. Because the cocoindex per-source-item function is memoised (@coco.fn(memo=True)), an unchanged re-ingest of a file whose byte-contents have not changed is correctly SKIPPED — declare_row is not re-invoked, so the row’s op_id retains the value from the run that last materially touched it (a no-op re-ingest does NOT re-stamp op_id). A full_reprocess run re-runs every function and DOES re-stamp every row with the current run’s op_id (initial full-corpus ingest stamps uniformly). This is the correct incremental-pipeline behaviour and is strictly better for provenance forensics (the stamp answers “which run last changed this row?”), but the original spec assumed flow-scope stamping that would have implied “every run stamps every touched row” — so the semantic is stated explicitly here for the Checker. Empirical basis: RESEARCH.md §R4 (memo-vs-op_id probe — run 2 with unchanged bytes leaves the run-1 op_id in place). RATIFIED by Liam (S265) — the Checker tests this as the final contract.

Inv-12. op_id round-trip via pipeline_runs. Given any op_id extracted from a content_items / q_a_extractions / source_documents row, an audit-forensics consumer can resolve back to the originating pipeline_runs row (start/end times, status, stage-level counters) via a single PK SELECT. Verifiable: pick any pipeline-produced corpus row → read its op_id → SELECT pipeline_runs WHERE op_id = <value> → exactly one row returned. Cross-ref: 02-data-flow.md §7.2 (recordPipelineRun() rollup); RESEARCH.md §3.3.

Inv-13. audit_log rows carry op_id for pipeline-driven writes. Every audit_log row produced by a Postgres AFTER INSERT / UPDATE / DELETE trigger firing on a pipeline-driven write to a governed table carries the op_id of the originating pipeline run, so audit forensics can GROUP BY op_id to enumerate every audited change a given run made. Verifiable: ingest a file via a pipeline run; query audit_log WHERE op_id = <run's op_id>; the result enumerates one row per governed-table write the run made.

v1 emission point (substrate note — contract unchanged). cocoindex 1.0.3 exposes NO public per-row UPSERT completion callback (TableTarget._apply_actions is private — RESEARCH.md §R8). The v1 per-write signal is therefore emitted at the declare_row call-site INSIDE the per-source-item function (the site KH controls), carrying {op_id, table, row_id, operation}. Two caveats follow: (a) the line logs that a row was declared, not that it was applied — the reconciler applies it privately on the next update cycle (“declared ≠ applied”); (b) it cannot distinguish INSERT from UPDATE — the reconciler decides that internally, so the operation value is “upsert” not a resolved INSERT/UPDATE. This refines only the mechanism; the Inv-13 contract (op_id correlation across a run’s audited writes) is unchanged, and remains the contract that the v1.1 audit_log-table substrate must satisfy (per P-OQ1 deferral below).

Open question — audit_log v1 surface. Per RESEARCH.md §3.3 + RLS-PATTERN P-5 [DEFERRED-v1.1], the v1 audit surface is RAISE LOG from triggers, NOT a populated audit_log table. Inv-13 reads natively against a v1.1 audit_log table populated by trigger-fanout. At v1, the equivalent observability is structured-log shipping (Inv-22). Decision to ratify at {28.3} TECH: does T8 ship with a placeholder audit_log table populated by a minimal trigger (so Inv-13 has a concrete substrate), OR rely on Inv-22 structured logs for v1 and defer audit_log table population to v1.1? Default at TECH dispatch: defer audit_log table population to v1.1, retain Inv-13 as the contract that v1.1 must satisfy.

Inv-14. Non-pipeline writes still produce audit_log rows. A direct UI edit, a governance-cron update, or any other non-cocoindex write path to a governed table also produces an audit_log row (without an op_id, since there is no cocoindex run to correlate against). The audit-log surface is coverage-complete across writers — cocoindex’s per-flow op_id is additive, not replacement. Verifiable: a direct UI edit to a content_items row produces an audit_log entry with NULL op_id (or absent op_id field) but populated table-name / row-id / operation-type / invoking-role. Cross-ref: 02-data-flow.md §5.2 (why trigger-driven, not app-stamped — coverage completeness).

Inv-15. No-op writes do not produce audit-log noise. When the pipeline’s idempotency short-circuit fires (Inv-4 — content-hash matches stored hash), no content_items UPDATE statement is executed against Postgres, and consequently no audit_log row is produced for that no-op cycle. Verifiable: ingest a file twice unchanged; audit_log row count for that row is the same after the second run as after the first. Cross-ref: 02-data-flow.md §3.2 (@coco.fn(memo=True) memo-hit semantics).

Area D — recordPipelineRun rollup

Inv-16. One pipeline_runs row per pipeline invocation. Every pipeline invocation — regardless of whether it succeeded, failed, or short-circuited at the memo-hit check (when the pipeline ran but produced no derivation work) — produces exactly one pipeline_runs row with a stable op_id, start timestamp, end timestamp, and terminal status. Verifiable: trigger N pipeline invocations; pipeline_runs row count increments by exactly N. Cross-ref: 02-data-flow.md §7.2 (recordPipelineRun() retained per N6 RESOLVED).

Inv-17. Per-stage counters. Each pipeline_runs row exposes per-stage observability — at minimum a count or boolean of stage-completion for the six canonical stages (source walk, binary conversion, LLM extraction, embedding, entity resolution, Postgres UPSERT). Verifiable: query any pipeline_runs row and resolve a per-stage rollup (e.g. stage_counts.binary_conversion = N, stage_counts.llm_extraction = M). Cross-ref: 02-data-flow.md §7.2; RESEARCH.md §3.3 (op_id surface inventory).

Inv-18. Pipeline runs go through recordPipelineRun(), not raw insert. All pipeline_runs writes from the cocoindex pipeline go through the recordPipelineRun() helper from @/lib/pipeline/record-run (per CLAUDE.md “Cron pipeline_runs inserts” gotcha). Verifiable: a code-level grep / ts-morph query confirms zero raw supabase.from('pipeline_runs').insert(...) calls in pipeline code. (This is a code-discipline invariant; the behavioural consequence is that all pipeline_runs rows have the schema-required fields populated correctly — an invariant the helper guarantees by construction.)

Inv-19. pipeline_failures table does NOT exist. No KH-owned pipeline_failures table is created at v1 — the pipeline_runs.status='failed' rollup + structured logs + cocoindex’s internal LMDB failure tracking together cover the failed-run surface, so the planned KH table is RATIFIED-DO-NOT-BUILD per COCO.7. Verifiable: \d pipeline_failures returns “relation does not exist” against prod and staging. Cross-ref: 02-data-flow.md §7.3 + §10.4 (pipeline_failures ratified DO-NOT-BUILD); RESEARCH.md §R7 (dead-letter substrate reality — supersedes the §1.5 “cocoindex DLQ” framing).

Area E — ExtractByLlm contract per Q-EX2

Inv-20. ExtractByLlm invocations honour the Q-EX2 contract. Every LLM extraction call the cocoindex pipeline makes supplies output_type=ExtractionOutput (or one of its three discriminated-union variants — q_a_form, entity_mention, classification) per the ratified Q-EX2 contract. The pipeline produces no untyped or dict[str, Any] extraction outputs. Verifiable: integration test ingests one file of each extraction-kind and asserts the resulting database rows match the discriminator-keyed Pydantic shape from docs/specs/id-36-cocoindex-extraction-contract/PRODUCT.md Inv-1..Inv-14. Cross-ref: docs/specs/id-36-cocoindex-extraction-contract/PRODUCT.md Inv-1 (discriminated-union root); RESEARCH.md §1.6 (sample call shapes).

Open question — content_type routing finalisation. Per docs/specs/id-36-cocoindex-extraction-contract/PRODUCT.md invariant 7 + [GAP-Q-EX2-001], the exact set of routing content_types for the q_a_form variant depends on the form_types CV instance table landing in T2 of the canonical-pipeline PLAN. At T8 dispatch, T2 is DONE-S246 + DONE-S247 per PLAN.md §4.2 — the routing set finalises post-T2 from form_types. No T8-level open question; T8 inherits the post-T2 routing closure.

Inv-21. Memoisation deterministic per content-hash. Re-invocations of an ExtractByLlm call against the same content-hash + same output_type + same instruction string + same LLM model return the same output (modulo cocoindex’s memo cache; LLM stochasticity is bounded by the cache layer). Verifiable: ingest a file, capture the q_a_extractions rows; bump no inputs; re-ingest; assert the q_a_extractions rows are byte-identical. Cross-ref: docs/specs/id-36-cocoindex-extraction-contract/PRODUCT.md Inv-15..Inv-18 (memoisation rules); RESEARCH.md §4.2 (memo-hit benchmarks).

Inv-22. Validation failures produce a structured failure record, NOT a partial write. When an ExtractByLlm response fails Pydantic parsing (missing required field, invalid enum, invalid discriminator), the pipeline writes a structured failure to pipeline_runs (with error_class, extraction_kind, content_items_id, raw LLM response redacted of PII per Q-EX2 Inv-13) AND writes no partial extraction row to q_a_extractions / entity_mentions / content_items. Verifiable: ingest a contrived input that produces an invalid LLM response (mock the LLM to return a malformed discriminator); assert pipeline_runs shows a failure record AND q_a_extractions / entity_mentions show no new rows. Cross-ref: docs/specs/id-36-cocoindex-extraction-contract/PRODUCT.md Inv-13 (validation-failure shape).

Area F — Failure-mode behaviour

Inv-23. Transient LLM failures retry per the layered policy. A transient failure on a KH-issued LLM extraction call (HTTP-429 / 503 / network blip mid-anthropic.messages.create) triggers automatic retry; the pipeline does NOT short-circuit on the first transient error of a retryable class. The retry is layered, not single-policy (RESEARCH.md §R7): (a) KH’s own tenacity wrapper (_anthropic_retry in extraction.py, with a before_sleep hook bumping the flow-scope retry counter) is the OPERATIVE retry for KH Path A direct-anthropic extraction calls — cocoindex’s native LLM retry does NOT cover these; (b) cocoindex-native HTTP-429 auto-retry applies ONLY to LLM/embedding calls cocoindex itself issues (via ops.litellm), which KH does not use for extraction; (c) Postgres writes have ZERO per-row retry — cocoindex exposes no public per-row retry primitive, so a transient PG failure surfaces as a component exception that cocoindex’s cross-update durability re-attempts on the next update cycle, not as an in-run per-row retry. Verifiable: inject a one-shot 503 from the Anthropic endpoint mid-extraction; the KH tenacity wrapper retries and the extraction eventually succeeds (or escalates per Inv-25 if retries exhaust). Cross-ref: RESEARCH.md §R7 (layered retry reality); 02-data-flow.md §7.3 (no KH-side pipeline_failures table); docs/research/cocoindex-1.0.3-extractbyllm-spec-reality-investigation.md (Path A precedent).

Layered retry policy (RATIFIED — OQ-E, S265). The OPERATIVE LLM retry is KH-tenacity-owned in extraction.py (KH’s choice of count + backoff), cocoindex-native for any ops.litellm calls, and ABSENT for per-row Postgres writes. Ratified v1 policy: KH-tenacity 3 attempts / exponential backoff for Path A LLM calls; cocoindex-native only for cocoindex-issued LLM/embedding; ZERO per-row PG retry for v1. RATIFIED by Liam (S265). Documented in TECH.md §P-8.

Inv-24. Persistent failures land in a dead-letter surface. A failure that survives the configured retry policy lands in a dead-letter surface observable by the pipeline operator. The dead-letter surface enumerates: op_id, content_items_id, last-attempted stage, error class, last-attempted timestamp. Verifiable: inject a persistent failure (e.g. mock the LLM to return malformed JSON on every call); after retry exhaustion, the failure is enumerable from the dead-letter surface. Cross-ref: 02-data-flow.md §7.3 (no KH-side pipeline_failures table — COCO.7); RESEARCH.md §R7 (dead-letter substrate reality).

Open question — dead-letter surface substrate. Per 02-data-flow.md §7.3, KH does NOT build a pipeline_failures table (COCO.7 RATIFIED-DO-NOT-BUILD). The v1 dead-letter surface is the composite of: (a) the pipeline_runs.status='failed' rollup row (Inv-25); (b) structured failure logs emitted per invocation (Inv-26); and (c) cocoindex’s own internal failure tracking in its LMDB ops-DB. Note there is NO per-row Postgres retry queue feeding this surface — the failed-row signal originates from the run-level rollup + the structured logs, not from a per-row PG retry mechanism (RESEARCH.md §R7). Decision to ratify at {28.3} TECH: how does the operator query the failed-run surface from outside the sidecar? Default at TECH dispatch: structured-log shipping + the pipeline_runs.status='failed' rollup row. No new KH table needed; the existing pipeline_runs surface covers the rollup view.

Inv-25. Failed pipeline runs roll up to pipeline_runs.status='failed'. When a pipeline invocation fails persistently (after the layered retry policy exhausts per Inv-23 — KH-tenacity for Path A LLM, cocoindex-native for cocoindex-issued calls, no per-row PG retry), the corresponding pipeline_runs row has status='failed', an error_class field naming the failure category (e.g. extraction_validation_failed, extraction_provider_unavailable, postgres_write_failed, binary_conversion_failed), and a non-NULL end-timestamp. There is no “silent partial-completion” state — a row in pipeline_runs with status='in_progress' past its expected SLA is observable as a hung run, not as success-with-warnings. The pipeline_runs.status='failed' rollup is the operator-facing terminus of the dead-letter surface (Inv-24). Verifiable: inject a persistent failure; the corresponding pipeline_runs row resolves to status='failed' with a populated error_class within the retry-exhaustion budget. Cross-ref: Q-EX2 Inv-13 (validation-failure shape); 02-data-flow.md §7.2 (rollup); RESEARCH.md §R7 (layered retry — what “retries exhaust” means).

Inv-26. Structured log emission per failed invocation. Every failed pipeline invocation emits at least one structured log line (Cloud Run log surface, ingested via the KH observability stack per RLS-PATTERN P-5 [DEFERRED-v1.1] precedent) containing: op_id, stage, error_class, content_items_id (if known), and a redacted error message. The log MUST be machine-parseable (JSON or equivalent structured format). Verifiable: trigger a failure of each error-class category; for each, exactly one structured log line is emitted with the required fields. Cross-ref: 02-data-flow.md §5.3 (v1 audit surface is RAISE LOG via Cloud Run sidecar log ingest); RESEARCH.md §3.3.

Inv-27. No silent partial writes. A failure mid-pipeline does NOT leave the corpus in a partial-write state where one downstream table (e.g. q_a_extractions) has the failed run’s rows but another (e.g. content_items) does not, OR where content_items is updated but q_a_extractions is not. Either both side-effects land (success) or neither does (failure). Verifiable: inject a failure at the postgres-UPSERT stage (mock a transient PG connection refusal); assert that the failed run leaves no partial-write rows in any downstream table. Cross-ref: 02-data-flow.md §10.5 anti-patterns table (silent partial completion is a rejected pattern); RESEARCH.md §3.3 (PG rejections surface as asyncpg.PostgresError — no engine-side suppression).

---

§3. Out of scope

These are explicitly NOT specified at the PRODUCT level — they are either (a) implementation choices that belong in {28.3} TECH.md, (b) deferred to v1.1, or (c) owned by sibling specs.

Item	Status	Owner
Exact Cloud Run Service config (CPU / memory / min_instances / max_instances shape)	TECH.md {28.3} decision	T8 TECH
Specific retry count + backoff schedule	TECH.md {28.3} decision	T8 TECH (Inv-23 OQ)
audit_log table shape, populating triggers, and v1 vs v1.1 split	TECH.md {28.3} decision (Inv-13 OQ)	T8 TECH + RLS-PATTERN v1.1
ExtractByLlm Pydantic class declarations (the actual ExtractionOutput subclasses)	Already ratified	docs/specs/id-36-cocoindex-extraction-contract/{PRODUCT,TECH}.md (T1.1+T1.2)
TS-facing API over the cocoindex per-flow-run ledger	DEFERRED-v1.1 per RATIFIED-S243 Item 11	docs/specs/id-36-cocoindex-ledger-api/TECH.md (v1.1 substrate)
Q1.3-Q1.N content-model invariants (per-row shape post-ingest, chunking-boundary rules)	T1.4	docs/specs/id-56-content-model-invariants/
source_documents.op_id ALTER migration SQL	TECH.md {28.3} (per O-Q1 — T8 absorbs as internal slice)	T8 TECH
Container image build details (Dockerfile, requirements.txt pins)	TECH.md {28.3} decision	T8 TECH
Specific cocoindex source-binding path (which folder gets watched at v1)	TECH.md {28.3} decision per O-Q8	T8 TECH (T8 ships empty; T7 stages files post-T8 stable)
CocoInsight on-prem deployment posture	DEFERRED-v1.1 per O-Q7	Re-evaluate when self-host scenario surfaces
LiteLLM Anthropic prompt-cache passthrough verification	T13 prompt-cache verification spike	PLAN.md §4.13 (not T8 scope)
Per-MIME extractor function declarations (Docling wrappers + pullmd HTTP client + the per-MIME @coco.fn chain)	TECH.md {28.3} decision	T8 TECH
Embedding cost projection (1024-dim vectors × corpus size)	T13 cost-projection slice	PLAN.md §4.13 (not T8 scope)
Phew Q&A first-ingest cutover sequencing	T7 dependency	PLAN.md §4.7 (T7)
audit_log v1.1 surface populating mechanism (fan-out helper vs trigger-driven population)	DEFERRED-v1.1 per RLS-PATTERN P-5	RLS-PATTERN spec v1.1

---

§4. Open questions for Liam ratification before {28.3} TECH dispatch

Each OQ here is a behaviour-level uncertainty that surfaced during PRODUCT.md authoring. None blocks PRODUCT ratification; all need a recommended default to be applied at TECH.md dispatch (which can be amended by Liam in the ratification gate). RESEARCH.md §6 already ratified 8 OQs at S252 Wave 1 close; the OQs here are NEW ones that surfaced during PRODUCT invariant decomposition.

ID	OQ	Inv where surfaced	Recommended default for TECH dispatch	Cost of deferral
P-OQ1	What is the v1 substrate for Inv-13 (audit_log rows carry op_id for pipeline writes)? audit_log table at v1, or RAISE LOG structured-log shipping only with Inv-13 deferred to v1.1?	Inv-13	Defer audit_log table population to v1.1 — at v1 retain Inv-13 as a behaviour-contract that v1.1 must satisfy; the v1 substrate is structured-log shipping (Inv-22 + Inv-26). This aligns with RLS-PATTERN P-5 [DEFERRED-v1.1].	LOW — structured logs already satisfy operator-side observability at v1; v1.1 audit_log is an upgrade not a blocker.
P-OQ2 (RATIFIED — OQ-E)	What retry-count + backoff schedule does T8 ship at v1 (Inv-23)? Note the retry is layered, NOT a single cocoindex-native policy (RESEARCH.md §R7).	Inv-23	KH-tenacity-owned LLM retry policy in extraction.py (KH Path A direct-anthropic) at 3 attempts / exponential backoff, 1 s base; cocoindex-native retry covers ONLY cocoindex-issued LLM/embedding (ops.litellm, which KH does not use for extraction); ZERO per-row Postgres retry for v1. Documented in TECH.md §P-8. RATIFIED by Liam (S265).	LOW — KH-tenacity defaults are sane for the v1 corpus scale; configurable in extraction.py if operational evidence demands.
P-OQ3	What substrate satisfies Inv-24 (persistent failures land in a dead-letter surface)? A composite of pipeline_runs.status='failed' rollup + structured-log shipping + cocoindex’s internal LMDB failure tracking, OR a KH-owned dead-letter table?	Inv-24	pipeline_runs.status='failed' rollup + structured-log shipping + cocoindex’s internal LMDB tracking. No new KH table, and NO per-row Postgres retry queue feeding the surface (RESEARCH.md §R7 — the failed-row signal originates from the run-level rollup + structured logs, not a per-row PG retry mechanism). Aligns with COCO.7 RATIFIED-DO-NOT-BUILD per 02-data-flow.md §10.4.	LOW — pipeline_runs.status='failed' + structured logs is sufficient operator surface; a KH-side table would duplicate the rollup.
P-OQ4	Should Inv-2’s “latency budget” use the 35-file canonical corpus 120 s budget as the v1 acceptance number, or a per-file budget (e.g. 30 s per typical document)?	Inv-2	Both — 35-file corpus end-to-end ≤ 120 s AND per-file p95 ≤ 30 s. The 35-file corpus matches the S2 spike benchmark; the per-file p95 protects against pathological inputs.	LOW — both numbers are operator-observable; per-file p95 is a quality-gate threshold, not a hard fail.

---

§5. Glossary

New terms introduced in this PRODUCT.md (already-defined terms inherited from upstream specs are not restated):

Term	Definition
Sidecar Service URL	The stable HTTPS endpoint at which the Cloud Run-hosted cocoindex Service is reachable from the orchestrator Vercel host. Used by Inv-6 + Inv-7.
Extractor binary	Either Docling (PDF/DOCX/XLSX) or pullmd (HTML). Markdown direct ingest does NOT pass through an extractor binary.
Pipeline invocation	A single end-to-end run of the cocoindex pipeline triggered by either a source-binding file-change event, an explicit operator-initiated run, or a cocoindex-internal cycle (e.g. a memo-hit poll). Each invocation produces exactly one pipeline_runs row (Inv-16).
Memo-hit poll	A pipeline cycle that touches a file but short-circuits at the content-hash check (Inv-4). Produces a pipeline_runs row (per Inv-16) but no derivation work. Distinct from a “true” run that does derivation work.
Dead-letter surface	The operator-observable enumeration of persistently-failed pipeline invocations. v1 substrate: pipeline_runs.status='failed' rollup + structured-log shipping + cocoindex’s internal LMDB failure tracking (Inv-24 + P-OQ3). No per-row Postgres retry queue feeds it (RESEARCH.md §R7).
Op_id round-trip	The audit-forensics property that any pipeline-stamped op_id (on content_items, q_a_extractions, source_documents, or audit_log) resolves back to its pipeline_runs row via a single PK SELECT (Inv-12).

---

§6. Coverage matrix

Required area (from dispatch brief)	PRODUCT.md invariants	Status
A. cocoindex flow observability — localfs → 6-stage → content_items UPSERT	Inv-1, Inv-2, Inv-3, Inv-4, Inv-5 (5 invariants)	Covered
B. Sidecar deploy contract — Docling + pullmd at engine boundary	Inv-6, Inv-7, Inv-8, Inv-9, Inv-10 (5 invariants)	Covered
C. op_id propagation contract — N7 hybrid pattern	Inv-11, Inv-12, Inv-13, Inv-14, Inv-15 (5 invariants)	Covered
D. recordPipelineRun rollup per §7.2	Inv-16, Inv-17, Inv-18, Inv-19 (4 invariants)	Covered
E. ExtractByLlm contract per Q-EX2	Inv-20, Inv-21, Inv-22 (3 invariants — cross-links to upstream cocoindex-extraction-contract PRODUCT.md for the full Q-EX2 detail)	Covered
F. Failure-mode behaviour — retry / dead-letter / observability	Inv-23, Inv-24, Inv-25, Inv-26, Inv-27 (5 invariants)	Covered

Total: 27 numbered Behaviour invariants distributed across 6 areas. 4 new Open Questions surfaced during PRODUCT decomposition (P-OQ1 through P-OQ4) — all carry recommended-default ratifiable in Liam pre-flight before {28.3} TECH dispatch.

---

§7. References

§7.1 Primary sources (read in full for this PRODUCT)

• docs/specs/id-28-cocoindex-flow-scaffolding/RESEARCH.md — ratified WP1 RESEARCH (S252 Wave 1).
• docs/specs/id-31-canonical-pipeline-implementation-plan/PLAN.md §4.8 — T8 detail.
• docs/plans/phase-0-investigation/architecture/02-data-flow.md §1-§10 (full read; especially §3 six-stage topology + §4 sidecar + §5 op_id hybrid + §7.2 recordPipelineRun rollup + §10 anti-patterns).
• docs/plans/phase-0-investigation/0.9-decision-graph.md §11.4.1 (N7 op_id hybrid).
• docs/specs/id-36-cocoindex-extraction-contract/PRODUCT.md (ratified S241; Q-EX2 contract — Inv-20/Inv-21/Inv-22 cross-link directly).
• CLAUDE.md — cocoindex 1.0.3 Gotcha; Supabase & Schema; Deployment; Silent failures in Supabase calls; Cron pipeline_runs inserts.

§7.2 Cross-spec relations

• Predecessor: RESEARCH.md (this directory) — Liam-ratified S252 Wave 1.
• Successor (fresh Planner per Q-PLANNER-2): TECH.md {28.3} — implements one Proposed change per PRODUCT invariant, with the 4 P-OQ defaults applied (or amended at Liam ratification gate).
• Decomposition (conditional, after TECH.md): PLAN.md {28.4} — decomposes ratified PRODUCT + TECH pair into TM-shape implementation Subtasks {28.5+}.
• Sibling specs (already ratified, cross-linked): docs/specs/id-36-cocoindex-extraction-contract/{PRODUCT,TECH}.md (T1.1+T1.2); docs/specs/rls-pattern/{PRODUCT,TECH}.md (RLS-PATTERN P-1..P-5 + T-1..T-5).
• Downstream consumer: PLAN.md §4.7 T7 (Phew Q&A first-ingest depends on T8 source-binding being stable per O-Q8).

---

§8. Verification (S265 ID-28.20 amendment — OQ-3 empirical import-and-call check)

Per the OQ-3 pre-ratification forcing function, every external-library symbol the S265 amendments touch was re-checked against the installed pin before this amendment was returned. The full empirical log lives in RESEARCH.md §R1; this section records the subset load-bearing for the PRODUCT substrate corrections.

• Date: 25/05/2026.
• Pinned version: cocoindex[postgres]==1.0.3 (requirements.txt); installed cocoindex.__version__ == "1.0.3". Verified via python3 -c "import cocoindex; print(cocoindex.__version__)" (sandbox-disabled per CLAUDE.md LMDB gotcha).

Symbol / claim	Bears on	Result
cocoindex.connectors.postgres.TableTarget.declare_row(*, row)	Inv-11 / Inv-13 emission point	PRESENT — keyword-only row arg; no public per-row completion callback (_apply_actions private) ⇒ Inv-13 “declared ≠ applied” caveat holds.
cocoindex.flow / DataSlice / bind_target / collect / export	(removed substrate)	ABSENT — confirms the memo-respecting op_id semantic (Inv-11) is the reactive+memo reality, not flow-scope stamping.
@coco.fn(memo=True) memoisation skip on unchanged bytes	Inv-11 (OQ-A memo-respecting op_id)	PRESENT + BEHAVIOUR-CONFIRMED — RESEARCH.md §R4 probe: run 2 with unchanged source bytes leaves the run-1 op_id in place (no re-stamp).
cocoindex.ops.litellm importable	Inv-23 (cocoindex-native LLM retry scope) / P-OQ2	ABSENT (ModuleNotFoundError: No module named 'litellm') — cocoindex-native LLM retry covers only ops.litellm calls, which are not even importable; reinforces KH-tenacity (Path A) as the operative LLM retry.
cocoindex.functions.ExtractByLlm / LlmSpec / LlmApiType	Inv-23 substrate (Path A precedent)	ABSENT — confirms S256 record; KH Path A direct-anthropic is the extraction path, so KH-tenacity owns its retry.

No ABSENT / SIGNATURE_DRIFT finding contradicts any retained PRODUCT behaviour invariant — every Inv-N contract is unchanged; only substrate prose was corrected to the installed reality.

---

End of PRODUCT.md. S265 ratification complete: OQ-A + OQ-E RATIFIED (Inv-11 / Inv-23 / P-OQ2); OQ-C OVERRIDDEN — entity resolution in scope at v1 (Inv-3; subtask 28.29). Re-grounded substrate (this amendment) consumed by implementation Subtasks per Q-PLANNER-2.