Producer operations

Producer operations

Mechanic 5 of 5. How the producer is triggered, where it runs, the provenance model that guarantees every citation is real, and how the hand-authored CONFORMANCE.md interacts with it. Source of truth: scripts/cocoindex_pipeline/producer/trigger.py (trigger + manual run); scripts/cocoindex_pipeline/producer/enrich.py (the BI-17 minting model); scripts/cocoindex_pipeline/producer/git_sync.py (publish/reconcile); the {132.15} journal; DR-018 (trigger posture); DR-016 (producer-override model).

---

How the producer runs

The automatic trigger (DR-018)

The producer is chained off a successful ingest walk, not scheduled (DR-018). The rationale (id-132 TECH.md, S436 D4): concepts are the map over the knowledge, so it is NEW/UPDATED source_documents rows that should drive new/updated concept files — not a calendar.

trigger_producer_post_walk (trigger.py:131-159): a walk that touched (created/updated) one or more source_documents rows chains one producer run; a walk that touched none is a no-op (if not deltas: return False, :145). A module-level reentrancy guard (_FIRED_OP_IDS, :128) ensures the same op_id fires at most once. The hook is in-server (flow.py’s app_main() calls it directly from its finally block using the same pool and op_id already in scope), not a second webhook hop — the op_id-scoped source_documents deltas are already the precise signal.

Delta-only + memoisation together scope a run: enrich_concept is @coco.fn(memo=True) keyed on ConceptKey (enrich.py:50-59), so a concept whose backing records are unchanged should not be re-drafted. BI-18 delta-determinism is UNPROVEN — a memo-delta defect is specced as {132.38} (specs/id-132-okf-concept-producer/MEMO-DELTA.md) and blocks the re-proof. The DR-060 delta-determinism contract lands WITH that fix: invalidation is a manual version= bump logged in log.md, NOT deps= auto-invalidation. Do not describe delta-determinism as proven until {132.38} lands and {132.35} re-proves it.

Manual invocation

run_producer_now (trigger.py:162-173) is the retained operator surface — it calls the same entry point but bypasses both the delta gate and the reentrancy guard, so an operator can always force a run. The full flow it drives (flow_def.run_producer_flow) is: LRecordsSource.list_concepts() → enrich_concept (Pass-1) → optional run_web_pass (Pass-2) → write_bundle → embed → publish-gate / git_sync.sync_bundle (trigger.py:36-42).

Idle-mode safety

default_producer_entry_point no-ops whenever OKF_BUNDLE_DIR is unset or does not point at an existing directory (trigger.py:48-55) — true in every deployed environment today, since nothing sets that env var yet. Wiring the hook into app_main therefore cannot spend Anthropic tokens or touch the filesystem until an operator deliberately configures the bundle location.

Deploy posture

The VPS producer deploy is {132.35} — DONE: deployed, GLM-5.2 wired (slices B–E), BI-18 proven 17/07/2026. The producer runs on ca-pipeline-platform-staging (Coolify, Server B / canonical-platform, uuid slh75m2si7yt60t42tfeajbq) as one route (POST /producer-run) inside the SAME aiohttp process that also serves /walk and /extract (server.py — one web.Application, one container). The automatic walk-chained trigger (DR-018, above) is wired in production; the manual forced-run surface (POST /producer-run, bearer-gated, DR-055) is the BI-18 proof mechanism — it runs in a dedicated coco.App (kh_pipeline_producer_forced) with its own memo namespace, distinct from the automatic walk’s KH_PIPELINE_APP namespace. The BI-18 proof itself must run through ONE surface (3 forced runs: cold draft-all, unchanged = memo-hit no-op, single-touch = one re-draft) — never mix automatic + forced runs when interpreting memo-hit/miss behaviour. Determinism (BI-18 no-op re-run) lives in the real cocoindex runtime, so it is proven by the VPS flow deployment, not by the standalone harness — the harness re-drafts every run ({132.15} journal, v4 caveat). BI-18 was PROVEN 17/07/2026 on image sha-35e1a9c1 (4-run forced-surface protocol): cold full-corpus draft 17/18 via GLM-5.2/OpenRouter (1 transient GLM JSON failure, re-drafted clean next run); PURE unchanged re-run = zero drafting calls AND zero bundle churn (“No changes (no-op re-run)”); memo-hits persist across container restarts (LMDB on the volume); single-touch (one form_instances.updated_at) = exactly one re-draft. Zero Anthropic drafting spend.

Two operational caveats from the proof (tracked as follow-up code items):

• RETIRED ({132.46}, S487): /producer-run was single-use per container lifetime — a second POST failed App name already registered: kh_pipeline_producer_forced because the handler constructed a fresh coco.App per call. Fixed: the forced-run App is memoised for the container’s lifetime (_FORCED_PRODUCER_APP singleton; the App name stays constant so the LMDB memo namespace is preserved) — repeated /producer-run calls reuse it safely, no restart required. Effective on the deployed app from the first image carrying main f9cb8016; on an older image the docker restart workaround still applies.
• Runs containing a real re-draft also re-render cross-linking concepts (log.md “Changed” count exceeds drafting calls — memo-hit concepts that cite the changed concept re-render their links; arguably correct). A PURE no-op run has zero churn. Residual question tracked code-side: is the cascade radius bounded?

Deployed-runtime env contract

Set on the Coolify application (names only — never print values):

Var	Purpose
OKF_BUNDLE_DIR	persistent-volume clone of the client-owned bundle repo (DR-055); idle-mode no-op when unset/missing
OKF_BUNDLE_DEPLOY_KEY_PATH	RW deploy-key path (GIT_SSH_COMMAND) for the publish push-lane (producer/publish.py); unset = clean no-op
OKF_BUNDLE_CLASS	bundle-class discriminator ({132.37}, OV-10/DR-054/DR-079) — showcase for this synthetic-corpus Platform bundle; unset would hard-reject any discovered ontology-overlay.json (base-only runs unaffected)
OKF_CLIENT_ID	unset for the platform bundle — context.jsonld ({132.44}) emits base-only; advisory diagnostic only, never aborts the run (IRI-6/9)
PRODUCER_MODEL	(ID-132 {132.35} slice B) producer-package-scoped model override, read once at import (producer/agent_loop.py) — default model for BOTH enrich_concept (Pass-1) and run_web_pass (Pass-2); unset/empty preserves the extraction lane’s ANTHROPIC_MODEL value unchanged. A PRODUCER_MODEL value change is drafting-config under DR-060 — same manual @coco.fn(..., version=N) bump + bundle log.md note contract as a literal ANTHROPIC_MODEL edit.
PRODUCER_BASE_URL / PRODUCER_AUTH_TOKEN	(slice C, 86c7bb08) producer-scoped endpoint/auth pair consumed by the shared producer_async_client() factory (producer/agent_loop.py) — the factory only passes base_url=/auth_token= when the matching var is set, so both-unset is byte-for-byte a bare AsyncAnthropic() (extraction lane’s process-wide ANTHROPIC_BASE_URL/ANTHROPIC_AUTH_TOKEN never leak in — isolation-proof test). Set to https://openrouter.ai/api + the OpenRouter key for GLM routing. In override mode the factory also passes api_key="" (slice E) so the SDK never falls back to the env ANTHROPIC_API_KEY / sends it off-Anthropic as x-api-key. DR-060 drafting-config contract applies to value changes.
PRODUCER_PROVIDER_ORDER	(slice D, 764f0957) comma-separated OpenRouter provider slugs (staging: z-ai), read once at import; when set the single producer messages.create site sends extra_body={"provider": {"order": [...], "allow_fallbacks": true}} — REQUIRED for any non-Anthropic model through OpenRouter’s Anthropic-skin (see gotcha below); unset omits the field entirely. DR-060 drafting-config contract applies.
ANTHROPIC_API_KEY / OPENAI_API_KEY	extraction lane + producer Pass-1/2 default credentials (Anthropic Anthropic-native calls; both lanes share these unless GLM routing below is wired)

Note on Coolify’s env-var listing: this is a docker-compose app with two services (cocoindex-platform-staging + bid-worker-platform-staging) under one application UUID — each application-scoped env var the Coolify API returns shows up as two records with different UUIDs but the same key/value (one per service). This is benign per-service materialisation, not orphaned drift — confirmed empirically ({132.35} G-DEPLOY-PROOF session, 17/07/2026) by creating OKF_BUNDLE_CLASS once and observing two list entries appear.

GLM-5.2 routing (DR-079 non-client-bundle rule) — wired via slices B+C+D

S481 ratified running this bundle’s Run-1 full-corpus draft on OpenRouter GLM-5.2 rather than Anthropic (DR-079: non-client bundles run GLM-5.2; client bundles stay Anthropic). Four distinct wiring problems surfaced in the {132.35} deploy-proof sessions (17/07/2026) — all four now CLOSED by producer-scoped slices (the pattern: read-once-at-import, unset = byte-identical legacy behaviour, extraction lane untouched):

1. Model slug — CLOSED by slice B (PRODUCER_MODEL, above). The producer agent-loop (producer/agent_loop.py’s run_tool_use_loop) had no env indirection for the model string at all; it was the literal ANTHROPIC_MODEL = "claude-opus-4-6" constant (extraction.py:119) with zero override path through flow.py → trigger.py → flow_def.py → enrich_concept/run_web_pass. PRODUCER_MODEL closes this in isolation from the extraction lane’s own ANTHROPIC_MODEL.
2. Endpoint/auth — CLOSED by slice C (PRODUCER_BASE_URL/PRODUCER_AUTH_TOKEN + producer_async_client(), 86c7bb08). The hazard it closed: anthropic.AsyncAnthropic()’s base_url/auth_token ARE SDK-env-readable (ANTHROPIC_BASE_URL/ANTHROPIC_AUTH_TOKEN) — but those are PROCESS-WIDE, and producer/enrich.py + producer/web_pass.py used to construct their client bare, exactly like extraction.py’s four call sites, with BOTH lanes in the SAME container process (server.py registers /walk, /extract, AND /producer-run on one web.Application). Setting the process-wide pair at the Coolify application level would have redirected the extraction lane through OpenRouter too — a live risk, not theoretical: /walk fires both on a schedule AND via the fire-and-forget nudge from lib/intelligence/pipeline.ts on every gate-passed feed_articles row. Slice C gives the producer’s two call sites their own factory-constructed client; extraction.py is untouched and an isolation test proves a leaked process-wide ANTHROPIC_BASE_URL cannot reach the producer client. Never set ANTHROPIC_BASE_URL/ANTHROPIC_AUTH_TOKEN app-wide on this app — the producer-scoped pair is the only sanctioned lever (DR-079).
3. OpenRouter Anthropic-skin provider default — CLOSED by slice D (PRODUCER_PROVIDER_ORDER, 764f0957), discovered by the first live Run-1 (18/18 drafts 404). OpenRouter’s Anthropic-compatible /v1/messages endpoint silently defaults requested_providers=['anthropic'] when the request body carries no provider field — and the anthropic provider does not serve z-ai/glm-5.2, so every draft returned 404 "No allowed providers are available for the selected model" before inference (zero billed). The docubot precedent did NOT prove GLM through the skin (docubot defaults to a Claude model; its own comment warns non-Anthropic models are pass-through, not guaranteed). Probe-proven fix: provider: {order: ["z-ai"], allow_fallbacks: true} in the body → 200 with a correct Anthropic-shaped GLM reply. Slice D injects exactly that via the Anthropic SDK’s extra_body passthrough when PRODUCER_PROVIDER_ORDER is set.
4. SDK x-api-key fallback overrides body routing — CLOSED by slice E (api_key="" in producer_async_client() override mode), discovered by Run-1 #2 (18/18 identical 404 WITH slice D deployed and verified reaching OpenRouter). The anthropic SDK falls back to env ANTHROPIC_API_KEY (set in-container for the extraction lane) and sends it as an x-api-key header alongside the OpenRouter Bearer; OpenRouter sees the Anthropic-shaped key and pins requested_providers=['anthropic'], overriding the body provider field. Probe chain: Bearer + x-api-key → 404 (exact repro); api_key="" (empty x-api-key, ignored by OpenRouter) + slice-D routing → 200, z-ai/glm-5.2. This is the same ANTHROPIC_API_KEY='' guard docubot’s run-agent.ts documents; the factory omitted it. Side benefit: the producer no longer transmits the real Anthropic key to a third-party endpoint on every draft call. (default_headers={"x-api-key": None} is NOT a viable alternative — the SDK raises TypeError on a None header value.)

The 15/07/2026 §7/§8/§11 conformance wave (log.md date-grouping, numbered-link citations, the okf_version stamp) is a set of bundle-shaping changes that landed pre-{132.35} by design: they change the on-disk shape a re-draft will emit, so they are batched INTO the single {132.35} cold re-draft rather than each triggering its own regeneration — the “pay-the-cold-draft-once” hold. That re-draft RAN 17/07/2026 (proof Run-1); its output — 19 concept files in the new citation format — sits staged-uncommitted in the VPS bundle working tree awaiting the owner-gated publish (BI-20/21). Until that publish, the PUBLISHED bundle’s concept trailers keep the legacy citation format.

The durable run recipe

The reproducible standalone run lives in .user-scratch/:

• preflight-list-concepts.py — DSN + grain enumeration check, no LLM calls. Instantiates LRecordsSource(pool) and prints list_concepts() totals by top-level directory. Run this first to confirm the L-records surface enumerates the grains you expect.

• run-synth-okf.py — the {132.15} G-SYNTH-OKF harness: runs the full producer flow over the Platform staging L-records via run_producer_now, writing the bundle to OKF_BUNDLE_DIR and UPSERTing record_embeddings after the run. It swaps in a real-write localfs (declare_file writes immediately, since no cocoindex ComponentContext exists outside App.update) and a ShimReTarget for the embedding rows.

  Required env: COCOINDEX_DB_DSN, OKF_BUNDLE_DIR, OKF_REPO_PATH, OPENAI_API_KEY, ANTHROPIC_API_KEY, KH_REPO.

BI-17: the provenance-minting model

This is the model that makes a bundle trustworthy: every citation must resolve to a row the run actually read. Format alone is not proof of provenance — a well-formed but never-issued canonical://source_documents/<random-uuid> FAILS validation (enrich.py:24-48).

The seen_anchors ledger

Every run holds a seen_anchors set — the anchors THIS run actually minted into a tool result (enrich.py:616). _mint (enrich.py:290-296) is the one function every mint site routes through: it records the anchor into seen_anchors and returns it unchanged, so no anchor can enter a tool result without also being recorded as “actually minted this run”. _validate_citation (enrich.py:400-455) then rejects any record-anchor citation that is not a member of seen_anchors (and any cross-link not in the list_concepts catalogue).

The mint sites — every row-id surface is a mint site

The lesson from the first Platform run (below) is that every tool-result surface that exposes a real row id must mint that row’s anchor — otherwise the model reads a real id it cannot legally cite. There are three such surfaces, all in enrich.py:

Surface	Function	What it mints
read_concept_raw	_annotate_raw_with_anchors (:232-287)	each source_documents/reference_items row gets its BI-6 per-row anchor; a top-level qa_resource (BI-8) when the key carries a topic locator
sample_rows	_sample_rows (:335-370)	sampled source_documents-backed grain rows (company/certification) get their per-row anchor minted — a sampled row is real provenance
entity_mentions payload rows	_annotate_raw_with_anchors (:272-279)	each mention row carries build_source_document_uri(source_document_id) — its context_snippet is genuinely-read content from that parent sd, so the parent sd is citable; the mention row’s OWN id stays unadorned (entity_mentions is not a BI-6 citation table)

Rows with no legal anchor form stay unadorned: q_a_pairs, record_lifecycle, entity_relationships, workspaces, form_templates — the model must cite their evidence via concept cross-links, never a bare id (enrich.py:266-282).

Why this took three runs to get clean (S463)

The first Platform producer run surfaced the model as a sequence of provenance failures — the gate refusing citations the tooling had set it up to trip:

• v3 (15/18): two SSCM certifications cited a real staging sd uuid that reached the model via sample_rows, whose executor returned raw rows with ids but no minted anchor — read_concept_raw was the only mint site, so a row genuinely read via sampling was never legally citable. Fix 83bf58db: route sampled company/certification rows through the same mint path.
• v4 (17/18): one residual — a certification cited a real sd that reached the model via an entity_mentions payload row (carrying source_document_id + a context_snippet) with no minted anchor. Fix b6b3f6c3: mint the parent-sd anchor onto each mention row.
• v5 (18/18 CLEAN): zero drafting failures; 18 record_embeddings upserted; every canonical:// anchor verified to resolve on staging (7/7 sd uuids published; 3/3 q_a_pairs?scope_tag= query anchors have published rows); zero broken cross-links. Published as commit 3f66483 at the owner-gated BI-20/21 publish gate.

The takeaway for future producer work: when you add a tool surface that exposes a row id, it is a mint site. An un-minted real id leaks provenance the gate then refuses. See the {132.15} journal (bun scripts/ledger-cli.ts journal 132.15) for the full run narrative.

Publish + git-sync (DR-016)

The bundle medium is a client-owned git repo (DR-016). git_sync.sync_bundle reconciles the freshly-written bundle dir against the repo: it stages each managed path individually (git add -- <path>, never git add -A), makes one commit only when at least one managed path changed, and returns HEAD unchanged on a genuine no-op. Approved human edits are captured as producer overrides (keyed by concept-path/section) re-applied on every regeneration — never direct file mutations (DR-016, reference/decision-register.md).

CONFORMANCE.md and the producer

Until 15/07/2026 the bundle repo carried a hand-authored root README.md (agent-facing — ID-71 agents read the bundle repo, not this docs-site). The OKF v0.1 conformance wave changed this: the published bundle’s root README.md was removed (owner decision, 15/07/2026) and its agent-facing content moved to a hand-authored CONFORMANCE.md (frontmatter type: reference) at the bundle root — which also carries the machine-facing conformance narrative (the deliberate divergences: closed type taxonomy, opaque canonical://, and the ontology.json artefact).

Both README.md and CONFORMANCE.md are now producer-reserved — they joined _RESERVED_BUNDLE_FILENAMES (bundle_writer.py:174-189, S464 rider R1). The producer only ever declare_files concepts + its generated bundle files; cocoindex’s localfs reconcile prunes only files it declared, and the git-sync layer re-reads the whole bundle dir into its “desired state”, so a hand-authored reserved doc resolves as unchanged — never staged, never git rm’d, never flagged.

The old cosmetic wrinkle is now RESOLVED (recorded, not silently dropped). Previously, because _existing_concept_paths (bundle_writer.py:799-818) is a recursive rglob("*.md") minus the reserved names, a root README.md — then NOT reserved — landed in each run’s RunSummary.removed and surfaced in log.md as a false - Removed (N): README.md line even though the file physically persisted. Now that README.md/CONFORMANCE.md are in the reserved set, the concept scan skips them (:817) and that false audit line can no longer be emitted. The wrinkle was real; it is now closed by the same reserved-set change, not merely left undocumented.