BI-18 deploy-proof evidence (S485)

BI-18 Deploy Proof — Evidence Log ({132.35} G-DEPLOY-PROOF)

Executor: BI-18 deploy-proof session. READ-ONLY on canonical code. Bundle-repo commits allowed. Deployed image: sha-86c7bb088cade936e3a8c4840873146c2ccc744c Coolify app: ca-pipeline-platform-staging (uuid slh75m2si7yt60t42tfeajbq), Server B. Container: cocoindex-platform-staging-slh75m2si7yt60t42tfeajbq-143737001757. Staging DB: rbwqewalexrzgxtvcqrh. VPS: root@87.106.72.59 (ssh -i ~/.ssh/kh_ionos_ed25519).

BASELINE (pre-Run-1 snapshot)

• Container: Up (healthy). OKF_BUNDLE_DIR=/cocoindex-state/okf-bundle, OKF_BUNDLE_CLASS=showcase, OKF_CLIENT_ID unset.
• Producer env: PRODUCER_MODEL=z-ai/glm-5.2, PRODUCER_BASE_URL=https://openrouter.ai/api, PRODUCER_AUTH_TOKEN SET, OKF_BUNDLE_DEPLOY_KEY_PATH SET, CRON_SECRET SET, PIPELINE_TRIGGER_SECRET SET.
• Bundle repo HEAD: 54d3bb7de8f84104d011cecc5ea248905debf4fd (“docs: agent-facing bundle README {132.33}”). Prior producer commit: 3f6648370758034c4c2af61248ecb305a0df2a23 (“okf producer git-sync 2026-07-11T15:50:42Z”). Working tree CLEAN.
• Remote: git@github.com:ai-solution-hub/canonical-okf-platform.git
• Existing bundle = 18 concepts (from 2026-07-11 run, log.md “Added (18)”):
  • topics(4): data-protection, procurement—tender-evaluation, quality-management, social-value
  • products(3): greencycle-furniture-programme, rapidrestock-managed-print-service, stocksense-ordering-portal
  • company(1): overview
  • certifications(4): facilities-assurance-standard-fas-2100, iso-9001, sscm, sustainable-supply-chain-mark-sscm
  • case-studies named-client(5): corvedale-academies-trust, northgate-borough-council, ridgeway-commercial-services-ltd, st-aldhelm-s-nhs-foundation-trust, wyndale-metropolitan-borough-council
  • case-studies won-bid(1): won-bid/northgate-borough-council
• Persistent LMDB memo/target-state store: /cocoindex-state/lmdb (744K) — survives restarts. Run-1 uses forced-App namespace kh_pipeline_producer_forced; version=1 bump + distinct namespace => Run-1 expected FULL cold re-draft of all 18.

RUN-1 #3 RESULT (requestId 0c0faf746f0649aea3acc4afe304887a) — SUCCESS 17/18 (slice-E fix works)

• POST 16:33:48Z -> 202. Completed 16:47:52Z (~14 min): ran=True, embedded=17. All httpx to openrouter = 200 (GLM-5.2). ZERO Anthropic drafting spend.
• 1/18 FAILED: case-studies/corvedale-academies-trust.md — “enrich_concept: terminal text was not valid JSON … Invalid control character at line 1 column 545”. GLM emitted an unescaped control char in its JSON output. MODEL-OUTPUT quirk, NOT routing. (DEFECT B — GLM output-robustness; parse-hardening belongs to a code slice, escalate.)
• Bundle content CORRECT: index.md has okf_version: "0.1" frontmatter + 17 markdown-link entries; log.md date-grouped newest-first (“Run 2026-07-17T16:47:42Z Changed (17)” + “Removed (1): corvedale”); concepts use markdown-link citations + # Citations trailer, base-only paths (/topics/…). context.jsonld base-only (OKF_CLIENT_ID unset).
• RECONCILE CONCERN (DEFECT B side-effect): corvedale (which has a GOOD July draft) was logged “Removed (1)” purely because this run’s GLM draft failed — a transient draft failure de-indexes a previously-good concept. Same reconcile-on-failure family as attempt-1. Flag: publish should not drop a concept on a transient draft miss.

PUSH-LANE ASSESSMENT — NOT reachable via /producer-run (BY DESIGN); container also lacks ssh

• Deployed run_producer_flow (flow_def.py:398-405) calls git_sync.sync_bundle(..., stage_only=True) — STAGES the bundle for review only. The ONE commit-and-push is a SEPARATE, human-gated action (publish.publish_bundle / producer publish CLI, BI-21 accept/edit/reject gate) — NEVER fired by the forced run. So Run-1 correctly left HEAD unchanged (54d3bb7d) with 17 concepts modified-but-uncommitted; zero git-sync/commit/push log lines. This is CORRECT, not a defect.
• => The “previously-untested push-lane” CANNOT be exercised through the /producer-run BI-18 proof surface. It requires the separate publish CLI + human gate. OUT of this proof’s scope.
• DEPLOY GAP (flag for the eventual publish action): the container has NO ssh binary (command -v ssh = NO_SSH; only git 2.34.1). OKF_BUNDLE_DEPLOY_KEY_PATH is set but git needs ssh to reach git@github.com:ai-solution-hub/canonical-okf-platform.git. git ls-remote origin failed “cannot run ssh: No such file or directory”. Unless publish_bundle uses GIT_SSH_COMMAND/a bundled ssh or an HTTPS-token transport, the push will fail. Needs verification before the publish lane can succeed live.

DEFECT C (BLOCKER, worked around) — /producer-run is SINGLE-USE per process

• Step-2 retry (requestId ae3c9e14…) FAILED instantly (ERROR, 0 draft calls): ValueError: failed to initialize app 'kh_pipeline_producer_forced' — Invalid Request: App name already registered: kh_pipeline_producer_forced (traceback: server.py:1147 _run_producer_forced -> :1138 _build_forced_producer_report -> forced_app.update_blocking() -> cocoindex _internal/app.py:263 core.App(…)).
• Root cause: _build_forced_producer_report constructs a NEW coco.App(AppConfig(name=_PRODUCER_FORCED_RUN_APP_NAME)) on EVERY call (server.py:1129-1132); cocoindex registers the App name process-globally and rejects a duplicate. So the 1st /producer-run per process works; every subsequent one fails. The BI-18 multi-run proof (Run-1/2/3 as separate POSTs) is UNREACHABLE in-process.
• FIX (code slice, out of scope): cache/reuse a module-level forced App singleton across calls (mirror KH_PIPELINE_APP’s process-wide caching), or unregister/tear down the App after each run, or use update_blocking on a persisted handle. Escalate.
• WORKAROUND applied (operator action, not a code patch): docker restart the container between runs to clear the process App-registry. The memo/target-state store is the PERSISTENT LMDB (/cocoindex-state/lmdb) which survives restarts, so memo comparability across runs is preserved — EMPIRICALLY CONFIRMED by Run-2 below (memo hits after restart).

RUN-2 (memo-hit) — requestId ff56f4d1056b4b449ae3845c5020af53 — MEMO WORKS

• Container restarted 17:01:45Z (healthy). POST 17:01:58Z -> 202. Completed 17:02:40Z (~42s vs Run-1’s ~14min).
• ONLY 4 OpenRouter calls total (all 200) = corvedale’s re-draft alone. The 17 previously-drafted concepts made ZERO drafting calls => MEMO HIT (unchanged content_version, BI-18 no-op proof for 17/18). Memo cache PERSISTED across the restart (LMDB).
• Corvedale (the Run-1 failure, never memoised) re-drafted and this time SUCCEEDED (no failure line; file written 17:02, 2858 bytes) => now memoised.
• embedded=6 / log.md “Changed (6)”: corvedale + 5 concepts that cross-link it (ridgeway, st-aldhelm, wyndale, company/overview, greencycle). These 5 made ZERO drafting calls — the change is a downstream bundle-RENDER cascade from corvedale’s removal(Run-1)->re-addition(Run-2) restoring cross-links, NOT re-drafts. Drafting memo-hit holds.
• NOTE: a PERFECTLY clean zero-draft/zero-change no-op was confounded only by corvedale’s Run-1 failure (DEFECT B); the memo-hit-on-drafting contract itself is demonstrated (17/18 zero calls).

RUN-3 TOUCH DESIGN (scouted during Run-1 #3, staging DB rbwqewalexrzgxtvcqrh)

• form_instances.updated_at is read ONLY by the won-bid case-study version grid (_SQL_WON_BID_CASE_STUDY_VERSION, fi_max term) — appears in NO other concept type’s content_version. Grouped by form id.
• Two won forms exist: 53cde987-…(derived_qa_published=0, NOT a concept) and 28355292-ee21-4b1e-9a3f-21a69e2201a7 (derived_qa_published=4 => the single won-bid concept won-bid/northgate-borough-council). Original updated_at of 28355292 = 2026-07-15 17:37:52.649053+00.
• RUN-3 PLAN: UPDATE form_instances SET updated_at = now() WHERE id = '28355292-ee21-4b1e-9a3f-21a69e2201a7' (only after Run-2 passes). Changes fi_max -> won-bid concept content_version -> memo MISS on exactly that ONE concept; all 17 others memo-HIT. content_version is excluded from identity (MD-4) so this is a re-draft, not add/remove. Reversible (updated_at bump is benign; original value recorded above).

RUN-3 (single-touch) — requestId 08291cb8793442e4810397ddb4910f50 — EXACTLY ONE RE-DRAFT

• Touch: UPDATE form_instances SET updated_at=now() WHERE id=‘28355292-…’ (2026-07-15 17:37:52.649053 -> 2026-07-17 17:04:58.328147). Only won-bid grid reads form_instances -> only won-bid content_version changes.
• Container restarted 17:05:xx (DEFECT C). POST 17:05:22Z -> 202. Completed 17:05:55Z (~33s).
• ONLY 3 OpenRouter calls total (all 200) = the won-bid concept’s re-draft ALONE. The other 17 concepts (incl. now-memoised corvedale) made ZERO drafting calls => memo HIT. This is the “exactly one re-draft on a single content-version touch” proof. No drafting failures.
• embedded=7 / log.md “Changed (7)”: won-bid/northgate-borough-council (the re-draft) + 6 render-cascade concepts (corvedale, ridgeway, st-aldhelm, wyndale, iso-9001, greencycle) that made ZERO draft calls. Bundle-RENDER churn, not re-drafts.

RUN-4 (PURE NO-OP, lead step 4) — requestId 4e4edb3fd154468c9fd25b8ed1eb11bd — ZERO DRAFTS, ZERO CHURN

• Restarted 17:11:xx (DEFECT C). POST 17:11:10Z -> 202. Completed 17:11:11Z (<1s). embedded=0.
• ZERO OpenRouter calls — all 18 concepts memo-HIT (corvedale memoised in Run-2, won-bid in Run-3, rest in Run-1). Zero drafting failures.
• log.md recorded “Run 2026-07-17T17:11:11Z — No changes (no-op re-run)” — NO “Changed” entries, ZERO bundle churn. This is the DEFINITIVE clean BI-18 no-op: unchanged corpus => zero drafting AND zero bundle churn. (My earlier Run-2 was confounded by the outstanding corvedale; THIS is the pure zero-draft no-op the protocol wants.)

DEFECT D — DOWNGRADED to OBSERVATION (cross-link cascade on a REAL re-draft; a true no-op is CLEAN)

• Run-2: 4 draft calls (corvedale) but “Changed (6)”. Run-3: 3 draft calls (won-bid) but “Changed (7)”. BUT the PURE no-op (Run-4) had ZERO churn (“No changes”). So the extra “Changed” in Run-2/3 was cross-link RE-RENDER driven by the one REAL re-draft (corvedale/won-bid) whose changed title/links rippled into concepts that link it — arguably CORRECT (links to a changed concept should update), not spurious churn. The “no churn on a true no-op” half of BI-18 HOLDS (Run-4 proof). Residual question for the code lane: is the cross-link cascade radius bounded/intended? Not a blocker.

DR-060 NOTE + FINAL STATE

• DR-060 operator note prepended to bundle log.md + committed in the BUNDLE repo: HEAD now 35ebdc6c37eefe7135d3a8ebe3a579f781798661 (“docs(log): DR-060 drafting-config rationale”) on parent 54d3bb7d. log.md only (1 file, +15). NOT pushed (no ssh binary; push is the separate human-gated publish action anyway).
• Bundle working tree: 19 concept files modified-uncommitted (the proof’s GLM-5.2 producer output, staged for the separate BI-21 human publish gate). corvedale (2863B) + won-bid (3755B) present.
• App: Up (healthy), health=200. Single-flight lock released. Canonical repo: UNTOUCHED (read-only honoured throughout).
• Run-3 DB touch left in place (form 28355292 updated_at=2026-07-17 17:04:58; original 2026-07-15 17:37:52.649053 recorded above) — consistent with the now-memoised won-bid draft; reversible if a clean-staging reset is wanted.

═══════════ FINAL VERDICT: BI-18 PROVEN (with 3 defects to escalate) ═══════════

• Run-1 cold draft: 17/18 drafted+embedded via GLM-5.2/OpenRouter, ZERO Anthropic spend. (1 fail = DEFECT B corvedale GLM JSON quirk.)
• Run-2 memo-hit: 17/18 zero drafting calls (memo HIT); only the outstanding corvedale re-drafted. Memo persists across restart (LMDB).
• Run-3 single-touch: exactly 1 concept (won-bid) re-drafted on a single content_version touch; 17 memo HIT.
• Run-4 PURE no-op: ZERO drafting calls, ZERO bundle churn (log.md “No changes (no-op re-run)”). The clean unchanged-corpus no-op. => The content_version memo-delta mechanism WORKS: unchanged => zero re-draft + zero churn; one touched record => exactly one re-draft. BI-18 PROVEN.
• Restart-between-runs adaptation (DEFECT C workaround, lead-endorsed): memo-hits ACROSS container restarts (persistent LMDB) are a STRONGER proof than in-process — proven by Run-2 & Run-4.
• Slices A-E all confirmed live: version=1 (cold draft), PRODUCER_MODEL=GLM-5.2 + PRODUCER_BASE_URL + PRODUCER_AUTH_TOKEN + PRODUCER_PROVIDER_ORDER routing + api_key="" x-api-key suppression => GLM-5.2 drafting works end-to-end. DEFECTS TO ESCALATE:
• DEFECT B (GLM output-robustness): 1/18 cold-draft failed on an unescaped control char in GLM’s JSON. Parse-hardening (tolerate/repair control chars, or retry) belongs to a code slice. Side-effect: reconcile logged “Removed (1)” — a transient draft failure de-indexes a previously-good concept (publish should not drop on transient miss).
• DEFECT C (proof-blocker): /producer-run single-use per process (App-already-registered). Needed container restarts between runs. Code fix: reuse/cache the forced App or tear it down per run.
• DEFECT D (bundle-write churn): more concepts “Changed” than re-drafted; bundle-render determinism/coupling. NON-DEFECTS (by design): push-lane not fired by /producer-run (separate human-gated publish); container-lacks-ssh is a gap only for that separate publish lane.

═══ RESUME (post slice-E fix) — FULL PROTOCOL ATTEMPT #3 ═══

Slice E (main 35e1a9c1, checker PASS): producer_async_client() now passes api_key="" in OpenRouter override mode (deployed line 279) — byte-for-byte the FIX-A probe combination. NEW container: cocoindex-platform-staging-slh75m2si7yt60t42tfeajbq-163002365823, image sha-35e1a9c19ccce72b975a79c35330655c5c004cf3, Up (healthy). Pre-flight (16:33:27Z): PRODUCER_MODEL=z-ai/glm-5.2, PRODUCER_BASE_URL=https://openrouter.ai/api, PRODUCER_PROVIDER_ORDER=z-ai, PRODUCER_AUTH_TOKEN/OKF_BUNDLE_DEPLOY_KEY_PATH/CRON_SECRET/PIPELINE_TRIGGER_SECRET SET, ANTHROPIC_API_KEY SET (extraction lane), OKF_CLIENT_ID unset, OKF_BUNDLE_CLASS=showcase. Bundle tree CLEAN @ 54d3bb7d. health=200. Deployed agent_loop.py line 279 confirms kwargs["api_key"] = "".

═══ RESUME (post slice-D fix) — FULL PROTOCOL ATTEMPT #2 ═══

Slice D (main 764f0957, checker PASS): PRODUCER_PROVIDER_ORDER env (comma-sep) -> extra_body={“provider”:{“order”:[…],“allow_fallbacks”:True}} on the single messages.create site in run_tool_use_loop; unset omits the kwarg. NEW container: cocoindex-platform-staging-slh75m2si7yt60t42tfeajbq-153044529580, image sha-764f0957e009678904746d454bffbf3896be7c29, Up (healthy). Pre-flight (15:35:58Z): PRODUCER_MODEL=z-ai/glm-5.2, PRODUCER_BASE_URL=https://openrouter.ai/api, PRODUCER_PROVIDER_ORDER=z-ai, PRODUCER_AUTH_TOKEN/CRON_SECRET/PIPELINE_TRIGGER_SECRET/OKF_BUNDLE_DEPLOY_KEY_PATH all SET, OKF_CLIENT_ID unset, OKF_BUNDLE_CLASS=showcase. Bundle tree CLEAN at HEAD 54d3bb7d (lead reset confirmed). health=200.

RUN-1 #2 (requestId 59a2f235a6854afaaea28d466eada9ee) — DEFECT AGAIN: 18/18 drafts failed, IDENTICAL 404

• POST 15:36:05Z -> 202. Completed 15:36:30Z: ran=True, embedded=0. All 18 concepts: 404 No allowed providers ... requested_providers=['anthropic']. 18/18 hit openrouter.ai/api/v1/messages. Zero Anthropic drafting spend.
• Slice-D code IS deployed & correct (verified in-container /workspace/scripts/…/agent_loop.py): PRODUCER_PROVIDER_ORDER read at import; _provider_routing_extra_body() returns {“provider”:{“order”:[“z-ai”],“allow_fallbacks”:True}}; run_tool_use_loop sets create_kwargs[“extra_body”]. TEST A in-container: PRODUCER_PROVIDER_ORDER=(‘z-ai’,), extra_body correct. So slice-D is NOT the miss.

ROOT CAUSE #2 (DEFINITIVE, PROVEN) — the anthropic SDK sends x-api-key=ANTHROPIC_API_KEY, which OpenRouter routes to the anthropic provider

Diagnostic chain (all read-only, container’s own creds, never printed):

• SDK request capture (httpx event hook): body has provider at TOP LEVEL (correct), URL correct, anthropic-version 2023-06-01. So extra_body passthrough WORKS.
• Raw urllib POST with provider-in-body + Authorization:Bearer(OR token) => 200 repeatedly (even WITH anthropic-version + full x-stainless-* + anthropic user-agent). Simultaneous A/B x3: RAW=200, SDK=404 every round (not transient).
• The SDK sends BOTH authorization: Bearer <PRODUCER_AUTH_TOKEN> AND x-api-key: <ANTHROPIC_API_KEY> (ANTHROPIC_API_KEY IS set in-container for the extraction lane, DR-079). producer_async_client() (agent_loop.py:166-171) passes base_url+auth_token but NOT api_key, so the SDK falls back to reading ANTHROPIC_API_KEY from env and sends it as x-api-key.
• CONFIRM: raw POST Authorization:Bearer(OR) + x-api-key:<ANTHROPIC_API_KEY> + provider-in-body => 404 requested_providers=[‘anthropic’] — exact reproduction. OpenRouter sees the Anthropic-shaped x-api-key and pins the anthropic provider, ignoring the body provider routing.
• This is EXACTLY the docubot precedent’s documented guard (run-agent.ts:52-55 sets ANTHROPIC_API_KEY='' because “the runtime can fall back to authenticating against Anthropic directly”). Slice C/D never suppressed api_key for the producer client.

PROVEN FIX (probe, read-only)

• FIX A: anthropic.AsyncAnthropic(base_url=..., auth_token=..., api_key="") + the slice-D extra_body provider routing => 200 OK, model z-ai/glm-5.2. Passing api_key="" sends an EMPTY x-api-key which OpenRouter ignores → honours provider routing.
• FIX B (default_headers={"x-api-key": None}) => TypeError (SDK rejects None header) — NOT viable.
• => CODE FIX (out of read-only scope; hand to code lane 127/128/135): in producer_async_client() add kwargs["api_key"] = "" whenever OpenRouter mode is active (PRODUCER_BASE_URL or PRODUCER_AUTH_TOKEN set), so the SDK never sends the real ANTHROPIC_API_KEY as x-api-key. Process-local (does NOT mutate shared env → extraction.py keeps ANTHROPIC_API_KEY, DR-079 preserved). Keep slice-D provider routing too (FIX A used both; provider-routing-necessity-without-x-api-key is untested). This is “slice E”.
• NOTE for code lane: api_key="" is falsy — verify the AsyncAnthropic constructor accepts empty string without falling back to env again (probe used the SYNC client Anthropic(api_key="") and it worked → empty string suppressed the env key). Recommend the fix pass api_key="" explicitly and add a unit test asserting no non-empty x-api-key is sent in OpenRouter mode.

VERDICT (attempt #2): BI-18 STILL NOT-PROVEN — Run-1 blocked by ROOT CAUSE #2 (x-api-key). Runs 2 & 3 NOT attempted. No code patched, /producer-run NOT re-fired past the one attempt.

RUN-1 #2 (cold full-corpus draft) — see above

VERDICT (attempt #1): BI-18 NOT-PROVEN — Run-1 blocked by a deploy defect (SUPERSEDED by attempt #2 above). Runs 2 & 3 correctly NOT attempted (failure discipline).

RUN-1 (requestId 017ddcacd7ad4583a7f766c13dbcfb26) — DEFECT: 18/18 drafts failed

• POST /producer-run -> 202 accepted at 14:45:13Z. Completed 14:45:21Z (ran=True, embedded=0).
• ALL 18 concepts failed drafting. Every call routed to https://openrouter.ai/api/v1/messages (18/18) — ZERO calls to api.anthropic.com. So base_url routing WORKS and NO Anthropic drafting spend occurred (all 404s, zero tokens billed; zero OpenRouter inference billed either — 404 before inference).
• Per-concept + summary error (identical for all 18): 404 - {'error': {'message': 'No allowed providers are available for the selected model', 'code': 404, 'metadata': {'available_providers': [...,'z-ai','fireworks','together',...], 'requested_providers': ['anthropic']}}}
• Log line: producer flow: 18/18 concepts failed drafting this run: [...] then /producer-run completed (...): ran=True embedded=0.

ROOT CAUSE (definitive) — producer omits OpenRouter provider routing on the Anthropic-skin endpoint

• Client factory producer_async_client() (agent_loop.py:137-171) sets base_url/auth_token from PRODUCER_BASE_URL/PRODUCER_AUTH_TOKEN correctly. The Anthropic SDK POSTs to {base_url}/v1/messages = OpenRouter’s Anthropic-compat (“Anthropic Skin”) endpoint.
• run_tool_use_loop’s client.messages.create(...) (agent_loop.py:394-403) passes ONLY model/max_tokens/system/messages/tools/tool_choice — NO extra_body provider routing. web_pass.py Pass-2 call is the sibling (same shape, would fail identically).
• OpenRouter’s Anthropic-skin /v1/messages defaults requested_providers: ['anthropic'] when no provider field is in the body; z-ai/glm-5.2 isn’t served by the anthropic provider -> 404.
• The cited docubot precedent (docs-site harness/scripts/docubot/run-agent.ts:43-62) does NOT actually prove GLM through the skin: it DEFAULTS to claude-opus-4-7 (an Anthropic model) and its own comment warns “non-Anthropic models are pass-through, NOT guaranteed — canary a run before trusting a scheduled lane to one.” The producer pinned GLM-5.2 as always-on and hit exactly that unguaranteed path. Neither passes provider routing.

PROVEN FIX (diagnostic probes, read-only, using container’s own PRODUCER_AUTH_TOKEN — token never printed)

• PROBE A: POST openrouter.ai/api/v1/chat/completions, model z-ai/glm-5.2 -> 200 OK (served by provider “WandB”). Proves token+account+model all work; rules out account-level allowlist / bad token.
• PROBE B: POST openrouter.ai/api/v1/messages (Anthropic skin) WITH body provider: {order: ['z-ai'], allow_fallbacks: true}, model z-ai/glm-5.2 -> 200 OK, correct Anthropic-message-shaped response from GLM-5.2.
• => FIX: add OpenRouter provider routing to BOTH producer messages.create call sites via the Anthropic SDK extra_body kwarg, e.g. extra_body={"provider": {"order": ["z-ai"], "allow_fallbacks": True}} (agent_loop.py run_tool_use_loop ~L395; web_pass.py Pass-2 call). Exact code fix is OUT of executor scope (READ-ONLY on canonical code) — escalated.

SIDE-EFFECT (flag): total-draft-failure produced a DESTRUCTIVE (but unpublished, uncommitted) reconcile

• Bundle working tree now dirty (HEAD still 54d3bb7d, NO commit, NO push): M index.md (stripped to okf_version frontmatter + empty header — ALL 18 concept links removed), M log.md (new entry “Run 2026-07-17T14:45:21Z — Removed (18): …”), M ontology.json, ?? context.jsonld (base-only, OKF_CLIENT_ID unset — this part is CORRECT).
• Concept .md bodies themselves are NOT deleted from disk (only de-indexed). The publish/push gate correctly held (embedded=0 -> no git_sync), so nothing destructive was published. Fully recoverable: git -C /cocoindex-state/okf-bundle checkout -- index.md log.md ontology.json && rm /cocoindex-state/okf-bundle/context.jsonld. RECOMMEND orchestrator reset the tree before the post-fix re-run (a successful run regenerates these fresh, so reset is advisable-not-mandatory).

PUSH-LANE PROOF: NOT obtained (gated off by the draft failure — embedded=0, no git_sync fired). Still UNTESTED live.

DR-060 log.md note: MOOT until fix lands — Run-1 wrote no successful drafting entry; adding a version-bump/model-change note now would annotate a failed run. Defer to the post-fix real run.

PRE-FLIGHT

/producer-run handler (server.py ~1160-1254)

• Auth: dual-accept bearer (PIPELINE_TRIGGER_SECRET OR CRON_SECRET). 503 if both unset, 401 wrong bearer.
• Rate-limit guard (shared /walk+/extract window) -> 429.
• Idle-bundle reject: OKF_BUNDLE_DIR unset/missing -> 400.
• Single-flight threading.Lock non-blocking -> 409 if in flight.
• On success: spawns daemon worker thread _run_producer_forced, returns 202 + {status:accepted, requestId} IMMEDIATELY.
• Completion observed ONLY via server log:
  • success: “/producer-run completed (requestId=%s): ran=%s embedded=%d”
  • failure: “/producer-run failed (requestId=%s)” + exception traceback
  • accepted: “/producer-run accepted (requestId=%s)”
  • entering: “/producer-run (requestId=%s) entering forced-run App ‘kh_pipeline_producer_forced’”
• Forced-run App namespace: kh_pipeline_producer_forced (DISTINCT from KH_PIPELINE_APP). Run entirely through ONE surface (/producer-run) for memo comparability. NEVER trigger /walk.

Memo-delta mechanism ({132.38} G-MEMO-DELTA, enrich.py + sources/l_records.py)

• enrich_concept = @coco.fn(memo=True, memo_key={‘source’: None}, version=1).
• content_version = per-backing-table count(*)+max(updated_at), combined by _combine_content_version in fixed table order.
• Unchanged content_version -> memo HIT (skip drafting). Changed -> memo MISS (re-draft that concept).
• version=1 (S481) already bumped in image; NOT yet consumed by any deployed run -> Run-1 is a full cold draft of the whole corpus.
• Concept types + backing grids:
  • topic (scope_tag or domain/subtopic): qa, sd, ri, rl, em, er
  • product (canonical_name): sd, qa, ri
  • company (singleton): sd, ri, em
  • certification (canonical_name): sd, ri (shared) + em (per name)
  • case_study named-client (canonical_name): sd, qa, ri
  • case_study won-bid (form_instances.id where outcome=won): qa, form_instances
• RUN-3 TOUCH CANDIDATE: form_instances is read ONLY by the won-bid grid, grouped by form id -> bumping ONE form_instances.updated_at (outcome=won) flips exactly ONE won-bid concept’s content_version. Alternatively bump updated_at on an isolated single-scope_tag qa. FINAL choice deferred until Run-1 reveals the actual concept set.