AI Telemetry Instrumentation Implementation Plan

AI Telemetry Instrumentation Implementation Plan

Version: 1.1 Date: 2026-04-27 Author: Claude Code (S204 Wave 1, fixes by S204 Wave 3) Status: RATIFIED Source spec: docs/specs/ai-telemetry-instrumentation-spec.md v1.1 RATIFIED Roadmap ref: docs/reference/product-roadmap.md §3.7 (AI Telemetry Instrumentation (S203 WP-B)); supersedes §3.5.5 placeholder. Target sessions: S204 (Phase 0 + Phase 1 + Phase 2); Phase 3 defaults to S205 (per spec §3.7.3 priority Should not Must; OQ-PL2 reverses if Liam wants Phase 3 same-session as 1+2).

For implementation agents: This is the plan that follows S203’s ratified spec. The spec is the source of truth for what and why; this plan is the source of truth for how and in what order. Where they disagree, the spec wins — file an Open Question and stop, do not diverge.

---

0. Change log

Version	Date	Author	Notes
1.0	2026-04-27	Claude Code (S204 Wave 1)	Initial draft from spec v1.1.
1.1	2026-04-27	Claude Code (S204 Wave 3)	Verifier findings applied (docs/audits/wp-b-plan-verifier-findings-2026-04-27.md, 25 findings 2C/5H/8M/6L/4I). All 25 addressed by ID; full mapping in §14 self-review + this row’s bullets. Critical: C-1 entity-storage block boundary corrected 1517-1620 → 1518-1721 (+101 lines, verified awk against lib/ai/classify.ts); C-2 OQ-PL1 withdrawn (source_file IS on content_items:585 Row, :661 Insert, :737 Update — verifier was right, plan v1.0 was wrong). High: H-1 partial-failure mode addressed (plan v1.1 uses option 3 — move only validateEntities(), NOT the whole 1518-1721 block; preserves the existing post-UPDATE invariant; AC1.2-AC9 + OQ-PL8 for spec ratification); H-2 Table A line refs re-verified + recolumnised into 3 unambiguous columns (generateEmbedding call / .insert()-.update() call / payload-construction start); H-3 backfill-classify-content-items.ts mirror claim re-worded honestly (parseArgs scaffold + env-loader carry over; CLI flags are bespoke); H-4 cron auth uses verifyCronAuth() helper from @/lib/cron-auth per all existing cron routes; H-5 effort totals recomputed honest (~23h cumulative with verifier+merge overhead). Medium: M-1 batch-route double-embed defense-in-depth caveat + AC1.3b-AC7 + OQ-PL12 cleanup; M-2 subsumed by C-2 (OQ-PL1 withdrawal); M-3 explicit env-flip cache test pseudocode in §5.2; M-4 embed rollup uses created_at instead of classified_at (+ OQ-PL13 for embedding_created_at column); M-5 numbered Task 3.1 staging-then-prod sequence (12 steps) + AC3.1-AC4..8; M-6 AC0.3-AC11 partial-mode CSV behaviour; M-7 out-of-scope list re-grepped (smoke-test confirmed exists; shared.ts factory at 220-221 added); M-8 Phase 3 default reworded as defaults-confirm. Low: L-1 dispatch waves renamed Dispatch Wave N (with Group A/B/C/D for sub-categorisation); L-2 reembed-missing-embeddings line numbers (28, 99); L-3 Phase 0c re-runs moved to operational; L-4 in/out totals reconciled with spec §5.6.1; L-5 OQ-PL9 for AC4.2 follow-up location; L-6 v1.1 changelog row added (this entry). Info: I-1..I-4 noted, no action needed. Spec drift escalations: OQ-PL10 (block-boundary), OQ-PL11 (Table A line refs) — Liam decides whether to patch spec to v1.2 or accept plan-v1.1 as documented divergence.

---

1. Goal

Wire the seven AI-telemetry columns on content_items (classification_model, classification_tokens_in, classification_tokens_out, classification_cache_creation_tokens, classification_cache_read_tokens, embedding_model, embedding_tokens) so that:

1. Every NEW item written by either pipeline (TS or Python) lands with the columns populated atomic with the existing content_items write.
2. Existing rows with a known provenance receive a best-effort backfill (classification_model, embedding_model, metadata.telemetry_source = 'backfill'); rows of unknown provenance (583 of 606 prod rows) are triaged by Liam/client BEFORE any model is recorded — no auto-default.
3. A weekly cost-aggregation cron rolls up tokens × per-token rates by model and call-type so Liam can answer “how much did the KB cost this week” without per-item N+1 reads.

---

2. Architecture summary

2.1 Data model (no schema change for Phase 1+2)

The seven telemetry columns already exist on content_items (verified spec §4.1 against supabase/types/database.types.ts:535-553). All nullable. No CHECK constraints. No triggers. No indexes.

Phase 3 adds one new table: cost_aggregations (see §6.3.1 below).

2.2 Write path — atomic single-statement UPDATE

For TS (classify.ts), the existing UPDATE at lib/ai/classify.ts:1463-1466 is extended in-place with the new columns; the embedding’s tokens + model are passed in via a new helper return shape so the same updateData object carries both classification and embedding telemetry. No second UPDATE round-trip is added. This satisfies AC1.1.

For Python (scripts/kb_pipeline/pipeline.py:209-262), the existing INSERT payload is extended with seven new keys when the pipeline ran a classify and produced an embedding.

For the 10 non-classify-time embed callsites, each existing INSERT or UPDATE on content_items is extended to include embedding_model + embedding_tokens. A new helper generateEmbeddingForItem() (in lib/ai/embed.ts or a new lib/ai/embed-with-telemetry.ts) returns { embedding, embedding_model, embedding_tokens } so the callsite spreads three keys instead of one.

2.3 Pass 2 ordering — partial reorder per OQ-5 path (a) refined

validateEntities() currently runs AFTER the Pass 1 UPDATE. Verifier finding C-1 + H-1 (Wave 3): the entity-storage block opening at if (result.entities?.length) actually spans lines 1518-1721 of classify.ts (re-verified via awk '/^ if \(result.entities\?\.length\)/{found=1; start=NR} found && /^ }$/{print start"-"NR; exit}' lib/ai/classify.ts), NOT the 1517-1620 range previously cited (and inherited from spec §5.2). The block contains:

• Deterministic-filter step (1518-1525)
• validateEntities() Pass 2 call (within ~1530-1620)
• Holder-derivation logic (~1620-1680)
• entity_mentions upsert with conflict-retry (~1680-1721)

Refined approach (verifier H-1 option 3): Move ONLY the validateEntities() call itself BEFORE the Pass 1 UPDATE. Capture the returned validated-entity list + Pass 2 token usage in local variables. Run the Pass 1 UPDATE with FOLDED tokens. Then run the delete-existing + entity_mentions upsert (lines 1503-1516 + remainder of the entity-storage block) AFTER the UPDATE — preserving the existing post-UPDATE invariant from the comment block at lines 1483-1502 (“Classification has already succeeded at this point (content_items has been updated above)”). This satisfies AC1.1 (single UPDATE round-trip with all telemetry folded) WITHOUT introducing a new partial-failure mode where entity_mentions is wiped before the Pass 1 UPDATE persists. (Liam’s S203 Wave 4 ratification confirmed default path (a) — this refines (a) per verifier H-1; documented as OQ-PL8 for explicit spec ratification.)

2.4 Cache-hit semantics (embedding_tokens = 0, NOT -cache suffix)

Spec §5.3 H-3 explicitly rejects a synthetic text-embedding-3-large-cache model string. Cache hits return embedding_tokens = 0 and the original model name. Cost-rollup queries SUM embedding_tokens × rate; cache hits contribute zero, which is correct (no API call was billed). The cache data structure is extended to store the model name alongside the embedding so the returned model is stable across AI_EMBEDDING_MODEL env flips mid-process.

2.5 Backfill — per-column idempotency + triage gating

Backfill writes columns ONE-AT-A-TIME per row: a row may receive classification_model and not embedding_model, or vice versa. The candidate query selects rows where EITHER column is NULL; per-row logic decides which columns get set. Rows with NULL metadata.ingestion_source (583 of 606 prod rows) skip BOTH writes and emit to the Phase 0 triage CSV. (Liam’s S203 Wave 4 OQ-3 ratification: NO unknown_provenance auto-tag; NO default-model fallback for NULL-provenance rows.)

2.6 Tech stack reused

• TypeScript: Next.js 16 App Router, supabase-js, sb()/tryQuery() helpers from @/lib/supabase/safe, recordPipelineRun() from @/lib/pipeline/record-run.
• Python: kb_pipeline package, anthropic + openai SDKs.
• Tests: Vitest (unit + integration with dangerouslyDisableSandbox), pytest, existing pipeline-parity guard at __tests__/validation/pipeline-parity.test.ts.
• Migrations: Supabase CLI (/opt/homebrew/bin/supabase migration new + db push --linked), dangerouslyDisableSandbox: true.
• Cron: Vercel cron (existing pattern in app/api/cron/*), recordPipelineRun(), CRON_SECRET env guard.

---

3. Phase overview + dispatch order

Phase	Scope	Effort	Sessions	Blocks downstream?
0	Provenance triage CSV scaffold + Liam/client review (async) + targeted SQL UPDATE batch	~1h script + Liam review (days)	S204 (script + dispatch)	YES — Phase 1 backfill of 583 rows blocked until Phase 0c lands
1	Wire telemetry at 11 callsites + Python pipeline; Pass 2 partial-reorder per H-1; embed.ts return-shape change	~10-10.5h (was 6-8h; bumped per H-5 effort recompute + C-1 Task 1.2 bump)	S204	NO — Phase 1 ships independent of Phase 2; Phase 1 unblocks new items getting telemetry from the moment of merge
2	Backfill 23 attributed rows ('markdown_file') + (post-Phase-0c) backfill remaining attributed rows	~3.5h (script 2.5h + staging 0.5h + initial prod 0.5h; Phase 0c re-runs are operational, see L-3)	S204	NO — Phase 2 ships incremental as Phase 0c attributions land
3	cost_aggregations table migration + weekly rollup cron + initial backfill	~3-4h	Defaults to S205 (deferred per spec §3.7.3 priority Should not Must); OQ-PL2 is now a defaults-confirm (not an open question) — reverses to S204 if Liam wants Phase 3 same-session as 1+2	NO — Phase 3 is strictly additive; depends only on Phase 1 having shipped

Cross-phase dependency:

   Phase 0a (triage CSV) ──→ Liam/client async review ──→ Phase 0c (SQL UPDATE batch)
                                                          │
                                                          ▼
   Phase 1 (instrumentation, parallel)  ────────────────► Phase 2 (backfill)
                                                          │
                                                          ▼
                                                       Phase 3 (cron + table)

Phase 1 does NOT block on Phase 0 — they ship in parallel. Phase 2’s backfill of the 583 NULL-provenance rows is gated on Phase 0c, but the 23 known-markdown_file rows can be backfilled the moment Phase 1 is in prod and the script is ready.

---

4. Phase 0 — Provenance triage workflow

S203 Wave 4 OQ-3 ratification (27/04/2026): No row should have unknown provenance. Backfill emits a CSV; Liam/client reviews row-by-row; targeted SQL UPDATE applies the reviewed metadata.ingestion_source per row; standard backfill re-runs. NO unknown_provenance auto-tag. NO default-model fallback for NULL-provenance rows.

4.1 Triage CSV emission (Phase 0a)

Task 0.1: New script scripts/wp-b-triage-report.ts

File ownership.

File	Mode	Notes
scripts/wp-b-triage-report.ts	NEW	Reads NULL-provenance rows from content_items, writes CSV to scripts/output/wp-b-provenance-triage-{date}.csv.
scripts/output/.gitkeep	NEW (if dir absent)	Output directory does not exist today (ls scripts/output returns “no output dir”). Create directory + add .gitkeep so the script can write into it. Do NOT create a new top-level dir per feedback_sandbox_new_directories.md — scripts/output/ is below the existing scripts/ tree.
__tests__/scripts/wp-b-triage-report.test.ts	NEW	parseArgs, candidate-query shape, CSV row structure (no real DB writes — mock client).

Acceptance criteria.

AC	Description
0.1-AC1	Script writes a CSV with columns: id, title, source_url, source_file, created_by, created_at, current_classification_model, current_embedding_model. Column order locked (per spec §6.7).
0.1-AC2	Sort order: created_at ASC (chronological review).
0.1-AC3	Default output path: scripts/output/wp-b-provenance-triage-{YYYY-MM-DD}.csv. CLI flag --output=<path> overrides.
0.1-AC4	CLI flags: --output, --limit N (default unlimited; useful for dry-run inspections), --env=prod (per CLAUDE.md staging-default convention).
0.1-AC5	Fails fast on missing NEXT_PUBLIC_CLIENT_ID, SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY (per feedback_branding_client_id_env).
0.1-AC6	Prints resolved Supabase URL host portion at startup so operator confirms targeting.
0.1-AC7	Read-only — no content_items writes. Re-runnable any number of times.
0.1-AC8	Counts NULL-provenance rows; prints summary at end. Expected ~583 on prod r at first run (Wave 3 verified count); decreases as Phase 0c lands.
0.1-AC9	Rate-limit unnecessary (read-only SELECT).

Candidate query (verified Wave 3 against prod r — see spec §6.1.1):

SELECT id,
       title,
       source_url,
       source_file,
       created_by,
       created_at,
       classification_model AS current_classification_model,
       embedding_model AS current_embedding_model
FROM content_items
WHERE metadata->>'ingestion_source' IS NULL
ORDER BY created_at ASC
LIMIT $1;

Schema confirmation (re-verified Wave 3, 2026-04-27): source_file IS a typed column on content_items — supabase/types/database.types.ts line 585 shows source_file: string | null inside the content_items Row type (also at 661 in Insert + 737 in Update). Wave 1 plan-time caveat (drafted in v1.0) was based on a misreading; v1.1 withdraws that caveat. The 8-column CSV per spec §6.7 ships as specified — no substitution needed.

Test surface.

• Vitest unit: parseArgs default + --limit + --output + --env=prod; candidate-query .from().select() chain shape; CSV row builder produces correct column order; sentinel “no rows” path emits header-only CSV.
• Manual (one-off): run with --env=prod --output=/tmp/wp-b-triage.csv post-merge; spot-check row count against SELECT COUNT(*) FROM content_items WHERE metadata->>'ingestion_source' IS NULL (live).

Effort: ~1h.

Dispatch: S204 Dispatch Wave 1 single agent (no worktree — main repo is fine for a NEW script that does not touch shared code).

Verification: Single verifier reviews the new script against AC0.1.* + parallel-script idiom (matches backfill-classify-content-items.ts CLI shape) + sandbox concerns (dangerouslyDisableSandbox: true for Bun fetch on writes — though this is read-only so probably unnecessary).

4.2 Liam/client triage review (Phase 0b)

This is NOT a code task. Phase 0b is async human review of the CSV output by Phase 0a. Process:

1. Phase 0a output (CSV) is dispatched to Liam (and onward to client if needed).
2. Liam/client reviews each row, fills in the CORRECT metadata.ingestion_source value per row in a free-text column added to the CSV during review. Acceptable values are the canonical enum from spec §6.1.1 ('manual', 'upload', 'upload_autosplit', 'url_import', 'markdown_file', 'markdown_import', 'stage2_markdown', 'bid_library', 'bid_library_import').
3. Liam returns the reviewed CSV to be processed by Phase 0c.

Estimated wall-clock duration: Days, not hours. Plan must NOT block Phase 1 dispatch on this.

4.3 Targeted SQL UPDATE batch (Phase 0c)

Task 0.3: New script scripts/wp-b-apply-triage.ts

File ownership.

File	Mode	Notes
scripts/wp-b-apply-triage.ts	NEW	Reads reviewed CSV from disk, applies SQL UPDATE per row (single statement: UPDATE content_items SET metadata = COALESCE(metadata, '{}'::jsonb) || jsonb_build_object('ingestion_source', $1) WHERE id = $2).
__tests__/scripts/wp-b-apply-triage.test.ts	NEW	parseArgs, CSV parsing, ingestion_source value validation against canonical enum, UPDATE statement shape (mocked DB).

Acceptance criteria.

| AC | Description | | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | ------------ | | 0.3-AC1 | Reads reviewed CSV by path (--input=<path>). | | 0.3-AC2 | Validates each row’s ingestion_source value against the canonical enum (spec §6.1.1). Rejects unknown values; prints rejected rows + asks operator to fix CSV. | | 0.3-AC3 | Per-row single-statement raw UPDATE merging metadata.ingestion_source; preserves all other metadata keys ( | | operator). | | 0.3-AC4 | --dry-run prints summary + first 5 example UPDATEs without writing. | | 0.3-AC5 | --limit N (cap concurrent updates at, say, 100/batch with 100ms rate-limit between rows; full 583 rows ≈ 60s). | | 0.3-AC6 | --env=prod per CLAUDE.md staging-default convention. | | 0.3-AC7 | Fails fast on missing NEXT_PUBLIC_CLIENT_ID, SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY. | | 0.3-AC8 | Idempotent: re-running on a row already attributed produces no-op (the UPDATE is a write, but the value is the same; spec accepts cost). | | 0.3-AC9 | Emits pipeline_runs row via recordPipelineRun() with pipeline_name='ai_telemetry_provenance_triage', cost=0, items_processed=count. | | 0.3-AC10 | Prints sanity-check counts post-run: SELECT COUNT(*) FROM content_items WHERE metadata->>'ingestion_source' IS NULL. Expected: 0 (or near-0 per Liam discretion). | | 0.3-AC11 | Behaviour on validation failure (verifier M-6). Default: script halts, no rows applied (all-or-nothing — risk-averse default). Flag --allow-partial opts into apply-valid-skip-invalid mode. Rejected rows are written to <input>.errors.csv for re-review; valid rows still pending application (in halt mode) are written to <input>.pending.csv. The script prints exact filenames at exit so the operator can re-run cleanly after fixing the rejected rows. Re-runnability: --allow-partial re-runs against the full input CSV (idempotent per AC0.3-AC8); halt-mode re-runs against .pending.csv after the operator fixes .errors.csv rows + re-merges. |

Test surface.

• Vitest unit: parseArgs; CSV parser handles the column added by Liam during review; ingestion_source validator rejects 'fake_value' + accepts each canonical enum value; UPDATE statement shape matches single-statement pattern (no merge_item_metadata RPC); rate-limit timer fires between rows.
• Vitest integration (__tests__/integration/wp-b-apply-triage.integration.test.ts, dangerouslyDisableSandbox: true): apply against staging fixture rows (3 rows with NULL ingestion_source seeded into staging); confirm metadata merge preserves other keys; confirm row count of NULL-provenance rows decreases.
• Manual: dry-run on prod with reviewed CSV; spot-check 3 rows; commit; live-run; sanity-check post-run.

Effort: ~1h.

Dispatch: S204 Dispatch Wave 1 single agent (or Dispatch Wave 2 — Phase 0a’s CSV must be emitted first so this script can be tested against a realistic input). NO worktree (no shared-code touch).

Verification: Single verifier: enum-match logic, single-statement UPDATE, idempotency (re-run safety), recordPipelineRun call.

4.4 Phase 0 checkpoint

After Phase 0a + 0c land:

• scripts/wp-b-triage-report.ts exists, has tests, dry-run printable.
• scripts/wp-b-apply-triage.ts exists, has tests, dry-run printable.
• First triage CSV dispatched to Liam (or queued for dispatch).
• Phase 1 dispatch can proceed in parallel — does NOT depend on Phase 0c finishing.

---

5. Phase 1 — Wire telemetry at 11 callsites + Python pipeline

5.1 11-callsite re-verification (BEFORE dispatch)

The spec §6.7 Table A enumerates 11 callsites. Re-grep verification (this plan, 2026-04-27 v1.1 fix-pass):

$ grep -rn "generateEmbedding\|\\.insert(insertData)\\|\\.update" lib/ app/ scripts/ --include="*.ts" \
   | grep -v node_modules | grep -v "__tests__"

Re-verified Table A (Wave 3 fix-pass). Three columns separate the ops cleanly so implementation agents do not confuse the embed call line with the insert/update call line. The “payload-construction start” is the line where the literal insertData / updateData object is declared — agents extending the payload edit there.

#	File	generateEmbedding call	.insert() / .update() call	Payload-construction start	Op
1	lib/ai/classify.ts	1432	1463-1466 (UPDATE)	1389 (updateData)	regen-on-classify
2	lib/mcp/tools/content.ts	383	432-435 (INSERT)	401 (insertData)	mcp create_content_item; draft branch (!isDraft at 380) writes both fields absent
3	lib/mcp/tools/governance.ts	422	442-452 (UPDATE)	444 (inline)	mcp publish
4	app/api/items/route.ts	75	175 (INSERT, single line .insert(insertData))	129 (insertData)	web-form create
5	app/api/items/batch/route.ts	282	285-286 (post-INSERT UPDATE) — separate from the main INSERT @ 245 (which extends insertData at 243)	285 (inline .update({ embedding: ... }))	batch Q&A autosplit; the redundant double-embed is flagged in M-1 / 1.3b acceptance below
6	app/api/items/[id]/route.ts	524	540-543 (UPDATE; the .update(updateData) is at line 542)	line 503 region (where updateData is built)	publish-from-review
7	app/api/items/[id]/route.ts	779	780-783 (UPDATE; the .update({ embedding: ... }) is at line 782)	782 (inline)	regenerate after edit
8	app/api/ingest/url/route.ts	108	181 (INSERT, .insert(insertData))	152 (insertData)	url_import (TS route)
9	app/api/upload/route.ts	666	667-670 (post-extract UPDATE, .update({ embedding: ... }) at line 669)	669 (inline)	docx/file upload — separate from the main UPDATE @ 587 that writes content + dedup
10	app/api/bids/[id]/outcome/integrate/route.ts	193	199-228 (INSERT, the .insert({...}) literal at 201-226)	201 ({...} literal inline)	bid integrate (create new)
11	app/api/bids/[id]/outcome/integrate/route.ts	282	283-286 (post-UPDATE, the .update({ embedding: ... }) at line 285)	285 (inline)	bid integrate (re-embed update); separate from content UPDATE @ 255-263

Verified out-of-scope embed callers (do NOT touch):

File	Line(s)	Reason out of scope
lib/content/chunk-store.ts	55	content_chunks not items
lib/intelligence/relevance-scorer.ts	29	scoring no item write
lib/intelligence/pipeline.ts	157	profile embed no item write
lib/mcp/tools/search.ts	108, 248, 486 (3 distinct invocations)	query embed (search-time)
lib/mcp/tools/shared.ts	220-221 (getGenerateEmbedding factory)	indirect — factory returns the function; consumers receive whatever shape generateEmbedding returns
app/api/embed/route.ts	35	search embed
app/api/search/route.ts	34	search embed
app/api/bids/[id]/questions/match/route.ts	114	match-time embed
scripts/batch-reclassify.ts	1093	re-classify CLI; spec §2.2 #2 if extension needed, file separate WP
scripts/embedding-smoke-test.ts	519 (file confirmed exists Wave 3 fix-pass — verifier was wrong)	smoke test
scripts/catalogue-standard-sq.ts	2210	one-off catalogue script
scripts/reembed-missing-embeddings.ts	28 (import), 99 (call)	one-off backfill; supersede with helper if WP-B catches it

Total in-scope: 11. Total out-of-scope = 9 distinct files (counting the 3 search.ts invocations as 1 file). Total out-of-scope invocations = 13 (counting search.ts × 3) — matches spec §5.6.1’s “Total out-of-scope = 9 counting search.ts triple as 1, or 11+ counting each invocation separately”. The 24-callsite count cited in 1.1-AC1 is the upper bound including the shared.ts factory pass-through; bunx tsc --noEmit catches any miss.

5.2 Helper extraction (Task 1.1)

Task 1.1: New helper generateEmbeddingForItem()

File ownership.

File	Mode	Notes
lib/ai/embed.ts	EXTEND	Change generateEmbedding() return type from Promise<number[]> to Promise<{ embedding: number[]; tokens: number; model: string }>. Cache stores { embedding, model, createdAt }. Cache hits return tokens: 0.
lib/ai/embed.ts	EXTEND	Add new exported helper generateEmbeddingForItem(text: string): returns { embedding: string; embedding_model: string; embedding_tokens: number } (embedding stringified for direct DB write; embedding_* field names match content_items columns).
__tests__/lib/ai/embed.test.ts	EXTEND	Cache-hit path returns tokens: 0 + original model name (NOT a -cache suffix); fresh API call returns positive token count + current model; cache stores model alongside embedding so env-flip mid-process returns the original model name.
__tests__/lib/ai/embed-with-telemetry.test.ts	NEW	generateEmbeddingForItem composes correctly; JSON-stringifies embedding; passes through tokens + model.

Acceptance criteria.

AC	Description
1.1-AC1	generateEmbedding() return type changes to { embedding, tokens, model }. All 24 callers (in + out of scope) updated to consume new shape.
1.1-AC2	Cache-hit semantics: embedding_tokens = 0 (NOT a -cache suffix on the model string).
1.1-AC3	Cache stores model alongside embedding (cache structure: { embedding: number[]; model: string; createdAt: number }).
1.1-AC4	New helper generateEmbeddingForItem(text) returns { embedding: string; embedding_model: string; embedding_tokens: number } for direct spread into INSERT/UPDATE payloads.
1.1-AC5	OpenAI usage shape verified at node_modules/openai/resources/embeddings.d.ts:41-50 — read response.usage.total_tokens (prompt_tokens === total_tokens per L-1 note).
1.1-AC6	All 13 out-of-scope callsites continue to use generateEmbedding() (NOT generateEmbeddingForItem). Compile-time-correctness — they consume { embedding } destructure.

Test surface.

• Vitest unit (embed.ts): 6-8 cases — fresh API call returns positive tokens + correct model; cache hit returns 0 tokens + cached model; cache stores model; env-flip post-cache returns the cached (NOT new) model; OpenAI 400 error propagates; empty-string input handled.
• Vitest unit (embed-with-telemetry.ts): 3 cases — composes correctly; JSON-stringifies embedding; tokens + model pass through.

Cache-hit cross-env-flip test pseudocode (verifier M-3 — explicit spec). The env-flip test verifies that a cached entry computed under old AI_EMBEDDING_MODEL=text-embedding-3-large is returned with that ORIGINAL model name even after the env flips to a different value. The cache stores { embedding, model, createdAt }; reads return the cached model (NOT a fresh getEmbeddingModel() call):

it('cache hit returns originally-cached model after env flip', async () => {
  vi.stubEnv('AI_EMBEDDING_MODEL', 'text-embedding-3-large');
  const r1 = await generateEmbedding('test text');
  expect(r1.model).toBe('text-embedding-3-large');
  expect(r1.tokens).toBeGreaterThan(0);

  // Env flip mid-process — does NOT invalidate the cache entry
  vi.stubEnv('AI_EMBEDDING_MODEL', 'text-embedding-3-small');

  const r2 = await generateEmbedding('test text'); // cache HIT
  expect(r2.tokens).toBe(0); // cache-hit semantic, NOT a fresh API call
  expect(r2.model).toBe('text-embedding-3-large'); // STILL the original
});

Why this matters: A subtle bug where generateEmbedding() reads getEmbeddingModel() on EVERY call (instead of returning the cached model) would silently mis-attribute cache hits to the post-flip model. This test catches that. Implementation agents must NOT mock getEmbeddingModel — that would mask the env-flip resilience semantic.

Effort: ~1.5h. Mechanical — change one return type, add one helper, update tests. The 13 out-of-scope callers re-typecheck because TypeScript infers the new return shape.

Dispatch: S204 Dispatch Wave 2 single agent (after Phase 0a script). Dispatch-Wave-3-style worktree NOT required — this is a single-file change with cascading test updates.

Verification: Strict — every caller of generateEmbedding() updated to either destructure { embedding } (out-of-scope) or use the new generateEmbeddingForItem() helper (in-scope). Verifier runs bunx tsc --noEmit to catch any missed callsite.

5.3 TS classify.ts — Pass 1 instrumentation + Pass 2 reorder (Task 1.2)

Task 1.2: Extend classify.ts Pass 1 UPDATE + reorder Pass 2

File ownership.

File	Mode	Notes
lib/ai/classify.ts	EXTEND	(a) Extend updateData at lines 1389-1401 with 5 classification telemetry fields. (b) Move ONLY the validateEntities() call (NOT the whole entity-storage block which spans lines 1518-1721, verified Wave 3) BEFORE the Pass 1 UPDATE at 1463-1466. Capture returned validated-entity list + Pass 2 token usage in local variables. (c) After the early validateEntities(), fold Pass 2 token totals into updateData.classification_tokens_in/out. (d) Run the Pass 1 UPDATE with folded tokens. (e) Then run the existing delete-then-upsert entity_mentions block (1503-1721) AFTER the UPDATE, against the previously-captured validated-entity list (skip the now-redundant validateEntities call inside it). (f) Use new generateEmbeddingForItem() helper at line 1432; spread into updateData. NB: The relationships block at 1722+ does NOT call validateEntities and is unaffected by this work — it stays in its current post-UPDATE position.
lib/ai/classify.ts	EXTEND	Capture model from getAIModel() at line 1186 into a local; pass to updateData.classification_model.
lib/ai/classify.ts	EXTEND	Modify validateEntities() to return usage?: { input_tokens, output_tokens } (spec §5.2). Function signature change cascades through all callers.
__tests__/lib/ai/classify.test.ts	EXTEND	(a) Pass 1 telemetry columns populate updateData with correct values; (b) validateEntities() invocation occurs BEFORE the UPDATE call AND entity_mentions delete+upsert occurs AFTER (mock supabase, assert call ordering); (c) Pass 2 fold logic — when validate=true and Pass 2 emits non-zero tokens, classification_tokens_in/out are SUMs (Pass 1 + Pass 2); (d) cache fields write ?? null when SDK returns null; (e) partial-failure mode test: Pass 1 UPDATE failure does NOT wipe entity_mentions (validateEntities-only-reorder option 3 invariant).

Acceptance criteria.

AC	Description
1.2-AC1	updateData extended with: classification_model, classification_tokens_in, classification_tokens_out, classification_cache_creation_tokens, classification_cache_read_tokens. Values read from pass1Usage (input_tokens, output_tokens, cache_creation_input_tokens ?? null, cache_read_input_tokens ?? null) + model from getAIModel().
1.2-AC2	Single UPDATE round-trip preserved (AC1.1 from spec). No merge_item_metadata RPC call.
1.2-AC3	Embedding telemetry written via spread of generateEmbeddingForItem() return value into updateData at line 1432-1433 (replaces existing JSON.stringify(embedding) line).
1.2-AC4	Pass 2 fold: validateEntities() call moved BEFORE the UPDATE (capturing validated-entity list + token usage in locals); usage folds into updateData.classification_tokens_in/out BEFORE the UPDATE. The entity_mentions delete+upsert (lines 1503-1721 minus the validateEntities call) STAYS in its current post-UPDATE position to preserve the existing comment-block invariant (“Classification has already succeeded at this point”). (feedback_silent_failure_prevention: still uses sb() / tryQuery() for the UPDATE.)
1.2-AC5	If generateEmbedding() throws (the existing try/catch at 1414-1445), updateData does NOT receive embedding_* fields — partial telemetry is never persisted (AC1.8 from spec).
1.2-AC6	Pre-existing race condition with metadata.ai_temporal_references is NOT amplified (spec §5.1 M-2). Telemetry writes go to dedicated columns, not into metadata.
1.2-AC7	TS↔Python parity: same 5 column names written. (Phase 1 wave-end parity guard verifies this; see Task 1.5.)
1.2-AC8	feedback_silent_failure_prevention: existing sb()-wrapped UPDATE preserved.
1.2-AC9	Partial-failure invariant preserved: Pass 1 UPDATE failure does NOT wipe entity_mentions (which would leave the row with stale classification metadata + missing entities — strictly worse than either pre- or post-state). The delete-existing call at 1503-1506 must NOT fire until AFTER the UPDATE has succeeded. Test 1.2-test-(e) asserts this.

Test surface.

• Vitest unit: ~8-10 cases — Pass 1 telemetry shape; Pass 2 fold shape; Pass 2 fold when validate=false (no fold); Pass 2 fold when validateEntities() throws (best-effort: no fold, Pass 1 tokens only); cache field nulling; entity-storage block runs BEFORE UPDATE (mock call ordering); embedding failure preserves Pass 1 telemetry; model recorded matches getAIModel().
• Vitest integration (__tests__/integration/ai-telemetry-classify.integration.test.ts, with dangerouslyDisableSandbox: true per feedback_test_runners_split and feedback_sandbox_proxy_breaks_python_sdk analogue): classify a fixture item against staging Supabase + real Anthropic; assert all 5 columns populated post-call. Marked REQ_HIGH cost — limit to pre-merge sweep, NOT Stop hook.

Effort: ~2.5-3h (bumped from 2h per verifier C-1: the entity-storage block is 200+ lines not 100+, increasing dependencies + test mocks + semantic links to flag during the partial-reorder). Largest single task in the plan; budget the upper bound for careful test verification + the partial-failure invariant assertion (1.2-AC9).

Dispatch: S204 Dispatch Wave 3 single agent in isolation: "worktree" (because of cascading classify.ts test updates and the reorder risk).

Verification: Adversarial verifier focused on: (1) the partial-reorder didn’t break entity-storage semantics (entities still get persisted post-classify); (2) the single UPDATE still fires with all telemetry; (3) Pass 2 fold math (SUM across Pass 1 + Pass 2); (4) validateEntities() signature cascade — every caller updated; (5) the delete-existing entity_mentions call at 1503-1506 stays AFTER the UPDATE (1.2-AC9 invariant); (6) the relationships block at 1722+ is not modified.

5.4 11 in-scope callsite refactors (Tasks 1.3a-1.3k)

Each callsite refactors to use the new helper. Tasks 1.3a-1.3k may be dispatched as parallel sub-agents in isolation: "worktree" after Task 1.1 (helper) lands, and after Task 1.2 (classify.ts) lands (because classify.ts is callsite #1 in the inventory).

To keep the plan bite-sized, callsite refactors are grouped into 4 groups (A/B/C/D) within Dispatch Wave 3, based on file proximity:

Group A (Dispatch Wave 3 sub-wave DW3.1) — TS API routes (Tasks 1.3a-1.3e — 5 in parallel)

Task	File	Callsite #	Effort
1.3a	app/api/items/route.ts	4	~30 min
1.3b	app/api/items/batch/route.ts	5	~30 min
1.3c	app/api/items/[id]/route.ts (publish)	6	~30 min
1.3d	app/api/items/[id]/route.ts (regen-after-edit)	7	~30 min
1.3e	app/api/ingest/url/route.ts	8	~30 min

Per-task pattern (illustrative — Task 1.3a, file app/api/items/route.ts):

// BEFORE (line 75)
embeddingArray = await generateEmbedding(embeddingText);
embeddingValue = JSON.stringify(embeddingArray);

// AFTER
const embeddingResult = await generateEmbeddingForItem(embeddingText);
embeddingValue = embeddingResult.embedding;
embeddingArray = JSON.parse(embeddingResult.embedding); // dedup expects number[]
// NEW: Capture telemetry for the insert payload
const embeddingTelemetry = {
  embedding_model: embeddingResult.embedding_model,
  embedding_tokens: embeddingResult.embedding_tokens,
};

Then in insertData at line 129-170, ADD:

...(embeddingValue && {
  embedding: embeddingValue,
  embedding_model: embeddingTelemetry.embedding_model,
  embedding_tokens: embeddingTelemetry.embedding_tokens,
}),

Acceptance criteria (uniform per task):

AC	Description
1.3-AC1	Replace generateEmbedding() with generateEmbeddingForItem().
1.3-AC2	embedding_model + embedding_tokens fields added to the existing INSERT or UPDATE payload (spec §6.7 Table A “Atomic-write target” column).
1.3-AC3	Atomic write — no separate UPDATE round-trip beyond what already exists. (Tasks 1.3b, 1.3d, 1.3e, 1.3i — those rows ALREADY have a separate post-INSERT UPDATE — keep the UPDATE structure unchanged; add telemetry to the existing UPDATE payload.)
1.3-AC4	Failure path preserved — if generateEmbeddingForItem() throws, telemetry fields are NOT included in the payload (matches spec AC1.8).
1.3-AC5	Test extended: 1 case asserting telemetry fields populate the payload; 1 case asserting failure path skips telemetry.
1.3-AC6	feedback_silent_failure_prevention: writes via sb() / tryQuery() preserved.

Task 1.3b double-embed defense-in-depth caveat (verifier M-1): The batch route at app/api/items/batch/route.ts currently double-embeds — the post-INSERT UPDATE @ 285-286 generates an embedding then immediately calls classifyContent({force:true}) at line 294 which regenerates the embedding inside classify.ts:1432 and writes both via the Pass 1 UPDATE (Task 1.2’s path). After all Dispatch Wave 3 group A tasks land, classifyContent’s UPDATE will overwrite Task 1.3b’s telemetry fields with the fresher classify-pipeline values.

AC

Description

1.3b-AC7

Task 1.3b telemetry write is defense-in-depth only: if classifyContent later succeeds, classifyContent’s UPDATE wins. If classifyContent fails (e.g. transient error before its own UPDATE fires), Task 1.3b’s telemetry persists for the brief window. Test must NOT assert “post-classify telemetry came from 1.3b” — only “telemetry shape on 1.3b’s UPDATE is correct in isolation”. A follow-up cleanup task (tracked as OQ-PL12) may remove the redundant initial embedding from the batch route entirely.

Test surface (per task):

• Vitest unit: 2 cases — happy-path (telemetry populates payload), failure-path (telemetry skipped on embed error).

Effort: ~30 min each = 2.5h cumulative across 5 tasks (parallelisable to ~30 min wall-clock).

Dispatch: S204 Dispatch Wave 3 group A — 5 parallel sub-agents in isolation: "worktree". Each agent’s first action is git reset --hard main (per CLAUDE.md worktree-stale rule). Cherry-pick (not merge) each branch sequentially after Task 1.2 lands.

Group B (Dispatch Wave 3 sub-wave DW3.2) — MCP tools (Tasks 1.3f-1.3g — 2 in parallel)

Task	File	Callsite #	Effort
1.3f	lib/mcp/tools/content.ts	2	~30 min
1.3g	lib/mcp/tools/governance.ts	3	~30 min

Special case for Task 1.3f (callsite #2 — MCP create with !isDraft gate):

The existing code skips generateEmbedding() entirely when isDraft is true. The plan preserves this; the embedding_model + embedding_tokens fields are only added to insertData when !isDraft (i.e. when the embedding was actually generated). The DRAFT branch leaves both fields absent from the INSERT payload (which means they’ll be NULL on the DB row, by default). This is the correct null-as-discriminator semantics (per spec §5.6.2 M-1):

// Skip embedding for drafts — no telemetry to write either
let embedding: number[] | null = null;
let embeddingTelemetry: { embedding_model: string; embedding_tokens: number } | null = null;
if (!isDraft) {
  try {
    const result = await generateEmbeddingForItem(args.title + ' ' + args.content.slice(0, 5000));
    embedding = JSON.parse(result.embedding);
    embeddingTelemetry = {
      embedding_model: result.embedding_model,
      embedding_tokens: result.embedding_tokens,
    };
  } catch (error) {
    console.error('Failed to generate embeddings:', error);
  }
}

// In insertData (lines 401-420):
...(embedding && embeddingTelemetry && {
  embedding: JSON.stringify(embedding),
  embedding_model: embeddingTelemetry.embedding_model,
  embedding_tokens: embeddingTelemetry.embedding_tokens,
}),

The discriminator embedding_tokens IS NULL unambiguously means “no embedding generated” (in contrast to embedding_tokens = 0 which means cache hit; spec §5.3 H-3).

Acceptance criteria (per task):

AC	Description
1.3f-AC1	When !isDraft, embedding_model + embedding_tokens populate insertData.
1.3f-AC2	When isDraft=true, both fields ABSENT from insertData (NULL on DB row).
1.3f-AC3	Test extended: 1 case (isDraft=true) asserts both fields absent; 1 case (isDraft=false) asserts both present.
1.3g-AC1	Embed-then-commit ordering preserved (publish gate).
1.3g-AC2	UPDATE payload extended with both fields.

Effort: ~1h cumulative.

Dispatch: S204 Dispatch Wave 3 group B — 2 parallel sub-agents in worktrees.

Group C (Dispatch Wave 3 sub-wave DW3.3) — Bid integration (Task 1.3h-1.3i)

Task	File	Callsite #	Effort
1.3h	app/api/bids/[id]/outcome/integrate/route.ts (create new)	10	~30 min
1.3i	app/api/bids/[id]/outcome/integrate/route.ts (re-embed update)	11	~30 min

Note: Both tasks are in the same file. Single agent does both tasks to avoid worktree branch collision (spec §11.1 cross-ref to feedback_post_merge_parallel_edit_drift). Effort: ~1h cumulative.

Group D (Dispatch Wave 3 sub-wave DW3.4) — Upload flow (Task 1.3j)

Task	File	Callsite #	Effort
1.3j	app/api/upload/route.ts	9	~30 min

Effort: ~30 min.

Dispatch: S204 Dispatch Wave 3 group D — single sub-agent in worktree.

5.5 Python pipeline instrumentation (Task 1.4)

Task 1.4: Extend Python kb_pipeline/classify.py + pipeline.py

File ownership.

File	Mode	Notes
scripts/kb_pipeline/classify.py	EXTEND	(a) Add model: str field to ClassificationResult dataclass at lines 660-681. (b) Capture CLASSIFICATION_MODEL (the constant, line 738 passes to messages.create()) into the result. Population: model=CLASSIFICATION_MODEL in the dataclass construction.
scripts/kb_pipeline/pipeline.py	EXTEND	Extend record dict at lines 209-244. (a) When cls is not None (line 232 if cls:), add 5 classification keys to record.update({...}). (b) When embedding is non-None (line 259 if embedding:), extend record["embedding"] block at line 260 with record["embedding_model"] = EMBEDDING_MODEL and record["embedding_tokens"] = result.embed_tokens.
scripts/tests/test_classify.py	EXTEND	1 case — ClassificationResult.model populated correctly.
scripts/tests/test_pipeline.py	EXTEND	1 case — record dict at insert time contains all 5 classification + 2 embedding telemetry keys. Use existing test fixture pattern.

Acceptance criteria.

AC	Description
1.4-AC1	ClassificationResult dataclass has model: str field.
1.4-AC2	classify() populates model=CLASSIFICATION_MODEL.
1.4-AC3	Pipeline record dict at insert time contains: classification_model, classification_tokens_in, classification_tokens_out, classification_cache_creation_tokens, classification_cache_read_tokens.
1.4-AC4	Pipeline record dict at insert time contains: embedding_model, embedding_tokens (when embedding generated).
1.4-AC5	TS↔Python parity preserved: same 7 column names written by both pipelines (Task 1.5 verifies).
1.4-AC6	cache_creation_tokens / cache_read_tokens populate from existing getattr(usage, "cache_creation_input_tokens", 0) or 0 (line 753-754) — preserves current null-safety.

Test surface.

• Pytest unit (test_classify.py): 1 case — ClassificationResult.model == CLASSIFICATION_MODEL.
• Pytest unit (test_pipeline.py): 1 case — record dict at insert time contains all 7 keys (use mocked insert_content_item to capture record).
• Pytest (existing — re-run): test_embed.py confirms no behavioural change.

Effort: ~1h.

Dispatch: S204 Dispatch Wave 2 — single sub-agent in worktree (Python files; isolated from TS work). NB: Python pipeline (Task 1.4) lands in DW2 alongside classify.ts (Task 1.2), NOT alongside the 11-callsite refactors.

Verification: Verifier confirms (a) parity check — Python writes the same column names as TS; (b) cache fields populate from the existing getattr defaults; (c) the test fixture for test_pipeline.py includes the new keys in expected record dict.

Run env note: Python pipeline scripts run with dangerouslyDisableSandbox: true per feedback_sandbox_proxy_breaks_python_sdk — the sandbox ALL_PROXY env breaks the anthropic SDK’s httpx client. Document this in the task brief.

5.6 Pipeline parity guard (Task 1.5)

Task 1.5: Extend pipeline-parity guard test

File ownership.

File	Mode	Notes
__tests__/validation/pipeline-parity.test.ts	EXTEND	Add new assertions: (a) TS classify writes the same 5 telemetry column names that Python writes. (b) Both record embedding_model + embedding_tokens in their insert payloads. (c) Per-pipeline classification_model value resolves correctly (TS: getAIModel(); Python: CLASSIFICATION_MODEL).

Acceptance criteria.

AC	Description
1.5-AC1	Test fails if TS classify omits any of the 5 classification telemetry column names.
1.5-AC2	Test fails if Python pipeline omits any of the 5 classification telemetry column names.
1.5-AC3	Test fails if TS or Python omits embedding_model / embedding_tokens.
1.5-AC4	Test confirms TS resolves classification_model to a value matching getAIModel() (default claude-sonnet-4-6); Python to CLASSIFICATION_MODEL (default claude-opus-4-6). The TS↔Python divergence (spec §1.5 / OQ-1) is documented in the test as expected behaviour pending OQ-1 resolution.
1.5-AC5	Test runs as part of bun run test (NOT integration; no live DB).

Test surface.

• The guard test itself is the test surface. No new tests beyond it.

Effort: ~30 min.

Dispatch: S204 Dispatch Wave 3 sub-wave DW3.5 single agent (no worktree — small change).

Verification: Verifier confirms the test catches the failure mode by deleting one of the columns from the TS path or Python path and observing the test fails.

5.7 Phase 1 checkpoint

After all Dispatch Wave 3 groups A+B+C+D + Tasks 1.1, 1.2, 1.4, 1.5 land:

• bun run test passes (unit tests across all callsites + parity guard).
• bunx tsc --noEmit clean (no callsite missed in Task 1.1’s return-type cascade).
• Pipeline-parity guard passes (Task 1.5).
• Manual verification: ingest a new test item via /api/items POST; query SELECT classification_model, classification_tokens_in, embedding_model FROM content_items WHERE id = <new> — all 7 columns populated.
• Python pipeline manually verified: run a small ingest with --limit 1 against staging; assert telemetry columns populate.
• Sentry: no new error spikes during the first 24h post-merge.

---

6. Phase 2 — Backfill 23 attributed rows + (post-Phase-0c) remainder

6.1 Backfill script (Task 2.1)

Task 2.1: New script scripts/backfill-ai-telemetry.ts

File ownership.

File	Mode	Notes
scripts/backfill-ai-telemetry.ts	NEW	Reuses backfill-classify-content-items.ts’s parseArgs scaffold pattern (manual argv loop with named flags), env-loading approach (process.env.NEXT_PUBLIC_SUPABASE_URL + process.env.SUPABASE_SERVICE_ROLE_KEY), PIPELINE_SERVICE_ACCOUNT_USER_ID constant import, and recordPipelineRun() call structure. CLI flags are bespoke (--dry-run, --limit N, --env=prod — NO --workspace-id since this script is workspace-blind, and NO --content-type since per-column-idempotency does not need filtering). The DEFAULT_LIMIT=50 / MAX_LIMIT=1000 constants do carry over. The analogue’s --workspace-id “anti-runaway” guard is replaced here by the --limit 1000 cap + per-column gating that turns repeat runs into no-ops.
__tests__/scripts/backfill-ai-telemetry.test.ts	NEW	parseArgs (--dry-run, --limit N, --env=prod), candidate-query shape, per-row processing decision matrix (per-column gates + ingestion_source attribution), single-statement raw UPDATE assertion (NO merge_item_metadata RPC call).
__tests__/integration/backfill-ai-telemetry.integration.test.ts	NEW	Live DB integration test against staging fixtures (3 rows seeded with various per-column gating combinations); assert UPDATEs land correctly + idempotency on re-run. Run with dangerouslyDisableSandbox: true per feedback_test_runners_split.

Acceptance criteria (mirroring spec §3.2 + §6.4):

| AC | Description | | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | ---------------------------------------------------- | | 2.1-AC1 | CLI: --dry-run (counts + preview, no writes), --limit N (default 50, cap 1000), --env=prod (per CLAUDE.md staging-default convention). | | 2.1-AC2 | Fails fast on missing NEXT_PUBLIC_CLIENT_ID, SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY (per feedback_branding_client_id_env). | | 2.1-AC3 | Prints resolved Supabase URL host portion at startup. | | 2.1-AC4 | Candidate query — see spec §6.4 “Candidate query”: selects rows where EITHER classification_model IS NULL OR embedding_model IS NULL, with classified_at and embedding IS NOT NULL projections for per-column gating. | | 2.1-AC5 | Per-row processing logic (spec §6.4 steps 1-9) — see §6.1.2 below. | | 2.1-AC6 | Per-column idempotency: skip classification_model write where classification_model IS NOT NULL; skip embedding_model write where embedding_model IS NOT NULL. | | 2.1-AC7 | Single-statement raw UPDATE per row (per spec §6.2 / §4.4) — NO merge_item_metadata RPC call. | | 2.1-AC8 | metadata.telemetry_source = 'backfill' set via JSONB | | merge in the same UPDATE; preserves existing keys. | | 2.1-AC9 | Token columns left NULL (spec §6.3 / option (a)). NO classifyContent({force:true}) call (per feedback_classifier_eval_nondeterminism). | | 2.1-AC10 | Triage CSV emission for NULL-provenance rows: a row hitting “no metadata.ingestion_source” at EITHER decision point is added to scripts/output/wp-b-provenance-triage-{date}.csv; backfill skips it (no UPDATE emitted). | | 2.1-AC11 | recordPipelineRun({ pipelineName: 'ai_telemetry_backfill', cost: 0, itemsProcessed: count, status: 'completed' }) called at end. | | 2.1-AC12 | Rate-limit 100ms between rows (DB-only ops). | | 2.1-AC13 | Run with dangerouslyDisableSandbox: true per CLAUDE.md “Bun fetch hangs on HTTP 204 through sandbox proxy” — supabase-js writes return 204 by default. | | 2.1-AC14 | updated_at is NOT touched by the backfill UPDATE (spec §6.2 — backfill is not a content change). |

6.1.1 Per-row processing decision matrix

Following spec §6.4 steps 1-9, the per-row logic is:

INPUT: row with { id, metadata, classified_at, embedding }
CHECK 1: Is classification_model already NOT NULL? → skip classification_model
CHECK 2: Is metadata.ingestion_source NULL? → triage. Skip both writes.
         (If already on triage CSV, do not re-emit.)
CHECK 3: Is classified_at NULL? → classification_model_to_write = NULL
         (Item never classified — recording a model would be a lie.)
CHECK 4: Else (classified_at NOT NULL + ingestion_source present) →
         classification_model_to_write = §6.1.1 lookup table value
         (e.g. 'markdown_file' → 'claude-opus-4-6')

CHECK 5: Is embedding_model already NOT NULL? → skip embedding_model
CHECK 6: Is embedding NULL? → embedding_model_to_write = NULL
         (Item has no embedding — recording a model would be a lie.)
CHECK 7: Else (embedding NOT NULL + ingestion_source present) →
         embedding_model_to_write = 'text-embedding-3-large'
         (Universal per spec §6.1.2.)

If both writes resolved to "skip", emit no UPDATE (log only).
Else, emit single raw UPDATE with metadata merge + column writes.

6.1.2 Single-statement raw UPDATE template

UPDATE content_items
SET metadata = COALESCE(metadata, '{}'::jsonb)
              || jsonb_build_object('telemetry_source', 'backfill'),
    classification_model = $1,  -- NULL or resolved value
    embedding_model = $2        -- NULL or 'text-embedding-3-large'
WHERE id = $3;

Implemented via supabase-js .rpc() to a new apply_telemetry_backfill function OR via direct .update() with metadata constructed in TS as { ...row.metadata, telemetry_source: 'backfill' }. Recommendation: direct .update() with TS-side merge (simpler; no migration needed). Note: race condition with mid-flight metadata mutations is theoretical (spec §5.1 M-2) — accept-gap rationale is the same as for the classify path.

Test surface.

• Vitest unit: 12-15 cases — parseArgs (default + each flag); candidate query selects rows with EITHER NULL column; per-row decision matrix (rows for each cell of the 4×4 grid: per-column-already-set / per-column-NULL × ingestion-source-present / NULL); per-row UPDATE shape (single-statement, metadata merge correct, NULL handling); rate-limit timer; triage CSV emission gating; re-run idempotency.
• Vitest integration (*.integration.test.ts per feedback_test_runners_split): seed staging with 5 rows (NULL+NULL, NULL+SET, SET+NULL, SET+SET, NULL-provenance); run script; assert UPDATEs land + triage CSV emitted for the NULL-provenance row; re-run; assert no second UPDATE.
• Manual: dry-run on prod (--env=prod --dry-run); confirm count matches live NULL count (NOT hard-coded 606); inspect first 5 example assignments.

Effort: ~1.5h script + ~1h tests = 2.5h.

Dispatch: S204 Dispatch Wave 4 single agent (after Phase 1 lands; new items have telemetry, so the candidate query won’t churn between drafting + execution).

Verification: Adversarial verifier focused on: (a) per-column idempotency correctness; (b) single-statement UPDATE (NO merge_item_metadata); (c) triage CSV gating logic; (d) recordPipelineRun called.

6.2 Phase 2 staging exercise (Task 2.2)

Task 2.2: Run backfill on staging + spot-check

File ownership. No new files. CLI execution + manual verification.

Acceptance criteria.

AC	Description
2.2-AC1	Backfill script run on staging (turayklvaunphgbgscat) against seeded fixtures (3+ rows, per-column gates exercised).
2.2-AC2	Post-run sanity SQL: SELECT id, classification_model, embedding_model, metadata->>'telemetry_source' FROM content_items WHERE metadata->>'telemetry_source' = 'backfill' — confirms expected count + values.
2.2-AC3	Triage CSV emitted for any seeded NULL-provenance row.
2.2-AC4	Re-run produces no-op (per-column idempotency).
2.2-AC5	pipeline_runs row written with pipeline_name='ai_telemetry_backfill', cost=0, items_processed=<count>.

Effort: ~30 min.

Dispatch: S204 Dispatch Wave 4 single agent — after Task 2.1.

6.3 Phase 2 prod execution (Task 2.3)

Task 2.3: Run backfill on prod — staged

File ownership. No new files.

Acceptance criteria.

AC	Description
2.3-AC1	Step 1: Pre-flight check cat supabase/.temp/project-ref per feedback_supabase_cli_silent_apply_failure (even though this is a script, not a migration push, the same CLI-target-confirmation discipline applies).
2.3-AC2	Step 2: Dry-run on prod (--env=prod --dry-run); confirm count matches live NULL count (NOT hard-coded 606). Spot-check first 5 example assignments — 'markdown_file' rows should resolve to claude-opus-4-6.
2.3-AC3	Step 3: Live run with --limit 50 first; verify 50 rows updated. SQL: SELECT classification_model, embedding_model, metadata->>'telemetry_source' FROM content_items WHERE metadata->>'telemetry_source' = 'backfill' LIMIT 50 — expect populated values per spec §6.1.1 lookup table.
2.3-AC4	Step 4: Live run remainder (--limit 1000). Today’s expected: 23 attributed 'markdown_file' rows. (583 NULL-provenance rows are emitted to triage CSV; no UPDATE for them yet.)
2.3-AC5	Step 5: Triage CSV from prod run dispatched to Liam for Phase 0b review.
2.3-AC6	Step 6: Post-Phase-0c (after Liam returns reviewed CSV + Task 0.3 applies metadata.ingestion_source): re-run backfill against newly-attributed rows. Each pass writes only the rows attributed in the prior Phase-0c batch.

Effort: ~30 min initial (the 23 known rows) + 30 min per Phase 0c application batch.

Dispatch: S204 Dispatch Wave 4 single agent.

Verification: Verifier confirms: (a) --env=prod flag was passed; (b) pipeline_runs row written; (c) sanity SQL post-run; (d) triage CSV correctly emitted with 583 rows on first prod run (down to 0 after Liam’s review + Phase-0c application).

6.4 Phase 2 checkpoint

After Tasks 2.1, 2.2, 2.3 land:

• 23 attributed rows backfilled with 'claude-opus-4-6' + 'text-embedding-3-large' + metadata.telemetry_source='backfill'.
• Triage CSV with 583 rows dispatched to Liam.
• As Phase 0c applications land, re-run backfill applies attributions incrementally.
• Sanity SQL: SELECT COUNT(*) FROM content_items WHERE classification_model IS NOT NULL ≥ 23 (and growing as Phase 0c applications land).

---

7. Phase 3 — Cost telemetry hybrid architecture

7.1 cost_aggregations table migration (Task 3.1)

Task 3.1: New migration for cost_aggregations table

File ownership.

File	Mode	Notes
supabase/migrations/<timestamp>_create_cost_aggregations.sql	NEW	CREATE TABLE cost_aggregations (...) per spec §7.2 schema. NO CHECK constraint on model field today (spec §3.7.4 follow-up will tighten when Zod enum is hardened).
supabase/types/database.types.ts	REGEN (deliberate)	Run supabase gen types typescript --project-id rovrymhhffssilaftdwd --schema public post-push, redirect to file. NB: stripping the trailing CLI update notice per reference_supabase_gen_types_notice_leak — strip 2 trailing lines after } as const.

Migration SQL (illustrative — verify against spec §7.2):

CREATE TABLE cost_aggregations (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  period_start TIMESTAMPTZ NOT NULL,
  period_end TIMESTAMPTZ NOT NULL,
  granularity TEXT NOT NULL CHECK (granularity IN ('day', 'week', 'month')),
  model TEXT NOT NULL,
  call_type TEXT NOT NULL CHECK (call_type IN ('classify_pass1', 'embed', 'summary')),
  total_tokens_in BIGINT NOT NULL,
  total_tokens_out BIGINT NOT NULL,
  total_cache_read BIGINT NOT NULL,
  total_cache_creation BIGINT NOT NULL,
  total_cost NUMERIC NOT NULL,
  item_count INT NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE UNIQUE INDEX cost_aggregations_period_unique
  ON cost_aggregations (granularity, period_start, model, call_type);

ALTER TABLE cost_aggregations ENABLE ROW LEVEL SECURITY;

-- Admins read-only; cron service-role inserts via service-key bypass
CREATE POLICY "cost_aggregations_admin_read"
  ON cost_aggregations
  FOR SELECT
  USING (get_user_role() = 'admin');

Acceptance criteria.

AC	Description
3.1-AC1	Migration creates cost_aggregations table with all columns per spec §7.2.
3.1-AC2	Unique index on (granularity, period_start, model, call_type) to prevent duplicate rollup rows for the same period.
3.1-AC3	RLS enabled; admin SELECT policy. Cron writes via service role (bypasses RLS).
3.1-AC4	Migration file created via /opt/homebrew/bin/supabase migration new create_cost_aggregations (CLAUDE.md DDL-via-CLI gotcha — never use MCP execute_sql for DDL).
3.1-AC5	Migration applied via /opt/homebrew/bin/supabase db push --linked with dangerouslyDisableSandbox: true.
3.1-AC6	cat supabase/.temp/project-ref confirms target before push (per feedback_supabase_cli_silent_apply_failure).
3.1-AC7	TS types regenerated via supabase gen types; bunx tsc --noEmit clean (per feedback_db_check_ts_union_paired_widening — TS union widening must accompany DB CHECK widening).
3.1-AC8	Staging-then-prod sequence (per feedback_supabase_cli_silent_apply_failure + verifier M-5): apply to staging first, verify, relink to prod, apply to prod. Numbered steps below.

Numbered apply-to-staging-then-prod sequence (3.1-AC8 detail):

1. /opt/homebrew/bin/supabase migration new create_cost_aggregations (creates dated file).
2. Author SQL per spec §7.2 + this plan’s illustrative SQL above.
3. cat supabase/.temp/project-ref → expect turayklvaunphgbgscat (staging). If wrong, /opt/homebrew/bin/supabase link --project-ref turayklvaunphgbgscat.
4. /opt/homebrew/bin/supabase db push --linked → applies to staging (dangerouslyDisableSandbox: true).
5. Verify schema on staging via MCP execute_sql: SELECT 1 FROM cost_aggregations LIMIT 0 should succeed (DDL-only verification — no data writes).
6. /opt/homebrew/bin/supabase link --project-ref rovrymhhffssilaftdwd → relink to prod.
7. cat supabase/.temp/project-ref → expect rovrymhhffssilaftdwd (prod).
8. /opt/homebrew/bin/supabase db push --linked → applies to prod (dangerouslyDisableSandbox: true).
9. Regenerate types: SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co /opt/homebrew/bin/supabase gen types typescript --project-id rovrymhhffssilaftdwd --schema public > supabase/types/database.types.ts.
10. Strip 2 trailing CLI-update-notice lines per reference_supabase_gen_types_notice_leak: sed -i '' -e '$d' -e '$d' supabase/types/database.types.ts.
11. bunx tsc --noEmit → expect clean (no type errors). If errors surface, do NOT proceed — reconcile per feedback_no_midsession_type_regen discipline.
12. Commit migration file + regenerated types together as a single commit.

Test surface.

• Vitest unit: No unit tests for migration SQL (Supabase CLI applies it). Schema test in next-task instead.
• Manual: post-push, query SELECT * FROM cost_aggregations LIMIT 1 (empty result expected) + \d cost_aggregations to confirm columns.

Effort: ~30 min.

Dispatch: S204/S205 single agent (after Phase 1 lands; Phase 3 waits for column population).

Verification: Verifier confirms (a) migration applied to correct project (rovrymhhffssilaftdwd); (b) types regenerated correctly; (c) bunx tsc --noEmit clean (per feedback_db_check_ts_union_paired_widening); (d) \d cost_aggregations shape matches spec.

7.2 Weekly rollup cron route (Task 3.2)

Task 3.2: New cron route app/api/cron/cost-aggregations/route.ts

File ownership.

File	Mode	Notes
app/api/cron/cost-aggregations/route.ts	NEW	Vercel cron handler (existing pattern at app/api/cron/review-cadence/route.ts). Computes INSERT INTO cost_aggregations SELECT…GROUP BY for the prior 7-day period.
vercel.json	EXTEND	Add cron schedule: { "path": "/api/cron/cost-aggregations", "schedule": "0 2 * * 0" } (Sunday 02:00 UTC, weekly).
__tests__/api/cron/cost-aggregations.test.ts	NEW	Mock supabase client; assert SQL aggregation call shape; assert recordPipelineRun() called with pipeline_name='cost_aggregations_rollup'.

Cron SQL (illustrative aggregation — for the prior 7-day period):

INSERT INTO cost_aggregations
  (period_start, period_end, granularity, model, call_type,
   total_tokens_in, total_tokens_out, total_cache_read, total_cache_creation,
   total_cost, item_count)
SELECT
  date_trunc('week', NOW() - INTERVAL '7 days') AS period_start,
  date_trunc('week', NOW()) AS period_end,
  'week' AS granularity,
  classification_model AS model,
  'classify_pass1' AS call_type,
  COALESCE(SUM(classification_tokens_in), 0) AS total_tokens_in,
  COALESCE(SUM(classification_tokens_out), 0) AS total_tokens_out,
  COALESCE(SUM(classification_cache_read_tokens), 0) AS total_cache_read,
  COALESCE(SUM(classification_cache_creation_tokens), 0) AS total_cache_creation,
  -- Compute cost via inline pricing — duplicates lib/provenance/pricing.ts logic
  COALESCE(SUM(...), 0) AS total_cost,
  COUNT(*) AS item_count
FROM content_items
WHERE classified_at >= date_trunc('week', NOW() - INTERVAL '7 days')
  AND classified_at < date_trunc('week', NOW())
  AND classification_model IS NOT NULL
GROUP BY classification_model
ON CONFLICT (granularity, period_start, model, call_type) DO NOTHING;

-- Repeat for embedding rollup with call_type='embed', SUM(embedding_tokens):
INSERT INTO cost_aggregations
  (...)
SELECT
  date_trunc('week', NOW() - INTERVAL '7 days') AS period_start,
  date_trunc('week', NOW()) AS period_end,
  'week' AS granularity,
  embedding_model AS model,
  'embed' AS call_type,
  COALESCE(SUM(embedding_tokens), 0) AS total_tokens_in,
  0 AS total_tokens_out,
  0 AS total_cache_read,
  0 AS total_cache_creation,
  COALESCE(SUM(...), 0) AS total_cost,
  COUNT(*) AS item_count
FROM content_items
WHERE created_at >= date_trunc('week', NOW() - INTERVAL '7 days')
  AND created_at < date_trunc('week', NOW())
  -- NB: uses created_at (item-creation timestamp) as proxy for
  -- embedding-generation time, NOT classified_at. Reason: items can
  -- have an embedding without ever being classified (44 NULL classified_at
  -- rows in prod, per spec §1.1); using classified_at filters those
  -- out, under-counting embedding tokens. created_at is more inclusive
  -- and slightly less accurate (item could be created without immediate
  -- embed, or re-embedded later — both rare). Tracked as OQ-PL13 for a
  -- follow-up `embedding_created_at` column once tokens are populated.
  AND embedding IS NOT NULL
  AND embedding_model IS NOT NULL
GROUP BY embedding_model
ON CONFLICT DO NOTHING;

Cache-effectiveness caveat (M-5 from spec §7.2): TS Pass 1 rows have NULL cache fields (TS doesn’t use prompt caching today); Python Pass 1 rows have non-NULL cache fields. The aggregation query must filter on pipeline (or per classification_model) when computing cache effectiveness. This is correct in the per-model aggregation above (one row per model).

Acceptance criteria.

AC	Description
3.2-AC1	Cron route auths via verifyCronAuth(request) helper from @/lib/cron-auth per all existing cron routes (canonical example: app/api/cron/review-cadence/route.ts:28 import + :61 call). Do NOT write a one-off if (request.headers.get('authorization') !== ...) check — that creates per-route drift on CRON_SECRET rotation, audit-log changes, etc.
3.2-AC2	Aggregates the prior 7-day period (date_trunc('week', NOW() - INTERVAL '7 days') to date_trunc('week', NOW())).
3.2-AC3	One row per (model × call_type) — separate INSERT...SELECT for classify_pass1 and embed.
3.2-AC4	ON CONFLICT (granularity, period_start, model, call_type) DO NOTHING — re-run safety.
3.2-AC5	total_cost computed via cost helpers from lib/provenance/pricing.ts (NOT inlined; reuse).
3.2-AC6	recordPipelineRun() called with pipeline_name='cost_aggregations_rollup', cost=0, items_processed=<rows-aggregated>.
3.2-AC7	vercel.json cron schedule: weekly Sunday 02:00 UTC.
3.2-AC8	Granularity is 'week' only for now (spec §7.2 enum supports day/month for future).
3.2-AC9	Cache-effectiveness caveat documented in route file comment block.
3.2-AC10	Per-pipeline (TS vs Python) split deferred — aggregation is per-classification_model which separates them naturally (claude-sonnet-4-6 is TS-only; claude-opus-4-6 is Python-only today).

Test surface.

• Vitest unit: 4-6 cases — auth check (correct + incorrect bearer); aggregation SQL shape (mocked supabase); ON CONFLICT no-op on re-run; recordPipelineRun called.
• Vitest integration (*.integration.test.ts): seed staging with 3 fixtures (different models + classified_at dates); run cron handler; assert cost_aggregations rows match expected; re-run handler; assert no duplicate rows.

Effort: ~2h. Cron pattern is standard; SQL aggregation is moderate complexity.

Dispatch: S205 single agent (after Task 3.1 migration applied).

Verification: Verifier confirms (a) auth via CRON_SECRET; (b) aggregation SQL is correct (manual eyeball + integration test passes); (c) recordPipelineRun called; (d) vercel.json updated.

7.3 Initial backfill of cost_aggregations (Task 3.3)

Task 3.3: Run cron-handler backfill loop for historical periods

File ownership.

File	Mode	Notes
scripts/backfill-cost-aggregations.ts	NEW	One-off script that calls the cron-handler aggregation SQL for each 7-day window from MIN(classified_at) FROM content_items to NOW(). Runs once post-Phase-1.
__tests__/scripts/backfill-cost-aggregations.test.ts	NEW	parseArgs, period iteration, ON CONFLICT no-op.

Acceptance criteria.

AC	Description
3.3-AC1	Iterates 7-day windows from MIN(classified_at) to NOW().
3.3-AC2	Reuses the cron-handler aggregation SQL (extract to a shared helper, e.g. lib/cost/aggregate-period.ts).
3.3-AC3	--dry-run (count windows, no writes); --env=prod.
3.3-AC4	Idempotent — ON CONFLICT DO NOTHING ensures re-runs are no-op.
3.3-AC5	recordPipelineRun() called with pipeline_name='cost_aggregations_initial_backfill'.

Test surface.

• Vitest unit: parseArgs + window iteration logic + ON CONFLICT.
• Manual: run on staging first; spot-check cost_aggregations rows; run on prod (--env=prod).

Effort: ~1h.

Dispatch: S205 single agent — after Task 3.2.

7.4 Phase 3 checkpoint

After Tasks 3.1, 3.2, 3.3 land:

• cost_aggregations table created in prod.
• Cron route deployed; first weekly aggregation runs Sunday 02:00 UTC.
• Initial backfill applied for historical periods (one row per 7-day window × model × call_type).
• Sample query: SELECT model, SUM(total_cost) AS total_cost FROM cost_aggregations GROUP BY model ORDER BY total_cost DESC returns sensible values.
• Sentry: cron job runs without error first 2 cycles.

---

8. Cross-cutting concerns

8.1 Silent-failure prevention (feedback_silent_failure_prevention)

ALL Supabase write paths in this plan use sb() / tryQuery() from @/lib/supabase/safe. The atomic-write UPDATE in classify.ts is already wrapped (line 1463-1468 has explicit updateError check); preserve that. The 11 callsite refactors do NOT introduce new bare .update() calls — they extend existing payloads in-place.

The backfill scripts (Phase 0a, 0c, 2.1) write via sb() — if any UPDATE errors, the tryQuery Result-returning variant captures + logs. Best-effort swallows are NOT used here — backfill UPDATEs are user-visible and must not silently fail (a row backfill failure should surface).

8.2 BRANDING client-id env guards (feedback_branding_client_id_env)

Every NEW backfill script (Phase 0a, 0c, 2.1; Phase 3 if applicable) fails fast on missing NEXT_PUBLIC_CLIENT_ID:

if (!process.env.NEXT_PUBLIC_CLIENT_ID) {
  console.error('FATAL: NEXT_PUBLIC_CLIENT_ID is not set. Refusing to run.');
  process.exit(1);
}

This is defensive — these scripts don’t directly use BRANDING.organisationName, but matches the pattern of every other backfill script and prevents future drift if scripts grow entity-touching logic.

8.3 Classifier-eval non-determinism (feedback_classifier_eval_nondeterminism)

The Phase 2 backfill MUST NOT call classifyContent({force:true}) on prod. Doing so would re-derive holder metadata for certifications, potentially overwriting Liam’s manually-reviewed values. Backfill writes ONLY:

• classification_model (env-default name from spec §6.1.1 lookup table)
• embedding_model (universal 'text-embedding-3-large')
• metadata.telemetry_source = 'backfill' flag

Token columns left NULL per spec §6.3 option (a). New items post-Phase-1 get full per-item telemetry; historical aggregate cost remains in batch logs.

8.4 DB CHECK / TS union paired widening (feedback_db_check_ts_union_paired_widening)

Phase 3’s cost_aggregations table has CHECK constraints on granularity and call_type. After applying the migration:

# 1. Push migration
/opt/homebrew/bin/supabase db push --linked  # dangerouslyDisableSandbox: true

# 2. Regenerate types
SUPABASE_URL=https://rovrymhhffssilaftdwd.supabase.co \
  /opt/homebrew/bin/supabase gen types typescript --project-id rovrymhhffssilaftdwd \
  --schema public > supabase/types/database.types.ts

# 3. Strip trailing CLI update notice per reference_supabase_gen_types_notice_leak
# (2 trailing lines after `} as const`)
sed -i '' -e '$d' -e '$d' supabase/types/database.types.ts

# 4. Verify TS union widening (NEW row type for cost_aggregations should appear)
bunx tsc --noEmit
# Expected: clean exit (no type errors)

If bunx tsc --noEmit reports errors, the type regen has drifted and needs investigation per feedback_no_midsession_type_regen.

8.5 No mid-session type regen (feedback_no_midsession_type_regen)

Type regen is a deliberate task within Task 3.1, NOT a random “let me regen types” step. Outside Task 3.1, do NOT run supabase gen types — the schema is otherwise unchanged in this plan.

8.6 Supabase CLI silent-apply (feedback_supabase_cli_silent_apply_failure)

Before EVERY supabase db push:

cat supabase/.temp/project-ref
# Expected: rovrymhhffssilaftdwd (prod) or turayklvaunphgbgscat (staging)

# If wrong, relink:
/opt/homebrew/bin/supabase link --project-ref <correct>

This applies to Task 3.1 only in this plan (no other migrations).

8.7 Sandbox-proxy breaks Python httpx (feedback_sandbox_proxy_breaks_python_sdk)

Python pipeline scripts (Task 1.4 indirectly — running Python tests with real Anthropic SDK calls) MUST run with dangerouslyDisableSandbox: true. The sandbox ALL_PROXY env breaks the anthropic SDK’s httpx client.

What this means in practice for tests:

• pytest scripts/tests/ for pure-mock tests can run within the sandbox.
• pytest scripts/tests/integration/ (if any test calls real Anthropic API) — NOT in this plan (all Python tests are mocked unit tests).
• The actual Python pipeline (Task 1.4) when run via python3 scripts/ingest.py — disable sandbox.

8.8 CHECK constraint vs app enum drift — the 3.7.4 follow-up dependency

Spec §3.7.4 (out of scope for this plan but flagged) tightens the Zod ingestion_source enum at lib/validation/schemas.ts:247-249 to match the canonical 9-value enum from spec §6.1.1.

Today’s drift: Zod has 4 values; Python writes 5 more bypassing Zod via REST. The Phase 2 backfill is unaffected (it reads from existing metadata.ingestion_source strings, doesn’t validate them via Zod).

Future work: When Zod is tightened, no impact on this plan’s scripts — but new Phase 0c attributions through web-form would reject any value not in the tightened Zod enum. Confirm before tightening that Phase 0c script wp-b-apply-triage.ts uses direct supabase-js .update() (bypasses Zod), not app/api/items/... PATCH route (which would Zod-validate).

8.9 Python regex ASCII parity (feedback_python_regex_ascii_parity)

This plan’s Python tasks (Task 1.4) do NOT add new regex patterns. No re.ASCII flag work needed. If future tasks add regex matching content hashes that cross the language boundary, the flag becomes relevant.

8.10 Audit ALL pipeline entry points (feedback_audit_all_pipeline_entry_points)

The 11-callsite enumeration in §5.1 has been re-verified by grep -rn "generateEmbedding" against the live tree (2026-04-27). The spec’s S203 Wave 3 inventory is the canonical list. Do NOT skip a callsite because “the test passes” — every callsite has a test that asserts telemetry is in the payload. Task 1.5 (parity guard) is the structural guarantee that future callsite additions don’t slip through.

8.11 Worktree hygiene (feedback_worktree_overwrite,

feedback_worktree_branching_stale, feedback_post_merge_parallel_edit_drift)

Dispatch Wave 3 callsite-refactor agents run in isolation: "worktree" per agent. Each agent’s first command: git reset --hard main. Cherry-pick (NOT merge) each branch on main sequentially. After each cherry-pick, run git status on main and clean leaked files (git checkout -- . + git clean -fd).

Same-file collision (Task 1.3h-1.3i): Both touch app/api/bids/[id]/outcome/integrate/route.ts. Single agent does both tasks in one worktree to avoid branch collision.

8.12 Documentation updates (deferred per spec §11.2)

After Phase 1 lands:

• docs/reference/SCHEMA-QUICK-REFERENCE.md — add 7 telemetry columns to content_items schema reference. (This is the “S203 Wave 4 update task” flagged in spec §11.2.)
• docs/reference/state-of-the-product.md — update §3.7 narrative to reflect Phase 1+2 shipped.

These are NOT part of any phase’s task list — they’re Liam-owned via /update-docs skill at session close.

8.13 Test runners split (feedback_test_runners_split,

feedback_integration_test_location)

• Unit tests (under __tests__/lib/, __tests__/api/, __tests__/scripts/): run by bun run test (excludes *.integration.test.{ts,tsx}).
• Integration tests (under __tests__/integration/**): run by bun run test:integration with dangerouslyDisableSandbox: true.

This plan’s integration tests (ai-telemetry-classify, ai-telemetry-embed, backfill-ai-telemetry, wp-b-apply-triage, cost-aggregations cron) all live under __tests__/integration/**. Failure mode (per feedback_integration_test_location): files outside this dir are unreachable from both runners.

---

9. Dispatch Wave order + verification gates

Note (verifier L-1): Sections renamed “Wave N” → “Dispatch Wave N” to disambiguate from S203 Wave 1-4 (the session waves of the spec adversarial-verifier process). Within this plan, “Dispatch Wave N” refers exclusively to task-dispatch waves. Within S203/S204 prompt history, “Wave N” refers to session waves. Implementation agents reading “Dispatch Wave 2 (Pass 2 partial-reorder)” should not confuse this with “S203 Wave 2 (verifier-feedback application)”.

Dispatch Wave 1 (S204) — Phase 0 scripts + helper extraction

Sub-wave	Task	Effort	Type
DW1.1	Task 0.1 (wp-b-triage-report.ts)	1h	Single agent, no worktree
DW1.2	Task 0.3 (wp-b-apply-triage.ts)	1h	Single agent, no worktree
DW1.3	Task 1.1 (helper extraction in embed.ts)	1.5h	Single agent, no worktree

Verification gate after Dispatch Wave 1. Adversarial verifier reviews 3 commits in parallel. Focus: (a) helper return shape stable; (b) all 24 generateEmbedding callers (incl. shared.ts factory) re-typecheck; (c) Phase 0 scripts read-only + idempotent + correct sandbox + env-guards.

Dispatch Wave 2 (S204) — TS classify.ts core + Python pipeline

Sub-wave	Task	Effort	Type
DW2.1	Task 1.2 (classify.ts Pass 1 + Pass 2 partial-reorder per H-1 option 3)	2.5-3h (was 2h; bumped per C-1 + H-1)	Single agent, worktree
DW2.2	Task 1.4 (Python pipeline)	1h	Single agent, worktree

Verification gate after Dispatch Wave 2. Adversarial verifier reviews 2 commits in parallel. Focus: (a) Pass 2 partial-reorder didn’t break entity storage; (b) the entity_mentions DELETE+upsert stayed in its current post-UPDATE position (1.2-AC9 partial-failure invariant); (c) Python parity columns; (d) dangerouslyDisableSandbox flag on Python tests.

Dispatch Wave 3 (S204) — 11 callsite refactors + parity guard

Sub-wave	Task	Effort	Type
DW3.1 (group A)	Tasks 1.3a-1.3e (5 API routes)	2.5h cumulative	5 parallel agents, worktrees
DW3.2 (group B)	Tasks 1.3f, 1.3g (MCP tools)	1h cumulative	2 parallel agents, worktrees
DW3.3 (group C)	Tasks 1.3h+1.3i combined (bid integration, same file)	1h	1 agent, worktree
DW3.4 (group D)	Task 1.3j (upload route)	30 min	1 agent, worktree
DW3.5	Task 1.5 (pipeline-parity guard)	30 min	1 agent, no worktree

Verification gate after Dispatch Wave 3. Adversarial verifier reviews ~10 commits in parallel (cherry-picked sequentially). Focus: (a) no leaked files on main post-cherry-pick; (b) every callsite has a test asserting telemetry in payload; (c) parity guard catches future regressions.

Dispatch Wave 4 (S204) — Backfill

Sub-wave	Task	Effort	Type
DW4.1	Task 2.1 (backfill-ai-telemetry.ts)	2.5h	Single agent, no worktree
DW4.2	Task 2.2 (staging exercise)	30 min	Single agent, no worktree
DW4.3	Task 2.3 (prod execution)	30 min initial + 30 min/Phase-0c batch	Single agent, no worktree

Verification gate after Dispatch Wave 4. Verifier confirms (a) backfill ran on prod (23 attributed rows updated); (b) triage CSV emitted to Liam; (c) sanity SQL post-run.

Dispatch Wave 5 (S205, defaults) — Phase 3 cost telemetry

Sub-wave	Task	Effort	Type
DW5.1	Task 3.1 (migration)	30 min	Single agent (CLI work, no worktree)
DW5.2	Task 3.2 (cron route)	2h	Single agent, no worktree
DW5.3	Task 3.3 (initial backfill)	1h	Single agent, no worktree

Verification gate after Dispatch Wave 5. Verifier confirms (a) migration in prod; (b) cron runs first cycle; (c) initial backfill rows present.

---

10. Open Questions

• OQ-PL1: source_file column on content_items? WITHDRAWN (verifier C-2, 2026-04-27). The Wave 1 plan-time caveat that source_file may be feed_articles-only was based on a misreading of the schema. Re-verified Wave 3 fix-pass: source_file IS a typed column on content_items (supabase/types/database.types.ts:585 Row, :661 Insert, :737 Update). The 8-column CSV per spec §6.7 ships as specified — no substitution needed. No Liam decision required.

• OQ-PL2: Phase 3 deferral. CONVERTED TO DEFAULTS-CONFIRM. Per verifier M-8: plan now defaults Phase 3 to S205 unambiguously (deferred per spec §3.7.3 priority Should not Must). Liam optionally reverses if same-session Phase 3 in S204 is preferred. Default acts in the absence of a decision.

• OQ-PL3: Cron schedule cadence. Spec §7.2 specifies “weekly Sunday 02:00 UTC”. Plan adopts default. Liam confirms: is this the right cadence, or daily/monthly preferred?

• OQ-PL4: Phase 0c re-run backfill triggering. As Liam returns reviewed CSV batches, who re-runs Task 2.3 against newly-attributed rows? Plan defaults to “main session re-runs on each Phase 0c batch”. Liam confirms: dispatch a sub-agent to monitor + re-run on each batch, or main session orchestrates.

• OQ-PL5: Cache-effectiveness rollup query. Spec §7.2 M-5 caveats that TS Pass 1 has NULL cache fields (no prompt caching) while Python has populated cache fields. Plan’s Task 3.2 rollup is per-classification_model which separates the two naturally — but downstream “cost dashboard” queries aggregating across models must filter pipeline correctly. Liam confirms: is the per-model split sufficient for the cost-dashboard use case (next session), or do we need an explicit pipeline column on cost_aggregations?

• OQ-PL6: Pre-flight schema audit before Phase 1 dispatch. Spec §1.5 references git log -- lib/anthropic.ts for any AI_SUMMARY_MODEL env-default changes since the 606 items were ingested + vercel env ls for historical Vercel env var values. Dispatch Wave 2 verifier should request this audit before the 'claude-sonnet-4-6' attribution in spec §6.1.1 is applied to the TS-pipeline rows. Plan does NOT block on this — but flag for Liam.

• OQ-PL7: Test coverage of cache-hit semantics in integration test. The integration test ai-telemetry-embed.integration.test.ts should exercise the cache-hit path (call twice with same input; second call returns 0 tokens + cached model). Plan includes this as a Task 1.1 test case. Confirm: is the 1-hour cache TTL acceptable for the staging integration test, or should the test programmatically clear the cache between assertions?

• OQ-PL8: Pass 2 reorder strategy refinement (verifier H-1, NEW S204 Wave 3 fix-pass). Spec §5.2 says “Reorder the entity-storage block” but the recommended option (3) — move ONLY validateEntities() not the whole block — achieves the same Pass-2-fold semantics WITHOUT introducing a new partial-failure mode where entity_mentions is wiped before Pass 1 UPDATE persists. Spec §5.2 ratification of OQ-5 path (a) was ambiguous on whole-block-vs-validateEntities-only. Liam confirms: does the S203 Wave 4 OQ-5 ratification of (a) imply moving the whole entity-storage block (1518-1721), or only the validateEntities() call? Plan defaults to option 3 (validateEntities-only) per verifier H-1 recommendation. If the spec must update, note as spec-drift escalation OQ-PL10 below.

• OQ-PL9: AC4.2 follow-up location (verifier L-5, NEW S204 Wave 3 fix-pass). Spec §3.4 AC4.2 references docs/reference/product-roadmap.md §3.5.5 OR docs/reference/product-backlog.md “Provenance UI surfacing” depending on Liam’s preference. After plan ratification, the provenance-UI-surfacing follow-up needs a single home. Liam decides: roadmap §3.5.5 sub-note or new product-backlog item (status Could). Default: backlog item, status Could.

• OQ-PL10: Spec drift — entity-storage block boundary (verifier C-1, NEW S204 Wave 3 fix-pass escalation). Spec §5.2 cites the entity-storage block as “lines 1517-1620”; S204 Wave 3 fix-pass re-verified the actual range as 1518-1721 (off by 101 lines). Plan v1.1 carries the corrected range. Spec is RATIFIED; plan must NOT amend the spec. Liam to decide: patch spec to v1.2 with the corrected range, or let plan v1.1 carry the corrected range as a documented divergence (with a comment-block reference to this OQ).

• OQ-PL11: Spec drift — Table A line refs (verifier H-2, NEW S204 Wave 3 fix-pass escalation). Spec §5.6.1 carries some line refs that drift 2-31 lines from actuals (e.g. content.ts INSERT cited at line 411 in spec vs actual line 432; 5 of 11 callsites had drift). Plan v1.1 carries S204-Wave-3-re-verified Table A. Same disposition as OQ-PL10.

• OQ-PL12: Batch-route double-embed cleanup (verifier M-1, NEW S204 Wave 3 fix-pass). app/api/items/batch/route.ts calls generateEmbedding in a post-INSERT UPDATE @ 285-286 and then classifyContent({force:true}) at line 294 which also embeds (Pass 1). After Dispatch Wave 3 lands, the post-INSERT UPDATE’s telemetry is functionally redundant — classifyContent’s UPDATE wins. Liam decides: schedule a follow-up task to remove the redundant initial embedding from the batch route entirely (saves ~one OpenAI call per Q&A autosplit). Default: track as backlog item, status Could — not blocking WP-B.

• OQ-PL13: embedding_created_at column for Phase 3 rollup (verifier M-4, NEW S204 Wave 3 fix-pass). The embed rollup SQL uses created_at (item-creation timestamp) as a proxy for embedding-generation time because embedding_created_at doesn’t exist. Slightly imprecise: re-embedded items count under their original created_at window. Liam decides: schedule a Phase-3-follow-up migration adding embedding_created_at TIMESTAMPTZ populated by the helper at embed-time, then update the rollup to use it. Default: backlog item, status Could — accept the imprecision for v1 of the rollup.

---

11. Risk register

Risk	Severity	Mitigation
Pass 2 partial-reorder breaks entity-storage semantics.	High	Adversarial verifier focuses on this in Dispatch Wave 2 gate. Integration test exercises classify+entities flow end-to-end. Liam can rollback the single Dispatch Wave 2 commit. Per H-1, plan v1.1 uses option 3 (move only validateEntities(), NOT the whole entity-storage block) which preserves the existing post-UPDATE invariant.
Partial-failure mode: Pass 1 UPDATE fails, but entity_mentions already wiped.	High (NEW v1.1 per H-1)	Option 3 reorder ensures the DELETE+upsert at lines 1503-1721 stays AFTER the Pass 1 UPDATE. Test 1.2-AC9 explicitly asserts this invariant. If a future Dispatch Wave 2 agent inadvertently moves the DELETE to BEFORE the UPDATE, the invariant test fails.
11-callsite refactor misses a site (test passes but field is silently absent).	High	Task 1.5 parity guard test explicitly asserts column names; runs on every bun run test. New callsites added in future will be caught. ESLint rule deferred (spec §8.7) but considered.
Phase 0 triage CSV ambiguous to Liam (column meanings).	Medium	Triage CSV header includes column descriptions; spec §6.7 is the documentation. Plan’s Task 0.1 emits a clear startup banner explaining the workflow.
Phase 0c re-application loop drags on forever (Liam never finishes review).	Medium	Plan supports incremental re-runs — every Phase 0c batch unblocks Phase 2 backfill of those rows. Worst case: 583 rows stay unattributed and are NEVER backfilled — provenance UI continues to show env-default model name (acceptable, per spec §9).
Phase 1 deployment introduces regression on prod ingest path (item creation broken).	Critical	Each callsite refactor is a small, tested change. Cherry-pick sequentially with rollback at each step. Sentry monitoring catches first error spike.
cost_aggregations migration breaks build (bunx tsc --noEmit fails post-regen).	Medium	Task 3.1 includes explicit bunx tsc --noEmit step (per feedback_db_check_ts_union_paired_widening).
Cache structure change breaks existing tests (other tests rely on cache shape).	Low	Cache structure is internal to embed.ts; only __tests__/lib/ai/embed.test.ts reads it. Spot-grep confirms no external consumers.
Cron route fails silently (Vercel cron schedule typo).	Medium	recordPipelineRun row provides audit trail; Sentry alerts on cron failure (existing pattern).
Embed rollup undercount due to created_at proxy.	Low (NEW v1.1 per M-4)	Tracked as OQ-PL13. Items with embedding-but-no-classification (44 NULL classified_at rows in prod) now ARE counted via created_at proxy — slight imprecision for re-embedded items only. Acceptable for v1 of the rollup.

---

12. Effort summary

Phase	Tasks	Effort (recomputed Wave 3 fix-pass)
Phase 0	0.1 (triage CSV, ~1h) + 0.3 (apply triage, ~1h)	~2h
Phase 1	1.1 (helper, 1.5h) + 1.2 (classify.ts, 2.5-3h post-C-1 bump) + 1.3a-e (5 API routes × 0.5h = 2.5h) + 1.3f-g (2 MCP tools × 0.5h = 1h) + 1.3h+i combined (bid integration, 1h) + 1.3j (upload, 0.5h) + 1.4 (Python, 1h) + 1.5 (parity guard, 0.5h)	10-10.5h (was 9h)
Phase 2	2.1 (backfill script, 2.5h) + 2.2 (staging, 0.5h) + 2.3 (prod, 0.5h initial)	~3.5h (Phase 0c re-runs moved to operational, see L-3)
Phase 3	3.1 (migration, 0.5h) + 3.2 (cron, 2h) + 3.3 (initial backfill, 1h)	~3.5h
Wave verifier+merge overhead	~1h per Dispatch Wave × 4 waves (sequential cherry-pick + leak-check + verifier sweep)	~4h (NEW row per H-5)
Total cumulative		~23h

Spec’s effort estimate was 13-14h; plan v1.0 said ~18h; plan v1.1 (this fix-pass) recomputes honestly to ~23h cumulative with verifier+merge overhead included. Plan adds:

• New Phase 0 (not in spec’s effort table): ~2h.
• Task 1.2 bumped 2h → 2.5-3h per C-1 (entity-storage block 200+ lines, partial-reorder per H-1 option 3 needs additional invariant test).
• Per-task verification gates between Dispatch Waves: ~2h cumulative within tasks + ~4h cherry-pick + leak-check overhead across 4 dispatch waves (per CLAUDE.md feedback on agent collisions / re-runs).

Wave-parallelisation reduces wall-clock to ~10-13h across 1-2 sessions — but only IF the 5 Dispatch Wave 3 group A worktree agents land first-try. Per CLAUDE.md (feedback_worktree_agent_silent_fail, feedback_worktree_overwrite, feedback_post_merge_parallel_edit_drift), allow re-run budget for ~20-30% of dispatched agents. Treat the spec’s 13-14h as a lower bound; plan v1.1 lands at 16-18h with verifier+merge overhead realistic + ~20% retry budget, mapping to 1.5-2 sessions. This is a more conservative + honest estimate than v1.0’s “8-10h wall-clock” claim.

Sequence (Dispatch Waves — renamed per L-1):

1. S204 Dispatch Wave 1: Phase 0 scripts + helper (3.5h cumulative; ~3h wall-clock).
2. S204 Dispatch Wave 2: classify.ts + Python (3.5-4h cumulative; ~3-3.5h wall-clock).
3. S204 Dispatch Wave 3: 11 callsites + parity guard (5h cumulative; ~2-3h wall-clock with 5 parallel + retry budget).
4. S204 Dispatch Wave 4: Backfill (3.5h cumulative; ~3h wall-clock).
5. S205 (or late S204) Dispatch Wave 5: Phase 3 (3.5h cumulative; ~3.5h wall-clock).

Operational (NOT session task): Phase 0c re-runs sprinkled over days as Liam returns reviewed CSV batches. Track as ongoing operational task, not session effort. Each batch unblocks an incremental Phase 2 backfill of those rows.

---

13. Cross-references

• Spec: docs/specs/ai-telemetry-instrumentation-spec.md v1.1 RATIFIED.
• Verifier findings applied (this v1.1 fix-pass): docs/audits/wp-b-plan-verifier-findings-2026-04-27.md (25 findings 2C/5H/8M/6L/4I).
• Roadmap: docs/reference/product-roadmap.md §3.7 (supersedes §3.5.5).
• Backfill analogue (parseArgs scaffold + env-loader, NOT CLI flag set): scripts/backfill-classify-content-items.ts:74-128 (parseArgs); see §6.1 file ownership notes for what carries over.
• Existing cron pattern: app/api/cron/review-cadence/route.ts:28 (verifyCronAuth import) + :61 (call) — canonical example for §7.2 H-4.
• Pricing helpers: lib/provenance/pricing.ts (used by Phase 3 cron).
• Provenance fallback: lib/provenance/item-provenance.ts:134-147 (preserved per spec §9).
• Anthropic SDK Usage: node_modules/@anthropic-ai/sdk/resources/messages/messages.d.ts:1354-1387.
• OpenAI SDK Usage: node_modules/openai/resources/embeddings.d.ts:41-50.
• merge_item_metadata RPC body: supabase/migrations/20260416222851_fix_merge_item_metadata_stub.sql:13-16 (verifies why we use a raw UPDATE instead).
• Silent-failure pattern: docs/specs/silent-failure-prevention-spec.md.
• Local-development env: docs/runbooks/local-development.md §3 (--env=prod opt-in).
• C-2 schema verification: supabase/types/database.types.ts:585 (Row), :661 (Insert), :737 (Update) — source_file IS on content_items.
• C-1 + H-1 entity-storage block boundary: lib/ai/classify.ts:1518-1721 (verified awk '/^ if \(result.entities\?\.length\)/{found=1; start=NR} found && /^ }$/{print start"-"NR; exit}').

---

14. Self-review

Cross-checked against Plan ACs (in handoff prompt §“Acceptance criteria for the PLAN you write”):

Plan AC	Plan section	Status
1. Phase 0 triage workflow documented end-to-end	§4.1, §4.2, §4.3	DONE — script → CSV → review → SQL UPDATE → Phase 1 dispatch.
2. 11 callsites enumerated with file:line refs (re-grepped)	§5.1	DONE — re-grepped Wave 3 fix-pass; Table A recolumnised into 3 unambiguous columns (generateEmbedding call / .insert() or .update() call / payload-construction start).
3. TS+Python parity for column writes	§5.6 (Task 1.5 parity guard)	DONE.
4. Phase 1 atomic-write strategy (raw UPDATE, no merge_item_metadata RPC)	§2.2, §6.1.2, AC2.1-AC7	DONE.
5. Cache-hit pattern (embedding_tokens=0, no -cache suffix)	§2.4, AC1.1-AC2	DONE — explicit env-flip pseudocode test added per verifier M-3.
6. Phase 3 cron design specified	§7.2 (cadence: weekly Sunday 02:00 UTC; route: app/api/cron/cost-aggregations/route.ts; SQL: per-model GROUP BY; granularity: week, with day/month future-proofed; auth via verifyCronAuth() per H-4)	DONE.
7. Cross-reference §3.7.4 (Zod tighten OPS follow-up)	§8.8	DONE — flagged as out of scope but documented.
8. Each phase decomposed into tasks ≤2h	All Dispatch Wave tables	DONE — largest single task bumped to 2.5-3h (Task 1.2 classify.ts partial-reorder per C-1 + H-1). All other tasks ≤2h. The 2.5-3h budget is required for the partial-failure invariant (AC1.2-AC9) test work; splitting Task 1.2 across two agents would require interleaving with the entity-storage block which would risk worktree collision.
9. Each task has explicit file ownership table (NEW vs EXTEND)	All Task subsections	DONE.
10. Test surface enumerated per task — unit + integration; live-DB tests flagged	All Task subsections + §8.13	DONE — dangerouslyDisableSandbox: true flagged on integration tests.
11. Effort estimate per task + per phase + total	§12	DONE — recomputed Wave 3 fix-pass with verifier+merge overhead row added (~4h across 4 dispatch waves).
12. Open Questions section	§10	DONE — 12 OQs (was 7; new: PL8 partial-reorder ratification, PL9 AC4.2 follow-up location, PL10/PL11 spec-drift escalations, PL12 batch-route double-embed cleanup, PL13 embedding_created_at column for Phase 3). OQ-PL1 WITHDRAWN (verifier C-2).

Cross-checked against Critical Gotchas in handoff prompt:

Gotcha	Plan section	Status
feedback_silent_failure_prevention	§8.1	DONE.
feedback_branding_client_id_env	§8.2, AC0.1-AC5, AC2.1-AC2	DONE.
feedback_classifier_eval_nondeterminism	§8.3, AC2.1-AC9	DONE — backfill writes ONLY telemetry columns; no force re-classify.
feedback_db_check_ts_union_paired_widening	§8.4, AC3.1-AC6	DONE — explicit bunx tsc --noEmit step.
feedback_no_midsession_type_regen	§8.5	DONE — type regen only in Task 3.1.
feedback_supabase_cli_silent_apply_failure	§8.6, AC3.1-AC5	DONE — cat .temp/project-ref before push.
feedback_sandbox_proxy_breaks_python_sdk	§8.7	DONE.
feedback_check_constraint_app_enum_drift	§8.8	DONE — flagged as out of scope but linked.
feedback_python_regex_ascii_parity	§8.9	DONE — no regex work in this plan; flag preserved.
feedback_audit_all_pipeline_entry_points	§8.10	DONE — re-grepped 11 callsites; Task 1.5 parity guard prevents future drift.

---

15. Execution handoff

When this plan is approved by Liam, dispatch order:

S204 Session start:

1. Dispatch Wave 1: 3 single agents in parallel (Task 0.1, 0.3, 1.1) — no worktrees.
2. Verification gate (1 verifier).
3. Dispatch Wave 2: 2 single agents in parallel (Task 1.2, 1.4) — worktrees.
4. Verification gate (1 verifier).
5. Dispatch Wave 3: 9 sub-agents across groups A-D (5+2+1+1) — worktrees, cherry-picked sequentially.
6. Verification gate (1 verifier reviews all 9 commits).
7. Dispatch Wave 4: 1 agent (Task 2.1, 2.2, 2.3 sequential) — no worktree.
8. Verification gate (1 verifier).
9. Final S204 commit + handoff.

S205 Session start (or late S204 if OQ-PL2 reverses): 10. Dispatch Wave 5: 1 agent (Task 3.1, 3.2, 3.3 sequential) — no worktree. 11. Verification gate (1 verifier). 12. Final S205 commit + handoff.

Inter-session work (async):

• Phase 0b: Liam/client triage CSV review.
• Phase 0c re-application: each batch triggers Task 2.3 re-run.