AI Evaluation Phase 4: Human-in-the-Loop Refinement Spec

AI Evaluation Phase 4: Human-in-the-Loop Refinement Spec

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Review status: Updated 08/04/2026 per S155 WP1.3 adversarial review (docs/audits/s155-spec-review-ai-eval-phase4-hitl-refinement.md). Verdict: PASS WITH NOTES → all 17 findings (F-1 through F-17) applied. Phase 4 implementation is hard-dependent on Phase 3 (see §3.1); begins when Phase 3 infrastructure lands.

Goal: Generalise the “signal → prompt improvement” loop across every AI touchpoint in the platform. Aggregate quality signals (user flags, eval regressions, governance findings) into systematic pattern detection, generate concrete prompt change proposals, A/B test them against gold standards, and give every touchpoint a rollback path.

Motivation: The platform currently has one touchpoint with a working feedback loop (Sector Intelligence feed_prompts versioning — admin-only, versioned, rollback-capable) and one touchpoint with an in-progress specific refinement skill (the SI Prompt Refinement Skill — guided flag-to-prompt workflow). Every other touchpoint ships prompt changes by hand-editing skill files or lib/ai/*.ts with no systematic way to:

1. Detect that a touchpoint is producing bad output in aggregate (signal)
2. Translate that signal into a concrete prompt change proposal (improvement)
3. Measure whether the change actually helps before deploying it (A/B)
4. Undo the change cleanly if it regresses (rollback)

Phase 3 (§3.3) provides the regression-detection machinery — baselines, thresholds, CI gates, taxonomy/model/data-model change impact detection. Phase 4 adds the human-in-the-loop refinement layer on top of Phase 3: instead of waiting for a baseline to fail, it proactively mines real-world quality signals, proposes improvements, and gives humans a structured review-and- approve workflow for every touchpoint — not just SI.

Parent documents:

• docs/audits/ai-eval/ai-touchpoint-map-and-improvement-strategy.md §8 Phase 4
• docs/audits/ai-eval/ai-eval-gap-analysis-s149.md §2 Phase 4
• docs/reference/product-roadmap.md §3.4
• docs/reference/sector-intelligence-pathway.md §8 (Review and Refine pattern)

Related specs (prior-art instances of this pattern):

• docs/specs/si-prompt-refinement-skill-spec.md — SI feed flag instance
• docs/specs/entity-classification-prompt-tightening-spec.md — eval-delta instance (being written in parallel; see §3 Dependencies if file absent)

---

1. Overview

1.1 What the phase delivers

Phase 4 is the unified refinement layer for all prompt-driven AI touchpoints in the platform. It delivers four interlocking capabilities:

#	Capability	Roadmap item
1	Cross-touchpoint signal aggregation	§3.4.1 Flag pattern analysis
2	AI-assisted improvement proposal generation	§3.4.2 Suggestion system
3	Parallel A/B execution against gold standards	§3.4.3 A/B testing
4	Unified rollback across all touchpoint storage models	§3.4.4 Rollback

The phase is deliberately architectural: it establishes the shared data model, APIs, and conventions that let specific touchpoint instances plug in. Two specific instances already exist or are being written:

• SI Prompt Refinement Skill (docs/specs/si-prompt-refinement-skill-spec.md) is an SI-scoped instance: signal source = feed_flags; target = feed_prompts (DB-versioned); scope = a single workspace.
• Entity Classification Prompt Tightening (docs/specs/entity-classification-prompt-tightening-spec.md) is a classification-scoped instance: signal source = scripts/eval-entity-classification.ts failures; target = lib/ai/skills/classification.md and lib/ai/skills/classification-entity-types.md (git-versioned); scope = whole pipeline.

Phase 4 does not replace either spec. It describes the generalised pattern both are specific instances of, and delivers the shared infrastructure both can rely on.

1.2 The generalised “signal → prompt improvement” pattern

  ┌──────────────────────┐
  │   SIGNAL SOURCES     │   (any stream of quality evidence)
  │                      │
  │  - SI feed_flags     │   (user-surfaced false positives/negatives)
  │  - content review    │   (governance review rejects / flags)
  │  - eval-fixture      │   (gold-standard regressions from Phase 3)
  │    failures          │
  │  - classification    │   (low-confidence classifications)
  │    confidence        │
  │  - bid draft edits   │   (human edits to AI-drafted bid responses)
  └──────────┬───────────┘
             │
             v
  ┌──────────────────────┐
  │   NORMALISER         │   Maps each signal into a common
  │                      │   {touchpoint_id, signal_type, evidence[],
  │                      │    severity, first_seen, last_seen} shape
  └──────────┬───────────┘
             │
             v
  ┌──────────────────────┐
  │   PATTERN DETECTOR   │   Clusters evidence into recurring patterns:
  │                      │   - "16 flags in last 14d all mention X"
  │                      │   - "8 eval failures all in domain Y"
  │                      │   - "5 low-confidence items all Q&A pairs"
  └──────────┬───────────┘
             │
             v
  ┌──────────────────────┐
  │   PROPOSAL GENERATOR │   Claude-assisted: given a pattern and the
  │                      │   current prompt/skill, propose concrete text
  │                      │   changes with rationale and evidence refs
  └──────────┬───────────┘
             │
             v
  ┌──────────────────────┐
  │   A/B EVALUATOR      │   Runs current vs proposed variant against
  │                      │   the touchpoint's gold standard; reports
  │                      │   per-metric delta with statistical CI
  └──────────┬───────────┘
             │
             v
  ┌──────────────────────┐
  │   APPROVAL + DEPLOY  │   Human reviews proposal + A/B results,
  │                      │   approves → creates new version via the
  │                      │   touchpoint's adapter (DB row or git commit)
  └──────────┬───────────┘
             │
             v
  ┌──────────────────────┐
  │   ROLLBACK REGISTRY  │   Every deploy records a rollback entry
  │                      │   (previous version ID + restore action)
  │                      │   per touchpoint adapter
  └──────────────────────┘

Every stage is pluggable. SI and classification differ in their signal source, gold standard, A/B execution, and prompt storage — but the shape is the same. Phase 4 delivers the shared stages (normaliser, pattern detector, proposal generator interface, A/B runner interface, rollback registry) and defines the adapter contracts for new touchpoints.

1.3 What is NOT in scope for Phase 4

• Building the SI refinement UI. That is docs/specs/si-prompt-refinement-skill-spec.md. Phase 4 ensures the SI work slots into the generalised pattern, not the other way around.
• Re-implementing the entity classification tightening cycle. That is docs/specs/entity-classification-prompt-tightening-spec.md. Phase 4 ensures the classification work can register its signals and rollbacks in the shared registry.
• Pre-registering touchpoints that do not yet have a gold standard. Bid drafting (blocked on real bid data, §3.5.1/§1.4), digest generation (no eval, §3.5.3), and summarisation (has an eval but low-quality ROUGE thresholds, §3.2.4) are Phase 5 work. They will plug into Phase 4’s infrastructure once they have a gold standard; Phase 4 provides the adapter spec but does not build those adapters.

---

2. Motivation

2.1 Concrete risks the phase addresses

R1: Signal fragmentation. The platform already collects four different quality signals, each in its own silo:

Signal	Storage	Used by	Fed back into prompts?
SI feed flags	feed_flags table	SI article UI	Only if an admin manually edits the scoring prompt
Governance review flags	content_items.governance_review_status	/api/governance/review	No
Content review flags	speed-review cards in /review	Review UI	No
Eval fixture failures	__tests__/fixtures/eval-baselines/*.json	bun run eval:* CLI	Only if a human reads the output and edits prompts

Only SI has even the beginnings of a loop, and that loop is manual. There is no way to ask “what are users complaining about across all touchpoints this month?” because the signals live in four incompatible stores.

R2: Ad-hoc prompt improvement. When a touchpoint degrades, the current process is: a human notices, a human guesses at a prompt change, a human edits lib/ai/classify.ts or lib/ai/skills/classification.md, a human runs the eval, a human decides whether it helped. This is how entity precision stayed at 43.6% across three sessions (S140, S141, S148 — see docs/audits/ai-eval/ai-eval-gap-analysis-s149.md C2) — nobody ever measured whether specific changes helped, because the measurement loop was not instrumented.

R3: No A/B for eval-driven touchpoints. Phase 3 will catch regressions after they happen. But proactive experimentation — “I think adding this diagnostic question would raise precision by 3 points; let me prove it before I merge” — has no infrastructure. The SI feed_prompts versioning supports this for SI (via snapshot comparison), but not for classification, summarisation, bid drafting, or digest.

R4: Rollback is touchpoint-specific and inconsistent. Rolling back an SI prompt is a DB write (feed_prompts.rolled_back_to_version_id). Rolling back a classification skill is a git revert plus a redeploy. Rolling back a summarisation prompt is a code edit. Rolling back a bid-drafting skill is a code edit. There is no unified “undo the last prompt change for touchpoint X” command, and no history page that shows what changed when for any given touchpoint.

R5: The “build the thing, forget to turn it on” pattern. S150 found multiple cases of backend code shipped without wiring (see CLAUDE.md gotchas). Without a unified registry of which prompts are live for which touchpoints, it is easy to ship a new lib/ai/skills/*.md file that never gets loaded — as happened with classification-entity-types.md (C1 in the gap analysis: 338 lines of entity guidance, never sent to the LLM).

2.2 Why now

Phase 2 is substantially complete; Phase 3 is the next spec in flight; Phase 4 is blocked on Phase 3 (see §3). The timing is to have the Phase 4 architecture documented before the SI refinement skill lands, so the SI work can slot into the generalised shape from day one rather than being retrofitted later.

The SI Prompt Refinement Skill and the classification tightening work are both valuable on their own and do not need to wait for Phase 4 implementation. But they DO need to agree on the shape of the signals they emit and the adapters they plug into, so the Phase 4 infrastructure can harvest both when it lands.

---

3. Dependencies

3.1 Phase 3 dependency (explicit, hard)

Phase 4 depends on Phase 3 (docs/specs/ai-eval-phase3-regression-infrastructure-spec.md) being at least partially implemented. The dependencies are concrete:

Phase 4 item	Depends on Phase 3 deliverable
§3.4.1 Flag pattern analysis	The cross-touchpoint baseline and comparison infrastructure Phase 3 establishes — Phase 4 treats eval-fixture regressions as one signal source and needs Phase 3’s machinery to detect them
§3.4.3 A/B testing	Phase 3’s regression detection is essentially a one-sided comparison (current vs baseline). A/B is a two-sided comparison (variant A vs variant B), reusing Phase 3’s metric computation and statistical threshold logic
§3.4.4 Prompt rollback	Phase 3’s baseline lifecycle (save, load, compare) is the model for Phase 4’s prompt-version lifecycle (save, activate, rollback). Phase 4 should reuse lib/eval/baseline.ts patterns

If the Phase 3 spec is not yet in the repo when this spec is implemented, treat it as a prerequisite. Phase 4 cannot ship without Phase 3’s shared regression infrastructure, although the SI and classification specific instances CAN ship independently (they have their own localised measurement paths — SI via feed_prompts snapshots, classification via bun run eval:classification and bun run eval:entity).

3.2 Specific-instance specs (parallel, compatible)

Phase 4 is designed to NOT block either of the parallel specific-instance specs being written in S154. Both should land independently, and Phase 4 harvests them when it lands:

Spec	Status	Relationship to Phase 4
docs/specs/si-prompt-refinement-skill-spec.md	Exists in S154	Specific instance of §3.4.1 + §3.4.2 for SI only. Ships first; Phase 4 extracts the abstractions afterwards.
docs/specs/entity-classification-prompt-tightening-spec.md	Being drafted in S154 (may not exist yet)	Specific instance of §3.4.1 + §3.4.2 for classification only, eval-delta-driven. Phase 4’s §3.4.2 must accommodate its iteration protocol when it lands.

Note on the classification tightening spec: if the file does not yet exist when §3.4.2 implementation starts, the implementer MUST read whatever version exists at that time, extract the signal-emission contract (what evidence the classification tightening spec produces per iteration), and ensure the Phase 4 proposal generator can consume it. Do not reimplement classification tightening — it is a separate spec.

3.3 Storage abstraction dependency (internal)

§3.4.4 is blocked on an architectural decision about prompt storage abstraction. Current state:

Touchpoint	Prompt storage	Change mechanism	Rollback mechanism
SI relevance scorer	feed_prompts DB table	POST /api/intelligence/workspaces/[id]/prompts	DB row + prompt_version_id FK
Classification (TS)	lib/ai/skills/classification.md + classification-entity-types.md	Git commit	git revert + redeploy
Classification (Python)	docs/reference/classification-prompt.md	Git commit + bun run sync:taxonomy	git revert + sync
Summarisation (TS/Py)	Hardcoded string in lib/ai/summarise.ts + scripts/kb_pipeline/summarise.py	Code edit	Code revert
Bid drafting	lib/ai/skills/bid-writing.md + lib/ai/skills/uk-procurement.md	Git commit	git revert + redeploy
Digest generation	Hardcoded programmatic prompt in lib/ai/digest.ts	Code edit	Code revert
Quality check	2-line hardcoded string in lib/ai/quality-check.ts	Code edit	Code revert
Question extraction	Hardcoded strings in lib/ai/extract-questions.ts	Code edit	Code revert

Three different storage models (DB, skill files, inline strings), each with a different version/rollback mechanism. Phase 4 must either:

Option A: Unify storage. Migrate all prompts to the DB-versioned model (like SI feed_prompts). Pro: one mechanism, admin-editable, clean rollback. Con: huge migration; loses the benefits of git-versioned prompts (diff review, PR approval, blame); breaks the “skill file” model the context- engineering skills framework is built around.

Option B: Per-touchpoint adapters. Leave storage where it is. Define a PromptStorageAdapter interface with list_versions(), activate_version(), rollback_to(version_id), current_version() methods. Implement adapters for each storage model: FeedPromptAdapter, SkillFileAdapter, InlinePromptAdapter. Pro: respects each touchpoint’s natural storage; low migration cost. Con: three adapter implementations; the “rollback” UX has edge cases (e.g. rolling back a skill file requires a git commit, which can’t happen from a button click in a web UI without a robot PR mechanism).

Recommendation: Phase 4 should adopt Option B (per-touchpoint adapters) with the following caveat:

• DB-backed touchpoints (SI now, possibly summarisation/digest in future) get full “rollback from web UI” capability.
• Git-backed touchpoints (classification, bid drafting skill files, inline prompts) get a rollback proposal capability — the system generates the diff and rationale; an admin applies the git revert manually (or, if a robot-PR tool becomes available, automatically). The rollback registry tracks the intent and the human confirmation, even when the action is manual.

This is a proposal, not a commitment — Phase 4 implementation begins with finalising this decision in an architecture doc (see Task 4.4.1). The §3.4.4 deliverable is the decision AND the adapter interface, not just code.

3.4 MCP evaluation dependency

• §3.6 docs/specs/mcp-evaluation-spec.md Layer 4 (Functional Correctness) provides the only metric the platform currently has for MCP tool-selection quality. Phase 4 does not modify MCP tool descriptions itself (that is Phase 5 §3.5.6) but A/B testing (§3.4.3) is conceptually the same infrastructure the MCP audit will use to compare tool description variants.

---

4. Architecture

4.1 Shared data model

A new table or typed record store (see Task 4.1.1 for decision) holds normalised quality signals from every touchpoint:

-- Proposed schema
CREATE TABLE prompt_quality_signals (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  touchpoint_id text NOT NULL,   -- e.g. 'classification.ts', 'si.relevance-scorer', 'summarisation.ts'
  signal_type text NOT NULL,     -- 'flag' | 'eval_regression' | 'governance_finding' | 'low_confidence' | 'human_edit'
  severity text NOT NULL,        -- 'critical' | 'high' | 'medium' | 'low'
  evidence_ref jsonb NOT NULL,   -- {source: 'feed_flags', id: uuid} or {source: 'eval', fixture_id: '...', metric: 'precision', delta: -0.05}
  payload jsonb NOT NULL,        -- signal-specific payload (the flag reason, the failing fixture, the edit diff)
  workspace_id uuid,             -- nullable: SI signals are workspace-scoped; classification is not
  created_at timestamptz DEFAULT now(),
  processed_at timestamptz,      -- nullable: set when the signal is included in a pattern analysis
  CHECK (signal_type IN ('flag', 'eval_regression', 'governance_finding', 'low_confidence', 'human_edit'))
);

CREATE INDEX idx_quality_signals_touchpoint ON prompt_quality_signals(touchpoint_id, created_at DESC);
CREATE INDEX idx_quality_signals_unprocessed ON prompt_quality_signals(touchpoint_id) WHERE processed_at IS NULL;

And a separate prompt version registry that tracks which prompt is live for each touchpoint:

CREATE TABLE prompt_version_registry (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  touchpoint_id text NOT NULL,
  version_ref text NOT NULL,     -- adapter-specific: 'feed_prompts:abc-def' | 'git:sha123' | 'inline:v4.5'
  storage_adapter text NOT NULL, -- 'feed_prompts' | 'skill_file' | 'inline'
  activated_at timestamptz NOT NULL DEFAULT now(),
  activated_by uuid REFERENCES auth.users(id),
  rolled_back_from uuid REFERENCES prompt_version_registry(id),
  rolled_back_at timestamptz,    -- when this row was deactivated by a subsequent rollback
  file_sha256 text,              -- F-13: integrity hash captured at activation for git-backed adapters
  notes text,
  workspace_id uuid,             -- nullable
  UNIQUE (touchpoint_id, version_ref, workspace_id)
);

-- Concurrent-activation race protection (per S155 WP1.3 F-12).
-- The composite UNIQUE constraint above blocks duplicate version_refs but
-- does NOT prevent two different version_refs being activated concurrently
-- for the same touchpoint. This partial unique index enforces "at most one
-- live version per (touchpoint_id, workspace_id) at a time":
CREATE UNIQUE INDEX idx_prompt_version_registry_one_live
  ON prompt_version_registry (touchpoint_id, COALESCE(workspace_id, '00000000-0000-0000-0000-000000000000'::uuid))
  WHERE rolled_back_at IS NULL;

-- Activation flow MUST run inside a SERIALIZABLE transaction:
--   1. SELECT current live row WHERE rolled_back_at IS NULL FOR UPDATE
--   2. UPDATE that row to set rolled_back_at = now()
--   3. INSERT the new row with rolled_back_at = NULL
-- This serialises concurrent activations cleanly. The partial unique
-- index is the belt-and-braces backstop if step 2 is forgotten.

Both tables are optional to implement as DB tables — an interim implementation could use JSON files in __tests__/fixtures/ or a lib/refinement/registry.ts in-memory store. The spec requires the shape to be honoured, not necessarily the storage mechanism. DB is recommended for production; files are acceptable for bootstrapping.

4.2 Adapter contracts

lib/refinement/adapters/types.ts

export interface SignalSource {
  touchpointId: string;
  scanForSignals(since: Date): Promise<QualitySignal[]>;
}

export interface PromptStorageAdapter {
  touchpointId: string;
  currentVersion(): Promise<PromptVersion>;
  listVersions(limit: number): Promise<PromptVersion[]>;
  proposeChange(proposal: PromptChangeProposal): Promise<PromptVersionDraft>;
  activate(draftId: string, approvedBy: string): Promise<PromptVersion>;
  rollbackTo(
    versionId: string,
    reason: string,
    actorId: string,
  ): Promise<PromptVersion | RollbackProposal>;
}

export interface GoldStandardEvaluator {
  touchpointId: string;
  runAgainst(
    promptVersion: PromptVersion,
    fixtureSubset?: string[],
  ): Promise<EvalResult>;
  compare(a: EvalResult, b: EvalResult): Promise<AbComparison>;
}

Each touchpoint implements all three interfaces (or registers “not applicable” for storage adapters that cannot rollback without a human in the loop, e.g. git-backed skill files).

4.3 Phasing within Phase 4

Task 4.1: Shared data model + registry    ──┐
Task 4.2: Signal normaliser + scanners    ──┼─→ Task 4.4: Proposal generator
Task 4.3: Per-touchpoint adapters         ──┘       │
                                                     v
                                          Task 4.5: A/B runner
                                                     │
                                                     v
                                          Task 4.6: Rollback registry
                                                     │
                                                     v
                                          Task 4.7: Admin refinement dashboard

Tasks 4.1-4.3 establish the substrate. Tasks 4.4-4.6 implement the refinement loop. Task 4.7 is the cross-touchpoint view.

---

5. Per-item Sections

5.1 §3.4.1 Flag Pattern Analysis Pipeline

Goal: Aggregate quality signals across every touchpoint and surface recurring patterns that suggest prompt issues.

Current state:

• SI flags exist in feed_flags with reason, category, article context, and resolution status. Workspace-scoped. No cross-touchpoint view.
• Governance review findings exist in content_items.governance_review_status and related columns. No structured reason codes; mostly free-text.
• Content speed-review flags exist in the review UI state; NOT persisted for long-term analysis.
• Eval fixture failures are emitted to console by bun run eval:*. NOT persisted in any structured form after the run completes.
• Low-confidence classifications: content_items.classification_confidence (numeric, range 0–1) exists but is not tracked as a signal or surfaced for review. Column name verified against migration 20260326164302_security_performance_fixes.sql:400 per S155 WP1.3 F-1.
• Human edits to AI-drafted bid responses: no diff tracking exists for the AI-to-human edit delta.

Proposed design:

1. Create the prompt_quality_signals table (Task 4.1.1).
2. Implement SignalSource scanners for each signal type (Task 4.1.2). Each scanner runs periodically (cron or on-demand) and writes unprocessed signals into the table.
3. Implement PatternDetector (Task 4.1.3): takes unprocessed signals for a touchpoint and clusters them. Minimum viable clustering:
   • Text similarity clustering (embedding-based) for free-text reasons
   • Categorical grouping (domain, subtopic, entity type, fixture ID)
   • Temporal bursts (sudden increase in signals of the same kind)
4. Expose a workspace-level and global pattern summary via API: GET /api/refinement/touchpoints/[id]/patterns returns the top N patterns with evidence counts, example signals, and severity scores.

Diagnostic phase precursor (per S155 WP1.3 F-5). Phase 4 assumes that quality signals already exist for a touchpoint when iteration begins. The work of making signals exist — running a one-shot baseline measurement to decompose failure modes into buckets before iteration begins — is the “Diagnostic phase” pattern from docs/specs/entity-classification-prompt-tightening-spec.md §3 (D-Q1..D-Q9). This phase is explicitly each touchpoint’s responsibility, not Phase 4’s. Each touchpoint owner must ensure that:

• A baseline measurement run has been executed against the touchpoint’s gold standard (via bun run eval:<touchpoint> --live --validate or equivalent).
• The failure modes have been decomposed into named buckets before any refinement iteration starts.
• The diagnostic outcome is captured in a free-text iteration log file committed to git (so the registry’s structured rows have a sibling prose audit trail; the registry captures intent, the log captures rationale).

If a touchpoint enters Phase 4 without a Diagnostic phase, the PatternDetector will still cluster signals — but the resulting proposals will be weakly grounded and the human reviewer should reject them. This is not a failure mode Phase 4 prevents; it is a process precondition.

Pipeline parity note (per S155 WP1.3 F-6). The classification touchpoint has both a TS pipeline (lib/ai/classify.ts) and a Python pipeline (scripts/kb_pipeline/classifier.py). They share lib/ai/skills/classification.md via the file system, so a SkillFileAdapter change is picked up by both pipelines automatically. However, TS-only filter helper changes (e.g. lib/ai/classify.ts post-processing rules) are not covered by the adapter model — see §5.4 for the FilterHelperAdapter deferral.

Task breakdown:

• Task 4.1.1: Create prompt_quality_signals table + RLS (admin-only read; authenticated insert via adapters). Effort: 1h.
• Task 4.1.2: Implement SignalSource scanners:
  • SiFlagScanner — reads from feed_flags where unresolved
  • GovernanceFindingScanner — reads from content_items where governance_review_status = 'changes_requested'. The CHECK constraint on content_items.governance_review_status allows only pending|approved|reverted|changes_requested|draft (verified at supabase/migrations/20260326164302_security_performance_fixes.sql:435 per S155 WP1.3 F-3). Optionally also scan pending rows older than a configurable freshness threshold to catch stuck review queues.
  • EvalRegressionScanner — reads Phase 3’s baseline regression events
  • LowConfidenceScanner — reads content_items.classification_confidence < 0.5 (the threshold is tunable). Column name verified against migration 20260326164302_security_performance_fixes.sql:400 per S155 WP1.3 F-1.
  • HumanEditScanner — reads the bid draft edit diff table (new; §3.4.1 creates this if §3.5.1 has not yet) Effort: 3-4h.
• Task 4.1.3: Implement PatternDetector with three clustering strategies (text similarity, categorical grouping, temporal burst). Effort: 3h.
• Task 4.1.4: API + admin UI for pattern browsing. Effort: 2h.

Acceptance:

• A query for prompt_quality_signals where touchpoint_id = 'classification.ts' returns at least one signal of each type (flag, eval_regression, low_confidence) once the scanners run against current production data.
• The pattern detector produces clusters from the signals and each cluster has ≥2 evidence items (single-item clusters are suppressed).
• The pattern browsing API returns JSON matching a documented schema.

Dependencies: Phase 3 regression infrastructure (for EvalRegressionScanner). The other scanners do NOT depend on Phase 3.

Effort: 1-2 sessions (matches roadmap estimate).

---

5.2 §3.4.2 Prompt Improvement Suggestion System

Goal: Generate concrete, reviewable prompt change proposals from patterns detected in §3.4.1.

Current state:

• Zero automation. The closest precedent is the SI Prompt Refinement Skill (in progress) which does exactly this for SI only, via a Claude-assisted analysis API.

Proposed design:

A shared library lib/refinement/proposal-generator.ts that takes:

• A pattern cluster (from §3.4.1)
• The current prompt/skill for the touchpoint (via the PromptStorageAdapter)
• The relevant touchpoint’s gold standard subset (if available)

…and produces:

interface PromptChangeProposal {
  touchpointId: string;
  patternId: string;
  evidenceRefs: string[];
  proposedDiff: {
    filePath?: string; // for skill files
    promptField?: string; // for DB-backed prompts
    before: string;
    after: string;
  }[];
  rationale: string; // Claude-generated explanation
  expectedImpact: {
    metric: string;
    direction: 'increase' | 'decrease';
    magnitude: 'small' | 'medium' | 'large';
    confidence: number; // 0-1
  }[];
  risks: string[]; // e.g. ['May regress domain accuracy in X']
  reviewGuidance: string;
}

The proposal generator is Claude-assisted (uses the AI_ANALYSIS_MODEL tier). The prompt template for the proposal generator itself lives in lib/refinement/skills/proposal-generator.md.

SI instance (existing): the SI spec’s /api/intelligence/workspaces/[id]/flags/analyse endpoint produces the same shape. Phase 4 refactors (or wraps) it to emit PromptChangeProposal rather than a bespoke SI-only shape.

Classification instance (parallel spec): the classification tightening spec’s iteration protocol should emit the same shape. If it does not, Phase 4’s implementation of §3.4.2 must adapt.

Task breakdown:

• Task 4.2.1: Define PromptChangeProposal interface and Zod schema. Effort: 30 min.
• Task 4.2.2: Implement proposal-generator.ts core: pattern → Claude call → parsed proposal. Effort: 3h.
• Task 4.2.3: Write lib/refinement/skills/proposal-generator.md (the prompt the generator uses to generate proposals). Effort: 1h.
• Task 4.2.4: Refactor SI flag analysis API to emit PromptChangeProposal shape (or add a wrapper). Effort: 1h.
• Task 4.2.5: Verify classification tightening spec’s iteration protocol produces proposals Phase 4 can consume; adjust the schema if needed. Effort: 1h (more if schema mismatch is discovered).

Acceptance:

• Given a pattern cluster from the classification touchpoint, the generator produces a valid PromptChangeProposal with a non-empty proposed diff and rationale.
• The SI refinement skill can still function end-to-end after the refactor (run the SI skill tests).
• The classification tightening iteration protocol produces proposals that validate against the Phase 4 schema.

Dependencies: §3.4.1. Parallel spec compatibility (classification tightening).

Effort: 1 session (matches roadmap).

---

5.3 §3.4.3 A/B Testing for Classification Prompts (Generalised)

Goal: Run two prompts side-by-side against a touchpoint’s gold standard and report the per-metric trade-off with statistical confidence.

Current state:

• scripts/eval-classification.ts (and the sibling entity, summarisation, search, bid drafting eval scripts) each run ONE prompt against a gold standard and compare against ONE baseline. There is no mechanism for running TWO prompts in parallel.
• SI feed_prompts versioning stores multiple prompts but runs them against different article streams over time, so A/B there is before-vs-after rather than parallel.

Proposed design:

Two different A/B execution paths depending on the touchpoint’s gold standard model:

Path 1: Gold-standard A/B (classification, summarisation, digest, bid drafting). These touchpoints have fixture-backed gold standards. A/B executes both prompts against the same fixture set in parallel and diffs the metrics.

bun run eval:classification --ab --variant-a=current --variant-b=proposed-xyz.md

Implementation:

1. Extend scripts/eval-*.ts to accept --ab mode.
2. Add an optional --variant-a=... and --variant-b=... pair that overrides the default prompt source (skill file path, or a feed_prompts.id, or an inline string).
3. Run the eval loop once per variant, collect two EvalResult values.
4. Compute per-metric delta + 95% CI (Wilson interval for proportions, t-test for continuous metrics) — see llm-evaluation skill for the ABTest class pattern.
5. Report: per-metric delta table, effect size (Cohen’s d), pass/fail recommendation.

Path 2: Live-stream A/B (DEFERRED — SI only; Phase 5 follow-up). SI cannot use a gold standard A/B because its relevance scoring is per-article and the “correct” answer depends on human judgement. Earlier drafts of this spec proposed a “dual-scoring” mode (new articles scored by BOTH the current and a candidate prompt, with a feed_articles.candidate_score column added) so the SI refinement UI could compare the two score sets after N articles.

Phase 4 explicitly does NOT ship Path 2. Per S155 WP1.3 F-4, the SI Prompt Refinement Skill spec (docs/specs/si-prompt-refinement-skill-spec.md lines 1100–1102) lists “A/B testing of prompt versions — running two prompts simultaneously and comparing results” in its Out of Scope section. Quietly re-introducing it here would create a cross-spec scope conflict, require an SI schema migration not coordinated with the SI spec author, and add an SI-specific deliverable to Phase 4’s generalised architecture phase. Instead:

1. Phase 4 ships Path 1 (gold-standard A/B) only.
2. SI’s existing feed_prompts versioning continues to support before-vs-after evaluation via snapshot comparison. This is sufficient for the SI refinement skill’s Phase 1 needs.
3. SI live-stream dual-scoring is recorded as OQ-7 in §12 (Open questions) and deferred to a follow-up Phase 5 work package. Re-opening it requires: (a) coordination with the SI spec author to update SI’s Out of Scope section, (b) a migration adding feed_articles.candidate_score numeric, (c) a new acceptance criterion, and (d) a Phase 5 task allocation.
4. Phase 4’s live-data preview gap (the SI primitive of “re-score the last 20 articles with a candidate prompt and show the user the score deltas”) is captured separately as F-2: it is neither Path 1 (gold-standard A/B) nor Path 2 (parallel dual-scoring), but a third “live-data dry-run preview” variant. See OQ-8 in §12 for resolution direction.

Task breakdown:

• Task 4.3.1: Design the A/B interface (GoldStandardEvaluator.compare) and statistical method choice. Effort: 1h. Decision required (per S155 WP1.3 F-9): does --ab mode live in Phase 3’s scripts/eval-runner.ts wrapper (single-point integration, smaller surface area) or in each scripts/eval-*.ts script individually (more flexibility, more code)? The tasks below assume eval-runner.ts integration for shared statistical machinery, with each script gaining only a thin --variant-a/--variant-b CLI parser. Confirm at Task 4.3.1 design time.
• Task 4.3.2: Implement Path 1 (--ab mode) wiring in scripts/eval-runner.ts (Phase 3 wrapper) plus eval-classification.ts variant-source CLI parsing first. Effort: 2h. Sandbox note: any ad-hoc invocation that writes baselines or A/B results to Supabase from inside Claude Code’s sandbox must use dangerouslyDisableSandbox: true per CLAUDE.md “Bun fetch hangs on HTTP 204” gotcha — see §6.5.
• Task 4.3.3: Extend Path 1 to eval-summarisation.ts, eval-search.ts, eval-entity-classification.ts. Effort: 2h.
• Task 4.3.4: Implement Path 2 (dual-scoring mode) in SI relevance scorer. DEFERRED to Phase 5 follow-up per S155 WP1.3 F-4. Phase 4 ships Path 1 only. See §5.3 Path 2 prose, §12 OQ-7, and §9 Out of Scope for the deferral rationale. The feed_articles.candidate_score column is NOT added in Phase 4; no SI spec changes are required.
• Task 4.3.5: Admin UI for A/B result comparison (re-uses the refinement dashboard from Task 4.7). Effort: 1h.

Acceptance:

• bun run eval:classification --ab --variant-a=current --variant-b=tmp/new-skill.md runs both prompts, reports per-metric deltas, and exits 0 if variant B is not worse than variant A on any metric.
• Path 1 produces output that feeds into the refinement dashboard.
• Path 2 (SI dual-scoring) is NOT an acceptance criterion for Phase 4 per §5.3 Path 2 deferral and S155 WP1.3 F-4. SI continues to use its existing feed_prompts snapshot-comparison workflow.

Dependencies: §3.4.2 (to produce the variant B prompts). Phase 3 shared eval infrastructure (the EvalResult type, metrics library).

Effort: 1 session (matches roadmap).

---

5.4 §3.4.4 Prompt Rollback for All Touchpoints

Goal: Every touchpoint has a documented and (where possible) automated rollback path.

Current state:

• SI: DB rollback via feed_prompts.rolled_back_to_version_id. Full.
• Classification: git revert only. Manual.
• Summarisation: code edit only. Manual.
• Bid drafting: git revert only. Manual.
• Digest: code edit only. Manual.
• Quality check, question extraction, tender metadata, search query generation: code edit only. Manual.

Proposed design (per §3.3 storage abstraction decision):

The spec commits to Option B (per-touchpoint adapters) from §3.3. Implementation proceeds in two tiers:

Tier 1: DB-backed adapters (full automated rollback)

• FeedPromptAdapter for SI relevance scorer — wraps existing feed_prompts table.
• (Future) SummarisationPromptAdapter if/when summarisation prompts move to a DB-backed store.

Tier 2: Git-backed adapters (rollback proposal, human-applied)

• SkillFileAdapter for touchpoints that use lib/ai/skills/*.md (classification, bid drafting).
• InlinePromptAdapter for touchpoints that use hardcoded strings (summarisation, digest, quality check, question extraction).
• Git-backed adapters implement rollbackTo() by returning a RollbackProposal object: the diff to apply, the commit to revert, the affected files, the redeploy requirement, and a human-readable explanation. A human (or a robot-PR tool) applies it.

Skill file cache (per S155 WP1.3 F-8). lib/ai/skills/loader.ts:4 declares an in-process Map<string, string> skill cache that is never invalidated. A SkillFileAdapter.rollbackTo() that changes on-disk content will not affect the cached version until the process restarts. On Vercel, each cold start re-reads the file, but warm invocations continue using the stale cache. The implementation must address this, choosing one of:

• (a) Restart the process to clear the cache. On Vercel this means a redeploy; locally it means restarting bun dev. Cleanest but has latency.
• (b) Add an invalidation hook to the loader: a new invalidateSkill(name: string) API that the SkillFileAdapter calls after writing the file. Smallest code change but introduces a new public API.
• (c) File-system mtime check on every loadSkill() call. Reads the stat once per call (cheap on Vercel local FS). Highest backwards compatibility — no caller changes — but adds a syscall to every load.

Decision deferred to Task 4.4.4 implementation time. The chosen approach must be logged in the iteration notes and in the ADR (Task 4.4.1). Until the decision lands, document in §5.4 that Tier 2 skill-file rollbacks are “intent-recorded” only — the registry captures the intent, but the change is not effective in the running process until redeploy/restart.

Filter helper adapter gap (per S155 WP1.3 F-6). The classification touchpoint has both a shared skill file (lib/ai/skills/classification.md, loaded by both TS and Python pipelines) and TS-only filter helpers (lib/ai/classify.ts post-processing). The SkillFileAdapter covers the shared file, but TS-only filter helper changes are NOT a covered storage model in Phase 4. Two options for resolution:

• Option (i): Add a FilterHelperAdapter (TS-only code edit, git- backed, manual rollback) as a third Tier-2 sub-type. Effort: 2h. Gives the classification touchpoint full Phase 4 coverage.
• Option (ii): Document that filter-helper changes are out of scope for Phase 4’s adapter model and follow the entity classification iteration log directly (see docs/specs/entity-classification-prompt-tightening-spec.md).

Phase 4 ships Option (ii). The FilterHelperAdapter is recorded as OQ-9 in §12 and may be added in a follow-up if the entity classification team needs the structured registry coverage. Until then, filter helper edits are tracked in the entity iteration log only and do NOT appear in the prompt_version_registry.

The rollback registry (prompt_version_registry table from §4.1) records EVERY version activation and rollback — manual or automatic — so there is a single auditable history of “what prompt was live for X at time T” for every touchpoint.

Task breakdown:

• Task 4.4.1: Finalise the storage abstraction decision in an ADR (docs/adrs/prompt-storage-abstraction.md). Document why Option B over Option A. Effort: 1h.
• Task 4.4.2: Create the prompt_version_registry table. Effort: 1h.
• Task 4.4.3: Implement FeedPromptAdapter. Effort: 1.5h.
• Task 4.4.4: Implement SkillFileAdapter with RollbackProposal semantics. Effort: 2h.
• Task 4.4.5: Implement InlinePromptAdapter. Effort: 1.5h.
• Task 4.4.6: Wire every touchpoint’s production code to record activations/rollbacks via the registry. Effort: 2h.
• Task 4.4.7: Admin UI for rollback history per touchpoint (re-uses the refinement dashboard). Effort: 1h.

Acceptance:

• ADR documented and committed.
• SI can rollback from the admin UI (Tier 1).
• Classification rollback produces a valid RollbackProposal that an admin can apply via git revert; the registry logs the intent and subsequent human confirmation (Tier 2).
• Every active touchpoint has a registry entry for its current version.

Dependencies: §3.4.1-§3.4.3 (context), §3.3 architectural decision.

Effort: 1 session (matches roadmap, but is tight — may slip into 1.5 sessions if the SkillFileAdapter robot-PR path is built at all).

---

6. Cross-cutting concerns

6.1 Observability

Every Phase 4 component emits structured logs via the logBestEffortWarn pattern (see docs/specs/silent-failure-prevention-spec.md) or the Phase 5 cost-tracking wrapper (see docs/specs/ai-eval-phase5-full-coverage-spec.md §3.5.5). Minimum events:

• refinement.signal.created (touchpoint, signal_type, severity)
• refinement.scanner.run (touchpoint, scanner_name, signals_emitted, duration_ms, ok) — emitted on every scanner invocation, regardless of signal count, so a dead scanner is visible in observability. Per S155 WP1.3 F-14: a touchpoint with no refinement.scanner.run log entry in the last 24h should fire a warning alert.
• refinement.pattern.detected (touchpoint, pattern_id, evidence_count)
• refinement.proposal.generated (touchpoint, pattern_id, claude_tokens) — logs token counts only, NOT prompt text. Prompt content stays in the database.
• refinement.ab.executed (touchpoint, variant_a, variant_b, outcome)
• refinement.version.activated (touchpoint, version_ref, adapter)
• refinement.version.rolled_back (touchpoint, from_version, to_version, reason)
• refinement.signal.rejected (touchpoint, signal_id, reason) — emitted when a bad-quality signal is filtered out at the PatternDetector stage (see §6.6 below).

6.2 Cost

The proposal generator (§3.4.2) is a live Claude call per pattern. Budget: ~2k input tokens / ~1k output tokens per call. Per S155 WP1.3 F-17, the calculation against the canonical pricing table at lib/ai/pricing.ts is:

• Analysis tier (Sonnet 4.5): 2k input × $3/1M tokens = $0.006 + 1k output × $15/1M tokens = $0.015 → ~$0.021 per call
• Drafting tier (Opus 4.5): 2k input × $15/1M = $0.030 + 1k output × $75/1M = $0.075 → ~$0.105 per call
• Quality tier (Haiku 4.5): 2k input × $0.25/1M = $0.0005 + 1k output × $1.25/1M = $0.00125 → ~$0.0018 per call

Phase 4 uses the analysis tier (AI_ANALYSIS_MODEL, defaults to Sonnet 4.5) for the proposal generator → ~$0.02-0.03 per pattern. At a nominal rate of 5 patterns/day across all touchpoints this is ~$3/month — well under any rounding noise. Cost tracking (Phase 5 §3.5.5) is a dependency if the refinement loop’s volume scales materially.

A/B testing (§3.4.3) in Path 1 runs the eval suite twice. Classification eval is already ~25 minutes / ~$2 per run (cached mode). A/B doubles this. Gate A/B runs behind explicit human triggers; do NOT run them on every pattern detection.

6.3 Rollback semantics

If a Phase 4 component itself ships bad code and needs rollback, the standard git revert + redeploy path applies. There is no recursive refinement loop.

6.4 Access control

Phase 4 is admin-only. The signal tables contain flag reasons that may reveal reviewer identity; the proposal generator’s output may contain sensitive prompt text. All API endpoints use getAuthorisedClient() with admin-only role checks. See the authFailureResponse() helper pattern.

6.5 Sandbox / environment (per S155 WP1.3 F-7)

Any Phase 4 script that writes to Supabase from inside Claude Code’s sandbox MUST run with dangerouslyDisableSandbox: true per the CLAUDE.md “Bun fetch hangs on HTTP 204” gotcha. This affects:

• The new --ab mode in scripts/eval-runner.ts and the per-script CLI parsers (Tasks 4.3.2-4.3.3) — they write A/B baselines and registry rows.
• Any ad-hoc bun -e snippet that inserts into prompt_quality_signals, prompt_version_registry, or updates feed_prompts.
• Scanner cron entry points if invoked manually for testing.

Production (Vercel) is unaffected — the gotcha only fires through the sandbox HTTP CONNECT proxy. Do not add .select() workarounds in production code; the production code is fine.

Phase 3 spec §6.4 carries the same guidance and Phase 4 inherits it.

6.6 Bad-signal detection (per S155 WP1.3 F-11)

Quality signals can themselves be bad-quality: a bored reviewer flags 50 articles as false positives without reading them, an automated scanner double-counts the same regression, a stale eval-fixture failure recurs because the gold standard itself is wrong. The PatternDetector must reject or weight signals whose provenance cannot be trusted. Mitigations:

• Minimum-distinct-actors threshold. A pattern with all signals from one user/scanner is suppressed unless the pattern has ≥5 evidence items. Patterns with signals from ≥2 distinct actors are surfaced at any evidence count ≥2.
• Signal weight decay. Signals older than 30 days contribute 0.5× weight to the cluster-significance score; signals older than 90 days contribute 0.25×. Decay is recomputed at each PatternDetector run.
• Provenance reject. A signal whose evidence_ref.source cannot be resolved (e.g. the cited feed_flags.id no longer exists) is dropped with a refinement.signal.rejected log entry. The signal row is NOT deleted — only excluded from the current run — so backfill can recover.
• Human-judgement gate at the proposal stage. Even after clustering, the proposal generator’s output goes through human review before activation. The reviewer can mark a proposal as “based on bad signals” which removes the underlying signals from the unprocessed pool.

Configurable per touchpoint via prompt_quality_signal_thresholds rows (Phase 4 implementation may use a hardcoded constant initially and add the table in a follow-up if tuning is needed).

6.7 Rollback integrity check (per S155 WP1.3 F-13)

For git-backed adapters (SkillFileAdapter, InlinePromptAdapter), the on-disk file may be edited between activation and rollback (e.g. another commit lands that touches the same file). The RollbackProposal would then contain the diff against the current on-disk state, not the original activated state. To detect this:

• At activation time, compute and store SHA-256 of the affected file(s) in prompt_version_registry.file_sha256 (added in §4.1 schema above).
• At rollback time, recompute the SHA and compare to the stored value.
• Mismatch → surface a warning in the RollbackProposal: “On-disk state has drifted since activation. The proposed rollback diff may not restore the originally-active prompt. Manual review required.”
• The rollback still proceeds — the human reviewer decides whether to apply it — but the warning ensures the drift is visible.

---

7. Implementation phasing within the phase

Session A — Substrate (3-4h): Tasks 4.1.1-4.1.4 (data model + scanners

• pattern detector + API).

Session B — Proposal Generator (3-4h): Tasks 4.2.1-4.2.5 (interface + generator + skill file + SI refactor + classification-spec compat check).

Session C — A/B Runner (2-3h): Tasks 4.3.1-4.3.3 + 4.3.5 (design + Path 1 in eval-runner.ts + extend to other evals + UI hooks). Task 4.3.4 (Path 2 dual-scoring) is deferred per §5.3 Path 2 and S155 WP1.3 F-4, which reduces Session C effort.

Session D — Rollback (3-4h): Tasks 4.4.1-4.4.7 (ADR + registry + adapters + wiring + UI).

Session E — Dashboard stub + polish (2-3h): Task 4.7 ships a minimal /admin/refinement route as a stub per AC #12 (revised). It lists registered touchpoints, their current registry version, and the count of unprocessed signals — using existing Warm Meridian table primitives only, no new components, no design system extension. Verification, doc updates. The full cross-touchpoint refinement dashboard is explicitly deferred to a follow-up work package (see §12 OQ-10) because the original AC #12 (“renders real data for at least two touchpoints”) cannot be satisfied from a 2-3h allocation. Per S155 WP1.3 F-10, the choice was either to expand the session to 6-8h with a real component breakdown or to demote AC #12 to a stub; Phase 4 chooses the stub to keep Session E in scope and defer the visual design work to a follow-up that can engage the Warm Meridian implementation spec properly.

Sessions A and B must land sequentially (B depends on A). Sessions C and D can run in parallel after B. Session E is the merge-and-polish.

Expected total: 14-19 hours (3-4 sessions at the project’s typical 2-4h/session pace).

---

8. Test strategy

• Unit tests: Each SignalSource scanner, the PatternDetector, the proposal generator parsing logic, each PromptStorageAdapter. Mock Claude API calls with vi.mock('anthropic').
• Integration tests: End-to-end flow for one touchpoint (classification): seed signals → detect pattern → generate proposal → A/B run → activate → rollback. Run against a test Supabase project (not production).
• Eval compatibility: Confirm that existing bun run eval:* runs still work when the touchpoint has a registry entry (i.e. the registry is additive, not required).
• SI regression: Run the SI refinement skill’s test suite after the §3.4.2 refactor to confirm no behavioural regression.
• Rollback dry-run test: For each Tier-2 adapter, confirm RollbackProposal objects contain the correct git SHA, file paths, and diff — without actually performing the revert.

---

9. Out of scope

• Bid drafting rubric (§3.5.1) — Phase 5, blocked on real bid data.
• Summarisation skill conversion (§3.5.2) — Phase 5.
• Digest evaluation (§3.5.3) — Phase 5.
• Output mode standardisation (§3.5.4) — Phase 5.
• Cost tracking wrapper (§3.5.5) — Phase 5, but Phase 4 consumes it if available.
• MCP tool description audit (§3.5.6) — Phase 5.
• Building the SI refinement UI — specific-instance spec covers it.
• Writing the classification tightening iteration loop — specific- instance spec covers it.
• Automated robot-PR application of rollback proposals — requires a separate tooling decision (e.g. GitHub Actions + Octokit + a bot account). Phase 4 ships the RollbackProposal object and a manual-apply path; the robot-PR path is a follow-up.
• Multi-workspace refinement aggregation — Phase 4 is per-workspace for workspace-scoped touchpoints (SI) and global for global touchpoints (classification, summarisation, etc.). Cross-workspace aggregation (e.g. “what are all my clients’ SI pipelines complaining about?”) is out of scope.
• SI live-stream dual-scoring (Path 2) — explicitly deferred per §5.3 Path 2 and S155 WP1.3 F-4. The SI Prompt Refinement Skill spec (docs/specs/si-prompt-refinement-skill-spec.md lines 1100–1102) lists this in its Out of Scope section and re-introducing it would require an SI schema migration coordinated with the SI spec author. See §12 OQ-7.
• SI live-data preview as a third A/B variant — the “re-score the last 20 articles with a candidate prompt and show the user the score deltas” primitive from the SI spec is neither Phase 4 Path 1 (gold-standard) nor Path 2 (parallel dual-scoring). It is a separate primitive Phase 4 does not generalise. See §12 OQ-8 per S155 WP1.3 F-2.
• FilterHelperAdapter (TS-only filter helper changes) — per S155 WP1.3 F-6, classification has both shared skill files (covered by SkillFileAdapter) and TS-only filter helpers (lib/ai/classify.ts post-processing rules) which are NOT covered. Phase 4 ships Option (ii) from §5.4: filter-helper edits are tracked in the entity classification iteration log only and do NOT appear in the registry. See §12 OQ-9.
• Full Warm Meridian refinement dashboard — per S155 WP1.3 F-10, Phase 4 ships only the AC #12 stub. The full visual design with real-data rendering for ≥2 touchpoints requires a 6-8h follow-up that engages the Warm Meridian implementation spec (docs/design/warm-meridian-implementation-spec.md) and is out of scope for Phase 4. See §12 OQ-10.

---

10. Acceptance criteria

1. The prompt_quality_signals and prompt_version_registry tables (or equivalent typed stores) exist and have documented schemas, including the partial unique index from §4.1 enforcing one live version per touchpoint at a time.
2. At least three SignalSource scanners are implemented and collecting real signals from production data, using the verified column names (content_items.classification_confidence, not ai_confidence; and governance_review_status IN ('changes_requested', 'pending'), not 'needs_attention') per S155 WP1.3 F-1 and F-3.
3. A PatternDetector produces non-trivial clusters from signals (at least one cluster with ≥2 evidence items on real data) and applies the §6.6 bad-signal mitigations (minimum-distinct-actors, weight decay, provenance reject).
4. A PromptChangeProposal can be generated end-to-end for the classification touchpoint.
5. bun run eval:classification --ab runs two prompts in parallel and reports per-metric deltas. Note: Path 2 (SI dual-scoring) is explicitly NOT shipped in Phase 4 — see §5.3 Path 2 deferral, §12 OQ-7, and S155 WP1.3 F-4.
6. The SI FeedPromptAdapter supports full automated rollback.
7. The SkillFileAdapter produces valid RollbackProposal objects for classification and bid drafting, including the §6.7 SHA-256 drift- detection warning when the on-disk file has changed since activation.
8. Every currently-active touchpoint has a registry entry for its current version, with file_sha256 populated for git-backed touchpoints.
9. The SI Prompt Refinement Skill’s test suite passes after the §3.4.2 refactor (no regression in specific-instance behaviour).
10. An ADR exists at docs/adrs/prompt-storage-abstraction.md documenting the Option B decision AND the §5.4 skill-loader cache invalidation decision (per S155 WP1.3 F-8).
11. Every Phase 4 API endpoint is admin-only and uses authFailureResponse().
12. (REVISED per S155 WP1.3 F-10) A /admin/refinement route exists as a stub page that lists registered touchpoints, their current registry version, and the count of unprocessed signals — using existing Warm Meridian table primitives only. The full cross-touchpoint refinement dashboard with real-data rendering for ≥2 touchpoints is deferred to a follow-up work package (see §12 OQ-10) because Task 4.7’s 2-3h allocation cannot satisfy the original AC #12 wording. Until that follow-up lands, Phase 4 ships the stub plus per-touchpoint API endpoints (signals, patterns, proposals, version history) that an operator can query directly via curl.
13. (NEW per S155 WP1.3 F-15) All four Phase 3 §3.3 contracts are exercised by Phase 4 implementation, verified by an integration test that fails if any contract is missing:
    1. Eval regression scanner contract → exercised by EvalRegressionScanner.
    2. A/B statistical machinery → exercised by Path 1 --ab runs.
    3. Per-suite regression threshold model (severity and variance_band on EvalBaseline.thresholds) → exercised by the proposal generator’s expected-impact bounds.
    4. Baseline lifecycle hooks (promoteBaseline, baselineHistory, compareBaselines) → exercised by FeedPromptAdapter.activate() and SkillFileAdapter.rollbackTo() for their version-history calls.
14. (NEW per S155 WP1.3 F-7) All Phase 4 dev-time scripts that write to Supabase document the dangerouslyDisableSandbox: true requirement in their script header per §6.5.
15. (NEW per S155 WP1.3 F-14) The refinement.scanner.run log event is emitted on every scanner invocation. An integration test seeds a failing scanner and asserts that the failure is visible in the observability stream within 24h.

---

11. References

Document	Location
Parent strategy	docs/audits/ai-eval/ai-touchpoint-map-and-improvement-strategy.md
Gap analysis	docs/audits/ai-eval/ai-eval-gap-analysis-s149.md
Post-MVP roadmap	docs/reference/product-roadmap.md §3.4
SI pathway	docs/reference/sector-intelligence-pathway.md §8
SI refinement skill spec (prior-art instance)	docs/specs/si-prompt-refinement-skill-spec.md
Classification tightening spec (prior-art instance, parallel)	docs/specs/entity-classification-prompt-tightening-spec.md
Phase 2 classification skill spec (structural reference)	docs/specs/ai-eval-phase2-classification-skill-spec.md
Phase 3 regression infrastructure spec (hard dependency)	docs/specs/ai-eval-phase3-regression-infrastructure-spec.md (parallel S154 work)
Phase 5 full coverage spec (sibling)	docs/specs/ai-eval-phase5-full-coverage-spec.md
AI integration layers	docs/reference/ai-integration-layers.md
AI integration strategy	docs/reference/ai-integration-strategy.md
AI visibility policy	docs/reference/ai-visibility-policy.md
Structured logging spec	docs/specs/structured-logging-spec.md
Silent failure prevention spec	docs/specs/silent-failure-prevention-spec.md
MCP evaluation spec	docs/specs/mcp-evaluation-spec.md
Context engineering skills	.claude/skills/context-engineering-collection/skills/advanced-evaluation/
LLM evaluation skill	.claude/skills/llm-evaluation/

---

12. Open questions

This section was missing from the original spec (per S155 WP1.3 F-16) and is added here in line with the four other S154 specs which all have an Open questions section. Open questions OQ-1..OQ-6 are unresolved design decisions surfaced by the audit; OQ-7..OQ-10 are deferrals from findings F-2, F-4, F-6, and F-10.

• OQ-1. Does the proposal generator’s output need to be human-readable markdown, machine-parseable JSON, or both? §5.2’s PromptChangeProposal interface is TypeScript, but the proposedDiff.before/after fields are strings — are they raw prompt text, unified diff format, or something else? Resolution direction: machine-parseable JSON for the registry row (deterministic), with a markdown rendering layer for the admin UI. Confirm at Task 4.2.1 design time.
• OQ-2. When a RollbackProposal is generated for a git-backed touchpoint, who creates the actual git revert commit? A human? A gh pr create invocation? An automated bot? §3.3 says “or, if a robot-PR tool becomes available, automatically” but doesn’t decide for v1. Resolution direction: v1 = human applies the diff manually and marks the registry row as “applied”. Robot-PR is a follow-up that uses GitHub Actions + Octokit + a bot account; deferred to Phase 5+.
• OQ-3. What is the relationship between Phase 4’s prompt_quality_signals table and Phase 3’s regression detection events? Is every Phase 3 regression also written to prompt_quality_signals as a row? If so, who writes it — Phase 3’s CI runner, or a Phase 4 scanner that polls Phase 3’s output? §5.1 Task 4.1.2 mentions EvalRegressionScanner — reads Phase 3's baseline regression events but doesn’t say where Phase 3 writes those events. Resolution direction: Phase 3 writes to a single eval_regression_events table; Phase 4’s EvalRegressionScanner polls it. Confirm at Phase 3 implementation time.
• OQ-4. Multi-workspace touchpoints: SI is workspace-scoped, every other touchpoint is global. The schema’s workspace_id uuid column on prompt_quality_signals and prompt_version_registry is nullable. The CHECK constraint relationship between touchpoint_id and workspace_id is not specified — is it valid to have touchpoint_id = 'classification.ts' with workspace_id = abc? Probably not (classification is global) but the schema doesn’t enforce that. Resolution direction: add a CHECK constraint (touchpoint_id LIKE 'si.%' AND workspace_id IS NOT NULL) OR (touchpoint_id NOT LIKE 'si.%' AND workspace_id IS NULL) at Task 4.1.1 implementation time, after a touchpoint registry is finalised.
• OQ-5. How does the PatternDetector’s text-similarity clustering interact with multi-language content? UK English vs other locales could fragment clusters that should be merged. Resolution direction: Phase 4 is UK-English-only because the Knowledge Hub is UK-English-only per CLAUDE.md. Multi-locale support is a non-issue until the platform itself goes multi-locale.
• OQ-6. What is the threshold for “this pattern is significant enough to surface”? AC #3 says “≥2 evidence items” but real-world patterns may need ≥5 or ≥10 to be actionable. The threshold should be configurable per touchpoint. Resolution direction: start with a global default of ≥3 and a per-touchpoint override stored in prompt_quality_signal_thresholds (mentioned in §6.6). Tune empirically after the first month of operation.
• OQ-7. (per F-4) SI live-stream dual-scoring (feed_articles.candidate_score column + Path 2 in §5.3) is deferred. Re-opening it requires: (a) coordination with the SI spec author to update SI’s Out of Scope section, (b) a migration adding feed_articles.candidate_score numeric, (c) a new acceptance criterion, and (d) a Phase 5 task allocation. Defer decision to S156+ once the SI Phase 1 refinement skill has shipped and the team has data on whether before-vs-after snapshot comparison is insufficient.
• OQ-8. (per F-2) SI live-data preview as a third A/B variant (“re-score the last 20 articles with a candidate prompt”). Phase 4’s two variants (Path 1 = gold-standard, Path 2 = parallel dual-scoring) do not capture this. Resolution direction: add a Path 3 (“live-data dry-run preview”) in a follow-up. Path 3 is non-destructive (no writes to feed_articles, no SI schema change), so it does not have the cross-spec scope conflict that Path 2 does and can ship sooner.
• OQ-9. (per F-6) FilterHelperAdapter for TS-only classification filter changes. Phase 4 ships Option (ii) from §5.4 (out of scope for the adapter model). If the entity classification team needs structured registry coverage for filter-helper edits, add a FilterHelperAdapter in a follow-up (effort: ~2h).
• OQ-10. (per F-10) Full Warm Meridian refinement dashboard with real-data rendering for ≥2 touchpoints. Phase 4 ships the AC #12 stub only. Follow-up effort: 6-8h, with Tasks: refinement-dashboard.tsx, signal-list.tsx, proposal-list.tsx, version-history.tsx; Warm Meridian semantic tokens; WCAG AA contrast notes; query-key registration in lib/query/query-keys.ts; integration tests against real data. Defer to a S156+ work package.

---

Appendix A: Signal source mapping table

Touchpoint	Primary signal source	Storage	Scanner name
SI relevance scorer	feed_flags (user flags)	DB	SiFlagScanner
Classification (TS)	Eval fixture failures + low-confidence items	Phase 3 baselines + content_items.classification_confidence	EvalRegressionScanner + LowConfidenceScanner
Classification (Python)	Eval fixture failures + low-confidence items	Same as TS	Same scanners
Summarisation	Eval fixture failures (ROUGE thresholds)	Phase 3 baselines	EvalRegressionScanner
Bid drafting	Human edits to AI drafts	(new table, see §5.1 Task 4.1.2)	HumanEditScanner
Digest generation	Human review of generated digests	(no signal source yet — Phase 5 dependency)	—
Quality check	Eval-driven; no user signal	Phase 3 baselines	EvalRegressionScanner
Governance review	content_items.governance_review_status	DB	GovernanceFindingScanner

Appendix B: Storage adapter mapping table

Touchpoint	Adapter	Tier	Rollback mechanism
SI relevance scorer	FeedPromptAdapter	1 (DB)	Automated
Classification skill file	SkillFileAdapter	2 (git)	RollbackProposal
Classification entity types ref	SkillFileAdapter	2 (git)	RollbackProposal
Classification reference prompt (Python)	SkillFileAdapter	2 (git)	RollbackProposal (+ sync:taxonomy)
Bid drafting — bid-writing skill	SkillFileAdapter	2 (git)	RollbackProposal
Bid drafting — uk-procurement skill	SkillFileAdapter	2 (git)	RollbackProposal
Summarisation (inline)	InlinePromptAdapter	2 (git)	RollbackProposal
Digest generation (inline)	InlinePromptAdapter	2 (git)	RollbackProposal
Quality check (inline)	InlinePromptAdapter	2 (git)	RollbackProposal
Question extraction (inline)	InlinePromptAdapter	2 (git)	RollbackProposal