Prompt Improvement Methodology

Prompt Improvement Methodology

Purpose: Live reference for how Knowledge Hub improves AI prompts and skills. Extracted in S152B WP1 from two session-specific documents that have since been archived: systematic-prompt-improvement-approach.md (S141) and prompt-skill-engineering-vision.md (S141).

Status: Live reference. The Prompt Refinement Skill spec (docs/specs/si-prompt-refinement-skill-spec.md, roadmap §2.2 — PRE-LAUNCH BLOCKER) is the concrete implementation of this methodology for Sector Intelligence; future prompt improvement work should follow the same four-part structure described below.

---

1. Guiding principles

1. All AI prompts need evaluation baselines. Every touchpoint where Claude makes a judgement — classification, entity extraction, summary generation, relevance scoring, response drafting — must have a gold standard fixture and a reproducible eval script so regressions across model changes are caught before they reach production. See docs/reference/ai-integration-layers.md for the current touchpoint map and the AI Evaluation section of state-of-the-product.md for the shipped eval infrastructure (S144-S145).
2. Prefer skills over wall-of-text prompts. A prompt that does many things in one pass (domains + subtopics + entities + temporal + keywords
   • confidence) dilutes attention on each sub-task. Convert such prompts to skills with reference documents, examples, and chaining so each sub-task gets dedicated context. The entity classification prompt is the canonical example — converted to lib/ai/skills/classification.md in S147 via a restructure-and-consolidate pass (795 lines, single source of truth, six placeholder-based client configuration points).
3. Human-in-the-loop refinement, not human-as-editor. Users should flag content; a skill-based refinement pipeline should propose prompt changes based on the flagged patterns. Prompt editing remains admin-only — users never hand-edit prompts because the failure mode of human prompt drift is a much bigger risk than the failure mode of an under-refined prompt. The Sector Intelligence flagging feedback loop (feed_flags → prompt refinement) is the canonical example.
4. Proactive regression detection. When the taxonomy, data model, or workspace configuration changes, automatically surface an eval run and highlight regressions before the change is committed. The pipeline-parity test (__tests__/validation/pipeline-parity.test.ts) is the first piece of this and runs on every bun run test; further regression-CI work is tracked in roadmap §3.
5. Rollback capability. Prompt changes must be revertable without re-ingesting content. The feed_prompts table with version history and rollback audit trail (S141) is the canonical reference — every skill refinement that touches scoring should support the same version-and-rollback pattern.
6. AI is invisible infrastructure. The whole point of improving prompts is that users don’t have to think about them. The platform’s job is to make the data and the actions taken with it progressively better in the background, not to expose prompt engineering as a product feature. See docs/reference/ai-visibility-policy.md for the operational rule.

---

2. Four-part systematic prompt improvement pattern

When an AI touchpoint under-performs (precision/recall below launch target, eval baseline regression, user complaints), apply the four-part pattern described below. This is the pattern used for the S140-S147 entity classification prompt improvements and is the template every future prompt improvement work package should follow.

Part 1 — Taxonomy or category spec

Write a precise, testable spec for every category the prompt classifies into. For each category include:

• Inclusion criteria — what IS this category, described with at least three distinct examples.
• Exclusion criteria — what is NOT this category, described with at least three distinct counter-examples. Be explicit about the boundary cases that trip up the AI (“Information Security Policy is not a framework even though the word ‘framework’ appears in it”).
• Disambiguation rules — when two categories could both apply, which takes priority and why. “If something can be certified against (ISO 27001, Cyber Essentials), it’s a certification not a framework” is a canonical example.
• Boundary examples — cases that are genuinely ambiguous but should be resolved one way for consistency.

The entity type taxonomy spec at docs/reference/entity-type-taxonomy-spec.md is the canonical example. Source-of-truth for the spec lives in docs/reference/, not inside the prompt or skill file — the prompt is generated from the spec so there is no drift.

Part 2 — Prompt or skill restructure

Rewrite the prompt using the spec as the source of truth. Replace free-text guidance with:

• Explicit pattern-based rules — “Never extract items ending in ‘Policy’, ‘Plan’, ‘Register’, ‘Procedure’ as entities.” These are deterministic filters the AI can check mechanically.
• Few-shot examples — 6-10 worked examples covering the happy path, the most common failure modes, and the boundary cases. Examples should come from the eval suite (Part 3) so they stay synchronised.
• Structured output schemas — Use Claude’s tool-use / structured output API wherever possible. A JSON schema with per-type validation rules reduces the space of possible errors to “the AI chose the wrong tool” instead of “the AI produced freeform text we couldn’t parse.”
• Layered decomposition — If the prompt still exceeds one clear responsibility, split it into a multi-pass pipeline (e.g., the S149 two-pass entity validation — Pass 1 extracts, Pass 2 validates).

See lib/ai/skills/classification.md for the canonical restructure output.

Part 3 — Eval suite

Build a gold standard test suite of 50-100 content snippets with known-correct outputs. Measure:

• Precision — of the items the AI extracted/classified, how many were correct?
• Recall — of the items that should have been extracted/classified, how many were found?
• Type accuracy — when the category was correct, was the type / subtype also correct?
• F1 — harmonic mean of precision and recall, or equivalent for the specific metric family.

Save baselines to disk (evals/baselines/) and run the eval on every prompt change. The eval becomes a gate: prompt changes must match or improve the baseline before being deployed.

For NLP-style outputs (summaries, drafts), add ROUGE-L and BERTScore side-by-side because they measure different dimensions.

Infrastructure for this lives in lib/eval/ (types, metrics, baseline save/load, reporter). Runners are in scripts/eval-*.ts. Gold standard fixtures are versioned under the appropriate eval suite’s __fixtures__/ directory.

Part 4 — Iterative refinement loop

Run the eval, identify the worst-performing patterns, fix the prompt, re-run. Repeat until precision and recall are both above the launch threshold. Pattern failure clusters from production flagging feedback (e.g., feed_flags for intelligence, verification_history for content items) drive the iteration priority. Never ship a prompt change without the eval having run green at least once.

Rollback must be trivial — the feed_prompts table pattern (version, previous_version_id, rollback_reason) is the reference.

---

3. When to apply this methodology

Apply the full four-part pattern when any of the following is true:

• A new AI touchpoint is being introduced to the platform.
• An existing touchpoint’s eval baseline regresses.
• User flagging feedback identifies a systemic failure pattern (not a one-off mistake).
• A new model family is being introduced and needs regression testing.
• The taxonomy or data model changes in a way that affects prompt assumptions.

Apply just Parts 3-4 (eval + refinement, not a full restructure) when:

• A minor prompt tweak is being made in response to a specific failure.
• Adding new few-shot examples to an existing skill.
• Threshold tuning (e.g., raising the confidence floor).

---

4. See also

• docs/reference/ai-integration-strategy.md — high-level AI strategy
• docs/reference/ai-integration-layers.md — touchpoint map
• docs/reference/ai-visibility-policy.md — UI / UX rule
• docs/reference/classification-architecture.md — classification-specific architecture
• docs/reference/classification-prompt.md — current prompt version
• docs/reference/entity-type-taxonomy-spec.md — canonical Part 1 example
• docs/reference/two-pass-validation-architecture.md — canonical layered decomposition example
• docs/specs/si-prompt-refinement-skill-spec.md — PRE-LAUNCH BLOCKER spec implementing this methodology for Sector Intelligence