ID-71 — AI Tooling Surface: Synthesis & Plan of Approach

ID-71 — AI Tooling Surface: Synthesis & Plan of Approach

• Status: Pre-RESEARCH synthesis. This document consolidates the S309 read-only orientation (main_session_output.md), Liam’s review notes (supporting-ai-tooling-notes.md), the current tooling grouping (current-ai-tooling-grouping.md), the historic onboarding vision (historic-onboarding-suggestion.md), and a five-corpus documentation sweep (AI-integration docs, ai-eval specs, platform-direction docs, workspace specs, client evidence). It feeds — and proposes the structure of — {71.1} RESEARCH.
• Date: 2026-06-09
• Inputs reviewed this session: 30+ documents across knowledge-hub-docs-site, knowledge-hub-archive, and kh-client-content-archive.

---

1. The headline synthesis

1.1 ID-71 is bigger than its ledger record

The ledger frames ID-71 as “MCP tooling rationalisation — audit + consolidate + update + extend the 54-tool surface”. The actual scope, once Liam’s notes and the evidence base are combined, is the whole AI-consumption layer of the platform:

Surface	Current count	Canonical source
MCP tools	58 (ledger says 54 — stale)	docs/generated/mcp-inventory.md + scripts/mcp-eval/fixtures.ts
MCP resources	12 (3 templates, 5 static, 4 app)	inventory
MCP prompts	7	inventory
MCP Apps	4 (coverage-matrix, bid-dashboard, reorient-me, intelligence-feed)	lib/mcp/app-bundles.ts
Plugin commands	8 (/kb:*)	plugin bundle
Plugin skills	7–9 (docs disagree; bundle is truth)	plugin bundle
Inline AI touchpoints	classification, drafting (3-pass), summarisation, intelligence scoring, guide generation, crons	lib/ai/

Beyond the inventory, Liam’s notes add four scope dimensions the ledger record does not carry:

1. Outcome-first design method — research the most valuable workflows first, work backwards to tooling; the failure mode being corrected is precisely “tools accreted as the platform developed”.
2. Eval-everything discipline — every AI usage point must be evaluable, with automation (hooks forcing skill/eval updates on tooling change).
3. Grounding standards — Anthropic structured outputs + citations wherever possible; Ontology integration for grounded responses.
4. The headless-future thesis — help SMBs bridge from UI-habits to natural-language/agentic work; tooling must serve human-via-Claude AND headless agents (MCP-only, no skill layer) equally.

Implication: keep ID-71 as the umbrella Task, but correct its title/description and structure the RESEARCH so each dimension is explicitly covered (proposal §3). The concept-over-artefact principle from Liam’s notes (“it’s the outcome we’d retain, not necessarily the tool in its current form”) becomes the audit’s unit of analysis: verdicts attach to outcomes, not to tool names.

1.2 The client-evidenced outcomes the surface must serve (ranked)

From the gap-analysis trio, sector-intelligence pair, personas, use-case affordances research (s314/ID-75), and the Claude Integration Guide:

1. Answer this question well, with citations I trust, fast. James’s <30s metric; universal A4 (citation/provenance) requirement; confidence postures.
2. Assemble revenue documents — proposals now, renewal packs later. Matthew’s “output engine, not reference library”; selection + composition + export, not just retrieval.
3. Keep knowledge trustworthy with minimal effort — fact-check-everywhere, feature-ingest with supersession, case-study push, document control. All client-promised, all unbuilt (spec-pending as MCP prompts).
4. Brief me — what changed, what needs me, what’s happening in my sector.
5. Turn sector/competitor signal into action — sales triggers, marketing content pipeline, roadmap input. The consumption half is underserved; no marketing/sales tooling or personas exist.
6. Show me where we’re exposed — gaps, staleness, expiring certifications.
7. Find and verify a specific thing in under 2 minutes, cold-start (Tom).
8. Keep guides sales-argument-grade — section alignment, “IMPACT of If Not Now”, dual-tier, research feeds.

Cross-cutting: clients were explicitly told “you don’t need to remember the tools — just ask”. The surface should be organised by outcome, with few, smart entry points. This is direct support for the rationalisation thesis.

1.3 Ratified constraints that survive the canonical-pipeline pivot

These are pipeline-agnostic and carry forward as design law:

• Tool-count ceiling: ~30–40 tools for accurate selection; beyond that, split into multiple MCP servers rather than deferred loading (ai-integration-strategy). At 58 we are documented as over our own ceiling.
• Layer ontology + placement rules: tools = ability, skills = expertise, apps = visual density, commands = thin orchestrators (“standalone + supercharged”). “Duplicate across layers only where the layer adds genuine value.” The Skill Question: would Claude do noticeably worse without it?
• Tool description doctrine: describe what the user gets, not what the API does; dual content + structuredContent; actionable error hints.
• defineTool annotation discipline (four ToolAnnotations compile-enforced, five named constants); per-user RLS client; checkMcpRole on writes; fresh server + transport per request; prefix-free single-purpose connector.
• Two-step retrieval (list/preview metadata → verbatim fetch on accept) — ratified platform pattern, used identically by q_a_search and the composer.
• Headless agents are surface #4 — core workflows must be completable by MCP alone (no plugin/skill dependence).
• AI invisibility policy (4 rules + admin Provenance carve-out) and UX principles #1/#2/#3/#9 (tool disappears; coordinator directs; content anchors everything; honesty over optimism).
• Eval gates: L1/L3/L4 + fixtures.ts + guard tests must track every change; plugin bundle + inventory regenerate on change.
• Corpus model: corpus-level q_a_pairs with scope_tag relevance; polymorphic citations (version-on-cite); application_types + satellite pattern; external-folder-as-canonical-store ingestion.
• Mempalace: direct pattern ratified; wrapped pattern + user-memory tools deferred post-launch (PL-1/PL-3).

1.4 What is visibly stale (must not be inherited)

• Bid-domain saturation across tools/prompts/resources/apps/skills/commands (get_bid_question, kb://bids/{id}, ui://bid-dashboard, bid_briefing, bid_pipeline_review, “Bid Writing” skill, /kb:bid-* commands). Note the S330/S331 DB renames (bid_responses→form_responses, get_bid_*→get_form_* RPCs, cite_content arg → form_response_id) largely resolved the schema-level question from the S309 findings — what remains is MCP-layer naming/description drift, which is now fresh drift against the renamed DB.
• Pre-pipeline ingestion model — docs describing inline-enrichment-at-write (create_content_item doing classify+embed+chunk in-request), kb_pipeline parity. create_content_item’s source-binding shape is the flagged watch item against external-folder-canonical ingestion.
• Layer-model fork: technical.md silently re-baselines 4 layers to 5 (“Layer 0”), contradicting strategy/layers docs. Reconcile in PRODUCT.
• Counts drift everywhere (54 vs 57 vs 58 tools; 6/5 vs 8/9 commands/skills) — exactly the failure the Tier-0 generated-inventory rule predicted. Prose must stop hard-coding counts.
• Confirmed functional bug: lib/mcp/tools/workspaces.ts:29 — list_user_workspaces passes enum 'bid' raw to application_types.key (mapping to 'procurement' never applied → silently zero rows; 'content' → retired 'kb_section' also dead).
• Known rot: package.json eval:bid-drafting → renamed script (eval-procurement-drafting.ts); eval:all broken at that link; mcp-evaluation-spec counts stale.

1.5 Eval landscape: what exists vs what the ambition needs

Built and live: MCP eval L1 (42 protocol checks) / L3 (17 response-quality) / L4 (37 functional, live DB) + CI matrix + canonical fixtures + fixture-sync guard; lib/eval/ shared framework (P/R/F1, ROUGE, MRR, nDCG) + 4 committed baselines; prompt-improvement methodology (spec → skill → eval suite → iterate; eval as deployment gate).

Designed but unbuilt (the gap between “evals exist” and “evals fire”):

Asset	Spec	Status	Relevance to ID-71
scripts/eval-runner.ts + severity/variance threshold model + baseline lifecycle (promoteBaseline, history) + nightly lane	ai-eval Phase 3	Not implemented	The engine for “automate evaluation everywhere”. Without it the ambition has no motor.
Claude-as-judge tool-description rubric (~$3/run)	Phase 5 W6	Not implemented	Directly evaluates the selection quality half of the tool surface that L1/L3/L4 don’t touch.
recordAiCall() cost tracking + grep-enforcement guard test	Phase 5 W5	Not implemented	The enforcement-test pattern generalises to “tooling change ⇒ forced eval/fixture/skill update”.
Touchpoint registry + rollback adapters	Phase 4	Not implemented	Adopt the registry concept only: every tool/prompt/skill has an owner, eval, and version record.
Structured-outputs-default ADR	Phase 5 W4	Not implemented	Aligns with Liam’s structured-outputs note; fold into TECH standards.

Eval-blind today: the 7 MCP prompts, 12 resources, all plugin skills, and MCP Apps have no evals at all. The “every place AI is used must be evaluable” requirement is currently met only for tools (behaviourally) and 4 inline touchpoints (baselines).

1.6 OPS-30 — resolved

OPS-30 was never a document; it was deferred backlog from the S195/S196 structured-outputs research: add strict: true + recursive additionalProperties: false to the classification tool schemas (~1h, pure hygiene, never shipped). The citations angle lives separately: Anthropic Search Result Citations and Structured Outputs are API-incompatible, which is why drafting is a 3-pass pipeline (lib/ai/draft.ts). Durable lesson for ID-71 eval discipline: schema enforcement ≠ semantic correctness — constrained decoding cannot replace evals for valid-but-wrong outputs. Both the incompatibility and strict:true semantics need re-verification against the current API before being treated as constraints (research was SDK 0.78.0 / 4.6-era models).

1.7 The onboarding/content-gathering thread (historic doc)

The pre-repo Bid Manager onboarding vision maps strikingly well onto the current architecture — better than onto the architecture it was written for:

• “Tell us where your documents live” (connect, don’t upload) is the external-folder-as-canonical-store model the cocoindex pipeline now implements (LocalFS/SharePoint/Notion source-walk).
• “Parse → propose → confirm” with bulk-accept review is the review-queue/publication-status model.
• The “content gathering skill” with onboarding mode (broad discovery) → monitoring mode (change detection) is the agentic wrapper the pipeline lacks.
• “Library bootstraps through use” is the UC5 flywheel.

Gap identified: the current 58-tool surface has no onboarding/source- connection/discovery tooling at all. Day-one experience is the moment SMB clients form the “this is different” judgement (under-4-hours vs Loopio’s 2–4 weeks). This is the strongest new-capability candidate from part (d) of the audit, and also the natural home of the Companies-House-style programmatic context enrichment.

1.8 Evidence gaps to close during RESEARCH

• Personas: no marketing, sales/account-management, or finance personas — despite those functions being the primary intelligence beneficiaries and the output-engine pivot making proposals/renewals central. Renewal packs imply an unpersonified customer-success user.
• Use-cases without owners: marketing content creation (U13), general research (U15), sentiment (U16) — named, never specced.
• Unbuilt promised surface (client commitments): fact_check_content, feature_ingest, case-study push prompts; account_brief, proposal_shortcut, case_study_assembly prompts; Account Brief MCP App; sales-proposal tool set (10-tool arm-a draft, unratified).
• Client docs: Platform Overview + Claude Integration Guide both flagged “heavily outdated” (tracker E.2) — refresh is a natural ID-71 deliverable since the surface it documents is what’s changing.

1.9 Third-party candidates (research lane, not commitments)

Candidate	Angle	Lane
Ontology next-stage (nebula-graph articles)	Ontology → graph DB evolution	Platform grounding
MCPJam	MCP app/tooling dev + eval harness	Tooling quality
supermemory / memanto	Per-user memory (vs MemPalace)	Platform + dev-workflow
raindrop.ai	Agent observability (OSS suitability check)	Eval/observability
watchmen	Context management for workflow evaluation	Dev-workflow
osiris	Cyber-security awareness add-on	Platform add-on
knowhere / rowboat / mirage / html-anything	Extraction/transformation	Pipeline-adjacent
iii	Backend	Infrastructure
Anthropic SMB plugin	Ready-made SMB workflow patterns	Pattern source

---

2. What this means for the audit method

The S309 first-pass (a/b/c/d: drift/remove/consolidate/add) was tool-centric. The corrected method, per Liam’s notes:

1. Outcomes first. Build the workflow/outcome inventory (per workspace + cross-workspace + per persona) from §1.2 evidence, extended s314 use-case × affordance matrix, and the new personas.
2. Map the existing surface onto outcomes. Every tool/prompt/resource/app/ skill/command gets a verdict at the concept level: outcome still valid → keep concept (current form / refined form), outcome invalid → retire, outcome valid but unserved → gap.
3. Design the target surface within constraints (§1.3): ≤~40 tools or multi-server split, outcome-grouped, two-step retrieval, headless-complete, every touchpoint born evaluable.
4. Plan the migration (renames, deprecations, eval/fixture/bundle/guide lockstep).

---

3. Proposed plan of approach

Phase 0 — Corrections + hygiene (immediate, cheap, no design dependency)

Item	Action	Size
0.1	Ledger correction: ID-71 title/description — 58 tools, scope = AI tooling surface (tools+prompts+resources+apps+skills+commands+inline touchpoints)	trivial
0.2	Fix list_user_workspaces enum bug (map 'bid'→'procurement'; decide fate of dead 'content' value)	small
0.3	Fix package.json eval:bid-drafting/eval:all script rot	trivial
0.4	OPS-30 hygiene: verify current API semantics, then strict:true + additionalProperties:false on classification schemas	~1h

{71.1} RESEARCH — two lanes

Lane A — Outcome & workflow research (the differentiator work):

• Extend personas: marketing, sales/account management, finance, customer-success (renewals).
• Extend the s314 use-case × affordance matrix across all six application types
  • cross-workspace loops; absorb the day-one/onboarding journey (§1.7) as a first-class workflow family.
• Produce the ranked workflow inventory (human-via-Claude AND headless-agent variants of each).
• Map all current tooling (the full §1.1 table) onto workflows → keep-concept/refine/retire/gap verdict table.
• Sketch target surface shape: tool count + grouping, multi-server question, prompt/skill/app placement per the layer rules.

Lane B — Technical & ecosystem research:

• Structured outputs + citations: current API state (re-verify strict:true, citations×structured-outputs incompatibility, output_config) → standards proposal for all touchpoints.
• Ontology integration mechanics: how tools/prompts ground responses in the ontology; next-stage (graph DB) assessment.
• Eval reuse audit: what of Phase 3/4/5 specs to lift for the eval-infrastructure track (see sibling task below).
• Third-party evaluation sweep (§1.9 table) with adopt/watch/reject verdicts.
• outputSchema standard design (bl-26 as forward-standard, per Liam’s note — not retrofitted onto retiring tools).

RESEARCH.md consolidates both lanes; ends with the verdict table + target-surface options for Liam to ratify before PRODUCT.

{71.2} PRODUCT → {71.3} TECH → {71.4} PLAN

• PRODUCT: the target surface spec — outcome-grouped tooling with numbered behaviour invariants; new-capability specs (fact-check, feature-ingest, onboarding/content-gathering, briefing, marketing/sales consumption); per-persona × per-surface coverage matrix; deprecation/rename policy; client-doc refresh scope. Every touchpoint declares its eval requirement.
• TECH: migration strategy (rename lockstep: code + fixtures + plugin bundle
  • inventory + evals + client guide); structured-outputs/citations/outputSchema standards; ontology grounding implementation; forcing-function hooks (create-skill/update-skill invocation on tooling change — Liam to land the skills in-repo; fixture-sync-style guard tests extended to prompts/skills).
• PLAN: decomposition into implementation subtasks (waves: hygiene → renames → consolidations → removals → additions → docs/evals).

Sibling track — Eval infrastructure (recommend: own Task ID)

Phase-3 eval-runner + threshold/severity model + baseline lifecycle + nightly lane; W6 tool-description judge; W5 cost tracking + enforcement-test pattern; touchpoint registry. Rationale for separating: it is platform infrastructure serving all AI work (including ID-101/ID-45 canonicaliser parity and the intelligence pipeline), not just ID-71; and ID-71’s PRODUCT wants its contracts to exist so the new surface ships born-evaluable. Sequence: open now, build in parallel with {71.1}.

Sequencing vs the canonical track

ID-71 remains parallel pre-launch, NOT a re-ingest gate (ledger). Two coordination points only:

1. Tool renames should land as one coordinated client-contract break, ideally aligned with (not before) the prod DDL cutover wave, so clients/ eval fixtures break once, not twice.
2. create_content_item / ingest-adjacent tool redesign must track the ID-101/ID-45 external-folder-canonical reality, not the retired inline model.

---

4. Decision points for Liam

#	Decision	Recommendation
D1	Reframe ID-71 ledger record (scope + count)	Yes — umbrella “AI tooling surface” Task; correct 54→58
D2	Eval infrastructure as sibling Task vs ID-71 subtasks	Sibling Task, opened now, parallel build
D3	Rename strategy: hard rename vs alias-and-deprecate	Hard rename, single coordinated break in lockstep with plugin bundle + fixtures + client-guide refresh; alias layer only if Phew’s live usage demands it
D4	Onboarding/content-gathering tooling: ID-71 (d) item vs own Task	Research in Lane A; expect it to graduate to its own Task at PRODUCT (it is a product capability, not just tooling)
D5	Backlog absorption: bl-26 (outputSchema), #49 (marketing personas/prompts), #50 (schema hygiene), #52 (taxonomy resource)	Absorb #49/#50/#52 into ID-71; bl-26 as forward-standard in TECH (per your note)
D6	Memory direction (MemPalace direct vs supermemory/memanto)	Lane B research; decision deferred to PRODUCT; wrapped-mempalace stays post-launch-deferred
D7	Phase 0 items: run now (this/next session) or fold into ID-71 waves	Run 0.1–0.3 now; 0.4 after Lane B API verification

---

5. Proposed immediate next steps

1. Liam ratifies/amends §3 structure + §4 decisions.
2. Phase 0 items executed (one small wave).
3. {71.1} RESEARCH subtasks opened per the two-lane split; Lane B third-party sweep can run as background research (web-capable agents) while Lane A workshops the workflow inventory with Liam — Lane A needs his input most.
4. Eval-infrastructure Task opened (if D2 ratified) seeded from §1.5 table.
5. Liam lands the create-skill/update-skill (and related) skills in-repo so the forcing-function hooks can be designed against them in TECH.