Supporting ai-tooling review notes

All tooling predates the canonical pipeline pivot and so will be considerably outdated against the current platform implementation and direction. One aspect of the review is to indeed determine what we keep/refine/retire, but to be clear, it’s not necessarily that it’s the tool or skill or functionality that we would necessarily retain in its current form but rather the concept of what it is or the output or outcome that we’re looking to achieve with the tool.
As noted, there are 58 MCP tools - this is not by design, but instead due to tools being built incrementally as the platform originally developed, rather than being an informed, curated set of tools.
- The tooling that we proceed with should be efficient, effective, and measurable, minimising duplication, and reducing cognitive overhead for both Claude and the eventual platform users.
- Something that we didn’t do when the tooling was created was to start with the outcome that a user would actually want and then work backwards from that to how we should go about implementing it within our platform. To avoid the same situation, one item that we need to do will be to research firstly what the most valuable workflows would be given the purpose of our platform and the data that we will hold and make available for users and AI agents, and then map out use cases and workflows to ensure that the UX that we provide is seamless. This is a big part of our differentiation. We know that most applications will become headless to some degree and critical to what we want to be able to do is help SMBs prepare for a future which is heavily influenced by AI, especially agentic AI, and ensuring that their data is structured and useful and available to both humans and AI to support with day-to-day company activities and also to help bridge the gap between what users currently are familiar with which is the UI of an application, to becoming more used to working with natural language to complete tasks, triggering skills, commands, workflows, etc. to get their work done day-to-day, without a heavy reliance on a UI.
We have backlog-26 (MCP outputSchema full rollout), but this isn’t something that we need to immediately apply to our current tooling (in case any referenced tools are retiring/changing), but can be used in terms of providing a more standardised output schema that we use for all of our MCP tooling going forward, providing it fits the use case, of course.
Wherever possible, any of the AI tooling that we use should be using Anthropic structured outputs functionality and citations functionality to ensure that responses are in the correct shape and as deterministic as possible. I’m not certain, but I think we previously had an item which was OPS-30, which related to this. But that ID is pre the migration to the new ledgers and it’s entirely possible that the content that was there is now irrelevant. However, it would make sense to check.
The platform has an Ontology now - this will be key functionality that should be integrated within our AI tooling to help ensure that responses are grounded in real context.
One of the most important things here is that everywhere that we’re utilising AI, we need to be in a position where we can evaluate the outputs/outcomes. This is something that was highlighted in the prompt improvement methodology document and related documents, but it’s also something that we need to be able to automate wherever possible, both in terms of improving the AI functionality that we include within the platform and also for when we introduce new functionality. A key part of this will be ensuring that in the same way that when we update skills predominantly for our dev workflow, we have hooks that ensure that create skill or update skill are invoked for any changes, we do the same for things like plugin, command, platform-related skills and MCP tooling development. And whilst the skills may not currently be available in the repo, I’ll ensure that they are and can be used on an ongoing basis throughout the rework of AI-related tooling that we’re about to undertake.
- a lot of work was previously done under the heading of section 3 (ai-eval) on the old product roadmap, again pre-migration to the new ledgers and the focus there was on AI evaluations, but again, we need to be of the mindset that that approach may or may not still be valid and we need to be open to adapting that to ensure that we’re able to automate wherever possible when it comes to evaluations and continuous improvement. I’m also going to provide some links to sources that I’ve been collecting which may also provide us with some valuable insights in regards to this.

Key documents

These documents will provide additional context, but important to note is that almost all of the documents are historic and were created prior to the implementation of the CocoIndex Canonical Pipeline.

1
- `ai-integration-strategy.md`
2
- `ai-integration-layers.md`
3
- `prompt-improvement-methodology.md`
4
- `product-differentiation-audit.md`
5
- `01-vision.md` (and the wider intended architecture docs)
6
- `mcp-app-build-guide.md`
7
- `ai-visibility-policy.md`
8
- `ux-principles.md`
9
- `docs/product-functionality/ai-integration/`
10
- `docs/specs/ai-eval/`

Platform functionality:
- docs/specs/intelligence-workspaces/
- docs/specs/procurement-workspaces/
- docs/specs/product-guide-workspaces/
- docs/specs/sales-proposal-workspaces/
- docs/themes/workspaces/sales-proposals/
Located in the local development/kh-client-content-archive repo:
- Knowledge Hub — Platform Overview.md - extremely outdated
- Knowledge Hub — Claude Integration Guide.md
Client-related feedback:
- kb-hub-gap-analysis-liam.md
- kb-hub-gap-analysis-response-s188.md
- kb-hub-gap-analysis-action-tracker.md
- Sector-Intelligence-Brief-Liam-Final.md
- sector-intelligence-analysis.md
- client-personas.md - baseline personas, but should be expanded to include e.g., marketing, sales, finance etc.
  - This was created, specifically related to the intelligence workspace, but could easily be extrapolated to cover wider platform UC’s: s314-id75-usecase-affordances.md

Third-party Tooling

Platform:
- Ontology: https://jessicatalisman.substack.com/p/the-ontology-pipeline (this was the source that our current ontology approach is based on)
  - Are these the next stage?: https://nebula-graph.io/posts/ontology-and-graph-databases-the-missing-link-in-enterprise-ai + https://nebula-graph.io/posts/ontology-and-graph-databases-enterprise-ai-from-theory-to-production-reality
- Explicitly as an add-on for cyber-security awareness, given the expected, significant increase in cyber threats: https://github.com/simplifaisoul/osiris
- To enhance and improve our MCP Apps & tooling: https://www.mcpjam.com/?utm_source=linkedin&utm_medium=social
- Data extraction/transformation/utilisation:
  - https://github.com/Ontos-AI/knowhere
  - https://github.com/rowboatlabs/rowboat
  - https://github.com/strukto-ai/mirage
  - https://github.com/nexu-io/html-anything
- Anthropic plugin, providing ready-made SMB workflow examples and supporting tooling https://claude.com/solutions/small-business
- Backend: https://github.com/iii-hq/iii
Dev-workflow:
- Context management & tooling:
  - https://github.com/firstbatchxyz/watchmen - this looks like it could become a useful as part of our workflow evaluation setup
Both:
- Memory:
  - Up until now, the intention was to utilise MemPalace for platform user memory, but both of the below products look promising, for both platform, and our dev-workflow:
    - https://github.com/supermemoryai/supermemory - has a per-user setup
    - https://github.com/moorcheh-ai/memanto
- Agentic workflow:
  - https://www.raindrop.ai/ - potentially useful for agent observability in our dev-workflow, supporting the workflow evaluator setup. We will also look to introduce agents within the knowledge-hub platform, and need to understand if the open-source version of raindrop would be suitable.