Skip to content
Canonical

corpus-structure-feedback

Valuable Documentation

Section titled “Valuable Documentation”

Critical Note: All documentation may be outdated, empirical verification against the codebase, and current platform direction is required when viewing these documents, to avoid taking any previous context about decisions/direction for granted as being up-to-date. The documents contain extremely valuable context, but intent may have since evolved.

  • /knowledge-hub-docs-site/src/content/docs/themes/canonical-pipeline/intended-architecture/01-vision.md

    • /canonical-pipeline/intended-architecture/ - the wider, original, cocoindex pivot architecture.
    • project-plan.md - the plan we were working to, prior to v1-completion-sequence.md.

  • reference/platform-direction.md - key document for most up-to-date context.

  • id-71 and id-104 supporting documentation (along with their specs) - headless-requirement-refinement.md, lane-a-workshop-outcomes.md, surface-design-record.md, SYNTHESIS.md, RESEARCH.md (id-71 & id-104), and lane-a-workflow-inventory-strawman.md.

  • Ontology Docs: development/knowledge-hub-docs-site/src/content/docs/specs/wp6-ontology-harness/, *-docs/themes/phase-0-investigation/phase-b-prerequisite-1-onthology-pipeline and *-docs/themes/phase-0-investigation/phase-b-prerequisite-1-onthology-pipeline-feedback-investigation.

  • There are others, but these tasks provide further contrext - specs/id-80-forms-path-b/, id-117-unified-diff-surface/, id-120-cross-workspace-qa-dedup/.

  • cocoindex-write-model.md

  • decision-graph.md - used throughout the pre-cocoindex project planning phase to help keep track of platform decisions being made, but as with all older docs, this is for historic context and won’t necessarily reflect the current thinking or platform evolution.

  • content-project-docs.md - this was used to highlight key documents during the cocoindex planning work, but beware, even some docs which are tagged “current canonical” are no longer canonical references.

Feedback from reviewing id127-corpus-structure-research.md

Section titled “Feedback from reviewing id127-corpus-structure-research.md”

Document Feedback

Section titled “Document Feedback”
  • “every workspace-agnostic content_item the corpus mints is currently invisible to every workspace (zero view membership). This is not a folder-layout bug — it is a missing writer. The folder layout must stay workspace-agnostic; the writer is the fix.”
    • whilst the first part of this makes sense in principle, adjusting the terminology from workspace, and maybe even content_item, to data being used specifically for activities - completing a form, setting up a sector intelligence pipeline, creating and managing a new sales proposal etc., where each of these is an activity that can be completed using the structured data that the platform enables - so far example, a sales proposal would have a sales proposal ID, and the same for other application types, rather than each type having a workspace_id? (i.e., is this an unneccessary abstraction?)
    • currently procurement_workspaces, intelligence_workspaces, sales_proposal_workspaces, and product_guide_workspaces (and others) are placeholder/satelite until built out, but procurement and intelligence are the most developed so far.
  • “qa_sidecar mapping” is spoken about as though it is an option to implement, but I believe the foundation was already added for this - see .user-scratch/id59-sidecar-subtasks.md. Keep in mind that just because something is implemented, given we haven’t launched yet, that doesn’t mean it’s the correct approach, so here for example, if Q&A pairings shouldn’t be a sidecar, then that’s a reasonably straightforward decision to make.
  • “content_items has source_url” - it shouldn’t, see reference item - id-118, it’s likely source_url should have been dropped.
    • The evolution from content item to reference is shown through id-110, id-111, id-112, and id-118.
      • reference_items are ingested/extracted, but to my knowledge, never re-ingested/re-extracted.
  • “It walks every file” - my understanding is that only files with changes get walked - is that not the case?
  • Upload route is stale (pre-cocoindex pipeline pivot) and the final approach we use (if any) is yet to be determined - UI folder-drop was built more recently, with a view to being a mechanism for adding to the dedicated local fs, but as with Q&A sidecar, we are in a position to adapt right now.
  • Coolify is also S3 compatible.

Wider Implications

Section titled “Wider Implications”

We need to ensure that we stay grounded in who are initial clients will be, and the impact we could have for them. Below are some key points to keep in mind, but these are also very high-level: - SMBs will have some structured, but mostly unstructured data. - Disparate systems - some will have a CRM, others won’t know what a CRM is. - Unlikely to have anyone in the business focused on data governance. - Likely to be repeatedly completing the same activities inefficiently e.g., completing forms where the same data is re-used, and provenance, auditability, data quality, and ultimately, usage of “canonical” records are critical.

The corpus structure research doc is excellent, but we need to take a step back, to ensure we take the right steps forward here. The purpose of the pipeline is to keep data canonical, enabling AI agents and humans to be able to confidently utilise the data for various application types. It’s possible that with the introduction of OKF as the key underlying knowledge framework, this is being conflated with the client documentation corpus - understandably, as it’s clear this aspect isn’t clearly defined yet, and if I’ve understood correctly, there’s a huge, valuable opportunity here, which would also considerably simplify how we go from ingesting client content, to it be genuinely useful to agents and humans alike.

The section “3.2 Concrete example directory tree” I think shows the crux of the issue, whilst the structure may make sense, the question is how the data gets created/sourced, which I think the combination of company docs + OKF now answers.

It’s always been the intention that after v1 was complete, we start to look at how we can more widely help SMBs with “getting their data in order”, so that it’s not just enterprises who will benefit from agentic workflows. OKF appears to be that mechanism, especially when combined with our ontology pipeline, and cocoindex pipeline. Our initial thinking was to “gate keep” the documentation that we allow the pipeline to ingest, which I believe aligns to OKFs enrichment second pass, where the enrichment agent uses only authorititative documentation to enrich OKF concept files. We have documentation from our first client which essentially is their “authorititative” sources.

The Guides functionality in our platform (currently called Product Guides, but to be expanded to include both Product and Service guides) already has a considerable UI, but is due to be reworked as one of the core application types (as we’re doing with Procurement, Intelligence, Sales Proposals etc.), and so far has two unapproved specs (PRODUCT & TECH) + a spec which was created to align the previous implementation of the functionality to our first clients requirements (p0-product-guide-section-alignment-spec) - the relevance being that these show how client’s think about ordering their product/service knowledge, as the purpose of the guides is mutli-faceted e.g., both sales personnel, and clients, can use them to understand a companies products and/or services (this feeds into the “one record, many views” philosophy). The original purpose of the Intelligence functionality was to feed the guides, to enable management to make informed, data-driven product/service decisions, based on what’s happening in a particular sector. It would also be possible to use things like Support Tickets/Customer feedback, to help drive these decisions too. I’m mentioning the guides now, even though the procurement and intelligence domains have been prioritised first, as I believe there is likely value within the Guides functionality in terms of identifying the type of data that’s required to create a guide - our current setup already includes the concept of “coverage”, whereby a clients authorititave documentation may be able to populate some aspects of the guide, but it also identifies gaps, which of course are incredibly valuable in terms of creating actionable insights, useful for both AI agents and humans.

Given the above, something we therefore need to determine is “what” gets ingested (e.g., point a connector to Notion or a local FS and ingest ALL content, and use the platform to sort/identify/create authorititave content, or continue to gate keep, and ensure only authorititative content is targeted for ingestion?), and where the “what” is stored, in what format, and how is it maintained.

I’ve outlined a potential flow below, but this is me as a non-technical user, so there will likely be lots that I haven’t considered yet.

  • Potential Flow:

    • [Raw Sources] (Local FS, Slack, Notion, SQL) -> [CocoIndex Pipeline] (LLM Extraction & Delta Compute) -> [OKF Bundle] (Interlinked Markdown, saved in Git) -> [AI Agents / Canonical Platform UI]

  • High-level breakdown:

    1. CocoIndex = “Producer” - Upon first ingestion/change, map raw inputs directly into the OKF structure.

    2. Use CocoIndex custom targets to configure pipeline to write the .md files into our specific hierarechical directory tree.

    3. Use the cocoindex native entity relationship and dependency mappings to build a localised concept graph, before flushing files to storage.

    4. Host the files in a private git repo, creating a local directory cloned from the repo, enabling git commits/pushes for “knowledge sync”, creating the bulletproof single source of truth, plus gaining authorship and point in time version rollback natively.

    5. Consumers - Would knowledge then be a structured, cross-linked OKF repo, reducing (removing?) the requirement for complex, lossy RAG loops? If so, this may have implications for how the data gets utilised by the platform’s various application types.

      • We should look at how cocoindex MCP works currently, what the impact would be for our own upcoming improvements to Canonical MCP tooling (id-71 & id-104), and how this would impact are planned headless agents, starting with Goose.

  • As we have our ontology pipeline defined on the docs-site, presumably this allows us to ensure that ontology rules are followed during the extraction process (I imagine this is already in place, but things like keeping “allowed_types” and “allowed_relations” in sync), as well as the potential to be used as an “Ontology Semantic Linter”, ensuring there is governance for what gets added to the OKF directory.

  • I also believe that using these would be extremely valuable:

    • index.md for progressive disclosure.
    • log.md for chronological change history.

  • OKF provides reference implementations for - these could potentially be “lift and shift” rather than building from zero:

    • Enrichment agent, drafting an OKF concept doc for every table view, followed by second LLM pass for crawling authoritative docs and enriching each concept with citations, schemas, and join paths.
      • https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf/src/reference_agent
    • Static, interactive HTML OKF bundle visualiser https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf/src/reference_agent/viewer.
      • Cocoindex already provides the source code for something like this, which could point at our OKF directory: https://cocoindex.io/docs/examples/docs-to-knowledge-graph/
    • There is also already a Claude Code plugin for using OKF, which includes scripts for validation https://github.com/scaccogatto/okf-skills.
    • I also found the following tools, which use Google’s Knowledge Catalog for metadata management, but it wouldn’t take much to adapt for our use case: https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/toolbox.

  • “3.4 How workspace relevance is assigned instead (NOT folders)” - if we ensure that our underlying framework is OKF, and consider resolving the application/workspace conflation, does a content item ever need to write content_item_workspaces, or maybe even more pertinent is what exactly is a content_item? We have source documents, Q&A pairs, reference items - is there anything a content_item gives us that warrants it’s separate existence (NB. Q&A pairs and reference items previously sat within the content_items table, but have since been extracted to their own tables)? Conceptually, are content_items replaced by the OKF concepts - i.e., a source document (raw content) gets extracted, and then either updates/removes/adds to, the OKF directory? Something I would be fairly confident of, for example, is that content_item_workspaces probably isn’t needed - for procurement, at least, a form would have an ID, and Q&A pairs (each with their own unique ID) would be used to complete the form. If you take something like creating a new marketing campaign, an agent will presumably get more value traversing the OKF concept files, and the formed product/service guides, and reference_items, than trawlling through content_item records. id-117 provided a diff UI, which again, I think diffs documents, rather than content_items. For context on where the concept of a content_item came from, the Canonical Platform (and Knowledge Hub before it) where forked from a system you and I created called the IMS, which has a content ingestion pipeline which processes content e.g., LinkedIn/Reddit articles, Emails, webpages etc., to create a personal knowledge base - this concept remained in place when we forked the repo, but probably doesn’t fit how the platform has since evolved.

  • Our current pipeline should already be closely aligned to the final link I’m sharing here, but if not, here is cocoindexes native approach to creating a self-updating Wiki - ignoring that the use case they use is for codebases, as this can of course be transposed onto our setup, keeping OKF files updated: https://cocoindex.io/blogs/multi-codebase-summarization/