AST + Dataflow Tool — PRODUCT

Status: DRAFT-S1 (kh-ast-S1 first cut) ID: ast-dataflow-tool (kebab feature name; no GH/Linear ticket) Companion: TECH.md (to follow in WP2)

Summary

A TypeScript-aware AST + dataflow probe that KH agents (orchestrator, executor, checker, curator) and Liam can query during spec, plan, implementation, and debug phases of any work package. The tool answers semantic call-graph and data-flow questions — “where is digests referenced”, “who writes to content_items.summary”, “which callers pass a string-literal where a UUID is required” — with the same fidelity as ts-morph + the TypeScript language server, returning small structured results that fit in an agent context window.

The tool exists because cocoindex-code (text search) and gitnexus (git provenance + framework structural graph) do not resolve TypeScript symbols across files. The first-use cases (§ First use cases) are the cross-file rename sweeps queued in the canonical-pipeline §11.3 combined pre-launch PR — the work most likely to ship subtle “build the thing, forget to turn it on” bugs without semantic visibility.

Audience

The “user” of this tool is a caller asking a structural question about KH’s TypeScript code:

KH agents invoking the tool from a terminal (orchestrator scoping a workpackage, executor verifying a refactor’s blast radius, checker auditing whether a commit touched what it claimed, curator triaging an out-of-scope finding).
Liam (and any future human collaborator) invoking the tool from a terminal during planning or debugging sessions.
Other tooling (CI scripts, ad-hoc one-off audits) that needs a semantic query surface over the KH TS corpus.

The tool is not user-facing in the Knowledge Hub app sense — there is no UI, no end-user visible affordance, no Warm Meridian design surface. It is infrastructure for the people and agents building the Knowledge Hub.

Goals

Answer 8–12 named queries (§ Behavior) over the entire KH TypeScript corpus (app/, components/, lib/, mcp-apps/, scripts/*.ts, __tests__/, e2e/) with semantic, not textual, accuracy.
Be invocable as a local CLI (bun run ast-dataflow <query> <args>) so any agent or human can call it from any KH worktree. An MCP server registration is an optional add-on (§ Behavior, invariant 28); it does not have to ship for the tool to be useful.
Be cheap enough to invoke conversationally — cold-start one-shot under 30 s, warm-cache queries under 5 s P95 (§ Performance invariants).
Return small, structured results — JSON or JSONL with one row per finding, capped at a per-query limit so an agent does not accidentally pull tens of thousands of rows into context.
Be correct against a curated ground-truth fixture set drawn from real KH files (§ Correctness invariants).

Non-goals

Not a replacement for cocoindex-code. Text-pattern search (“find me every comment mentioning OAuth”, “where is the string legal-team@…”) remains cocoindex-code’s job. This tool does not index comments, JSDoc, or string-literal content for semantic search.
Not a replacement for gitnexus. Git provenance (“who changed this symbol last”, “which commit introduced this route”), framework-aware structural queries (HANDLES_ROUTE, FETCHES, QUERIES edge types), and process-flow traces remain gitnexus’s job. This tool does not consult git history.
Not a runtime profiler. It does not measure execution counts, latency, call frequency, hot paths, or any property only observable at runtime.
Not a SQL/DB analyser. It does not parse SQL strings, evaluate PostgreSQL parser output, or know what a CHECK constraint means. It can find TS-side references to a column name (passed as a string literal to supabase-js methods or stored in fixtures) but it does not validate that the column actually exists.
Not a Python pipeline analyser. scripts/kb_pipeline/, scripts/*.py, and the Cloud Run job are out of scope. A future sibling tool may cover Python; this one is TS/TSX/JS/JSX/MTS/CTS only.
Not a build-step validator. Type errors, lint, knip, and the test suite remain the build-step authorities. The tool may surface a likely type conflict as part of a query result, but it does not gate CI and never claims the role of tsc --noEmit.
Not an autofix tool. It answers questions; it never edits source. Renames, splits, extracts, and other transformations remain the caller’s responsibility (ts-morph + ast-grep, IDE refactors, or hand edits).
Not a documentation generator. It does not produce API docs, dependency diagrams, architecture overviews, or other narrative artefacts. It returns query-shaped data only.

Behavior

Each numbered invariant is independently testable. The query set deliberately covers both the cross-file rename sweep cases (§ First use cases) and the broader “I am about to edit this; what breaks?” surface the workflow harness needs.

Query surface

callers(symbol) — who calls a function or method? Given a fully qualified symbol — a file path plus a function/method name, or a unique exported name — the tool returns every call site across the corpus, with file path, line, column, the enclosing function/method (or “module top-level” if not in a function), and whether the call is direct, re-exported, aliased on import, destructured, or via a computed property. Results are semantic: a call to a same-named foo() in a different module does not appear.
callees(symbol) — what does a function or method call? Given the same shape of input, the tool returns every function/method called from inside the body of the named symbol, with caller-side context (file, line, column) and callee-side context (declared file, line). Method calls on a value of an inferred type are resolved through the type checker; indirect calls (variables holding a function reference, arrow-function parameters, etc.) are reported with a resolution: "indirect" flag rather than silently dropped.
references(symbol) — semantic find-references for any named symbol. Generalises callers to non-function symbols: types, interfaces, enums, enum members, classes, constants, variables, type aliases, JSX components, exported tuples like VALID_CONTENT_TYPES. Returns every read and write site, with each result tagged read | write | reexport | typeReference | typeOnly | jsxComponent. Definition sites are identified by an isDefinition: true boolean on the row rather than a dedicated declaration kind. Refs through re-export barrels are followed and reported with their import-time path so the caller can see both the source declaration and the alias that reached it.
importers(module-path) — who imports a module? Given a path like lib/supabase/safe.ts or a package specifier like @/lib/ai/change-reports, the tool returns every file that imports from that path, with the specific named imports drawn, the import style (named / default / namespace / type-only), and whether the import is unused (declared but no reference in the file). Re-export-only imports are flagged as such.
column-reads(table, column) — which functions read a Supabase column? Given a Postgres table + column name (both strings), the tool finds every call site in the TS corpus that reads that column via supabase-js query chains. Heuristic-but-grounded: detects .from('table').select('…col…'), .from('table').select('*') (flagged as wildcard), client.rpc('fn', …) parameters and return-type fields when the type checker resolves them, and destructuring on rows returned from those calls. Each result reports enclosing function, file, line, the matching call-chain expression, and a confidence tier (exact | wildcard | indirect).
column-writes(table, column) — which functions write a Supabase column? Same shape as column-reads, restricted to .insert(…), .update(…), .upsert(…), and equivalent RPC payload-object property writes. Object literals are inspected for the named property; spread expressions are followed one hop where the source object is statically resolvable. Wildcard / indirect writes are reported with the same confidence tier as reads.
type-evolution(type-name) — how does a type flow across the codebase? Given a type or interface name, the tool returns its declaration site plus every re-export, alias, intersection, generic specialisation, and import the type appears in. Useful when an interface is renamed or its shape changes — answers “did the new field reach every downstream consumer of this type?”.
reexport-chain(symbol) — trace through barrel files to the source. Given a symbol resolved to a barrel re-export (from '@/lib/bid'), the tool returns the chain of export … from … declarations that lead from the alias to the source declaration. Used to enforce the no-barrel-reexports invariant in CLAUDE.md by surfacing chains a caller can rewrite to direct imports.
dead-exports(scope) — exported symbols with zero references. Given a file, directory, or glob, the tool returns every exported symbol that has zero non-self, non-test references in the corpus. Test files are reported separately so a caller can choose whether tests count as “real” consumers. Useful for the “build the thing, forget to turn it on” gotcha and as a finer-grained complement to bun run knip.
string-literal-uses(needle, scope?) — semantic-context string search. Given a string literal value (e.g. 'project_id', 'digests'), the tool returns every TS/TSX file location where that exact literal appears as a string literal node — not in a comment, not as part of a longer string, not as a TypeScript type. Each result reports the enclosing statement kind: argument to a function call, object-literal value, JSX prop, type literal, array element, switch case, etc. This is the rename-sweep primitive: text-grep produces false positives in comments and unrelated string contexts; this query produces only structural matches.
fixture-uses(needle, kinds?) — references across data/fixture files. Given a string needle (column name, table name, magic literal), the tool returns every match in the project’s test/data fixtures (__tests__/**/*.json, __tests__/**/*.ts flagged as fixture by path or convention, e2e/fixtures/**, scripts/tests/fixtures/**, docs/ontology/*.md frontmatter, supabase/types/database.types.ts). Matches inside JSON object keys vs string values are reported separately so the caller can distinguish “this column is named in a fixture row” from “this string happens to appear as content”.
enum-member-uses(enum.member) — references to a specific enum member. Given BID_STATES.DRAFT or ProvenanceModel.client_defined, the tool returns every read of the named member as a property access, every type-position reference, and every string-literal use that resolves to the same value (where the enum is a const object or as const tuple). Used when retiring an individual enum value (e.g. kb_section retirement in §11.3 item 3) without renaming the whole enum.

Result shape invariants

Every query returns JSON or JSONL on stdout (CLI) or as a structured MCP tool response (MCP transport). The schema for each query is stable across versions — a minor version may add fields, not remove or rename them. Numeric line/column offsets are 1-based to match editor conventions.
Results are capped per query at a configurable maximum (default 200 rows). When the cap is reached, the response includes truncated: true and a total_estimated: N field so the caller can decide whether to refine the query or page through it. Truncation prefers spatial coverage — distinct files come first, multiple hits per file thin last.
Every result row carries a confidence tag of exact, wildcard, or indirect. exact means the type checker resolved the symbol to the queried declaration. wildcard means the result matched a .select('*') or similarly broad query that may include the target column but cannot confirm it without runtime data. indirect means the result depended on a structural heuristic (string literal, fixture grep) and the caller should treat it as a candidate, not a proof.
Result paths are repo-root-relative POSIX strings. The tool never emits absolute paths, never emits Windows-style separators, and never emits paths outside the indexed corpus.

Indexing and lifecycle invariants

The tool runs from any KH worktree (main, parallel tracks, agent worktrees) without explicit configuration. It discovers the corpus from tsconfig.json at the worktree root (the same configuration tsc and the IDE use) and indexes every file tsc would compile.
A cold-start invocation (no prior cache) completes any query within 30 s on the current KH corpus (roughly 1.4k production TS files plus tests, per docs/generated/codebase-stats.md). 30 s is the upper bound; the tool should be visibly faster on smaller scopes.
A warm-cache invocation completes the same query within 5 s P95 and 2 s P50 for the eight resolution-based queries (callers, callees, references, importers, type-evolution, reexport-chain, dead-exports, enum-member-uses). Heuristic queries (column-reads, column-writes, string-literal-uses, fixture-uses) are bounded at 10 s P95 / 5 s P50 because they fan out across the corpus rather than starting from a resolved symbol.
Cache invalidation is per-file by content hash. Editing one source file invalidates that file’s indexed facts and any transitively dependent semantic results; unrelated files are not reindexed. Switching git branches or worktrees does not require an explicit cache flush — the next invocation rebuilds whichever entries are stale.
A query never blocks waiting for a separate long-running daemon. The tool either starts a fresh ts-morph project per invocation (cold-path) or reuses a persisted index file from a previous invocation (warm-path). It is acceptable for the warm-path to detect cache corruption and fall back to cold-path on the same invocation, but the user-visible behaviour is “I asked a question; I got an answer within the latency budget”.
Stale-index detection is loud, not silent. If a query result depends on a file that has been deleted, renamed, or whose hash has changed since the cached entry was written and the rebuild does not complete in time, the tool returns the partial result with stale: true and the list of files whose cached entries were skipped. The caller can then decide whether to rerun or accept the partial answer.

Correctness invariants

For the eight resolution-based queries, the tool’s output set equals ts-morph’s output set on the curated ground-truth fixture. The ground-truth fixture is the canonical correctness oracle: every query has a small set of fixture files with hand-labelled expected results that the test plan in TECH.md will exercise. Equality means no missing rows, no extra rows.
For the heuristic queries (column reads/writes, string-literal, fixture), every result row that the ground-truth fixture marks confidence: exact is present in the tool’s output. False positives in the indirect tier are acceptable; false negatives in the exact tier are not.
Re-export aliasing does not produce false negatives. A reference to sb imported as import { sb } from '@/lib/supabase/safe' and a reference imported as import { sb as sbClient } from '…' both appear in callers('sb') / references('sb') results, with the alias name surfaced as metadata.
Type-only imports are reported as such, not collapsed into runtime references. import type { Foo } from '…' counts as a typeReference in references('Foo'), not as a runtime read.

Invocation invariants

The CLI surface is bun run ast-dataflow <query> [args…] [--json | --pretty] [--limit N]. With no arguments, the tool prints the query catalogue and exits zero. Unknown queries exit non-zero with a list of valid names. Malformed arguments exit non-zero with the offending argument and its expected shape. The CLI is the primary and only required invocation surface.
An MCP server registration is an optional add-on, not a requirement. If a registration ships, it exposes the same query catalogue as the CLI (either one tool per query, or a single dispatching ast_dataflow tool with a query argument — TECH.md decides). The CLI must remain independently usable whether or not the MCP surface exists, and no feature is reserved for the MCP-only path.
Errors are returned as structured failures, not crashes. A query against a non-existent file, a syntax-error file, or a symbol that resolves to multiple declarations returns a structured error row with error.kind (unknown_file | parse_error | ambiguous_symbol | out_of_corpus) and a remediation hint. The CLI exits zero with the error in result data (and a non-zero exit only for transport-level failures: missing binary, unreadable tsconfig.json).
The tool never edits source files, never writes outside its own cache directory, and never makes network calls. All inputs are read-only file system access scoped to the worktree; all outputs are stdout (or the MCP response channel if that surface is enabled). The cache directory lives at a single known path under the worktree (specified in TECH.md) and can be deleted at any time without breaking the tool.

First use cases

Three concrete questions the tool will be asked in the next 30 days, drawn from docs/specs/id-31-0.9-canonical-pipeline/PRODUCT.md §5 (the combined pre-launch PR list) and docs/specs/classifycontent-subtopic-contract-spec.md. These anchor the query surface above and act as living acceptance criteria — if the tool cannot answer them well, the surface is wrong.

digests → change_reports rename PR blast radius. Item 7 of the canonical-pipeline §11.3 combined PR. Before the rename lands, the workflow-orchestrator queries the tool with string-literal-uses('digests') + string-literal-uses('digest') + references('ChangeReport') + importers('@/lib/ai/change-reports') and produces the full TS-side change set: imports to rewrite, type aliases to update, JSX labels to keep (“Change Reports” — the UI label was already correct), and tests asserting on the old string. Text grep would return hundreds of false positives because “digest” appears in unrelated contexts (HTTP digest auth, message digests in security middleware, etc.). The tool must filter to TS-symbol-resolved hits.
project_id → workspace_id rename across 44 code files. Item 5 of the canonical-pipeline §11.3 combined PR — explicitly referencing “ts-morph + ast-grep” in PRODUCT.md line 199. The workflow executor queries the tool with column-reads('bid_questions', 'project_id') + column-writes('bid_questions', 'project_id') + column-reads('templates', 'project_id') + column-writes('templates', 'project_id') + string-literal-uses('project_id') to enumerate the 44 files, then references on every type symbol referencing the old name. The tool reports each result with confidence (exact vs indirect) so the executor knows which rewrites are safe and which need a human eyeball pass.
classifyContent userId contract enforcement. The CLAUDE.md gotcha “classifyContent userId must be a UUID” already has a spec at docs/specs/classifycontent-subtopic-contract-spec.md. The workflow-checker queries callers('classifyContent') to enumerate every call site, then for each enclosing function inspects the argument passed for the userId parameter. The tool’s result reports the argument expression kind (Identifier | StringLiteral | TemplateLiteral | MemberExpression | …) so the checker can flag every call passing a string literal where a UUID is required. Without semantic resolution this is a hand-grep job that runs against 1k+ files and produces noise.

Open questions

All six OQs below were resolved as design decisions during S1 WP2 (TECH.md authorship). Each resolution block records: (a) the decision, (b) the TECH.md section where the full rationale lives, and (c) the rejected alternative and why it was ruled out. The OQ register is closed as of S1; no further OQs are open.

OQ1 — Skill bridge surface. RESOLVED. CLI is the primary and only required surface; the MCP optional wrapper is deferred until agent usage of the CLI proves sufficiently clumsy to justify the additional build complexity. See TECH.md §Installation surface. Rejected alternative: ship CLI and MCP as co-equal peers from S1 — rejected because dual-build adds two failure modes (server boot, stdio transport bugs) and the context cost of an MCP tool call versus a Bash call is negligible at current query volume.
OQ2 — Cache anchoring. RESOLVED. Cache keys are per-file content hash only; the current git revision is recorded in index.json for staleness debugging but plays no role in invalidation. See TECH.md §Cache strategy. Rejected alternative: composite key (content hash + git rev) — rejected because it forces full cache rebuilds on every branch hop even when no file content has changed, which wastes the latency budget without improving correctness.
OQ3 — Heuristic query confidence tiers. RESOLVED. Un-typed .from('table') calls are accepted and downgraded to indirect confidence; the tool does not reject them. See TECH.md §Query implementations (row column-reads). Rejected alternative: require typed Supabase client usage and reject un-typed calls — rejected because a significant portion of pre-migration KH code uses untyped clients, so rejection would produce false negatives in the codebase the tool most needs to serve.
OQ4 — Index lifecycle across worktrees. RESOLVED. Each KH worktree maintains its own .ast-dataflow-cache/ directory; the cache is never shared across the three concurrent worktrees. See TECH.md §Cache strategy. Rejected alternative: globally shared cache — rejected because sharing would allow uncommitted edits in one worktree to surface in queries from another, producing semantically incorrect results on branch-divergent ASTs.
OQ5 — String-literal extraction scope. RESOLVED. string-literal-uses is default-on for test files, fixtures in __tests__/, and e2e/ specs; a --exclude tests flag allows opt-out. See TECH.md §Query implementations (row string-literal-uses). Note: the --exclude tests flag itself is S3 backlog, not S2 work. Rejected alternative: default-off for test scopes — rejected because rename sweeps need maximum reach (test fixtures are often the first place a stale string literal hides), and the caller can always post-filter; default-off would silently miss them.
OQ6 — LSP fallback. RESOLVED. No LSP fallback for V1; ts-morph alone is the resolver. See TECH.md §LSP fallback. Rejected alternative: fall back to an LSP request when ts-morph cannot resolve a dynamic import — rejected because ts-morph wraps the same TypeScript compiler and type checker an LSP server would consult, so an LSP adds a daemon boot and IPC round-trip without providing any additional semantic capability.

Amendments — id-375 productionisation (2026-07-27)

Grounded in specs/id-375-ast-dataflow-productionise/RESEARCH.md (5-agent research pass: measured baseline, market scan, validated fix designs). Amendments A2-A4 change invariant semantics; A1 and A5-A7 are additive. A2 and A3 RATIFIED by Liam 2026-07-27, with one rider on A3: MCP registration in .mcp.json is deferred to the extraction phase — canonical will install the extracted tool and register the MCP as an end user would, and the server setup gets an /mcp-builder best-practice audit then.

A1 — Invariants 2 (callees) and 11 (fixture-uses) move from DEFERRED to SHIPPED (id-375 waves 2-3). Design decisions adopted, D1-D5 in RESEARCH.md §7: fixture-by-convention = /fixtures/ path segment or *-fixture.ts basename; docs/ontology/*.md glob retained as a no-op (target moved to the private docs-site — scanning it would breach inv 30); database.types.ts PropertySignature names → key, union string literals → value; external callees excluded by default with top-level externalCount + --include-external (rows carry callee.file: null, never a node_modules path, honouring inv 16); new additive ErrorKind not_callable.
A2 — Warm path is a warm process, not a persisted index file (RATIFIED 2026-07-27). Invariants 19-22’s “warm-cache” language is reinterpreted: measurement showed 11 of 12 queries consume the live type-checked AST, so a per-file extracted-facts cache serves ~1 query (RESEARCH.md §6). The warm path is a long-lived process (MCP server) holding the ts-morph Project; inv 20’s per-file invalidation maps to per-file refreshFromFileSystem() on mtime+size change; inv 21 holds because the CLI remains the always-available cold path and the server is a client-owned per-session subprocess, not a shared daemon; inv 22’s stale-loud contract maps to a meta: { refreshedFiles, addedFiles, removedFiles, staleFiles } envelope field. No on-disk cache directory exists; inv 30’s “cache directory” clause is vacuous until one does.
A3 — Invariant 28 MCP wrapper: DEFERRED → shipped-optional (RATIFIED 2026-07-27 — registration deferred to extraction phase, see header). Shape per inv 28’s permitted option (b): a single dispatching ast_dataflow tool. Activation trigger (OQ-R4 “observed agent friction”) fired: rename-sweep chains 5 cold CLI calls ≈ 60-90 s; importers breached its P-19 budget in any state. CLI remains primary and independently usable.
A4 — exact tier hardening (detectIsTyped). Inv 15’s exact promise (“the type checker resolved the symbol”) was breached: untyped supabase-js clients echo the table-name literal into the builder generic, so text matching over-claimed isTyped. Proof of a typed client is now structural — the .from() return type carries a non-any Relation type argument with a concrete Row shape — with the client-binding explicit-type-argument check as secondary. No rows are dropped by the fix (inv 24 unaffected); false-exact rows demote to indirect.
A5 — Spatial truncation (inv 14) is implemented via a shared truncateSpatial helper — deterministic (file, line, column) total order, round-robin-by-file thinning. Exemptions, documented in TECH.md: flow-trace and reexport-chain rows are path/chain hops, not coverage sets; they keep traversal order.
A6 — New report surface: schema-coverage (ROADMAP extension, same status as enum-uses). Enumerates every table/column from the generated database.types.ts, one-pass-scans the corpus for all .from() chains (including one-hop const-resolved table names), and emits a per-column verdict: wired | read-only | write-only | undecidable | unwired. Verdict semantics are conservative by construction: wildcard/indirect evidence NEVER counts as wiring; a column is unwired only when its table also has zero wildcard/indirect/unattributable smoke; unknown tables and columns are structured errors (unknown_table / unknown_column), never silent empty results. The report states its blindness per table (SQL function bodies, api.* views, external PostgREST consumers). --report <path> is the only write path, honouring inv 30’s no-writes default.
A7 — Table-argument resolution widened. .from(X) where X is an identifier or property access whose type is a single string-literal type resolves to that literal (one hop, type-checker-backed). Plain-string arguments remain unattributable and are surfaced as counts in schema-coverage, never silently dropped.