Type-safety pipeline — TECH

Status: DRAFT-S8 (kh-ast-S8 Wave 1 — R-WP12 spec triple) PRODUCT.md: ./PRODUCT.md (WP-D only; numbered invariants D-1…30). Companions: ../investigations/R-WP12-type-safety-pipeline.md (S7 feasibility brief, source of WP decomposition); ../TECH.md (parent — the ast-dataflow tool surface this pipeline composes on top of); ../ROADMAP.md R-WP12 row. Scope: WP-D implementation detail (most of this doc) + WP-A/B/C/E/F technical sketches + ESLint rule design + JSONB inventory + Supabase types CI plan. The mechanical WPs (A/B/C/E/F) do not have their own spec triple per Liam OQ1 — they are sketched here and implemented directly from the sketches with TDD. ROADMAP ID map (Wave 5, S9): WP-D → R-WP17 (full report) · WP-B (KH-wide sweep) → R-WP18 · WP-C → R-WP19 · WP-E (5-tool scaffold) → R-WP20 · OPS-T1 decision gate → R-WP21. WP-A and the narrow S8 3-site WP-B fix ship in Wave 4 (S8) as R-WP12-WPA / R-WP12-WPB. WP-F (JSONB write-side validation) is a follow-up roadmap item gated on Liam’s review of the JSONB inventory below.

Context

The Knowledge Hub TypeScript codebase has the structural skeleton of a typed client/server boundary — Zod input validation, typed Supabase clients, named response interfaces in types/ — but the route ↔ fetcher edge is unchecked (per the tRPC evaluation, docs/plans/phase-0-investigation/trpc-evaluation.md, and the R-WP12 brief). The cumulative effect is five named gaps catalogued in the brief (§Current type-safety inventory):

Gap 1 — route/fetcher <T> drift. fetchJson<T> at lib/query/fetchers.ts:29 returns res.json() as Promise<T> with no structural link to the matching app/api/**/route.ts handler. 193 routes × 100+ fetchers = the largest unguarded boundary in the codebase.
Gap 2 — opaque-Json RPCs. 14 PL/pgSQL functions declared Returns: Json in supabase/types/database.types.ts (verified by grep -c "Returns: Json"). Call sites must cast manually; the type checker sees unknown-equivalent.
Gap 3 — unnecessary casts on typed RPCs. Structured RPCs return fully typed rows but call sites cast to Record<string, unknown> (e.g. lib/bid/bid-queries.ts:83-86, lib/mcp/tools/search.ts:152). 79 such casts in app/ + lib/ as of S8.
Gap 4 — MCP outputSchema absent. 58 MCP tools, zero outputSchema registrations (grep -rn "outputSchema:" lib/mcp/tools/ empty). toStructuredContent erases types via JSON round-trip.
Gap 5 — domain-typed JSONB columns. 17 distinct JSONB columns across 13 tables; writes are hand-cast with no shape-validation enforcement.

The brief proposes an MVP across five workpackages (WP-A through WP-F) using the existing AST tool surface. Liam’s S8 decisions reorder these:

Liam decision	Effect on this spec
OQ1 — start with WP-D (novel, audit-style)	WP-D gets its own PRODUCT.md + TECH.md section. Other WPs (A/B/C/E/F) sketched only.
OQ2 — migration sprint gated on WP-C	Sprint stays in roadmap-future state; WP-C produces the inventory that gates the decision.
OQ3 — CI gating, list-to-fix shape	WP-D `--ci` mode (per PRODUCT.md D-19); bundled with Supabase types CI in one workpackage.
OQ4 — multi-agent MCP outputSchema rollout deferred	WP-E scaffold is in scope; full rollout waits on main-track MCP finalisation.
OQ5 — JSONB preference + inventory	Inventory below; migration sketches per column; specific high-risk columns flagged.
OQ6 — Supabase types CI + Helper/Response patterns	CI plan below; adoption analysis for the two patterns.
Liam-add — ESLint rule paired with WP-A/B	New ESLint rule designed in this spec; ships in the same WPs.

Relevant existing code

File	Why it matters
`lib/query/fetchers.ts:29`	`fetchJson<T>` — the unchecked-cast root cause for Gap 1. ~100 fetcher call sites carry the `<T>` generic.
`lib/query/fetchers.ts:356`	`mutationFetchJson<T>` — mirror for POST/PATCH/DELETE fetchers.
`lib/query/fetchers.ts:71`	`fetchJson<TaxonomySyncStatus>` — canonical false-negative-tolerance fixture (PRODUCT.md D-18).
`app/api/review/stats/route.ts:76`	`statsResult.data as Omit<ReviewStatsResponse, ...>` — canonical opaque-Json RPC cast site.
`lib/bid/bid-queries.ts:83-86`	`(row as Record<string, unknown>).needs_sme_count` — canonical Gap 3 anti-pattern.
`lib/mcp/tools/search.ts:152`	`(r: Record<string, unknown>) => ...` on `hybrid_search` typed result — canonical Gap 3 cast on already-typed RPC return.
`lib/mcp/tools/content.ts:520`	`metadata as unknown as Json` — canonical double-cast on JSONB write (Gap 5).
`lib/mcp/tools/shared.ts:135`	`outputSchema?: OutputArgs` on `DefineToolConfig` — the unused affordance Gap 4 reactivates.
`lib/mcp/tools/shared.ts:213`	`toStructuredContent` — `JSON.parse(JSON.stringify(data))` round-trip.
`supabase/types/database.types.ts:3227`	`get_author_analysis: { Returns: Json }` — one of 14 opaque RPCs (Gap 2).
`lib/ast-dataflow/queries/references.ts`	Resolution primitive — WP-D’s main consumer.
`lib/ast-dataflow/queries/string-literal-uses.ts`	Secondary primitive — URL matching between fetcher and route.
`lib/ast-dataflow/queries/reexport-chain.ts`	Resolves type aliases across barrels (`types/index.ts` → `types/review.ts`).
`eslint-rules/index.js`	The existing local ESLint plugin entry — the new rule lands here.
`eslint-rules/no-silent-promise-catch.js`	Style reference for the new ESLint rule.
`.github/workflows/ci.yml`	The CI surface the gating step bolts onto.
`.github/workflows/schema-parity.yml`	Style reference for a Supabase-touching workflow (the `gen types` step mirrors its diff-and-fail shape).
`supabase/types/database.types.ts`	Source of truth for the JSONB inventory.
`docs/reference/SCHEMA-QUICK-REFERENCE.md`	Schema cross-reference for the JSONB inventory.

Proposed changes

WP-D — Route/fetcher type-drift detector (primary spec target)

Files touched:

lib/ast-dataflow/queries/type-drift-detect.ts (new — the detector query).
lib/ast-dataflow/queries/index.ts (modified — register new query).
scripts/ast-dataflow-cli.ts (modified — add type-drift-detect subcommand with --limit, --scope, --ci, --update-baseline, --interface-pattern, --json, --pretty flags).
docs/generated/type-drift-baseline.json (new — empty JSONL on first commit; the list-to-fix ledger).
docs/generated/type-drift-report.md (new — regenerated by --ci).
__tests__/lib/ast-dataflow/queries/type-drift-detect.test.ts (new — TDD fixture suite).
__tests__/lib/ast-dataflow/fixtures/type-drift/ (new fixture directory — see §Testing).

Algorithm overview: the detector composes three existing AST queries (references, string-literal-uses, reexport-chain) plus a small new classification layer. The compose-not-reinvent principle (per parent TECH.md “Data model” — “ts-morph project IS the graph”) keeps WP-D additive.

1
1. Enumerate candidate interfaces:
2
   - Walk types/**/*.ts, app/api/**/route.ts, lib/query/fetchers.ts.
3
   - For each TypeAliasDeclaration / InterfaceDeclaration whose name matches
4
     the candidate regex (Response$ | Payload$ | Result$ | Body$) — or any
5
     identifier used as a fetcher generic — capture it.
6

7
2. For each candidate:
8
   a. references(name) → fetcher uses + route uses + miscellaneous uses.
9
   b. Classify each reference:
10
      - Inside lib/query/fetchers.ts as a generic to fetchJson/mutationFetchJson
11
        → "fetcher use" with extracted URL (statically resolved string-literal
12
        argument; otherwise flagged 'unresolved-url' indirect).
13
      - Inside app/api/**/route.ts at a route handler return-type position
14
        → "route use" (NextResponse<X>, Promise<X>, helper-typed-X).
15
      - Inside __tests__/** → "test-only use".
16
      - Everything else → "miscellaneous use".
17
   c. Map fetcher use URL → candidate route file via the AST-tool's existing
18
      "app/api/<segments>/route.ts" convention. URL match is heuristic
19
      (indirect confidence per PRODUCT.md D-9 wildcard tier).
20

21
3. Classify the interface:
22
   - enforced: ≥1 fetcher use AND ≥1 route use that resolves to the same
23
     interface.
24
   - fetcher-only: ≥1 fetcher use, 0 route uses (or only candidate-routes
25
     that import but don't annotate).
26
   - route-only: ≥1 route use, 0 fetcher uses.
27
   - unused: 0 fetcher uses, 0 route uses (testOnly tag if test-only refs).
28

29
4. Emit JSONL row per interface; render markdown if --pretty.
30

31
5. If --ci: diff against baseline; exit non-zero on new fetcher-only rows.

The new query reuses the existing QueryContext, BaseResult, and QueryResponse<R> types from lib/ast-dataflow/types.ts. A new TypeDriftResult type extends BaseResult with the row shape from PRODUCT.md D-11.

1
// lib/ast-dataflow/types.ts — addition
2
export interface TypeDriftResult extends BaseResult {
3
  interface: string;
4
  declaredAt: { file: string; line: number; column: number };
5
  classification: 'enforced' | 'fetcher-only' | 'route-only' | 'unused';
6
  fetchers: Array<{ file: string; line: number; column: number; url: string | null }>;
7
  routes: Array<{ file: string; line: number; column: number; confidence: Confidence }>;
8
  candidateRoutes: Array<{
9
    file: string; line: number; column: number;
10
    matchReason: 'imported-not-annotated' | 'url-match' | 'naming-convention';
11
    confidence: Confidence;
12
  }>;
13
  remediationHint: string;
14
  testOnly?: boolean;
15
  allowlisted?: { reason: string };
16
}

Caveat — URL resolution heuristic. Fetcher URLs are typically string literals (fetchJson<X>('/api/review/queue')), but some are template literals with interpolations (/api/bids/${bidId}). The detector statically resolves the literal prefix and matches against route file paths by translating [id] segments to ${...} wildcards. Unresolvable URLs (computed, imported-from-const) are flagged confidence: 'indirect'. This is acceptable because the dominant fetcher pattern is a hard-coded prefix.

Caveat — anonymous payloads. Some routes return inline anonymous shapes (NextResponse.json({ x, y })). These have no name to classify and are out of scope (PRODUCT.md D-28). A future workpackage might add an “anonymous payload” detector, but it lives outside WP-D.

Caveat — multiple-fetcher-one-route and vice versa. When a single interface is used by multiple fetchers targeting different URLs, every candidate route is reported. When one route returns multiple shapes (e.g. a union type for error vs success), the most specific match wins and the others are recorded as candidateRoutes with matchReason: 'naming-convention'.

WP-A — Structured-RPC de-cast audit (Gap 3 detection)

Effort: ~4h. Source brief: §MVP proposal WP A.

Files touched:

scripts/type-safety-audit.ts (new — CLI wrapper over references + string-literal-uses queries).
docs/generated/type-safety-audit.md (new — regenerated report).
__tests__/scripts/type-safety-audit.test.ts (new — smoke + correctness).

Approach: thin shell over existing AST queries. For each structured-RPC return type name in database.types.ts (the ~46 non-opaque-Json functions), run references(typeName, scope: app/, lib/). Filter results to AST nodes where the parent is an AsExpression casting to Record<string, unknown> or Record<string, any>. Emit JSONL with { rpc, file, line, column, castShape, inferredType }.

Acceptance criteria (from brief, restated for verification):

Runs in < 2 minutes on KH corpus.
Reports every as Record<string, unknown> cast site on a structured RPC return, with file + line + inferred type.
Zero false-negatives for the three known sites: lib/bid/bid-queries.ts:83, lib/mcp/tools/search.ts:152, lib/mcp/tools/search.ts:191.

Pairs with the ESLint rule (see §ESLint rule below) — WP-A is the audit that produces the canonical cast-site list; the ESLint rule prevents new casts being added once WP-B has cleared the existing list.

WP-B — Structured-RPC cast removal (Gap 3 execution)

Effort: ~3h mechanical + ~1h ESLint rule.

Files touched:

lib/bid/bid-queries.ts — remove casts on get_bid_question_stats_batch rows.
lib/mcp/tools/search.ts — remove casts on hybrid_search rows (~8 sites).
lib/mcp/tools/content.ts — remove casts on structured table query rows (~4 sites).
Additional sites from the WP-A report (estimated ~13 sites in lib/mcp/
- ~34 in app/api/, but many of those will be JSONB-column legitimate casts and excluded).
eslint-rules/no-supabase-record-cast.js (new — see §ESLint rule).
eslint-rules/index.js (modified — register new rule).
eslint.config.mjs (modified — enable rule at error level in lib/** + app/api/**).
eslint-rules/tests/no-supabase-record-cast.test.js (new — rule unit tests).

Acceptance criteria:

All removed casts replaced with direct property access on the typed result.
bun run test lib/bid/ lib/mcp/tools/ passes.
bun lint passes with the new rule active.
Re-run of WP-A’s audit shows zero structured-RPC-return casts remaining.

WP-C — Opaque-Json RPC inventory (Gap 2 investigation only)

Effort: ~4h. Source brief: §MVP proposal WP C.

Files touched:

docs/specs/id-16-ast-dataflow-tool/investigations/R-WP12-opaque-json-rpcs.md (new — markdown inventory).

Approach: for each of the 14 opaque-Json RPCs identified in the brief (verified count: 14 Returns: Json entries in database.types.ts), run:

1
bun run ast-dataflow string-literal-uses '<function_name>' --scope app/ lib/

…to enumerate TS call sites. Then inspect each PL/pgSQL function via Supabase CLI:

1
/opt/homebrew/bin/supabase db pull --schema public | grep -A 20 'CREATE.*FUNCTION public.get_'

(Or read the function bodies from the most recent migration that defines them — supabase/migrations/ contains the canonical SQL.)

For each function, record:

TS callers (file:line list from ast-dataflow).
Current return shape (read from the function body).
Feasibility verdict: convertible (function returns a JSONB literal with a fixed schema — change to RETURNS TABLE(...) is mechanical), too-dynamic (function returns varying shapes per call), or no-ts-callers (Python-only / admin-only / unused).
Migration sketch (if convertible): the new RETURNS TABLE(...) signature
- the gen types regenerate command + the cast-removal sweep estimate.

Decision gate (Liam OQ2): the convertible verdict count determines whether to schedule the “opaque-Json migration sprint” in S10 or later. Until WP-C ships, the sprint stays roadmap-future.

WP-E — MCP outputSchema scaffold (Gap 4 partial)

Effort: ~4h. Status: scaffold only — full rollout deferred per Liam OQ4 until main-track finalises which MCP tools are retained.

Files touched:

lib/mcp/tools/search.ts — add outputSchema to search_knowledge_base (the highest-usage tool per bun run test:mcp-eval:fc hit count).
lib/mcp/formatters/search.ts — add a Zod schema derived from SearchResult interface.
4 more tools selected from a hit-count audit before implementation.
__tests__/mcp/output-schema-smoke.test.ts (new — verifies the MCP SDK validates a known-good and known-bad object).

Pattern (single example):

1
// lib/mcp/formatters/search.ts — new export
2
export const SearchResultSchema = z.object({
3
  id: z.string(),
4
  title: z.string().nullable(),
5
  excerpt: z.string().nullable(),
6
  rank_score: z.number(),
7
  primary_domain: z.string().nullable(),
8
  // …mirror the SearchResult interface exactly
9
});
10

11
// lib/mcp/tools/search.ts — defineTool call gains outputSchema field
12
defineTool(server, 'search_knowledge_base', {
13
  inputSchema: SearchInputSchema,
14
  outputSchema: z.object({ results: z.array(SearchResultSchema) }),
15
  // …
16
}, async (input) => { … });

Why scaffold only? Per Liam OQ4: some MCP tools may be retired or merged in the main track’s MCP cleanup; scaffolding the pattern on 5 high-usage tools validates the approach without paying the ~29h cost of all-58 rollout before that decision lands.

Re-evaluation trigger: main-track MCP finalisation (no fixed date — tracked in docs/reference/state-of-the-product.md or wherever the main MCP work surfaces). At that point, the remaining tools are a multi-agent parallelisation target (per Liam OQ4): ~58 × 30m ≈ ~29h, distributed across 4–6 worker sessions of ~5h each.

WP-F (brief’s framing for the JSONB write-side validation)

Effort: decision-pending. The brief’s open question OQ5 asked whether JSONB write-side validation is worth a separate WP or covered by existing parseBody(). Liam’s S8 response: prefer no new JSONB columns; for existing ones, inventory them (§JSONB inventory below) and decide per-column.

The Zod-at-write-time pattern (e.g. BidUpdateBodySchema in lib/validation/schemas.ts already validates domain_metadata as part of the body) is the cheap fix for the rich-write-side cases. For columns where the write path is not a route (e.g. MCP tool inserts, pipeline writes), Zod validation must be added at the write site. Specific actions live in the JSONB inventory below.

ESLint rule design — `no-supabase-record-cast`

File: eslint-rules/no-supabase-record-cast.js (new). Pattern reference: eslint-rules/no-silent-promise-catch.js (style + structure).

Detection scope

The rule flags TypeScript AsExpression (type assertion) and TSAsExpression nodes where the target type is Record<string, unknown> or Record<string, any> AND the expression being cast originates from a Supabase result row.

Origin-from-Supabase recognition is structural, not type-based (the ESLint rule runs without type information, mirroring no-silent-promise-catch.js). The rule recognises three patterns:

Direct chain: (.from('x').select(...).data as Record<string, unknown>) — flagged.
Identifier after destructure: the cast target is an identifier whose declaration is const { data } = await supabase... — flagged.
RPC result: (.rpc('fn', ...).data as Record<string, unknown>) — flagged.

Escape hatches (intentionally permitted):

JSONB columns: when the cast target is a member access into a JSONB column (e.g. (bid.domain_metadata as Record<string, unknown>)), the cast is allowed because the column is genuinely Json and downstream code needs a structural shape. Detected by member-name match against the JSONB column inventory (hard-coded list in the rule; see §JSONB inventory). Future improvement: read from a JSON file shared with the inventory.
Third-party API responses: casts immediately following a fetch() or axios.get() are allowed — these are not Supabase results.
Test fixtures: the rule is disabled for files under __tests__/**, e2e/**, scripts/tests/**, and *.test.ts / *.spec.ts filenames. Fixtures often need permissive casts.
Explicit suppression: // eslint-disable-next-line local/no-supabase-record-cast with a justification comment in the same block is documented as the legitimate escape hatch when none of the above apply.

Implementation sketch

1
'use strict';
2

3
const JSONB_COLUMNS = new Set([
4
  // From the JSONB inventory below. Format: 'table.column'.
5
  'workspaces.domain_metadata',
6
  'content_items.metadata',
7
  'content_items.summary_data',
8
  // …complete list at §JSONB inventory
9
]);
10

11
const TEST_FILE_REGEX = /(__tests__|e2e|scripts\/tests|\.(test|spec)\.[tj]sx?$)/;
12

13
module.exports = {
14
  meta: {
15
    type: 'problem',
16
    docs: {
17
      description:
18
        'Disallow casting Supabase query results to Record<string, unknown>. Use the typed row shape from database.types.ts directly.',
19
    },
20
    messages: {
21
      recordCast:
22
        'Casting a Supabase result to Record<string, unknown> discards the typed row shape from database.types.ts. Remove the cast and use direct property access (typed RPCs already narrow). If the cast is for a JSONB column, name the column in the JSONB allowlist. If unavoidable, suppress with an inline justification.',
23
    },
24
    schema: [],
25
  },
26
  create(context) {
27
    if (TEST_FILE_REGEX.test(context.getFilename())) return {};
28

29
    function isRecordStringUnknown(typeAnnotation) {
30
      // Match Record<string, unknown> or Record<string, any>
31
      if (!typeAnnotation) return false;
32
      if (typeAnnotation.type !== 'TSTypeReference') return false;
33
      const name = typeAnnotation.typeName?.name;
34
      if (name !== 'Record') return false;
35
      const params = typeAnnotation.typeParameters?.params;
36
      if (!params || params.length !== 2) return false;
37
      const [k, v] = params;
38
      if (k.type !== 'TSStringKeyword') return false;
39
      return v.type === 'TSUnknownKeyword' || v.type === 'TSAnyKeyword';
40
    }
41

42
    function isSupabaseOrigin(expression) {
43
      // Walk the expression looking for a Supabase chain.
44
      // Direct member access: x.from('y').select(...).data
45
      // Identifier from destructure: rely on closer analysis
46
      // RPC: x.rpc(...).data
47
      // Returns true if found, false otherwise.
48
      // (Implementation walks AST ancestors; see test fixture for shapes.)
49
      // …
50
    }
51

52
    function isJsonbAllowlisted(expression) {
53
      // If the expression is a MemberExpression whose property name maps
54
      // to a known JSONB column, return true.
55
      // (Without full type resolution this is best-effort but the column
56
      // names are distinctive enough to be useful.)
57
      // …
58
    }
59

60
    return {
61
      TSAsExpression(node) {
62
        if (!isRecordStringUnknown(node.typeAnnotation)) return;
63
        if (isJsonbAllowlisted(node.expression)) return;
64
        if (!isSupabaseOrigin(node.expression)) return;
65
        context.report({ node, messageId: 'recordCast' });
66
      },
67
    };
68
  },
69
};

The full implementation is owned by WP-B (not this spec). The sketch above is sufficient for an executor to TDD the rule.

Test plan for the rule

eslint-rules/tests/no-supabase-record-cast.test.js follows the existing no-silent-promise-catch.test.js pattern: a Mocha + RuleTester suite with valid and invalid cases. Coverage:

Invalid (rule fires): the three canonical sites (bid-queries.ts:83, search.ts:152, search.ts:191) and synthetic fixtures for .from()...as Record<string, unknown> direct chain, destructured-data variant, and .rpc(...).data variant.
Valid (rule silent): JSONB column member access (domain_metadata), third-party API response (fetch().then(r => r.json() as Record<string, unknown>)), test file (the rule is disabled there), explicit eslint-disable-next-line with justification.

Rationale for the rule scope

The rule scope is deliberately narrow — Supabase-result + Record-cast — for three reasons:

Concrete anti-pattern. The brief identifies three exact sites; the rule prevents the next occurrence.
Low false-positive rate. Restricting to Supabase chains avoids flagging legitimate Record casts (e.g. JSON envelope unwrapping).
Pairs with WP-B. Once WP-B clears the existing 79-ish Record casts (or whatever subset are Supabase-origin), the rule keeps the codebase clean.

Why not the broader as Record<string, unknown> ban? Because many casts are legitimate — JSON envelopes from external APIs, JSONB column unwrapping, MCP structured-content shaping. A broader ban would generate churn for no safety gain.

JSONB column inventory (OQ5)

Per grep -nE ': Json' supabase/types/database.types.ts, the codebase has 20 distinct JSONB columns across 14 tables (per actual count via awk against database.types.ts) plus 1 view (quality_issues_pending) and 5 view/function return fields (covered by WP-C’s opaque-Json inventory). Each table-level column is tagged with migration risk + value below. The cross-reference to docs/reference/SCHEMA-QUICK-REFERENCE.md is included where the JSONB key structure is documented.

Inventory

Table	Column	Nullable	Default	Domain semantics	Migration risk	Migration value
`bid_response_history`	`metadata`	yes	—	Snapshot of metadata at version time	low-risk	low-value
`bid_responses`	`metadata`	yes	`'{}'`	Drafting metadata (model, tokens)	low-risk	low-value
`classification_disputes`	`current_value`	no	`'null'::jsonb`	Snapshot of disputed classification	low-risk	medium-value
`classification_disputes`	`proposed_value`	yes	—	User-proposed correction	low-risk	medium-value
`company_profiles`	`competitors`	no	`'[]'`	Array of competitor records	high-risk	high-value
`content_history`	`metadata`	yes	—	Version metadata	low-risk	low-value
`content_items`	`metadata`	yes	—	Flexible KV store (legacy keys per SCHEMA-QUICK-REFERENCE §36)	high-risk	medium-value
`content_items`	`summary_data`	yes	—	Structured summary (key points, quotes)	high-risk	high-value
`digests`	`domain_summaries`	no	`'[]'`	Per-domain summaries	medium-risk	medium-value
`digests`	`theme_clusters`	no	`'[]'`	Thematic groupings	medium-risk	medium-value
`digests`	`metadata`	yes	—	—	low-risk	low-value
`entity_mentions`	`metadata`	yes	`'{}'`	Entity-level properties (cert version, expiry, etc.)	medium-risk	high-value
`feed_prompts`	`performance_snapshot`	yes	—	`{pass_rate, flag_rate, articles_scored}`	low-risk	high-value
`ingestion_quality_log`	`details`	yes	—	Structured issue details	low-risk	medium-value
`pipeline_runs`	`result`	yes	—	Execution results	medium-risk	low-value
`pipeline_runs`	`progress`	yes	`'{}'`	Step progress tracker (`step`, `steps_completed`, `steps_total`, `detail`)	low-risk	medium-value
`processing_queue`	`payload`	no	—	Job envelope per spec §3.1	high-risk	medium-value
`processing_queue`	`result`	yes	—	Job result data	medium-risk	low-value
`source_documents`	`extraction_metadata`	yes	`'{}'`	Page count, table count, etc.	low-risk	medium-value
`workspaces`	`domain_metadata`	yes	`'{}'`	Type-specific metadata (e.g. bid details, per `BidMetadata`)	high-risk	high-value
`quality_issues_pending` (view)	`details`	yes	—	Pass-through from `ingestion_quality_log`	n/a (view)	n/a

(Function/RPC return-shape JSONB columns — claim_next_job, get_entity_summary, get_item_workspaces, get_topic_layers, hybrid_search — are tracked in WP-C’s opaque-Json inventory, not here.)

Risk + value rationale

High-risk means: the column has a complex semi-structured schema, the schema has evolved over time without versioning, or the write paths are spread across many call sites. Migrating to a proper schema requires either (a) introducing a new typed table with a foreign key + backfill + cut over all writers, or (b) introducing a Postgres composite type / domain type which TypeScript would still see as Json. Both are sprint-scale.

High-value means: the column is read in user-facing surfaces, type mismatches cause runtime parse errors, and the existing JSONB-shape mismatch has caused at least one observable bug (per CLAUDE.md gotcha register).

Migrating off JSONB — sketch (not a plan)

For a hypothetical migration of workspaces.domain_metadata:

Identify the union of consumer types (BidMetadata, ProcurementMetadata, etc., from types/bid-metadata.ts).
For each variant, either:
- Hoist to typed columns. Add bid_round, bid_lot, etc., as proper columns; write a migration that backfills from JSONB; update write paths.
- Move to a side table. Create workspace_bid_metadata table with workspace_id FK + typed columns + 1:1 relationship. Backfill + update reads via JOIN.
Either way: deprecate the JSONB column with column COMMENT ON noting the migration path; eventually drop the column.

For each of the high-risk JSONB columns in the inventory, the migration sketch is similar but the exact strategy differs. Detailed per-column plans are out of scope for this spec.

Specific high-risk / low-value flags for Liam decision

The following columns are flagged for Liam’s pre-WP review:

processing_queue.payload (high-risk, medium-value): the job envelope is intentionally polymorphic (different job types have different payloads). Migrating off JSONB would require per-job-type tables or a payload_kind discriminator + per-kind validation in application code. The polymorphism is the design intent; converting to rigid columns would lose flexibility without proportional safety gain. Recommendation: keep as JSONB; add Zod validation at every enqueue site instead.
content_items.metadata (high-risk, medium-value): documented legacy keys in SCHEMA-QUICK-REFERENCE §36; many have been hoisted to typed columns (e.g. source_document keys per AC1.4). The migration shape is incremental hoisting, not a single sweep. Recommendation: continue the existing piecemeal hoisting; do not write off-platform migrations.
content_items.summary_data (high-risk, high-value): structured summary with key_points, quotes, etc. Read in user-facing summaries. Migration to typed columns or a content_item_summaries side table would be ~1 sprint. Recommendation: schedule a separate spec for content_item_summaries migration; flag as the highest type-safety leverage among the existing JSONB columns.
company_profiles.competitors (high-risk, high-value): array of competitor records with { name, website_url, notes, monitoring_priority }. Migration to a side table (company_competitors with FK to company_profiles) is well-shaped but moderate effort. Recommendation: consider a Wave 5 spec; high type-safety leverage for a small footprint.

The remaining high-risk flags (workspaces.domain_metadata, entity_mentions.metadata) are similarly evaluated case-by-case.

Policy going forward

Per Liam’s S8 preference:

No new JSONB columns. New tables and column additions must use proper types (regular columns, enum, text[], composite types). The production-readiness track’s migration review checklist should include “no new JSONB column” as a hard gate.
Existing JSONB columns are not retroactively migrated without a per-column spec.
Write-side validation is mandatory. Every JSONB write site MUST pass through a Zod schema (either via parseBody(Schema, raw) for HTTP routes or an explicit Schema.parse(payload) for MCP-tool / pipeline writes). This invariant is partially enforced today (HTTP routes go through parseBody); MCP and pipeline writers need a sweep.

The “write-side validation sweep” is left as a roadmap item — a candidate for Wave 6 or later, gated on (a) Liam reviewing the high-risk-low-value flags above and (b) WP-C completing the opaque-Json inventory.

Supabase types CI plan (OQ6)

Goal

Prevent the silent drift between supabase/migrations/**.sql (the source of truth for schema) and supabase/types/database.types.ts (the auto-generated TypeScript types) by running supabase gen types in CI and failing on diff.

Proposed workflow step

New job in .github/workflows/ci.yml (added to the existing 7-job topology):

1
jobs:
2
  # …existing 7 jobs…
3

4
  supabase-types-parity:
5
    name: Supabase generated types parity
6
    runs-on: ubuntu-latest
7
    environment: Staging
8
    # Mirrors schema-parity.yml — uses the persistent Staging branch.
9
    timeout-minutes: 8
10
    env:
11
      SUPABASE_ACCESS_TOKEN: ${{ secrets.SUPABASE_ACCESS_TOKEN }}
12
      STAGING_PROJECT_REF: turayklvaunphgbgscat
13

14
    steps:
15
      - name: Checkout
16
        uses: actions/checkout@v6
17

18
      - name: Install Supabase CLI
19
        run: |
20
          curl -fsSL https://github.com/supabase/cli/releases/latest/download/supabase_linux_amd64.deb -o /tmp/supabase.deb
21
          sudo dpkg -i /tmp/supabase.deb
22

23
      - name: Generate types from staging branch
24
        run: |
25
          supabase gen types typescript \
26
            --project-id "$STAGING_PROJECT_REF" \
27
            --schema public \
28
            > /tmp/database.types.generated.ts
29

30
      - name: Diff against checked-in types
31
        run: |
32
          if ! diff -q supabase/types/database.types.ts /tmp/database.types.generated.ts; then
33
            echo "::error::Generated Supabase types differ from supabase/types/database.types.ts."
34
            echo "::error::Run: /opt/homebrew/bin/supabase gen types typescript --project-id rovrymhhffssilaftdwd --schema public > supabase/types/database.types.ts"
35
            diff -u supabase/types/database.types.ts /tmp/database.types.generated.ts | head -200
36
            exit 1
37
          fi
38
          echo "Generated types match committed types."

Important: the staging project ref (turayklvaunphgbgscat) is used because (per CLAUDE.md “Environment”) that’s where the live schema is. A prod-targeted variant could be added once the schema-parity workflow confirms prod ↔ staging parity at the schema level. Per CLAUDE.md gotchas, the STAGING_PROJECT_REF should be a workflow-level constant to prevent the .temp/project-ref drift gotcha from biting CI.

Why diff, not regenerate-and-commit? The auto-commit pattern (per Supabase’s docs) bypasses code review for schema drift. The diff-and-fail shape forces a PR to explicitly include the regenerated types file, which is what CLAUDE.md’s supabase migration new + db push + gen types workflow already requires.

Bundling with WP-D `--ci` (OQ3)

Both CI checks share a structure: regenerate-then-diff. They go in one workpackage labelled “CI hardening — type-safety gates” with two jobs:

type-drift-check — runs bun run ast-dataflow type-drift-detect --ci against the checked-in baseline.
supabase-types-parity — the job above.

Both are wired into the existing ci-summary aggregator. Both report to Vercel via the existing repository-dispatch mechanism (per docs/runbooks/ci.md §2).

Helper types adoption (per Supabase docs)

Supabase recommends the Tables<>, TablesInsert<>, TablesUpdate<>, Enums<> shorthand types:

1
// Today (verbose):
2
let row: Database['public']['Tables']['workspaces']['Row'];
3

4
// With shorthand:
5
let row: Tables<'workspaces'>;

Current KH state: the verbose form is widespread (~20+ occurrences in lib/, app/, lib/mcp/tools/). The shorthand is not used.

Adoption recommendation: adopt incrementally, not as a sweep.

Sites that benefit most:

lib/mcp/tools/governance.ts (4+ verbose annotations).
lib/mcp/tools/content.ts (3+ verbose annotations).
app/api/bids/[id]/responses/[rId]/route.ts and similar route handlers with long generic chains.

Sites where shorthand may not help:

Per-table type aliases that already exist as local type X = Database['public']['Tables']['x']['Row'] declarations are clearer than inline shorthand at use sites.

Recommendation for the WP: ship a small refactor (~2h) that adopts Tables<>, TablesInsert<>, TablesUpdate<> in the top-5 most-verbose files. Do not do an all-of-codebase sweep — the verbose form is fine where it appears infrequently.

Response types for complex queries (per Supabase docs)

Supabase recommends QueryData<typeof query> for typing nested joins:

1
const query = supabase.from('countries').select('id, name, cities ( id, name )');
2
type CountriesWithCities = QueryData<typeof query>;

Current KH state: no usage of QueryData / QueryResult / QueryError helpers (verified by grep). Nested joins are typed by hand-rolling intersection types, or by destructuring with type assertions.

Sites that would benefit:

lib/bid/bid-queries.ts — multiple nested join queries.
lib/intelligence/pipeline.ts — complex feed-source + workspace joins.
app/api/intelligence/workspaces/[id]/route.ts — joined queries with related tables.
Any RPC call site that today casts to a hand-typed result type.

Adoption recommendation: adopt as a deliberate pattern in new nested-query code. Add to the agent-facing convention list (CLAUDE.md gotcha section or a new “TypeScript conventions” subsection) so future nested-join code uses QueryData rather than hand-rolling types.

Caveat — JSONB column nesting. When a nested query touches a JSONB column (e.g. domain_metadata), the QueryData type still includes Json for that field. Use the overrideTypes pattern (or as cast on the specific field) at the call site. The JSONB inventory above is the reference for where this is unavoidable.

Enhanced type inference for JSON fields (per Supabase docs)

Supabase’s v2.48.0 release supports custom JSON types via MergeDeep for fields read with -> and ->> operators:

1
// database.types.ts (overridden)
2
import { MergeDeep } from 'type-fest';
3
import { Database as DatabaseGenerated } from './database-generated.types';
4

5
type WorkspaceDomainMetadata = { /* …BidMetadata shape… */ };
6

7
export type Database = MergeDeep<DatabaseGenerated, {
8
  public: {
9
    Tables: {
10
      workspaces: {
11
        Row: { domain_metadata: WorkspaceDomainMetadata | null };
12
      };
13
    };
14
  };
15
}>;

Recommendation: adopt selectively for the high-value JSONB columns flagged above. Specifically:

workspaces.domain_metadata → typed as BidMetadata (already a domain type at types/bid-metadata.ts).
content_items.summary_data → typed as a new ContentItemSummaryData interface (requires a JSON-shape audit first).
feed_prompts.performance_snapshot → typed as { pass_rate: number; flag_rate: number; articles_scored: number }.

Caveat — the overridden Database type breaks supabase gen types round trips. The override layer must NOT be regenerated; it lives in a separate file (supabase/types/database-overrides.ts or similar) that extends the generated file. The CI parity check above must only verify the generated file, not the override.

Caveat — tsconfig.strictNullChecks requirement. MergeDeep requires strictNullChecks: true (or strict: true). KH’s current tsconfig.json should be verified before adoption.

Testing and validation

Vitest suite for WP-D

__tests__/lib/ast-dataflow/queries/type-drift-detect.test.ts covers PRODUCT.md D-1 through D-30. Fixture directory: __tests__/lib/ast-dataflow/fixtures/type-drift/.

PRODUCT.md invariant	Fixture / verification
D-1 CLI invocation	`cli.test.ts` — spawn `bun scripts/ast-dataflow-cli.ts type-drift-detect`; assert default scope behaviour.
D-2 output formats	`output-formats.test.ts` — assert JSONL parses, Markdown contains expected sections.
D-3 scope override	`scope.test.ts` — fixture with two routes; `--scope app/api/foo/**` only inspects one.
D-4 interface pattern flag	`interface-pattern.test.ts` — fixture with a non-default-named interface; `--interface-pattern '^Custom'` picks it up.
D-5 no-flag canonical	Implicit across other tests.
D-6 four classification states	`classification.test.ts` — fixture set covers all four: `enforced`, `fetcher-only`, `route-only`, `unused`.
D-7 fetcher generic recognition	`fetcher-recognition.test.ts` — fixtures for direct, useQuery-wrapped, re-exported-alias.
D-8 route return-type recognition	`route-recognition.test.ts` — `Promise<NextResponse<X>>`, `Promise<X>`, helper-typed-X.
D-9 confidence tiers	`confidence.test.ts` — `exact` (return-type annotated), `wildcard` (typed variable into `NextResponse.json(v)`), `indirect` (URL-match fallback).
D-10 one classification per interface	`dedup.test.ts` — fixture where interface is used by 2 fetchers and 1 route → reports as `enforced`.
D-11 JSONL row shape	`schema.test.ts` — Zod parse the JSONL output against the documented `TypeDriftResult` schema.
D-12 Markdown report shape	`markdown.test.ts` — snapshot the Markdown for a known fixture set.
D-13 capped output	`truncation.test.ts` — 600-interface fixture + `--limit 500` → `truncated: true`.
D-14 result paths	`paths.test.ts` — every result row’s paths pass `isRepoRootRelativePosix(x)`.
D-15 default candidate set	`discovery.test.ts` — types/, app/api/, lib/query/fetchers.ts all walked.
D-16 evidence rows on every finding	`evidence.test.ts` — every `fetcher-only` row has ≥1 fetcher site + ≥0 candidateRoutes + remediationHint.
D-17 allowlist	`allowlist.test.ts` — interface marked in allowlist.json moves to `allowlisted` bucket.
D-18 false-negative tolerance	`false-negative.test.ts` — assert `TaxonomySyncStatus` and `ReviewStatsResponse` appear in `fetcher-only` on the real KH corpus (integration test, runs against project source).
D-19 `--ci` mode contract	`ci-mode.test.ts` — fixture baseline + introduced gap → exit non-zero; baseline-matching → exit 0.
D-20 list-to-fix shape	Implicit in D-19 verification; assert baseline shrinks across a “fix gap, re-run” sequence.
D-21 no baseline auto-mutate	`baseline-immutable.test.ts` — `--ci` invocation does not modify baseline file; `--update-baseline` does.
D-22 latency budget	`performance.test.ts` — full-corpus cold scan <3 min, warm scan <90 s P95.
D-23 cache reuse	`cache-reuse.test.ts` — second invocation faster than first.
D-24 worktree portable	`worktree.test.ts` — temp-dir clone runs cleanly.
D-25 no fetchers found	`no-fetchers.test.ts` — fixture with empty fetchers.ts → exit 0 + informational row.
D-26 type checker resolution failure	`unresolved.test.ts` — fetcher with `fetchJson<T>` where `T` is a type parameter → `confidence: indirect` row.
D-27 multiple declarations	`aliases.test.ts` — interface re-exported via barrel → primary declaration in `declaredAt`.
D-28 inline route response types	`anonymous-payload.test.ts` — `NextResponse.json({...})` inline → not in report.
D-29 test-only references	`test-only.test.ts` — interface used only in `__tests__/**` → `unused` with `testOnly: true`.
D-30 structured failure	`errors.test.ts` — malformed allowlist.json → exit 0 with error row; missing tsconfig.json → exit non-zero.

Smoke test against real KH corpus

1
bun run ast-dataflow type-drift-detect --pretty

Expected (kh-ast-S9 first run on main):

≥5 fetcher-only rows surfaced (per brief D-18 minimum).
TaxonomySyncStatus and ReviewStatsResponse present.
Report regenerated at docs/generated/type-drift-report.md.
Baseline initialised at docs/generated/type-drift-baseline.json with the current count (after Liam reviews and accepts).

Validation for WP-A / WP-B / WP-C / WP-E

Each mechanical WP has its own acceptance criteria (listed inline above). Verification is the standard pattern:

Acceptance criteria explicit in each WP’s brief section.
TDD where applicable (the ESLint rule has its own test file).
The audit reports (WP-A docs/generated/type-safety-audit.md, WP-C docs/specs/.../R-WP12-opaque-json-rpcs.md) are reviewable artefacts in the merge PRs.

Validation for the ESLint rule

Beyond the unit tests above:

After WP-B’s cast removal sweep, bun lint MUST run cleanly on the full codebase. Any remaining as Record<string, unknown> cast is either flagged by the rule (and removed) or annotated with // eslint-disable-next-line local/no-supabase-record-cast plus a justification comment that survives code review.

Validation for the CI workpackage

New CI job supabase-types-parity passes on a known-good schema state (no drift) and fails when a deliberately-out-of-date types file is committed.
New CI job type-drift-check passes when baseline matches and fails when a new fetcher-only row is introduced (verified by adding a synthetic fetcher in a test PR).

Risks and mitigations

WP-D URL-resolution false negatives. Heuristic URL match between fetcher and route may miss cases. Mitigation: PRODUCT.md D-18 mandates zero false negatives on the canonical cases; integration test asserts this; the confidence tier surfaces uncertainty to the caller.
ESLint rule false positives on JSONB. Without type info, the rule cannot perfectly distinguish JSONB columns from typed columns. Mitigation: the JSONB allowlist (hard-coded column list) suppresses the rule on known JSONB sites; the inline-suppression escape hatch handles novel cases.
Supabase gen types flakiness in CI. The CLI occasionally times out under load. Mitigation: 8-minute timeout; the diff-only approach means a single retry is sufficient.
Staging schema drift. If staging schema drifts from prod, the CI types-parity check fails for the wrong reason. Mitigation: the existing schema-parity.yml workflow detects prod↔staging drift; encourage running it before merging types-touching PRs.
WP-E premature rollout. If main-track decides to retire MCP tools that WP-E added schemas to, the schemas are wasted work. Mitigation: scaffold only — 5 highest-usage tools; full rollout gated on main-track signal.
JSONB write validation gap. Until the write-side sweep ships, MCP and pipeline writes to JSONB columns remain unvalidated. Mitigation: no immediate change — this is the existing state; the inventory above documents the gap so the next WP scoping it has the data.
Custom JSON types break gen types round trips. If KH adopts MergeDeep overrides, the CI parity check must compare only the generated file, not the merged Database type. Mitigation: documented caveat above; override types must live in a separate file the CI does not regenerate.

Follow-ups

Opaque-Json RPC migration sprint — gated on WP-C inventory (Liam OQ2). Spec to be authored after WP-C reports.
Full MCP outputSchema rollout — gated on main-track MCP finalisation (Liam OQ4). Multi-agent parallelisation target (~58 × 30m ≈ ~29h).
JSONB write-side validation sweep — gated on Liam’s review of the high-risk/high-value flags above. Spec candidate for Wave 5+.
Helper types adoption (Tables<>, etc.) — incremental refactor of top-5 verbose files; not a formal workpackage, more a hygiene PR.
Adopt QueryData for new nested queries — convention update to CLAUDE.md or a new “TypeScript conventions” doc.
Custom JSON types via MergeDeep — selective adoption for high-value JSONB columns; requires the separate-file approach described above.
Cross-track MCP coupling note — WP-E’s deferral is the main cross-track dependency this spec creates. Surface this in the next main-track planning session so MCP-tool decisions explicitly reckon with the type-safety upgrade pipeline.

Decision log (Liam OQ responses, kh-ast-S8)

OQ from brief	Liam decision	Effect on this spec
OQ1 (sequence)	Start WP-D first (audit-style, novel). Others are mechanical and sketched here.	PRODUCT.md scoped to WP-D; other WPs in TECH.md sketches.
OQ2 (migration sprint)	Gated on WP-C findings. Stays roadmap-future until inventory lands.	WP-C scoped here; sprint scoping deferred.
OQ3 (CI integration)	Yes — list-to-fix shape; bundled with OQ6.	WP-D `--ci` mode + CI workpackage with two jobs.
OQ4 (MCP outputSchema parallelisation)	Yes — but deferred until main-track MCP finalises. Scaffold now, rollout later.	WP-E in scope as scaffold; full rollout in Follow-ups.
OQ5 (JSONB)	Strong preference: no new JSONB columns. Inventory existing. Sketch migrations. Flag risky sites.	Inventory above; policy stated; per-column flags surfaced.
OQ6 (Supabase types CI)	Yes. Also adopt Helper / Response patterns where they fit.	CI plan + adoption analysis above.
Liam add	ESLint rule paired with WP-A/B.	New rule designed in §ESLint rule.

Type-safety pipeline — TECH

Type-safety pipeline — TECH

Context

Relevant existing code

Proposed changes

WP-D — Route/fetcher type-drift detector (primary spec target)

WP-A — Structured-RPC de-cast audit (Gap 3 detection)

WP-B — Structured-RPC cast removal (Gap 3 execution)

WP-C — Opaque-Json RPC inventory (Gap 2 investigation only)

WP-E — MCP outputSchema scaffold (Gap 4 partial)

WP-F (brief’s framing for the JSONB write-side validation)

ESLint rule design — no-supabase-record-cast

Detection scope

Implementation sketch

Test plan for the rule

Rationale for the rule scope

JSONB column inventory (OQ5)

Inventory

Risk + value rationale

Migrating off JSONB — sketch (not a plan)

Specific high-risk / low-value flags for Liam decision

Policy going forward

Supabase types CI plan (OQ6)

Goal

Proposed workflow step

Bundling with WP-D --ci (OQ3)

Helper types adoption (per Supabase docs)

Response types for complex queries (per Supabase docs)

Enhanced type inference for JSON fields (per Supabase docs)

Testing and validation

Vitest suite for WP-D

Smoke test against real KH corpus

Validation for WP-A / WP-B / WP-C / WP-E

Validation for the ESLint rule

Validation for the CI workpackage

Risks and mitigations

Follow-ups

Decision log (Liam OQ responses, kh-ast-S8)

ESLint rule design — `no-supabase-record-cast`

Bundling with WP-D `--ci` (OQ3)