Authored S403 (2026-06-23) as the evidence basis for Task ID-128. Static method: read 4 runner configs + 12 workflow files, grep counts, no suite execution. Repo canonical @ test-audit-review HEAD 6798bb9b (fresh off main).
| Category | Claimed | Measured | Note |
|---|
| unit-vitest (excl integration) | 791 | 849 raw .test.ts(x) excl /integration/ (includes __tests__/scripts 58, __tests__/build, __tests__/eval) | claim’s 791 is after subtracting scripts/mcp-eval drivers; consistent |
| integration-vitest | 80 | 75 | DISCREPANCY: only 75 *.integration.test.ts under __tests__/integration/; zero strays elsewhere (confirmed) |
| e2e-playwright | 47 | 47 | confirmed; only 6 carry @smoke |
| pytest test_*.py | 69 | 69 | + 2 conftest confirmed |
| tools/ast-dataflow tests | 21 | 16 | DISCREPANCY: 16 .test.ts present |
| eslint-rules/tests | 3 | 3 | confirmed |
| vendored skill tests | listed | present (2 .ts + 1 .py under .claude/skills/...) | confirmed |
| mcp-eval harness scripts | 9 | 9 | confirmed |
| live-verify | 2 | 2 (live-verify.sh, verify_driver.py) | confirmed |
| Runner / glob | Selected by config | Executed in PR-blocking CI? | Where |
|---|
vitest.config include __tests__/** (excl *.integration.test) | yes | YES | ci.yml quality-test shard 1-4/4. PR = --changed origin/BASE only (diff-scoped); push = full |
vitest.config include tools/**/*.test | yes | YES (push full; PR only if diffed) | quality-test shards |
vitest.config include eslint-rules/tests/** | yes | YES (push full; PR only if diffed) | quality-test shards |
__tests__/build/*.test (env-substitution) | matched by vitest include BUT describe.skipIf(!RUN_BUILD_TESTS) | YES via RUN_BUILD_TESTS=1 | quality-precheck test:build step |
__tests__/eval/*.test (5 specs) | matched by vitest include BUT all 5 env-gated (EVAL_ENTITY etc., skip by default) | NO (skip in shards; gate vars unset in CI) | run by NO workflow; only local eval:* scripts |
vitest.integration.config __tests__/integration/** | yes | PARTIAL — see §2 | ci.yml integration job |
playwright.config --project=smoke (grep /@smoke/) | yes | YES (6 smoke specs only) | ci.yml e2e-smoke |
playwright chromium-desktop / chromium-mobile (full 47) | yes (projects exist) | NO | run nowhere in CI (no full/mobile lane anywhere) |
pyproject pytest scripts/tests/ (69) | yes | CONDITIONAL — only when python paths-filter matches (scripts/**/*.py, requirements.txt, scripts/tests/**) | quality-precheck |
mcp-eval L1/L3/L4 (scripts/mcp-eval/{protocol-compliance,response-quality,functional-correctness}.ts) | package scripts | YES (cost-gated, see §2) | ci.yml mcp-eval matrix |
mcp-eval harness sets headless-complete-set.ts, propose-write-set.ts, dual-runtime-connectivity-set.ts | NOT invoked by any test:mcp-eval* package script wired into a workflow | indirectly via 3 vitest wrappers in __tests__/scripts/mcp-eval-*-set.test.ts (run in quality-test shards) | partial |
eval-runner.ts --all (registry dispatcher) | nightly only | NO (not PR-blocking) | eval-nightly.yml cron 0 3 * * * |
deploy/onprem/verify/live-verify.sh | operator/cron on B1 host | NO | not in any workflow file |
| Tests | Gate | Intentional? | Evidence |
|---|
integration cocoindex suites (~41 files reference COCOINDEX_*) | COCOINDEX_STAGING_URL / COCOINDEX_SOURCE_PATH / COCOINDEX_FIXTURE_STAGING_URL unset (sidecar absent) | YES — bl-341, sidecar not in CI | ci.yml integration env comment; 41 files via grep |
| integration intelligence-golden-path | INTEGRATION_INTELLIGENCE unset (2 files) | YES — bl-338 real-API spend deferred | ci.yml comment “intentionally NOT set” |
| integration q-a-pairs + others | KH_RUN_INTEGRATION unset (5 files) | YES — parity w/ local test:integration | same comment block |
| ~128 integration cases total | above three flags | YES — bl-338/341, visibility notice emitted | ci.yml integration “Integration coverage summary” step |
| integration job entirely (push only) | detect-changes.outputs.relevant != true | YES — ID-91/bl-246 real-API cost gate; PR always runs | ci.yml integration if: |
__tests__/eval/* (5 specs) | EVAL_ENTITY / per-script env unset | gate is intentional, but see §4 ORPHAN — no CI runs them at all | — |
scripts/tests/oq/test_integration_live.py | OQ_LIVE_CMUX != 1 skipif | YES — NOT accidental; opt-in live-cmux, properly gated | skipif line 192 |
eval-runner.ts --all (eval-nightly.yml cron, soft); supabase advisors (cron + push main supabase/**); staging-reference-refresh (fortnightly cron); schema-parity (dispatch only); onprem cocoindex build/deploy (push main/staging); token-parity (push main paths, never PR-blocking, Inv 30); task-view-vendor-drift (PR paths, warning-only).
| Path | Issue | Verification |
|---|
__tests__/eval/{classification,summarisation,search,procurement-drafting,entity-classification}-eval.test.ts (5) | Matched by vitest include so they live in shards, but ALL env-gated (skip by default) → 0 assertions run in any CI. eval-nightly runs eval-runner.ts (registry dispatcher → scripts/eval-*.ts), NOT these vitest specs. Only local eval:* package scripts exercise them. | grep: all 5 reference EVAL_ENTITY/skip; grep eval: .github/workflows → only an identity-guard exclusion; eval-nightly references eval-runner.ts only |
deploy/onprem/verify/live-verify.sh + verify_driver.py | Not referenced by ANY .github/workflows/* (operator/cron on B1 host). Intentional (ID-62) but the ONLY path that exercises __tests__/integration/cocoindex/url-landing-set.integration.test.ts. | grep -r live-verify .github/workflows → none |
.claude/skills/.../tests/{evaluation,skills}.test.ts + interleaved_thinking/tests/test_models.py | Vendored third-party examples. Correctly EXCLUDED (vitest include __tests__//tools//eslint-rules/; pytest testpaths=scripts/tests). NOT orphans; flagged so not mistaken for repo suites. | vitest include globs; pyproject testpaths |
tools/ast-dataflow/__tests__/* (16) + eslint-rules/tests/* (3) | Correctly wired (vitest include) — on PRs run ONLY if the diff touches them (--changed); push runs full. Not orphaned. | vitest.config include |
scripts/tests/oq/test_integration_live.py | MISPLACED-ish: a live test inside the standard pytest dir, but properly OQ_LIVE_CMUX-gated so pytest scripts/tests/ skips it. | skipif line 192 |
| Area | Description | Severity | Recommended action | Owning subtask |
|---|
| E2E breadth | 41 of 47 Playwright specs (chromium-desktop/mobile full projects) run in NO workflow — only 6 @smoke gate PRs. | high | New nightly Playwright lane --project=chromium-desktop (+ mobile); or tag more @smoke. | 128.1 |
__tests__/eval/* vitest specs | 5 eval specs run nowhere: env-gated out of shards, eval-nightly uses the separate eval-runner.ts registry. False “we have eval tests” signal. | medium | Decide ownership: delete (superseded by eval-runner) or wire into a gated nightly vitest step. | 128.2 |
| cocoindex integration (~40 files) | Never executed in PR CI (sidecar env unset). Only live-verify.sh on B1 drives ONE spec. | high | Stand up the nightly cocoindex lane (bl-341) with COCOINDEX_* + sidecar; re-enable the 9 bl-227 stubs + bl-195 no-op assert as substrate lands. | 128.3 (folds bl-341, bl-227, bl-195) |
| intelligence-golden-path + q-a-pairs (~52 Supabase+AI cases) | Gated off (INTEGRATION_INTELLIGENCE/KH_RUN_INTEGRATION) for real-API spend; can run on existing CI creds. | medium | Nightly integration lane setting the two flags (no sidecar) to exercise the ~52 cases off the PR critical path. | 128.4 (folds bl-338) |
| pytest conditional execution | All 69 pytest files skip on any PR/push not touching scripts/**. | low | Acceptable (paths-filter intentional); optional nightly unconditional pytest scripts/tests/ for drift safety. | 128.5 (optional) |
- PR-blocking that ACTUALLY executes assertions: unit-vitest (full on push / diff-scoped on PR) incl tools/ + eslint-rules/ + build; ~30 ungated integration files (PR; push-gated by detect-changes); 6 @smoke e2e; mcp-eval L1/L3/L4 (cost-gated); pytest (Python-path-gated); plus structural gates (type-drift, supabase-types-parity, build-gate typecheck, api-grant-guard, identity-guard).
- Largest real gaps: 41 non-smoke E2E specs + ~40 cocoindex integration specs execute nowhere automated; 5
__tests__/eval specs are orphaned from CI.
