id-377 research — declarative write surfaces

Declarative write surfaces ({377.4} — the flow.py blind spot)

Problem

scripts/cocoindex_pipeline/flow.py expresses the pipeline’s primary Postgres writes as DATA, not code: seven TableSchema(columns={...}) constants (q_a_extractions 12 cols, source_documents 22, reference_items 12, entity_mentions 9, entity_relationships 6, content_chunks 7, record_embeddings 4), bound to table names by mount_table_target(DB_CTX, "<table>", <SCHEMA>), written through target.declare_row(row={...}). No SQL string, no fluent chain — so both ast-dataflow sides were blind: the TS schema-coverage listed the Python pipeline as a static caveat, and ast_dataflow_py detected only raw asyncpg SQL + supabase-py chains. Initiative 12’s census rule (“no wiring verdict until TS chains + Python + flow.py TableSchema declaratives PLUS a pg_proc/migrations scan”) was unsatisfiable by tooling.

Perspective (i) — shipped for this codebase

Branch ast-dataflow-declarative-writes (canonical repo):

1. tools/ast_dataflow_py/declarative_writes.py — collects TableSchema declarations (per-column line numbers), mount bindings (target var ↔ table string ↔ schema const; ctx args resolve away by name-matching against collected schema decls), and declare_row sites. Receiver→table resolution ladder: R1 mount-var name match (corpus-wide; the <abbrev>_target convention covers helper params like re_target); R2 payload-key ⊆ exactly-one mounted schema; R3 unresolvable → loud caveat, never dropped. row=<var> payloads (the entity_relationships dedup-map pattern) fall back to same-enclosing-scope dict literals whose keys fit the target’s columns; a resolved site with no recoverable keys emits a table-scoped * smoke row.
2. Evidence semantics (the load-bearing decision): a declare_row payload key is write PROOF (exact); a TableSchema column declaration alone is INTENT (indirect, method table-schema) — flow.py itself declares six source_documents classification columns “for schema completeness” that the producer deliberately leaves NULL. Mapping declaration→indirect reuses the existing verdict engine unchanged: declared-but-never-written = undecidable, never wired/write-only. No new confidence tier.
3. schema-uses bulk query (bun run ast-dataflow-py schema-uses) — ONE corpus walk emitting every attributable (table, column, direction, confidence) row across all three detectors (SQL via sqlglot, supabase-py chains, declaratives) as the v1 evidence sidecar. ~330 rows over scripts/ in ~350 ms. Loud-failure caveats: rpc payloads skipped (table-blind), unparsed SQL counts, sqlglot-absent SQL skips.
4. TS schema-coverage --evidence <sidecar.json> — merges external evidence rows into per-column verdicts; * rows land as wildcard-reads / indirect-writes (smoke); unknown tables/columns from a sidecar go to an evidenceUnknownTables caveat; the “Python pipeline” invisible-surface caveat drops only when an ast-dataflow-py sidecar was actually merged.

Perspective (ii) — generic design (other users’ codebases)

Other codebases hit the same class differently: SQLAlchemy Table() / declarative models, Django models, Prisma/Drizzle schema files, ORM-less config-driven ETL — all “the write is a data structure; the executor is in a library the scan never sees”. Three-layer answer, so unseen setups degrade loudly instead of silently mis-verdicting:

1. Evidence-sidecar contract (the extension point). {schemaVersion: 1, source, rows: [{table, column|"*", direction, confidence, method, file, line, source}]}. Any extractor in any language can produce one; schema-coverage --evidence merges N of them. The standalone package ships producers (Python detector now; migrations/pg_proc SQL scanner next) and third parties can add their own without touching the TS core. Verdict discipline is enforced at the merge, not per producer.
2. Declarative-write descriptors. The cocoindex detector’s engine (collect declarations → bind to table names → attribute row writes → R1–R3 ladder) is framework-agnostic; what varies is names/arg-positions: constructor (TableSchema), binder (mount_table_target, table arg, schema arg), writer (declare_row, payload arg). Extraction-phase shape: built-in descriptors per known framework + a user-supplied descriptor file for in-house patterns. NOT built until a second real framework demands it — the descriptor boundary is designed now so the cocoindex-specific constants live in one frozen set, trivially liftable.
3. Loud-failure floor. Anything schema-shaped that matches no descriptor surfaces as a caveat (unattributable declare_row sites, rpc payloads, unparsed SQL, sqlglot absence), and declaration-only evidence can never upgrade a column past undecidable. A tool that cannot see a surface must SAY so per column — this is the market.md “state its corpus scope per column” principle applied to the write side.

Verification

• scripts/tests/test_ast_dataflow_py_declarative_writes.py — resolution ladder, fallback, smoke rows, refuse-to-guess ambiguity, real-corpus pin (flow.py 7 mounts resolve completely, er dedup-map recovered), sidecar envelope, CLI arg contract.
• Full Python suite 2180 passed / 5 skipped (no regressions).
• TS-side merge tests in tools/ast-dataflow/__tests__/schema-coverage.test.ts.

Also shipped (found mid-session)

• SQL-as-named-constant resolution — a SECOND invisible declared-as-data shape: the entire l_records read layer (+ url_source) passes SQL via module-level _SQL_* constants (33 exec sites missed by literal-only detection). Name→const and f-string-over-const hops now resolve to exact SQL, including constants interpolating other constants (_QA_WON_COLUMNS). Sweep went 330 → 765 rows; 3 conditional tuple-assign sites remain, caveated (sqlSitesUnresolvedDynamic).
• Stored-proc FROM sources (SELECT ... FROM public.resolve_or_mint_...) emit nameless sqlglot Table nodes — skipped with a dedicated caveat (sqlFunctionSourceSites); that is the pg_proc surface, not a table.
• Cross-module schema refs (flow.RECORD_EMBEDDINGS_SCHEMA in server.py’s closure mount) resolve via attribute-name candidates.

End-to-end proof (real corpus): merging the Python sidecar re-verdicts 51 columns — unwired 85→78, undecidable 134→102, write-only 52→62, read-only 223→201, wired 313→364 — and drops the Python invisible-surface caveat. Join over-attribution (multi-table statements) is indirect by construction and lands in evidenceUnknownTables when the column is not on the table — worst case undecidable, never a false wired.

Full surface inventory (agent sweep, 25 hard cases, per-site file:line): ./declarative-surface-inventory.md.

Follow-ups (fold into {377.x} scoping)

• pg_proc / migrations-SQL sidecar producer (initiative-12 census needs it; sqlglot exists, surface is .sql files + pg_proc dump). The inventory’s zero-column-token proc calls (flow.py:2655, writer_fence, claim_next_job) are the acceptance cases.
• bid_worker’s supabase client uses ClientOptions(schema="api") — its chain evidence names api-VIEW mirrors. Correct today because DR-032 views mirror public names 1:1, but the sidecar has no schema field; add one if api ever diverges from a pass-through mirror.
• on_conflict= column strings and modifier-only mentions (bid_worker:505, :199) are not yet evidence; list-comprehension payloads emit table-scoped * smoke only.
• localfs.declare_file (bundle_writer, 6 sites) is a parallel declarative surface for FILE artefacts — out of scope for column lineage; noted for the standalone tool’s roadmap.
• Wildcard-write (*) semantics on the TS side are v1-conservative (indirect on every column); revisit if smoke drowns real signal.
• MCP server tool surface not yet extended with --evidence (extraction phase / /mcp-builder audit is the natural home).
• {377.1} blocker resolved: the old audit was deliberately removed as a point-in-time doc (owner, S509+1) — {377.1} struck from scope.