Synthetic Procurement Corpus — Scoped Design Proposal

⚠️ RE-SCOPED (bannered S462). The forms half (Need 1/Need 3: forms/procurement/ manifest mappings → walk-minted form_templates, open question Q-1) describes the pipeline forms-route RETIRED by DR-014 (S434) + id-136 — drop it. The DB-seed half (Need 2: outcome-spanning synthetic fixtures for win-rate/mint validation) remains the only non-degenerate metrics fixture design; seed forms via the manual-upload path or direct inserts. Consumer: ID-145 (staging debris purge + exemplar seeding is OQ-10 in its RESEARCH).

Synthetic Procurement Corpus — Scoped Design Proposal

Status: scope-only (do NOT build). For next session. Targets: Platform prod zjqbrdctesqvouboziae + persistent staging branch rbwqewalexrzgxtvcqrh. Satisfies 3 needs: (1) ID-127.4 corpus + manifest; (2) ID-130 {130.8}/T-B22 mint validation on data-sparse staging; (3) pre-ID-45 walk proof.

---

0. Grounding (what the code/specs actually require)

ID-127.4 (“Assemble synthetic corpus + mandatory root manifest”, pending, BI-3/7/9):

• Build under local-fs-platform/corpus (a LOCAL simulated source folder, NOT in-repo): ≥3 content shapes (Q&A, capability/methodology, sector intel, procurement) + ≥2 formats (md + PDF/DOCX); FILE-ONLY for this subtask (the feed slice is the separate S3/seed-platform-feed.ts).
• MANDATORY root .kh-workspace-map.json — flow.py:3403-3427 app_main loads it UNCONDITIONALLY and raises manifest_missing/manifest_invalid (status failed) if absent. Shape: {schema_version:1, mappings:[{path_prefix:'forms/procurement/', workspace_id:'<procurement-workspace-uuid>', route:'forms'}]}.
• A file literally named .kh-workspace-map.json is SKIPPED in the walk (flow.py:1781).
• Content/Q&A files match NO prefix → UnmappedPath soft-warn → route content (workspace-agnostic; content_items has no workspace_id).
• AVOID __qa__/ reserved prefix unless carrying an explicit {path_prefix:'__qa__/', route:'qa_sidecar'} mapping (else qa_sidecar_route_missing + ingested as JUNK — S297 BUG-B).
• BI-7 base-path stability: /cocoindex-state/corpus is a FIXED operator path (identity = uuid5 of rel_path; re-rooting forks every identity). No pipeline-code change.

The walk pipeline (entities a walk reads/writes):

• Trigger path (feed slice): feed_sources → feed_articles (passed=true) → FeedUrlSource (_PASSED_URLS_SQL: WHERE passed = true) enumerates the URL → /extract (trafilatura) → cleaned content. (seed-platform-feed.ts already seeds 1+1.)
• Trigger path (file walk): connectors.localfs.walk_dir(live=True, recursive=True) over COCOINDEX_SOURCE_PATH → per-file route via manifest.
• Writes: source_documents (upsert, flow.py:2159/2549/2957), content_items (:2195; route=content; no workspace_id), reference_items (:2987; the .body write), form_templates (route=forms; FK workspace_id NOT NULL → the procurement workspace from the manifest), and pipeline_runs (one in_progress row at flow start + one terminal row, via the /api/internal/pipeline-runs/record webhook → recordPipelineRun). pipeline_runs.result.stage_counts has 7 stages incl source_walk.
• form_templates walk-write columns (flow.py:1483): workspace_id, name, filename, file_size, mime_type (3 CHECK-permitted: pdf/xlsx/docx), storage_path, status, ingest_source, form_type, deadline, issuing_organisation, evaluation_methodology, field_count, mapped_count. Note: the walk mints form_templates from INGESTED files; {130.8} mints them from domain_metadata (different provenance — see Need 2).

ID-130 {130.8}/T-B22 data migration (the mint under validation): for each live procurement workspace, INSERT INTO form_templates one row lifting workspaces.domain_metadata:

• issuing_organisation ← domain_metadata->>'buyer'; deadline ← ->>'deadline'; submission_date ← ->>'submission_date'.
• workflow_state ← COALESCE(->>'status','draft').
• withdrawn transform: when ->>'outcome' = 'withdrawn' set workflow_state='withdrawn' AND outcome=NULL; only {won,lost} lift into form_templates.outcome; audit fields (outcome_recorded_at/by) lift alongside a lifted {won,lost} outcome.
• form_type ← default 'bid' (final-award set → enters win-rate denominator).
• Then T-B4 backfills form_questions.form_template_id. Win-rate parity test needs: one won final-award form + one not_shortlisted psq shortlist form, each with a citing form_response+citation. The TECH note explicitly says a live snapshot is degenerate (all 12 live workspaces carry NULL outcome/status) — a synthetic fixture is the real coverage.

domain_metadata shape (ProcurementMetadataSchema, lib/validation/schemas.ts:1019): buyer (string), status (enum incl won/lost/withdrawn), deadline (nullable iso), reference_number, estimated_value, tender_source, tender_document_ids[], submission_date (nullable iso), outcome (won/lost/withdrawn|null), outcome_notes, notes, outcome_recorded_at?, outcome_recorded_by? (uuid). .passthrough().

Existing seed harness to reuse: seed-platform-workspaces.ts (the PLATFORM_TARGETS prod/staging map, parseSeedArgs/resolveTarget/project-ref guard, PIPELINE_SYSTEM_USER_ID, lookup-then-insert idempotency, dry-run-default + --apply). seed-platform-feed.ts already binds to 'Platform — Procurement' by name.

---

(a) Entity set + concrete shapes/counts

The corpus is two coordinated halves: (A) DB rows (workspaces + domain_metadata + form_questions + the feed slice) and (B) the local-fs file tree + manifest (the walk’s COCOINDEX_SOURCE_PATH input).

A1. Procurement workspaces carrying domain_metadata — 6 synthetic workspaces

All bind to application_types.key='procurement', created_by = PIPELINE_SYSTEM_USER_ID, names prefixed Synthetic — First-Client … (clearly synthetic + removable + client-name-guard-safe). These are SEPARATE from the 6 BI-8 Platform — * workspaces (those are pipeline targets; these are {130.8} mint subjects).

#	name	domain_metadata.status	outcome	submission_date	purpose ({130.8} branch)
1	Synthetic — First-Client Open Tender (won)	won	won	set	won lift; enters win-rate numerator+denominator (form_type bid)
2	Synthetic — First-Client Framework Bid (lost)	lost	lost	set	lost lift; enters denominator only
3	Synthetic — First-Client RFP (withdrawn)	withdrawn	withdrawn	set	withdrawn transform: → workflow_state='withdrawn', outcome=NULL
4	Synthetic — First-Client SQ (in progress)	questions_extracted	null	null	COALESCE→ workflow_state questions_extracted; no outcome
5	Synthetic — First-Client Tender (draft, null status)	(omit status)	null	null	COALESCE(NULL,‘draft’) edge — proves the NOT NULL guard
6	Synthetic — First-Client PSQ shortlist (not_shortlisted)	lost	lost	set	shortlist case: gets a psq form_type override (NOT default bid) so win-rate test (b)/(c) — counts_toward_win_rate=false, ABSENT from denominator, present in shortlist pass-rate

Each row’s domain_metadata is a full ProcurementMetadataSchema-valid object (buyer = a generic UK public-body name e.g. “Northgate Borough Council”, “Westmere NHS Trust” — invented, never a real client; reference_number SYN-001…, estimated_value, notes flagged Synthetic corpus row — ID-127.4/ID-130 validation; safe to delete.).

Won/lost rows (1,2,6) carry outcome_recorded_at + outcome_recorded_by = PIPELINE_SYSTEM_USER_ID.

A2. form_questions — ~5 questions per workspace × 6 = ~30 rows

Keyed to workspace_id (current schema — NO form_template_id column yet; {130.8} T-B4 ADDS it). Reuse the SQ-style fixture set from seed-procurement-test-data.ts (Economic/Financial, Technical/Professional, Health & Safety, Data Protection sections). ~30 rows total matches the {130.8} “30-row UPDATE” backfill scale, so the mint-then-backfill is exercised at realistic scale.

A3. Win-rate parity fixtures — 2 form_responses + 2 citations

To make the {130.8} T-B7 win-rate rewrite testable post-mint (TECH “Risks” §, the synthetic-fixture parity test): on workspace #1 (won) one form_response + citation; on workspace #6 (not_shortlisted psq) one form_response + citation. These let the post-mint recompute prove numerator/denominator/shortlist-pass-rate behaviour. (Open Q-2: build now as data, or leave to the {130.8} integration test’s own fixture — see §d.)

A4. Feed slice — already covered by seed-platform-feed.ts (1 feed_sources + 1 feed_articles passed=true). The proposal REUSES it, does not duplicate it.

B. Local-fs file tree + manifest (the walk input)

Under local-fs-platform/corpus/ (LOCAL, not committed — gitignored sim source; mounted at the fixed /cocoindex-state/corpus on the Coolify app):

local-fs-platform/corpus/
  .kh-workspace-map.json            ← MANDATORY root manifest (skipped by walk)
  forms/procurement/
    synthetic-sq-officesupplies.pdf   ← route=forms → form_templates (workspace #1 uuid)
    synthetic-itt-groundsmaint.docx   ← route=forms → form_templates (2nd format)
  capability/
    synthetic-methodology.md          ← route=content → content_items
    synthetic-case-study.md           ← route=content → content_items
  sector-intel/
    synthetic-uk-procurement-brief.md ← route=content → content_items (sector-intel shape)
  qa/
    synthetic-qa-pairs.md             ← route=content (q_a shape; AVOID __qa__/ prefix)

• ≥3 shapes: forms/procurement, capability/methodology, sector-intel, q_a (4 shapes). ≥2 formats: md + pdf + docx (3 formats). File-only (per BI-3; feed slice is separate).
• Manifest: {schema_version:1, mappings:[{path_prefix:'forms/procurement/', workspace_id:'<#1 procurement workspace uuid>', route:'forms'}]}. Only the forms prefix is mapped; everything else soft-warns → content (the documented v1.1 behaviour).
• All file CONTENT is invented synthetic UK-procurement prose (no client IP, no sanitised client fixture — ID-68 IP separation). Filenames use synthetic-/first-client generic only.

---

(b) How it satisfies each of the 3 needs

Need 1 — ID-127.4 corpus + mandatory manifest (BI-3/7/9): The §B file tree + root manifest IS the BI-3 deliverable: ≥3 shapes, ≥2 formats, file-only, mandatory .kh-workspace-map.json mapping forms/procurement/ → the procurement workspace, __qa__/ hazard avoided, fixed base path documented. The §A1 workspaces give the manifest a real workspace_id target (the BI-8 Platform — Procurement workspace for the forms route; the §A1 synthetic workspaces are the {130.8} subjects). No pipeline-code change.

Need 2 — ID-130 {130.8}/T-B22 mint validation on data-sparse staging (0 forms today): The §A1 six workspaces deliberately span every {130.8} transform branch: won lift (#1), lost lift (#2), withdrawn → workflow_state + NULL outcome (#3), COALESCE-status fallback for in-progress (#4) and NULL→‘draft’ edge (#5), and the shortlist psq non-default-form_type case (#6). After {130.8}‘s migration runs on staging, an operator/test asserts: 6 form_templates rows minted, the withdrawn row has outcome=NULL workflow_state='withdrawn', won/lost rows carry lifted outcome + audit fields, #5 got workflow_state='draft', and (with §A3 fixtures) the win-rate recompute gives the documented numerator/denominator/shortlist split. This is the non-degenerate fixture the TECH “Risks” section says is the real coverage (vs the vacuous live snapshot where all workspaces carry NULL outcome).

Need 3 — pre-ID-45 walk proof: With §A + §B seeded and the COCOINDEX_SOURCE_PATH mounted at the corpus, an operator runs the walk on the Platform Coolify app BEFORE the first-client re-ingest. Success criteria: a pipeline_runs row pair (in_progress + terminal completed) lands via the webhook with stage_counts.source_walk ≥ 6; source_documents/content_items/reference_items rows appear for the content/qa/sector files; form_templates rows appear for the 2 forms/procurement files bound to the manifest workspace; manifest load succeeds (no manifest_missing). This proves the walk end-to-end on synthetic data, de-risking ID-45.

---

(c) Seeding approach

Mirror the seed-platform-workspaces.ts harness — a NEW idempotent TS script, NOT a migration (this is DATA, not DDL; per supabase/CLAUDE.md data≠migration, and it must NOT ship in the squash baseline).

Proposed: scripts/seed-synthetic-corpus.ts (DB half) + local-fs-platform/corpus/ tree (file half, authored once, kept local).

• Target both prod + staging: reuse PLATFORM_TARGETS/parseSeedArgs/resolveTarget/project-ref guard verbatim. One DB per run (--target=prod|staging), no “both at once” (credential-fat-finger guard). Run twice to cover both DBs.
• Idempotency: lookup-by-name-then-insert for workspaces (no unique constraint, same pattern as the BI-8 seed); for form_questions lookup by (workspace_id, question_text) (UNIQUE exists). Re-run = clean no-op. Dry-run default; --apply to write.
• Clearly-synthetic + removable: every name Synthetic — First-Client …; every domain_metadata.notes carries a Synthetic corpus row …; safe to delete. marker; created_by = PIPELINE_SYSTEM_USER_ID. Ship a --clean flag (delete all name LIKE 'Synthetic — %' workspaces + cascade), so the corpus is one-command removable post-validation.
• Client-name guard: generic invented buyer names + first-client/synthetic only — never a real client name in filenames, names, or commands (the guard-hook denylist).
• File half: authored by hand under local-fs-platform/corpus/ (gitignore-confirm needed — Q-3); the manifest’s workspace_id is filled from the seed script’s output (the Platform — Procurement BI-8 workspace uuid, NOT the synthetic #1 — open Q-1 which workspace the forms route should bind to). A small scripts/write-synthetic-manifest.ts (or a --emit-manifest flag) can stamp the resolved uuid into the manifest so it stays in sync per target.

Ordering: BI-1 (application_types verify) → BI-8 (seed-platform-workspaces.ts) → this seed (synthetic workspaces + questions) → seed-platform-feed.ts → author/mount file tree → run walk. {130.8} migration runs AFTER the synthetic workspaces exist (so its mint has subjects).

---

(d) Risks + open questions for Liam

1. Q-1 — which workspace does the forms-route manifest bind to? The BI-8 Platform — Procurement workspace, or synthetic #1? If forms-route walk writes land on a synthetic workspace, the {130.8} mint (which also writes form_templates per workspace) and the walk could both write form_templates to the same workspace — need to decide whether walk-minted and {130.8}-minted forms should be separated (recommend: forms route → BI-8 Platform — Procurement; {130.8} subjects → the 6 synthetic workspaces, kept disjoint).
2. Q-2 — build the §A3 win-rate fixtures (form_responses+citations) now, or rely on {130.8}‘s own integration-test fixture? The TECH “Risks” parity test seeds its OWN fixture in-test. If we seed them in the corpus too, the staging DB carries them persistently (good for manual win-rate UI checks; risk of double-counting if the test also seeds). Recommend: corpus seeds the workspaces + outcomes; leave the citing-response fixtures to the {130.8} integration test unless Liam wants a persistent staging win-rate demo.
3. Q-3 — is local-fs-platform/corpus/ gitignored / where does it physically live? ID-127.4 says “local, not in-repo”. Confirm the path + that the Coolify app’s COCOINDEX_SOURCE_PATH mount points at /cocoindex-state/corpus. The file tree authoring is an OPERATOR task (per the subtask) — scope: does this proposal’s next-session build include authoring the files, or only the DB seed + manifest emitter?
4. R-1 — form_templates schema timing. Pre-{130.8}, form_templates has NO workflow_state/outcome/submission_date/form_type columns (TECH: “0, neither column exists”). The walk’s form_templates write (§B forms files) only uses pre-{130.8} columns (form_type/deadline/issuing_organisation exist). The {130.8} mint adds the new columns. So: if the walk runs BEFORE {130.8}, the 2 forms files mint pre-{130.8}-shaped rows; that’s fine, but the corpus’s value for Need 2 only lands once {130.8} migrates. Sequencing: seed synthetic workspaces (Need 2 subjects) any time; {130.8} validates after its migration; walk proof (Need 3) works pre-{130.8}.
5. R-2 — prod pollution. Seeding 6 synthetic workspaces + ~30 questions into PROD zjqbrdctesqvouboziae (which is acting as prod+staging). Confirm Liam wants synthetic data in prod, or prod gets only the BI-8/feed minimum and the {130.8} validation happens staging-only. The --clean flag + Synthetic — prefix make removal safe, but recommend staging-first, prod only if explicitly wanted.
6. R-3 — schema-parity side workflow fires when ledger-adjacent schemas change; this is data-only so should be inert, but {130.8}‘s DDL (run separately) triggers it — not this corpus’s concern.
7. R-4 — feed_articles real public URL (gov.uk/.../procurement-act-2023-short-guide) is the existing feed seed’s choice; the walk’s /extract depends on that URL staying live. Already an accepted ID-127.3 risk, inherited.

---

Summary of the build (next session, NOT now)

1 new idempotent TS seed (scripts/seed-synthetic-corpus.ts) mirroring the seed-platform-workspaces.ts harness → 6 synthetic procurement workspaces (won/lost/withdrawn/null-status/in-progress/psq-shortlist) + ~30 form_questions, --target/--apply/--clean, prod+staging-capable; a manifest emitter; and the local-fs corpus/ file tree (4 shapes, 3 formats) + root .kh-workspace-map.json. Reuses seed-platform-feed.ts for the trigger slice. Resolve Q-1/Q-2/Q-3 + R-2 (prod opt-in) with Liam before building.