Born-in-DB Provenance — Decision Surface (P5 / OQ-ES-1)

Born-in-DB Provenance — Decision Surface (P5 / OQ-ES-1)

For: Liam (product owner) Question: When a completed procurement form/response that has an outcome (a won bid, an accepted proposal, a validated questionnaire) is promoted into the knowledge base, the resulting content is born in the database — its origin is the form-response record it came from, not an uploaded file. The platform is about to enforce a rule that every content_item must trace to a source_document. Born-in-DB content has no file. How should it satisfy provenance — decided from first principles, not from the current “a source is a file” habit? Date: 2026-06-20 Status: Decision surface for Liam. Recommends, does not pre-empt. The semantic call is Liam’s.

---

1. The problem, framed wider than bids

A “bid” in this platform is not a special object. It is a workspaces row bound to application_types.key = 'procurement'. The same workspace + form + state-machine pattern already underpins six application types — procurement, intelligence, sales proposals, product guides, competitor research, training/onboarding — each with its own terminal “outcome” states (app/api/procurement/[id]/outcome/integrate/route.ts:55-60; lib/workspace-types.ts:23-30).

So the real question is not “how does a won bid get provenance”. It is: whenever any completed form reaches an outcome state and its answers are curated back into the library, what is the provenance of that born-in-DB content? Sales proposals are explicitly the next domain, so whatever we choose must work for “accepted proposal” the same way it works for “won bid”, with no second migration.

This matters because it is the platform’s core value loop, not a side feature. Guiding Principle 5 — “the library and the applications feed each other” — is the flywheel: won answers, vetted summaries and validated proposals get curated back so that “the corpus that wins bid #20 is meaningfully better than the one that wrote bid #1” (platform-direction.md:37; 01-vision.md:60). Provenance for born-in-DB content is load-bearing for the single most strategically important flow in the product. If the provenance model makes that loop awkward — e.g. forces us to fake a file — we are taxing the flywheel.

Crucial framing point: we are NOT a bid-management tool. Our edge is data quality and cross-application reuse. That cuts two ways here. It means we should adopt the market’s content-first provenance instinct (below), but we should not copy bid-tool workflow mechanics. The provenance model should be generic to “form outcome”, never bid-specific.

---

2. What provenance must actually GUARANTEE here

The invariant “every content_item traces to a source” is really protecting four different things. Only one of them is genuinely threatened by born-in-DB content — and it is the one that does not apply.

#	What the invariant guarantees	Born-in-DB content can satisfy it?
1	Auditability / attribution — every fact can answer “where did this come from, who/what produced it”. The trust signal that lets AI reuse content confidently.	YES. The form-response record (won bid answer) is a real, auditable, citable origin.
2	Re-derivability — the pipeline can re-mint the exact same rows from the same input on a full re-walk (truncate-and-rebuild is safe because PKs are deterministic).	N/A. Born-in-DB content was never in the file walk. It is app-written, not pipeline-walked. This guarantee simply does not apply to it.
3	Relational integrity — a non-orphan FK target so lineage and re-ingestion-diff queries do not break.	YES. A real provenance row keyed on the form-response gives a valid, non-orphan target.
4	Legal / quality attribution per tenant	YES. The originating workspace + response carry this.

Evidence: platform-direction.md:34,37; source_documents column comment “lineage tracking and re-ingestion diffing” (squash:351); the deterministic-PK seeds in the pipeline (flow.py:2039 sd:{rel_path}, flow.py:2775 sd:{url}).

The category error driving the whole P5 dilemma: the file-centric framing conflates guarantee #1 (auditability — satisfiable) with guarantee #2 (re-walk-derivability — not applicable). The “never invented” principle (ES-1) does not mean “must be a file”. It means the provenance must trace to something real the creator supplied. A won bid answer is real provenance. The only thing born-in-DB content cannot do is be re-derived from a file walk — and it does not need to, because it is not a file.

Two facts that demolish the “a source is a file” assumption

These are not opinions — they are already true in the live schema and verified against the migration:

1. source_documents is already a non-file provenance table. The reference_ingest RPC mints a source_documents row for a URL source (no file on disk): it sets storage_path = source_url, extraction_method = NULL, status = 'processed', and synthesises filename/mime_type/file_size/content_hash from the fetched bytes (squash:4585-4596, verified — the comment literally reads “storage_path = source_url for URL-sourced provenance”). A source_documents row with no physical file behind it exists in production today. The bid-outcome case is the same shape, one step further from a file.

2. The platform already has the born-in-DB answer for its newer knowledge unit. q_a_pairs — the corpus’s primary reusable unit — has no source_document_id column at all. It models born-in-DB provenance as typed lineage FKs: source_form_response_id, source_question_id, source_workspace_id, plus origin_kind with the value 'derived_from_form_response' (squash:7128-7165, verified; column comment: “UC5 promotion lineage: the form_responses(id) this Q&A draft was promoted from… ON DELETE SET NULL”). This is already ratified (S229): UC5 (form-response → Q&A promotion) is “a KH-DB-only operation… not a write-back operation” (05-qa-flow.md:279).

So the platform already holds two divergent, both-legitimate provenance models: file/URL → source_documents (the file-shaped path) and form-response → typed lineage FKs (the born-in-DB path). content_items is the lone table being forced into the file-shaped model. The whole question is really: which existing model should born-in-DB content_items content align with?

Where the born-in-DB content actually originates today (the single seam)

Verified at app/api/procurement/[id]/outcome/integrate/route.ts:209-246: the new_entry insert writes content_items with source_url: null, platform: 'extraction', ingestion_source: 'bid_outcome_integration', and a metadata blob holding source_bid_id, source_bid_name, source_question_id, source_question_text, integrated_at — and no source_document_id key at all. Everything a faithful provenance record needs is already in local scope at the moment of the write. This is the only born-in-DB content_items mint site. The Q&A promotion routes write to q_a_pairs (out of scope). So whatever we choose is applied at exactly one place.

---

3. The design options

Synthesised and de-duplicated across the five researchers. The original spec framed this as a binary (mint-a-source-doc vs carve-out). The evidence surfaces a richer space. The options below are ordered to make the decision tree clear.

Option A — Mint a synthetic source_documents row per form-outcome

Treat the form outcome as a kind of source document. Mint a real source_documents row keyed uuid5(sd:form_outcome:{form_response_id}) (generic — not the bid-specific sd:bid_outcome:{question_id} the spec drafted), carrying the bid/form provenance in extraction_metadata. Derive the 5 NOT-NULL columns from the answer bytes (content_hash = sha256(content), file_size = len(content), mime_type a form-outcome marker, extraction_method = NULL, storage_path = the synthetic key). Set content_items.source_document_id to it.

• Schema impact: none. Reuses the existing table and a proven copyable RPC pattern (reference_ingest).
• Honours “never invented”? Yes — every column derives from the real won bid the user produced; nothing is a placeholder.
• Re-walk safe? Yes — born-in-DB rows are not walked; the cross-target FK ordering hazard (R1/OQ-45-9) is a pipeline problem and does not apply to this app route (single transaction, sd-first).
• Cost of the call: Liam must be willing to assert “a form outcome IS a source document.” The synthesised file-shaped columns are the only thing that can feel invented.

Option B — Name the provenance KIND honestly (source-kind abstraction)

Add a kind discriminator to source_documents (uploaded_file | url | derived_from_platform_record) and make the 5 file-shaped columns conditionally-required per kind. The form-response record becomes a first-class, truthfully-named source kind.

• Schema impact: HIGH — new column, relax 5 NOT-NULL columns to per-kind CHECKs, regen the api view + database.types.ts, and every consumer that reads filename/mime_type must handle the derived kind.
• Honours “never invented”? Best of all. It stops pretending a bid is a file and names the source kind truthfully — the most first-principles-honest answer.
• Wider than bids? Natively yes — derived_from_platform_record covers every born-in-DB surface (sales proposals next) under one named kind.
• Cost: highest, and it reopens the NOT-NULL discipline that ID-45 is tightening. But: the ID-45 zero-row re-ingest window (INV-6) is the one cheap moment this column-relaxation will ever happen — retrofitting later means a backfill. This decision is time-boxed to the cutover.

Option C — Align with q_a_pairs: typed record-lineage FKs (recommended core)

Give content_items the same typed lineage columns q_a_pairs already has (source_form_response_id, source_question_id), and treat a non-null lineage to a form-response as co-equal provenance with a file source_document_id. Born-in-DB content is anchored to the record it was born from, not to a file-shaped intermediary. This is the platform’s own already-ratified answer for the identical data.

• Schema impact: medium — add two FK columns (mirroring q_a_pairs), and the provenance constraint becomes “every content_item has SOME provenance” (file source_document_id XOR born-in-DB form-response lineage) rather than source_document_id NOT NULL.
• Honours “never invented”? Yes, and most faithfully — it records the truth (the record IS the provenance) with no file masquerade.
• Wider than bids? Yes — keyed on form_response, type-generic across all six application types. Composes cleanly with the citation graph (citing_entity_kind = 'form_response') and the win-based-surfacing roadmap signals (bl-138/139), which share the form-response pivot.
• Cost: changes the shape of the constraint ID-45 is landing (a discriminated “has provenance” rule, not a single NOT-NULL FK). This is a meaningful re-spec of the CONSTRAINT subtask. It also raises the question of whether P5 should write content_items at all (see §6).

Option D — Materialise a real sidecar file for the outcome (reject for cutover)

Emit the outcome answer as a real .md file in the source folder, so it is walked like any other file and source-backed by construction.

• The platform has already rejected this for exactly this content. The Q&A sidecar spec (id-59) explicitly defers derived_from_form_response pairs because they are “KH-DB-native (born from a form response, no source file), so a sidecar would be inventing a file identity as a side effect — exactly the source-layout-freeze hazard” (PRODUCT-qa-sidecar-canonical.md:217, verified). It also inverts the born-in-DB premise, imports the entire sidecar emit/route/skip hazard surface, and couples P5 to an unfinished convention. Disproportionate as a cutover mechanism. Keep only as a possible long-term “v2”.

Option E — Sanctioned carve-out (nullable escape hatch / sentinel)

Exempt born-in-DB content from the constraint: either source_document_id stays nullable for the bid_outcome_integration class, or a single shared sentinel source_documents row.

• The sentinel variant violates ES-1 (“not a placeholder” — a row invented solely to satisfy the constraint is exactly what’s forbidden).
• The nullable-class variant violates ES-7/ES-8 — it reintroduces the nullable escape hatch ID-45 exists to close, and creates the “third category of source-less creation” ES-8 forbids.
• Lowest DDL, highest principle cost. Once examined, the honest version of this instinct is just Option B without the schema work — so it collapses into B.

Options table

Option	What it is	Schema cost	”Never invented”?	Wider than bids?	Re-walk safe?	ID-45 fit
A Mint synthetic sd	Form outcome IS a source document; mint a real sd row	None	Yes (mild file-masquerade)	Yes if keyed on form_response	Yes	Keeps NOT-NULL uniform; lowest-risk cutover mechanism
B Name the kind	Add kind discriminator; conditional NOT-NULLs	High (must ride zero-row window)	Best	Native	Yes (cleaner)	Reopens NOT-NULL but at the only free moment
C Typed lineage FK	Mirror q_a_pairs; provenance = form-response record	Medium	Yes (most faithful)	Yes	Yes	Reshapes constraint to “has SOME provenance”
D Sidecar file	Emit a real .md, walk it	None	Yes (+re-derivable)	Yes	Perfect	Rejected — explicitly refused for this content
E Carve-out	Exempt born-in-DB from the constraint	Low DDL	No (sentinel) / weakens guarantee (nullable)	Yes	Fine	Punctures the headline ES-7/ES-8 guarantee

Cross-checks

• CocoIndex / re-walk: The biggest technical objection (the sd-before-ci cross-commit race) is a pipeline-only hazard. P5 is an app route in a single transaction, so it does not carry that risk. ID-45 reinstates a real NOT-NULL + ON DELETE RESTRICT FK (verified, CONSTRAINT subtask: ALTER COLUMN source_document_id SET NOT NULL + REFERENCES source_documents(id) ON DELETE RESTRICT). This means any option relying on a NULL/sentinel (E) fights the very constraint being added; and any option that points source_document_id at a row (A) must mint that row first, in the same transaction (trivially satisfiable app-side).
• “Real provenance, never invented”: every option except the sentinel form of E passes. The most honest are B and C (they name/record the truth); A is honest but makes the bid wear a file-shaped costume.
• Competitors (adopt vs avoid): Every dedicated bid/proposal competitor (Responsive, Loopio, Qvidian, Altura, Brainial) treats the library answer record — not a file — as the unit of reusable knowledge, and reuses at Q&A-answer granularity. “AI-generated responses can be added to the library” with no file ancestor is the norm, not the edge case. The original RFP is consumed by a separate transient “shred/analyse” stage and is never the durable source of a library answer. Adopt: content/record-as-provenance, outcome-as-metadata, Q&A granularity. Avoid: AutogenAI’s file-only model (explicitly the laggard pattern) and any bid-workflow coupling — we are not a bid tool. Competitors also confirm a clean separation we must honour: provenance (“where it came from” → the form-response record) is distinct from outcome/performance (“how well it did” → win-status, eval score, “Used: 6, Won: 4”). The win is metadata on the content item, not part of the source layer. Trying to make the source row encode the win is over-modelling.
• Wider form-outcome surface: Only options keyed on form_response (not the bid) generalise to sales proposals and the other four application types without a second migration — which matters because the ID-71 bid → forms rename is in flight and sd:bid_outcome:{question_id} is on the wrong side of it.

---

4. Recommendation

Recommended: Option C (typed record-lineage on content_items, keyed on the form-response) as the destination, delivered via the Option E-hybrid seam now — i.e. mint through the shared content_item_ingest_with_source RPC with a named derived_from_platform_record provenance kind, deferring the file-column relaxation until the zero-row window.

In plain terms: anchor born-in-DB content to the record it was born from, not to a fake file. That is what the platform already does for its newer, primary knowledge unit (q_a_pairs), it is what the entire competitor market does, and it is the most faithful answer to “never invent provenance”. The form outcome is genuinely the origin — record that truth directly.

Why not A (mint a synthetic source_documents row), even though it is the cheapest and the TECH’s informational lean? A is the safest cutover mechanism and is genuinely zero-new-data. But it asks Liam to assert “a form outcome IS a file-shaped source document”, and it synthesises file columns that the data does not have — walking right up to the “invent provenance to satisfy a constraint” line that ES-1 forbids. It is also bid-shaped unless carefully generalised, and it diverges content_items from q_a_pairs, which already solved this without a source_documents row. A is the right fallback if the schema appetite for C is zero — but it optimises for “least change now” over “right shape forever”.

Why C over B? B (a kind column on source_documents with conditional NOT-NULLs) is the most architecturally pure, but it relaxes the very NOT-NULL discipline ID-45 is tightening, and it keeps born-in-DB content in the file-shaped table. C puts the provenance where the platform already proved it belongs — typed lineage on the content unit — and aligns content_items with q_a_pairs instead of forking a third pattern.

The pragmatic delivery path (lowest regret): Build the shared source-required write seam (the content_item_ingest_with_source RPC the amendment already proposes) with a provenance-KIND parameter from day one. Route P5’s born-in-DB write through it with kind derived_from_platform_record, keyed on form_response_id. This:

• gives constraint-satisfaction now without blocking the cutover;
• records the provenance kind as a queryable marker so the eventual promotion to true typed-lineage columns (C) or a named source-kind (B) is a migration, not a re-architecture;
• defers the hard, expensive-to-reverse schema decision from the cutover to a cheap-to-evolve metadata decision — while still landing the honest shape.

One thing is unambiguous regardless of which option wins: key the provenance on the FORM OUTCOME / form-response record, never on “the bid”. That is the single mistake the original sd:bid_outcome:{question_id} framing makes, and the ID-71 bid → forms rename will immediately contradict it.

---

5. Decisions for Liam

These are genuine product/architecture calls. The recommendation justifies a direction, but each of these is yours.

1. THE SEMANTIC CRUX. Is a completed-form-with-outcome ontologically: (A) a source document → mint a synthetic sd row; (B) a distinct source KIND → add a named kind, relax file columns; (C) the provenance record itself → typed lineage FK to the form-response (recommended); (D) a file-to-be → emit a sidecar (rejected for cutover); or (E) not a source at all → carve-out (weakens the guarantee)? Every option is just one answer to this one question.

2. Does P5 content even belong in content_items? Given the Q&A two-tier model, born-in-DB outcome promotion arguably belongs in q_a_pairs (which already has typed lineage, origin_kind='derived_from_form_response', and no source_document_id) — the legacy 395 q_a_pair content_items are already being soft-archived in its favour. If P5 re-routes to q_a_pairs, OQ-ES-1 may largely dissolve. This is a routing decision with the biggest leverage of all.

3. How many born-in-DB surfaces do you expect? Sales proposals are explicitly next; product guides, competitor research, intelligence all have outcome states. If you expect many, the per-case synthetic mint (A) accrues N copies of synthesised-column logic and the named-kind / typed-lineage abstraction (B/C) pays off. If you expect only bids for the foreseeable future, A’s “least change now” is more defensible. Only you can scope expected breadth.

4. Spend the zero-row window, or not? The ID-45 cutover (empty-table, INV-6) is the one cheap moment to do any schema relaxation/addition (B or C) without a backfill. After cutover, the same change needs a data migration. Do you want to pay the abstraction cost once, now, while it is free — or take the lowest-change path (A) and accept a possible later retrofit?

5. Outcome vs provenance separation. Confirm the win/loss/score lives as metadata/usage on the content item (composing with the citation graph and win-rate surfacing), not inside the source/provenance layer. The recommendation assumes this; competitors validate it; please confirm it is the intended model.

6. Generic key now. Confirm the provenance key is authored in the generalised form_response vocabulary (post bid → forms rename), not the bid-specific sd:bid_outcome:{question_id} the draft used. (Low-controversy, but it is a deliberate choice to make now rather than migrate later.)

---

6. Downstream consequences of the recommended direction

How the recommendation reshapes the surrounding ID-45 work.

ID-45 source-backing decomposition (P5 / CONSTRAINT deps)

• The SEAM subtask (content_item_ingest_with_source RPC) gains a provenance-kind parameter — a small extension to its already-planned shape, not a new subtask. It must accept derived_from_platform_record keyed on form_response_id, alongside the existing file/URL provenance.
• P5 (currently BLOCKED — awaits OQ-ES-1) unblocks and routes its single insert (route.ts:209) through the seam. It writes the provenance kind, not a fake file.
• CONSTRAINT subtask: under the recommended C-shaped direction, the constraint is not a bare source_document_id NOT NULL. It becomes a discriminated “every content_item has SOME provenance” rule (source_document_id NOT NULL XOR born-in-DB form-response lineage NOT NULL). The spec already flags that CONSTRAINT → P5 dependency must be wired “once OQ-ES-1 lands” — under C, CONSTRAINT must depend on P5 (born-in-DB rows must be backed before the rule lands), and the migration text changes from a single SET NOT NULL to a CHECK admitting the born-in-DB kind. If Liam instead picks A, the original bare-NOT-NULL CONSTRAINT stands unchanged and CONSTRAINT depends on P5 only to ensure the mint exists.
• ES-7 (“zero null source_document_id”) is re-expressed under C as “zero content_items with no provenance of any kind” — same guarantee strength, broader vocabulary. Under A/B it stays literally as written.

Q&A-sidecar relationship

• Confirms and reuses the id-59 ruling: derived_from_form_response content is born-in-DB, no source file, and the sidecar approach is the wrong model for it (it “invents a file identity”). The recommendation is fully consistent — born-in-DB provenance is a record pointer, not a materialised file. No conflict; the two specs reinforce each other.

Schema work that must ride the ID-45 zero-row window

• Option C: add content_items.source_form_response_id + source_question_id FKs (mirroring q_a_pairs, ON DELETE SET NULL), and land the discriminated provenance CHECK. Best done in the empty-table window.
• Option B (if chosen instead): add source_documents.kind + relax the 5 NOT-NULL columns to per-kind CHECKs + regen the api view and database.types.ts. Must be in the window — otherwise a backfill.
• Option A: no schema work — purely the seam + route change. (Its main appeal.)
• The citing_entity_kind enum (currently single-valued 'form_response', deliberately extensible) and any new provenance key should be authored generically now so sales proposals and the other application types reuse one mechanism.

Re-derivability footnote (worth a deliberate check)

Born-in-DB content minted via A/B/C/E is not reproduced by the truncate-and-re-walk (it is not a file). So on the next full re-ingest, confirm born-in-DB outcome rows sit in a RETAIN-class table across the ID-45 truncate (they are app-written, not a pipeline target, so they should survive) — otherwise they must be re-promoted from the still-existing form_responses rows. Only Option D (sidecar) makes them natively re-walkable, which is one of the few arguments for ever revisiting D as a long-term v2.

---

All claims in this surface were cross-checked against the live migration (supabase/migrations/20260617130000_squash_baseline.sql), the P5 route (app/api/procurement/[id]/outcome/integrate/route.ts:209-246), the ID-45 amendment (specs/id-45-full-corpus-reingest/PRODUCT-source-backing.md), and the id-59 sidecar spec. The reference_ingest URL-source precedent (a real fileless source_documents row in production), the q_a_pairs typed-lineage model with no source_document_id, and the sidecar deferral of derived_from_form_response content were each read directly in source, not paraphrased.