Skip to content
Canonical

ID-59 {59.25} PRODUCT — Q&A markdown-sidecar canonical model (sidecar-emit promoted to v1)

ID-59 {59.25} PRODUCT — Q&A markdown-sidecar canonical model

Section titled “ID-59 {59.25} PRODUCT — Q&A markdown-sidecar canonical model”

Status: PRODUCT draft — authored 20/06/2026 by a FRESH planner instance (Q-PLANNER-2). This is a SEPARATE feature from the two sibling PRODUCT specs in this dir (PRODUCT.md per-UC write-back + edit_intent; PRODUCT-qa-corpus-promotion.md route-i promotion); it does NOT overwrite either. Inputs read in full: PRODUCT.md (esp the three “Q&A file-sidecar materialisation — DEFERRED-v1.1” notes at :140, :189, :147), the ratified PRODUCT-qa-corpus-promotion.md (INV-1..16) + TECH-qa-corpus-promotion.md (R1 promoteCorpusExtractions() as-built), the built code (lib/q-a-pairs/promote-corpus.ts, lib/edit-intent/write-back.ts, app/api/q-a-pairs/[id]/route.ts), the cocoindex flow (scripts/cocoindex_pipeline/flow.py, adapters.py, extraction.py), and the canonical architecture/decision docs (05-qa-flow.md §4–§5, decision-graph.md Q2.9/Q3.2/:43).

This PRODUCT is a RATIFIED REVERSAL (Liam, S380). Markdown sidecar-emit was always the INTENDED canonical Q&A form but is currently ratified DEFERRED-v1.1 (decision-graph.md:43 records “Markdown sidecar emit = DEFERRED-v1.1, not reopened to v1”; 05-qa-flow.md §4.4 reaffirms the deferral). Liam has ratified promoting sidecar-emit to v1 and specifying it NOW, before ID-45. The required doc-corrections are surfaced as INTENTS in §Doc-update intents (the Orchestrator routes them to the Curator / docs-site lane).

Sequence note (load-bearing): the sidecar file convention + walk skip/route gate (INV-1..INV-6) is the gate that ID-45 {45.3} TECH references (the re-ingest walk must know to skip/route Q&A sidecars, not mint junk content_items). The convention MUST be frozen by this PRODUCT before {45.3} TECH is ratified. {45.3} is being authored in parallel — this is a cross-Task ordering dependency tracked at Task level, not a sibling dep.

Summary

Section titled “Summary”

The markdown sidecar file becomes the canonical form of a Q&A pair: an in-platform Q&A edit, a corpus promotion, and a re-walk all converge on one .md file in the source folder, and the q_a_pairs row + q_a_extractions row are DERIVED projections of that file. This realises the Wikipedia Principle for Q&A (one golden record, many views) and resolves the multiple-representations problem — one canonical file projects to two DB rows, it does not add a third representation. The change promotes sidecar-emit from DEFERRED-v1.1 to v1.

Problem

Section titled “Problem”

The platform has shipped the two-tier Q&A model DB-side but left the sidecar FILE — the form the model was designed around — unbuilt, so today the DB rows ARE the canonical Q&A form with no file behind them, which is the multiple-representations hazard the sidecar pattern was meant to dissolve:

  • Corpus promotion ends at the DB. promoteCorpusExtractions() (lib/q-a-pairs/promote-corpus.ts:170) inserts the q_a_pairs row, CAS-links the extraction, embeds, publishes — and STOPS. There is no file emit. (gitnexus_context: promoteCorpusExtractions — 1 direct caller app/api/q-a-pairs/promote-corpus/route.ts:POST; outgoing calls retireSupersededPairs, embedAndPublish, tryQuery, safeErrorMessage — no filesystem write in the call graph.)
  • UC6 user-direct Q&A revision is explicitly KH-DB-only by design. app/api/q-a-pairs/[id]/route.ts:11-13 carries the comment “KH-DB-ONLY (INV-4): this revision path writes ONLY to Postgres. The file-sidecar materialisation for Q&A is DEFERRED-v1.1 — there is no file write here, by design.” The route UPDATEs q_a_pairs and the trigger snapshots history; no .md is written.
  • File write-back exists ONLY for content_items. writeBackFileFirst() (lib/edit-intent/write-back.ts:131) is the file-first + compensating-restore adapter for the UC1/UC4 content-item leg — it writes the edited bytes to the EXACT source_documents.storage_path (the uuid5 PK seed) and restores on DB-leg failure. (gitnexus_context: writeBackFileFirst — 3 direct callers patchHandler / runSweep / rollbackSweep; participates in flows proc_85_post, proc_86_post.) This is the proven pattern the Q&A sidecar leg REUSES; it has no Q&A caller today.
  • The pre-walk hazard is the reason sidecar-emit was deferred, and it is real. A Q&A .md dropped in COCOINDEX_SOURCE_PATH matches _TEXT_EXTENSIONS = {".md", ".markdown", ".txt"} (scripts/cocoindex_pipeline/adapters.py:62) → convert_binary_to_markdown → _ingest_content_branch (flow.py:2003), which runs the FULL content branch (extract_classification + extract_qa_form + extract_entity_mentions + extract_relationships, flow.py:2016-2027) and mints content_items / source_documents / content_chunks / entity_mentions. So an un-gated Q&A sidecar would mint junk content_items — the S297 BUG-B hazard, the exact reason edit_intent was put on a column not a sidecar (per {59.1} RESEARCH + Liam 08/06).
  • The fork that the gate slots into already exists. _ingest_file_body (flow.py:1900) already computes a route ONCE before either write path runs: resolve_route(manifest, rel_path) returns route ∈ {content, forms} (flow.py:1917-1962), dispatching to _ingest_form_branch or _ingest_content_branch. A Q&A sidecar is a natural THIRD route at this fork — extraction-only, no content_items mint. There is also a precedent for a reserved file being skipped from the content walk: _WORKSPACE_MANIFEST_FILENAME = ".kh-workspace-map.json" (flow.py:1619).
  • The inner two-tier idempotency contract is already half-built. extract_qa_form(content_text: str) -> QAFormExtraction (extraction.py:1062) is the inner, content-hash-keyed extractor the 05-qa-flow.md §5.2 layered shape mandates (extract_q_a(content_text: str) → ExtractedQa); it already satisfies the COCO.10 idempotency gate (memo keyed on content_text, not FileLike). What is missing is the outer sidecar-file tier and the emit legs that write the file.
  • q_a_pairs has NO linkage anchor to a sidecar today. The live schema (supabase/migrations/20260617130000_squash_baseline.sql:7128-7165) has id, question_text, answer_standard, answer_advanced, scope_tag, anti_scope_tag, source_workspace_id, origin_kind, publication_status, superseded_by, valid_from/valid_to, created_at/updated_at, alternate_question_phrasings, question_embedding, edit_intent, source_form_response_id, source_question_id. There is NO source_document_id and NO sidecar_storage_path. The linkage column is greenfield (INV-8). q_a_pairs live count = 0 (the zero-row window the column rides).

gitnexus orientation: existing symbols promoteCorpusExtractions/writeBackFileFirst/extract_qa_form are the three integration points — the sidecar-emit legs attach to the first, reuse the second's pattern, and feed the third. Not greenfield at the symbol level; greenfield only for the new linkage column and the walk-routing gate.

Goals / Non-goals

Section titled “Goals / Non-goals”

Goals. Promote markdown sidecar-emit to v1. Make the sidecar .md file the canonical Q&A form, with q_a_pairs + q_a_extractions as derived projections. Freeze the sidecar file convention + walk skip/route gate (the ID-45 {45.3} gate). Add the q_a_pairs↔sidecar linkage anchor. Classify the DB-only lifecycle columns that a re-walk MUST NOT clobber. Emit sidecars on corpus promotion and on the user-facing Q&A write paths. Prove the round-trip (in-platform edit → file → re-extract → DB reconverges with no drift). Reconcile sidecar FILES on UC8 merge (not just the DB superseded_by).

Non-goals (v1). Sidecars for content_items (that is the existing writeBackFileFirst content leg — UC1/UC4 — already shipped; this spec only REUSES its pattern for Q&A). The edit_intent arbitration substrate (sibling PRODUCT.md — sidecar-emit is single-actor on the promotion path and rides the existing UC6 arbitration on the user-direct path). The DDL/migration mechanics (a {59.26} TECH concern) — this PRODUCT specifies the behaviour the columns + convention enable. SharePoint/Notion sidecar sync (the source binding is LocalFS for v1 per Q2.10). A user-facing “sidecar files” management UI — the sidecar is AI-invisible infrastructure (INV-22), not a surfaced feature.

Figma: none provided. This is backend + pipeline + data-model work with no new visual surface; the only adjacent UI is the existing Q&A revision-history surface (bl-304, classified stays-backlog in §Satellite disposition).

Behavior

Section titled “Behavior”

“User” here is fourfold. (a) The operator running corpus promotion / re-ingest (who must see sidecars emitted and the round-trip reconverge). (b) The admin/editor revising a Q&A pair in-platform (whose edit must reach the canonical file). (c) The cocoindex walk (the “code that reads the model”) which must route a Q&A sidecar correctly and never mint junk content_items. (d) The downstream consumer of q_a_search (who must see no drift after a round-trip). Behaviour is specified so each can rely on it.

A. The canonical model (the Wikipedia Principle for Q&A)

Section titled “A. The canonical model (the Wikipedia Principle for Q&A)”

  1. The sidecar file is the canonical Q&A form. For every Q&A pair that has a sidecar (INV-7 scope), the markdown .md file in the source folder is the single golden record. The q_a_pairs row and the q_a_extractions row are DERIVED projections of that file: one canonical file → exactly two DB projections. The platform MUST NOT treat the DB rows as a third independent source of truth — on any conflict between the file content and the round-trippable DB fields (INV-9), a re-walk of the file is authoritative for those fields.

  2. The sidecar carries the round-trippable Q&A content, nothing else. A sidecar file’s canonical payload is exactly the fields a re-walk reconstructs (INV-9 “carried” set): question_text, answer_standard, answer_advanced (when present), alternate_question_phrasings, scope_tag, anti_scope_tag. It MUST NOT carry the DB-only lifecycle state (INV-9 “not-carried” set) — those are never written to the file and never read back from it.

  3. Re-walk converges, does not duplicate. Re-walking a sidecar that has already been promoted reconstructs/updates the SAME q_a_pairs projection (via the existing promoted_to_pair_id idempotency anchor on the extraction, plus the new linkage anchor INV-8), never a duplicate pair. This is the existing promoteCorpusExtractions() idempotency contract (PRODUCT-qa-corpus-promotion.md INV-5/INV-8) extended to the file-seeded re-walk.

B. Sidecar file convention + the walk skip/route gate (THE ID-45 GATE — freeze first)

Section titled “B. Sidecar file convention + the walk skip/route gate (THE ID-45 GATE — freeze first)”

  1. A Q&A sidecar is identified by a frozen, reserved convention — NOT by being a generic .md. The convention MUST be unambiguous to the walk so a Q&A sidecar is NEVER routed down the content branch. The convention is frozen by this PRODUCT as: a reserved sub-directory prefix under the source root (recommended __qa__/ — a double-underscore sentinel dir, mirroring the existing .kh--prefixed reserved-file precedent at flow.py:1619), with the Q&A sidecar files inside it. A path resolving under the reserved prefix routes to the Q&A-sidecar branch; a path outside it is generic content as today.

    • Why a reserved subdir (recommendation + justification). It is the lowest-ambiguity, lowest-collision option and slots directly into the EXISTING resolve_route(manifest, rel_path) fork (flow.py:1921) as a third route value alongside content / forms — no new interception point, no per-file content sniff. A reserved extension (e.g. .qa.md) would still match the .md suffix in _TEXT_EXTENSIONS and require changing the suffix-matching in adapters.py:62, which is shared with generic content routing — a riskier blast radius. A frontmatter discriminator requires READING file bytes before routing (defeating the pre-walk skip and re-introducing the content-sniff the reserved dir avoids). The reserved subdir wins on all three axes.
    • Open question (OQ-25-1): confirm the reserved-prefix string (__qa__/ proposed) and whether it is manifest-declared (an entry in .kh-workspace-map.json routing the prefix to qa_sidecar, consistent with the existing manifest-prefix routing) or a hard-coded sentinel. Needs Liam ratification — this is THE string ID-45 {45.3} freezes against.

  2. A Q&A sidecar NEVER mints content-branch rows. When the walk encounters a path under the reserved prefix, it MUST NOT create content_items, content_chunks, or entity_mentions for it (the S297 BUG-B junk-content hazard), nor any content embedding. It mints exactly one source_documents row — the INV-8 linkage anchor (PK uuid5("sd:" + rel_path), per TECH P1) — and otherwise runs ONLY the Q&A-extraction tier (INV-6). This is the load-bearing skip-rule: a re-ingest run over a folder containing N Q&A sidecars adds ZERO content_items for those N files.

  3. A Q&A sidecar runs the two-tier extraction shape (COCO.10-safe). The walk routes a reserved-prefix path to an outer sidecar-file tier (keyed on the file handle / byte change) that calls the EXISTING inner extract_qa_form(content_text: str) extractor (extraction.py:1062), which is content-hash-keyed and hits memo on metadata-only edits (the 05-qa-flow.md §5.2 layered-fn requirement, COCO.10). A metadata-only touch of a sidecar (mtime/owner change, no content change) MUST NOT re-run the LLM extraction. The inner tier UPSERTs q_a_extractions exactly as the content path’s extract_qa_form call does today; the difference is only the absence of the content_items legs (INV-5).

C. Linkage anchor on q_a_pairs

Section titled “C. Linkage anchor on q_a_pairs”

  1. Which origin_kinds get sidecars. Sidecar emit is v1 for origin_kind='extracted_from_corpus' (corpus promotion, the promoteCorpusExtractions() leg) and origin_kind='curated_explicit' (manual authoring / UC6 user-direct revision — the app/api/q-a-pairs/[id] route). It is NOT v1 for origin_kind='derived_from_form_response' (UC5 promotion, app/api/q-a-pairs/promote) — form-response-derived pairs keep their existing KH-DB-only resting state (draft, no file) and are evaluated for sidecar materialisation in v1.1 once the corpus + curated legs are proven. origin_kind='imported_legacy' is the one-shot Phew migration and gets no sidecar (historical). This scopes the emit leg to TWO sites: the promote-corpus loop and the user-direct Q&A route — NOT the UC5 form-response route.

    • Rationale. extracted_from_corpus pairs are FILE-SEEDED by definition (their extraction came from a walked source), so emitting a sidecar closes their round-trip. curated_explicit pairs are the ones a human edits in-platform — the canonical-file promise only holds if their edit reaches a file. derived_from_form_response pairs are KH-DB-native (born from a form response, no source file), so a sidecar would be inventing a file identity as a side effect — exactly the source-layout-freeze hazard UC1’s source-less guard already refuses (write-back.ts:178). Deferring them to v1.1 keeps v1’s emit legs to the two paths where a file is genuinely canonical.

  2. The linkage anchor is a source_document_id uuid on q_a_pairs (uuid5-derived, FK-LESS). A new nullable q_a_pairs.source_document_id uuid column links a pair to its sidecar’s source_documents row, without a real cross-target foreign key — consistent with the landed cocoindex-write-model R1 (the content_items→source_documents FK was already DROPPED, migration 20260602073942 / BUG-E, because the cocoindex autocommit write model cannot satisfy cross-target FKs — see write-back.ts:96). The id is the uuid5 of the sidecar’s source-relative path (the same derivation the content path uses, flow.py:1915 uuid5(_KH_PIPELINE_DOC_NS, "ci:" + rel_path)-analogue for the sidecar namespace), so the linkage is stable across re-walks. source_document_id IS NULL means “no sidecar yet” (pre-emit, or a derived_from_form_response pair).

    • Why this over a sidecar_storage_path text. A uuid5-derived source_document_id is (a) FK-pattern-consistent with the rest of the cocoindex write model, (b) stable: it does not break if the human-readable path is later reorganised within the reserved dir (the 05-qa-flow.md §5.5 path-stability caveat — UUID-keyed naming survives folder reorg), and (c) it reaches the actual storage_path via the source_documents row (one indirection) rather than duplicating a mutable path string on q_a_pairs. A bare sidecar_storage_path text would denormalise a path that the source-binding can rename, re-introducing the drift the UUID key avoids. Recommend source_document_id.
    • This column rides the zero-row window: q_a_pairs live count = 0, so the ALTER is zero-backfill (mirrors the edit_intent pre-cutover discipline in sibling PRODUCT.md INV-13). The DDL itself is a {59.26} TECH concern.

D. Lifecycle columns NOT round-tripped (the S233 gate)

Section titled “D. Lifecycle columns NOT round-tripped (the S233 gate)”

  1. DB-only lifecycle state is reconstructed-but-not-clobbered on re-walk. A re-walk of a sidecar reconstructs the carried set (INV-2: question_text, answer_standard, answer_advanced, alternate_question_phrasings, scope_tag, anti_scope_tag) from the file, but MUST NOT overwrite the not-carried DB-only set: publication_status, superseded_by, source_workspace_id, question_embedding, edit_intent, valid_from/valid_to, created_at/updated_at, and the linkage source_document_id itself. A re-walk that finds an existing pair (via INV-3/INV-8) updates ONLY the carried fields and leaves every not-carried field untouched. This is the explicit S233 SIDECAR-REOPENED gate (“lifecycle-only columns explicitly classified as not-round-tripped”, decision-graph.md:190) — re-ratified here as INV-9.

    • Confirmation against the brief’s proposed set: the brief proposed {publication_status, superseded_by, source_workspace_id, question_embedding, edit_intent, valid_from/valid_to, created_at/updated_at}. This PRODUCT confirms that set and ADDS source_document_id (the linkage anchor must not be clobbered by its own re-walk) — the decision-graph’s historical example named confidence / governance_review_status which are not live columns on the current q_a_pairs schema, so they are dropped; the live equivalents are question_embedding (the expensive derived signal) and publication_status.

  2. question_embedding is a derived signal, never written to the file and never read back. Because a re-walk reconstructs question_text but MUST NOT clobber question_embedding, a metadata-only or carried-field-only re-walk does NOT re-embed (the embedding stays valid). A re-walk that CHANGES question_text (a real content edit) leaves the stale embedding in place and the existing self-heal eligibility (linked-but-unembedded, or re-embed-on-question-change) is the path that refreshes it — embedding refresh is a derived-projection concern, not a sidecar-content concern. The platform MUST NOT publish a pair whose question_embedding is stale-but-non-NULL in a way that makes q_a_search return a wrong-vector match; the re-embed trigger on a question_text change is the {59.26} TECH’s concern to wire, surfaced here as the invariant it must satisfy.

    • Open question (OQ-25-2): on a re-walk where question_text changed, is the embedding refreshed eagerly in the same walk, or marked-stale for the next promotion/embed pass? Recommend mark-stale-then-reconverge (consistent with the existing linked-but-unembedded self-heal in promote-corpus.ts), so the walk stays cheap. Needs Liam ratification.

E. Emit legs (where the file gets written)

Section titled “E. Emit legs (where the file gets written)”

  1. Corpus-promotion emit leg. When promoteCorpusExtractions() promotes an extraction to a new extracted_from_corpus pair, it ALSO emits the sidecar .md file to the reserved prefix at the pair’s derived storage_path, writing the carried set (INV-2), and sets the pair’s source_document_id linkage (INV-8). The emit reuses the writeBackFileFirst-pattern (file-first + compensating-restore, write-back.ts): the file write and the q_a_pairs projection are presented as one atomic promotion outcome — on file-write failure the pair is not left published-without-a-file; on DB-leg failure the file is restored. A promotion that cannot write its sidecar is a recorded promotion failure in the batch summary (extending the existing PromotionSummary shape), NOT a silent DB-only pair.

    • Open question (OQ-25-3): does emit run inside the existing promote loop (per-pair, adding a sidecar_failed count to PromotionSummary), or as a separate post-promote pass over freshly-promoted pairs? Recommend in-loop per-pair (the linkage + file are written in the same logical step as the pair, mirroring the existing CAS-then-embed ordering). Needs Liam ratification — bounds the PromotionSummary shape change.

  2. User-direct emit leg (UC6 / curated_explicit). When an admin/editor revises a Q&A pair via PATCH /api/q-a-pairs/[id], and that pair has a sidecar (INV-7 scope + source_document_id IS NOT NULL), the route ALSO writes the revised carried-set bytes back to the sidecar file — reusing the writeBackFileFirst file-first + compensating-restore adapter exactly as the content-item route does. The DB UPDATE (with edit_intent, via the existing arbitration path) and the file write are one atomic save: on either leg’s failure the user sees one failure state and neither leg is left applied in isolation. This REPLACES the current explicit “NO file write (DEFERRED-v1.1)” behaviour (app/api/q-a-pairs/[id]/route.ts:11-13) — that comment + behaviour is the v1.1 deferral this spec reverses.

    • A curated_explicit pair with NO sidecar yet (source_document_id IS NULL) — e.g. a manually-authored pair created before this feature — is handled per INV-13.

  3. First-edit materialisation for source-less curated pairs. When an admin/editor revises a curated_explicit pair that has no sidecar (source_document_id IS NULL), the platform materialises a sidecar on that first edit: it writes a new .md to the reserved prefix and sets source_document_id, so the pair becomes file-canonical from that edit forward. This is DISTINCT from the content_item source-less guard (write-back.ts:178, which refuses to auto-create a source_document for a content_item) because a Q&A pair under the reserved prefix does NOT mint a content_items identity (INV-5) — the source-layout-freeze hazard the content guard protects against does not apply to a Q&A sidecar. The user experience is a normal save; the sidecar appears transparently.

    • Open question (OQ-25-4): alternatively, defer materialisation of pre-existing source-less curated_explicit pairs to a one-shot backfill pass (leave first-edit as DB-only). Recommend materialise-on-first-edit (no backfill, no separate pass; the pair becomes canonical exactly when a human next touches it). Needs Liam ratification.

F. UC8 file-side merge semantics (fold {59.15})

Section titled “F. UC8 file-side merge semantics (fold {59.15})”
  1. Merging two pairs reconciles their sidecar FILES, not just the DB superseded_by. When UC8 cross-workspace dedup merges two q_a_pairs (the surviving pair references the superseded one via superseded_by, as the existing retirement pass does), the merge MUST also reconcile the two sidecar FILES so the folder does not retain a live file for a retired pair. The surviving pair keeps its sidecar; the superseded pair’s sidecar is retired so a subsequent re-walk does not resurrect the merged-away pair as a fresh extraction.
  • The file-side retirement is delete-on-merge (recommended), with archive-rename as the considered alternative. Deleting the superseded sidecar is the clean realisation of “one golden record” — the merged-away content now lives in the survivor. The risk (losing the pre-merge file) is covered by q_a_pair_history (the DB snapshot of the superseded pair survives) + the existing superseded_by lineage, so the file deletion is not a data-loss event. Leave-and-mark is REJECTED — a left-behind sidecar would be re-walked into a fresh extraction → a fresh pair, resurrecting the duplicate UC8 just merged (the exact loop the merge exists to close). - Open question (OQ-25-5): delete-on-merge vs archive-rename (move the superseded sidecar out of the reserved prefix into a non-walked __qa_archive__/). Recommend delete-on-merge (history table is the durable archive). Needs Liam ratification + is gated on UC8 ratification + Spike#10 — see INV-21.
  1. {59.15} is folded into this amendment. The previously-parked {59.15} (UC8 dedup file tie-in) is subsumed by INV-14: its scope is the file-side merge semantics specified here. It inherits INV-14’s gating (UC8 ratification + Spike#10, INV-21) and does not start until those clear.

G. Round-trip proof (UC1-Candidate-A, COCO.10 idempotency)

Section titled “G. Round-trip proof (UC1-Candidate-A, COCO.10 idempotency)”

  1. An in-platform Q&A edit round-trips with no DB drift. The canonical-model contract is provable end-to-end: an admin/editor edits a Q&A pair in-platform (INV-12) → the carried bytes are written to the sidecar file → a subsequent cocoindex re-walk of that file re-extracts via extract_qa_form → the q_a_pairs projection reconverges to the same carried-field values WITHOUT clobbering any not-carried field (INV-9). The observable invariant: for a metadata-only-equivalent edit (the carried fields unchanged after the write→re-walk loop), the re-walk is a no-op on q_a_pairs (the COCO.10 memo short-circuits the inner extractor); for a real carried-field edit, the post-re-walk pair equals the in-platform edit, byte-for-byte on the carried set, with all not-carried fields preserved.

  2. The round-trip is idempotent N times. Running write→re-walk→write→re-walk over an unchanged pair produces a stable fixpoint: the q_a_pairs projection does not oscillate, accumulate history-row churn beyond the genuine edits, or drift the embedding. This is the COCO.10 contract on metadata-only edits (05-qa-flow.md §5.2) made testable at the pair level.

H. Roles, scoping, AI-invisibility

Section titled “H. Roles, scoping, AI-invisibility”

  1. Authorisation. Sidecar emit on corpus promotion follows the existing promoteCorpusExtractions operator guard (getAuthorisedClient(['admin','editor']), failures via authFailureResponse). Sidecar write-back on the user-direct path follows the existing PATCH /api/q-a-pairs/[id] admin/editor guard. No new public route; no addition to proxy.ts publicRoutes. Viewers/reviewers cannot trigger an emit or a write-back.

  2. Workspace scoping. Q&A sidecars are corpus-level (the Q&A corpus is shared across workspaces, 05-qa-flow.md §1.1); the sidecar files live in the corpus source folder, not a workspace-partitioned location. An emit MUST NOT leak a pair across a workspace boundary the acting user cannot otherwise reach; the existing q_a_pairs RLS and q_a_search scope-tag overlap filter continue to govern visibility identically to a sidecar-less pair.

  3. Path stability across reorganisation. A sidecar’s source_document_id (INV-8) is the stable identity; if a sidecar file is renamed/moved within the reserved prefix during a folder reorganisation, the re-walk re-keys on the new path’s uuid5 and the promoted_to_pair_id / source_document_id linkage must reconcile rather than mint a duplicate pair (the 05-qa-flow.md §5.5 path-stability caveat). The naming convention under the reserved prefix SHOULD preserve the UUID-keyed name so a reorg does not trigger spurious re-extraction.

  4. UC8 file-merge gating. The UC8 file-side merge (INV-14/INV-15) is gated on UC8 ratification + Spike#10 (dedup substrate) — tracked as cross_doc_links at Task level, NOT sibling subtask deps. The corpus-promotion emit (INV-11), user-direct emit (INV-12), convention/gate (INV-4..6), linkage column (INV-8), lifecycle classification (INV-9), and round-trip proof (INV-16/17) do NOT depend on UC8 and can land first.

  5. AI-invisible infrastructure. The sidecar file, the extract_qa_form LLM extraction, and the embedding are infrastructure — they are NOT surfaced as a user-facing “AI feature” or “sidecar files” management surface. A consumer of q_a_search and an admin editing a Q&A pair both experience the pair exactly as if the data was always there (per the AI-visibility policy); the only neutral provenance signal remains origin_kind. The user editing a Q&A pair does not see “this writes a markdown file” — they see a save.

Open questions (collected)

Section titled “Open questions (collected)”
  • OQ-25-1 (INV-4): Confirm the reserved-prefix string (__qa__/ proposed) + whether it is manifest-declared or a hard-coded sentinel. This is the string ID-45 {45.3} freezes against — highest-priority ratification.
  • OQ-25-2 (INV-10): On a re-walk where question_text changed — refresh embedding eagerly in-walk, or mark-stale-then-reconverge? Recommend mark-stale.
  • OQ-25-3 (INV-11): Corpus-promotion emit in-loop (per-pair, sidecar_failed count) vs a separate post-promote pass? Recommend in-loop.
  • OQ-25-4 (INV-13): Source-less curated_explicit pairs — materialise-on-first-edit (recommended) vs one-shot backfill pass?
  • OQ-25-5 (INV-14): UC8 superseded sidecar — delete-on-merge (recommended) vs archive-rename to a non-walked dir? (Gated on UC8 ratification + Spike#10.)

Doc-update intents (for the Orchestrator → Curator / docs-site lane)

Section titled “Doc-update intents (for the Orchestrator → Curator / docs-site lane)”

These docs currently record the DEFERRED-v1.1 state this PRODUCT reverses. They MUST be corrected when this amendment is ratified (the Planner does NOT edit them — surfaced as intents):

  • 05-qa-flow.md §4.4 (“Markdown sidecar on promotion — deferred v1.1”) — update from DEFERRED-v1.1 to v1, sidecar-emit promoted per this PRODUCT; cite the S380 ratification.
  • 05-qa-flow.md §4.1 — corrects a STALE reference: it cites the sidecar path as source_documents.original_path; the live column is source_documents.storage_path (there is no original_path column — same correction the sibling PRODUCT.md made as S-a). Update to storage_path.
  • decision-graph.md:43 (Q1.3/Q3.1/Q3.2 row: “Markdown sidecar emit = DEFERRED-v1.1, not reopened to v1”) — crosswalk to RATIFIED-v1-S380; the sidecar-emit reversal.
  • decision-graph.md:190 (Q2.9 SIDECAR-REOPENED-S233, Q3.2 trace) — record that the S233 gates (OQ10 markdown shape locked + S9 idempotency confirmed + lifecycle columns classified not-round-tripped) are RE-RATIFIED and SATISFIED: markdown is the v1 input/output shape (OQ10), extract_qa_form is content-hash-keyed (S9/COCO.10, extraction.py:1062), and the not-round-tripped column set is classified at INV-9.

End of PRODUCT. Behaviour invariants INV-1..INV-22. Companion {59.26} TECH (write-tech-spec) is the next chain step: the walk skip/route gate (a third resolve_route branch + reserved-prefix routing), the q_a_pairs.source_document_id migration, the two emit legs (promoteCorpusExtractions + PATCH /api/q-a-pairs/[id] reusing writeBackFileFirst), the not-round-tripped column enforcement on re-walk, and the round-trip golden-fixture proof — one Proposed-change per invariant. Freeze INV-4 before {45.3} ratifies.