EP8 — Q&A .docx Import UI Surface Spec

EP8 — Q&A .docx Import UI Surface Spec

Status: v5 — major rewrite reflecting S192-S198 ships (S209, 28/04/2026) Severity: pre-launch blocker (non-developer client handover path) Author: Claude Code (main session, S189; rewritten S209) Target impl: S210+. All upstream dependencies are now SHIPPED — EP8 is unblocked.

---

Changelog

• v5 (28/04/2026, S209). Major rewrite reflecting S192-S198 ships. Stripped: §1.5 sequencing in full (C-9 gate cleared at S194 when P0-BM Phase 3 Parts 1-4 + the TS extractor set landed); §1.5.1 Q&A ContentEditor “downstream” framing (shipped S198 — dependency direction is now reversed, EP8 writes INTO a fully-mature Q&A editor surface); §1.5.5 5-path POST create-path alignment (shipped S198 — app/api/items/route.ts:193, app/api/items/batch/route.ts:259-264, lib/mcp/tools/content.ts:512-515, parity test qa-editor-create-post-populates-answer-standard.integration.test.ts); §3.4 Python-primitive rows for extract_qa_from_docx, dedup_across_files_by_title, exact_dedup, find_near_duplicates (all shipped or covered by existing helpers); §3.4 sandbox-new-directory warning for lib/bid-library-ingest/ (directory now exists); §11/§12 OQ section compressed to one-line “all 5 OQs resolved at S189”; §13 v3-vs-v4 effort table duplication (replaced with single rebaselined v5 table); §1.5.7 reference to main_session_output.json. Updated: §7.1 INSERT payload with three new required columns (ingest_source: 'qa_import', content_owner_id, publication_status: 'in_review') per S201/S205-S207 schema widening; §3.4 dedup helpers reframed as “use existing helpers”; §13 effort rebaseline 18-21h → ~16-19h; §10/§14 acceptance baselines to drop stale 10,364-test number; §5.3 UI route concrete pattern citing existing settings sections; §6.4 supersession path verified at lib/supersession/set.ts; §8 NFR SUPABASE_SERVICE_ROLE_KEY env-var name confirmed (renamed S201 WP-FU.1); §7.2 footnote citing recordPipelineRun actual signature; §15 references updated post-archive.
• v4 (22/04/2026, S189 close-out). Added §1.5 Sequencing & Dependencies recording Option C decision (P0-BM Phase 3 ships first → delivers TS markdown extractor → EP8 consumes it). Effort revised down by 3-4h on extract-qa-pairs.ts since P0-BM Phase 3 owns the canonical TS port. §4.1 importer/PATCH reconciliation confirmed as promoted to P0-BM Phase 3 scope.
• v3 (22/04/2026, S189). Liam OQ sign-off — all 9 OQs resolved; fuller run-post-insert TS port scoped; Vercel Pro confirmed; Supabase Storage presigned-URL upload flow adopted.
• v2 (22/04/2026, S189). Post-verifier rewrite — C1/C2/H1-H4/M1-M5/L1-L3 findings addressed.
• v1 (22/04/2026, S189). Initial draft (520 lines, 15 sections, 6 OQs).

---

1. Problem Statement

The client’s primary bid-library ingestion path today is python3 scripts/import_bid_library.py <dir>. This is a developer-only workflow — Liam runs it, monitors stdout, and inspects stored records via dashboard. The client, post-handover, has no programmatic access and no Python environment. They receive new Q&A .docx files from their internal bid team (typically 1-10 documents per batch, some DRAFT, some final, sometimes with Track Changes unresolved, sometimes near-duplicate revisions of previously- stored content) and must be able to ingest those files via the Knowledge Hub web UI alone.

Today none of this is UI-surfaced. There is a partial EP3 upload route at app/api/upload/route.ts (single-file PDF / DOCX / Markdown / TXT upload that produces a single content_items row), but it is not Q&A-aware: it produces one item per file, not one-per-Q&A-pair, and does not emit Track Changes / draft-vs-final / cross-file dedup signals the CLI importer surfaces.

This spec defines an admin-only web UI + server route that wraps the existing TS upload pipeline with Q&A-bid-library semantics, mirroring the CLI importer’s functional behaviour while surfacing the four required UI behaviours identified in the original re-ingestion gap-analysis audit:

1. Track Changes detection (warn / block based on user choice).
2. Duplicate detection at upload (content-hash + title-normalised).
3. Draft-vs-final detection (filename heuristic → supersession prompt).
4. Clear messaging when something is skipped (non-silent per-file result with stored / skipped / error counts and reasons).

1.5 Dependency status (28/04/2026)

The v4 sequencing block (§1.5 Option C / C-9 gate) is removed in v5. All upstream items have shipped:

Dependency	Status	Evidence
P0-BM Phase 3 Parts 1-4 (TS markdown extractor + CLI importer markdown emission + PATCH reconciliation)	Shipped S192-S194	lib/bid-library-ingest/{docx-to-markdown,extract-qa-pairs,extract-answer,resolve-question}.ts exist; scripts/import_bid_library.py:467 calls extract_qa_from_docx(filepath, emit_markdown=True); SPEC-SEQUENCE C-9 gate cleared at S194; state-of-the-product L1143-1156 confirms
Q&A ContentEditor Upgrade (Tiptap markdown editor on Q&A surfaces)	Shipped S198	components/qa/qa-answer-display.tsx mounts dynamic-imported ContentEditor (Tiptap markdown). Spec/plan archived at .planning/.archive/.specs/qa-contenteditor-upgrade-{spec,plan}.md
5-path POST create-path alignment (answer_standard populated for q_a_pair)	Shipped S198	app/api/items/route.ts:193, app/api/items/batch/route.ts:259-264, lib/mcp/tools/content.ts:512-515; parity test at __tests__/integration/qa-editor-create-post-populates-answer-standard.integration.test.ts
Parity-test scaffold for the TS extractor	Shipped	__tests__/integration/bid-library-ingest/parity.integration.test.ts + fixtures at __tests__/fixtures/qa-docx-parity/{audit-format.docx,draft-format.docx}

Reversed dependency direction. The v4 spec framed the Q&A ContentEditor upgrade as “downstream of EP8”. That assumption is no longer valid: the editor is shipped and stable. EP8 now writes content INTO a fully-mature Q&A editor surface. There is no remaining sequencing constraint between EP8 and any other Q&A workstream — EP8 may build immediately. Q&A pairs ingested via EP8 will render via the same ContentRenderer / ContentEditor pair that all other Q&A rows use.

2. Goals and Non-Goals

Goals

1. Enable an admin user to upload 1-10 Q&A .docx files via the web UI and have them ingested into content_items with content_type = 'q_a_pair' using the same quality gate the CLI importer enforces.
2. Surface Track Changes, duplicates, and draft-vs-final supersession as explicit pre-flight choices (not silent behaviours).
3. Return a structured per-file result summary: how many pairs extracted, how many new / duplicate / superseded / skipped / errored.
4. Admin-only access (getAuthorisedClient(['admin'])).
5. Ship a runnable minimum — it is acceptable that the UI be a single admin settings section (rather than an integrated tab in an existing flow), if that is the shortest path to functional parity.

Non-Goals

1. Not a replacement for scripts/import_bid_library.py. The CLI remains for developer / operator use. Both paths MUST call the same underlying TS helpers so behavioural parity is guaranteed.
2. Not a full markdown-ingest UI (EP2). That is a sibling WP in a later session.
3. Not a drag-and-reorder pre-flight UI for Q&A pairs. The UI exposes per-file signals only; per-pair review / merge UI is the separate OPS-3 Phase 2 near-dedup surface on the roadmap.
4. Not a redesign of the quality-gate or soft-block model. Reuse the existing dedup_status column, the supersession helper, and the quality-gate.ts audit-content profile as-is.
5. Not client-side file parsing. All DOCX parsing happens server-side (auth-gated) — the client uploads the raw file bytes (via Supabase Storage presigned URL — see §8).

3. Current State Evidence

3.1 CLI importer behaviour (canonical reference)

scripts/import_bid_library.py is a 9-step pipeline:

Step	Action	Library
1	Find .docx files in dir	pathlib.Path.glob
1b	Detect Track Changes per file	scripts/docx_utils.py::has_tracked_changes()
2	Extract Q&A pairs (markdown-emitting) + cross-file title dedup	extract_docx_tables.py::extract_qa_from_docx(emit_markdown=True), dedup.py::dedup_across_files_by_title
3	Exact dedup (MD5 of normalised question)	dedup.py::exact_dedup
4	Near-duplicate detection (flag only)	dedup.py::find_near_duplicates
5	Keyword classification	keyword_classifier.py
6	Quality validation (empty / fragment)	validate_content_quality
7	Idempotency check (existing question ILIKE in DB)	check_question_exists
8	Embed	kb_pipeline/embed.py
9	Store with ingest_source='qa_import' (and content-hash soft-block + optional auto-supersede)	kb_pipeline/store.py, kb_pipeline/post_insert.py

Key flags exposed by CLI: --dry-run, --batch-tag, --require-clean, --auto-supersede / --auto-supersede-dry-run, --entities.

3.2 TS upload pipeline (reuse scaffolding)

app/api/upload/route.ts already provides:

• Auth (getAuthorisedClient) + admin/editor check.
• 50 MB max file size (single-file legacy path; EP8 uses presigned URLs — §8).
• Magic-byte MIME validation (PDF, DOCX).
• Mammoth HTML extraction + Turndown → markdown for DOCX.
• PDF text extraction via @/lib/extraction/pdf.
• pipeline_runs row creation for progress tracking.
• Service-client writes (createServiceClient).

This is the skeleton EP8 extends. The spec introduces a new Q&A-aware route group at app/api/ingest/qa-docx/{presign,analyse,import}/route.ts rather than overloading /api/upload, because:

• /api/upload creates one content item per file; Q&A import creates N items per file.
• The pre-flight signals (Track Changes, draft-vs-final) are Q&A-specific.
• Changing /api/upload response shape would break existing callers.

3.3 Existing TS primitives to reuse

Primitive	Location	Use
extractQaFromDocx (markdown-emitting)	lib/bid-library-ingest/extract-qa-pairs.ts	Q&A row extraction (shipped S192-S194)
docxToMarkdown	lib/bid-library-ingest/docx-to-markdown.ts	DOCX → markdown (shipped)
extractAnswer	lib/bid-library-ingest/extract-answer.ts	Per-answer normalisation (shipped)
resolveQuestion	lib/bid-library-ingest/resolve-question.ts	Question title resolution (shipped)
normaliseTitleForDedup	lib/dedup-normalise.ts	Title-level dedup
classifyContent	lib/ai/classify.ts	Q&A classification
sb() / tryQuery()	lib/supabase/safe.ts	Fail-fast Supabase calls
setSupersession	lib/supersession/set.ts	Supersession (same helper the MCP supersede_content_item tool wraps)
getAuthorisedClient	lib/auth.ts	Admin gate
resolveContentOwnerId	lib/auth/owner-default.ts	Owner resolution for content_owner_id (S205+ widening)
recordPipelineRun	lib/pipeline/record-run.ts	Final-state pipeline_runs record
Audit-content quality-gate profile	scripts/quality-gate.ts	Post-ingest validation (optional)
PDF / Turndown extraction	lib/extraction/pdf.ts, lib/extraction/turndown.ts	Reuse from /api/upload route
Mammoth convertToHtml() + Turndown	(existing dependency chain)	DOCX HTML extraction (per CLAUDE.md “mammoth convertToMarkdown() drops tables”)

3.4 Remaining TS port surface

Helper	TS path	Notes
docx_utils.py::has_tracked_changes	lib/extraction/docx-tracked-changes.ts (new)	Inspect OOXML body XML for <w:ins> / <w:del> tags. ~30 lines.
dedup.py::should_auto_supersede (filename heuristic)	lib/bid-library-ingest/filename-heuristic.ts (new)	Filename “final” vs “DRAFT” heuristic. ~20 lines.

Helpers no longer needed (covered by existing surface):

• extract_docx_tables.py::extract_qa_from_docx — shipped at lib/bid-library-ingest/extract-qa-pairs.ts (S192-S194).
• dedup.py::dedup_across_files_by_title — wrap existing normaliseTitleForDedup from lib/dedup-normalise.ts. Thin wrapper only.
• dedup.py::exact_dedup — TS crypto.createHash('md5') over normalised question text. Inline in the route handler.
• dedup.py::find_near_duplicates — defer to “flag via content-hash soft block” for v1; true near-dupe detection is OPS-3 Phase 2 scope. No new TS port required.
• keyword_classifier.py::classify_pairs — use existing classifyContent (AI-based). No TS keyword shortcut.
• kb_pipeline/post_insert.py::run_post_insert — see §6.3 (genuinely unbuilt; lib/pipeline/ currently contains only record-run.ts).

lib/bid-library-ingest/ already exists in main; it is no longer a “new top-level directory” warning case.

Cross-system dedup reference. The CLI importer’s dedup model is documented in the cross-system-dedup spec. That spec is being archived in S209 to .planning/.archive/.specs/cross-system-dedup-spec.md (current path docs/specs/cross-system-dedup-spec.md until archive completes); the §1.7 roadmap entry will replace the live reference after archive lands. EP8’s dedup behaviour mirrors the CLI’s — re-read whichever copy is authoritative at build time.

4. User Flow

4.1 Pre-flight

User (admin) opens /settings?section=bid-library-import
  ↓
Drop-zone: drag-and-drop or file-picker
  Accept: .docx only; multi-select up to 10 files; max 10 MB each.
  Invalid files rejected with inline error.
  ↓
Client requests presigned URL: POST /api/ingest/qa-docx/presign
  Body: { files: [{ filename, size, mime }] }
  Response: { uploads: [{ filename, presigned_url, storage_key }] }
  ↓
Client uploads each file directly to Supabase Storage via presigned URL.
  ↓
Client calls: POST /api/ingest/qa-docx/analyse
  Body: { storage_keys: string[] }
  Response: per-file analysis (Track Changes, pairs detected, draft/final
    heuristic, cross-file collisions, existing-in-DB matches).

4.2 Decision prompt

UI renders analysis table:
  | File | Pairs | Track Changes | Duplicates | Supersession hint |
  | foo-final.docx | 42 | 0 | 3 exact | Replaces foo-draft.docx? [y/n] |
  | foo-draft.docx | 40 | 15 w:ins / 8 w:del | 2 exact | --- |
  ↓
User chooses:
  (a) "Import all" — proceed with default behaviour (soft-block dedup, leave
      Track Changes as-is, do not supersede).
  (b) "Import with supersession" — admin checks supersession hints; on import,
      draft rows are superseded by final.
  (c) "Abort — files need Track Changes resolved first" — user aborts, resolves
      externally, re-uploads.
  ↓
User clicks "Import" after choosing options.

4.3 Import

POST /api/ingest/qa-docx/import
  Body: {
    storage_keys: string[],
    options: {
      auto_supersede: boolean,
      batch_tag: string,
      require_clean: boolean,
      force: boolean,         // admin bypass of idempotency
      skip_dedup: boolean     // admin bypass of content-hash soft block
    }
  }
  Server response: { pipeline_run_id: string }
  ↓
  UI polls GET /api/pipeline-runs/:id every 1-2s
    Final state: status ∈ {completed, completed_with_errors, failed}
  ↓
UI shows per-file progress + running totals from `progress.detail` and
`result.results_summary`. Poll stops on final state.

4.4 Post-flight

Final summary card:
  Files processed: N
  Total pairs extracted: N
  New pairs stored: N
  Duplicates flagged (soft-block): N
  Pairs superseded: N
  Skipped (existing): N
  Errored: N
  Pipeline run ID: <link to pipeline_runs dashboard>
  Quality-gate result: link to automated audit-content profile run (optional).

5. API Design

5.1 POST /api/ingest/qa-docx/presign

Purpose: Issue short-lived presigned URLs so the client can upload directly to Supabase Storage without traversing the Vercel route body.

Auth: getAuthorisedClient(['admin']).

Request:

{
  files: Array<{
    filename: string;
    size: number;
    mime: string;
  }>;
}

Response:

{
  uploads: Array<{
    filename: string;
    presigned_url: string;
    storage_key: string;
    expires_at: string; // ISO 8601, +5 min from issue
  }>;
}

Errors: 400 invalid mime, 413 too large, 401/403/500 via authFailureResponse.

5.2 POST /api/ingest/qa-docx/analyse

Purpose: Pre-flight analysis (no DB writes). Reads files from Storage by key.

Auth: getAuthorisedClient(['admin']).

Request: { storage_keys: string[] } (1-10).

Response:

{
  analyses: Array<{
    filename: string;
    storage_key: string;
    size_bytes: number;
    mime_type: string;
    magic_bytes_valid: boolean;
    has_tracked_changes: boolean;
    tracked_changes_stats: { insertions: number; deletions: number };
    pairs_detected: number;
    draft_or_final: 'draft' | 'final' | 'unknown';
    cross_file_title_collisions: Array<{
      question: string;
      collides_with_filename: string;
    }>;
    existing_in_db: {
      content_hash_matches: number;
      title_matches: number;
      sample_matches: Array<{ content_item_id: string; title: string }>;
    };
  }>;
  warnings: string[];
  errors: Array<{ filename: string; error: string }>;
}

5.3 POST /api/ingest/qa-docx/import

Purpose: Execute the ingestion pipeline.

Auth: getAuthorisedClient(['admin']).

Request:

{
  storage_keys: string[];
  options: {
    auto_supersede: boolean;       // default false
    batch_tag?: string;            // default '' (empty)
    require_clean: boolean;        // default false (warn, don't block)
    force: boolean;                // default false (admin-only; bypass idempotency)
    skip_dedup: boolean;           // default false (admin-only; bypass content-hash soft block)
  };
}

Response: { pipeline_run_id: string } — the route creates the pipeline_runs row, kicks off the import work, returns immediately. The UI polls GET /api/pipeline-runs/:id (1-2s interval) until terminal state.

5.4 UI route

Location: app/settings/page.tsx — add a new 'bid-library-import' section enum value. The existing SettingsSidebar (components/settings/settings-sidebar.tsx) shows this section only to admins (via useUserRole) and lazy-loads @/components/settings/bid-library-import-section.tsx.

Rationale for extending /settings rather than creating a new top-level /admin/bid-library-import route:

• The existing /settings page already has lazy-loaded admin-only sections (e.g. team-section.tsx, governance-section.tsx, taxonomy-section.tsx, entities-section.tsx) behind role gating. Adding one more matches the pattern.
• Avoids introducing a new top-level route the non-developer client would need to discover. Settings is already the admin “back-office” surface.
• Reduces duplication of the settings chrome (sidebar, mobile layout, auth gating).

Component tree (new):

components/settings/
  bid-library-import-section.tsx          (entry point, lazy-loaded)
  bid-library-import/
    upload-dropzone.tsx                   (drag-and-drop + file-picker; uploads via presigned URL)
    analysis-table.tsx                    (pre-flight results table)
    import-options-form.tsx               (checkboxes for auto_supersede, etc.)
    import-progress-poller.tsx            (TanStack Query polling renderer)
    import-summary-card.tsx               (final summary)

6. Architecture

6.1 TS port vs Python subprocess (decision standing — Option A)

The v3/v4 decision (TS port) stands. Vercel Node.js serverless does not host Python; the operational cost of a sidecar dominates the maintenance cost of the TS port. The lib/bid-library-ingest/ extractor set is already in TS as of S192-S194. EP8 consumes it.

6.2 Streaming progress (polling — decision standing)

Polling via existing GET /api/pipeline-runs/:id pattern. SSE on Vercel serverless has known termination issues at maxDuration boundaries. Polling updates progress JSONB per-file (granularity: “file 3/10 — 18/40 pairs stored”).

Per-pair event fidelity is not a user requirement. SSE is an explicit non-goal for v1 to protect the Vercel timeout budget.

6.3 Shared post-insert helper — fuller TS port (genuinely unbuilt)

lib/pipeline/ currently contains only record-run.ts. The Python run_post_insert helper (scripts/kb_pipeline/post_insert.py, S185 WP-D) codifies the 8-step canonical order that every ingest path must run. There is no TS equivalent today; each TS ingest surface re-implements steps ad-hoc.

This WP ships the fuller TS port at lib/pipeline/run-post-insert.ts mirroring the Python helper one-to-one. It wraps every step the Python version runs, gated by per-caller flags with safe defaults:

1. History insert (content_history) — gated by writeHistory, default true. Per CLAUDE.md, change_reason is mandatory on content_history inserts (see feedback_content_history_change_reason_mandatory).
2. Layer inference (via lib/ai/layer-inference.ts) — inferLayerFlag, default true.
3. Chunk + embed — writeChunks, default true.
4. Classification (via lib/ai/classify.ts::classifyContent) — gated by classification parameter being passed.
5. Entity extraction — storeEntitiesFlag, default true.
6. Temporal metadata bridge — bridgeTemporal, default true.
7. Temporal metadata write — writeTemporal, default true.
8. Progressive-depth generation (S189 WP3 Python side) — generateProgressiveDepthFlag, default false.

Signature shape (mirrors Python):

export interface RunPostInsertOptions {
  itemId: string;
  title: string;
  content: string;
  contentType: string;
  ingestionSource: string; // mirrors content_items.ingest_source
  classification?: ClassificationResult;
  historyChangeSummary?: string;
  historyChangeReason?: string; // REQUIRED if writeHistory = true
  writeHistory?: boolean; // default true
  writeChunks?: boolean; // default true
  storeEntitiesFlag?: boolean; // default true
  writeTemporal?: boolean; // default true
  bridgeTemporal?: boolean; // default true
  inferLayerFlag?: boolean; // default true
  generateProgressiveDepthFlag?: boolean; // default false
  questionText?: string;
  answerStandard?: string;
  answerAdvanced?: string;
  logPrefix?: string;
  logger?: Logger;
}

export interface PostInsertResult {
  historyWritten: boolean;
  layerInferred: string | null;
  chunksStored: number;
  entitiesStored: number;
  relationshipsStored: number;
  temporalReferencesStored: number;
  progressiveDepthGenerated: boolean;
  errors: string[]; // non-blocking failures captured per step
}

Non-blocking error capture: each step’s failure is recorded in errors[] rather than thrown, matching the Python helper’s posture.

Tests: 6-8 unit tests per step covering happy-path + failure-path, plus 4-6 integration tests gated by KH_RUN_INTEGRATION=1. Effort: 3h.

Risk — divergence from Python: parity integration test (§9.3) running both helpers against the same fixture and asserting structural equality of the output is part of build acceptance.

Effort estimate: 5-6h including tests.

6.4 Supersession integration

When the user selects “auto-supersede” in §4.2 and the content-hash soft-block fires during ingest (§4.3), the route imports and calls setSupersession directly from lib/supersession/set.ts (verified path; same helper the MCP supersede_content_item tool wraps).

The filename heuristic (should_auto_supersede from scripts/kb_pipeline/dedup.py) is ported to TS at lib/bid-library-ingest/filename-heuristic.ts:

• Incoming filename contains “final” (case-insensitive) AND existing source_file contains “DRAFT” (case-insensitive) → supersede.
• Any other combination → do not auto-supersede.

7. Data Shapes

7.1 content_items insert payload

Mirrors the Python build_content_record() function plus three columns added since v4 by S201/S205-S207 schema widening:

{
  title: string;              // truncate_at_word_boundary(question_text, 120)
  content: string;            // markdown shape — see CLI emit_markdown=True
  answer_standard: string | null;
  answer_advanced: string | null;
  content_type: 'q_a_pair';
  platform: 'extraction';     // semantically "extracted from a document"
  source_url: '';
  source_domain: '';
  primary_domain: string | null;
  primary_subtopic: string | null;
  secondary_domain: string | null;
  secondary_subtopic: string | null;
  classification_confidence: number;
  classified_at: string;      // ISO 8601 now
  summary: string;            // truncate_at_word_boundary(answer, 200)
  ai_keywords: string[];
  source_file: string | null;
  dedup_status: 'clean' | 'suspected_duplicate' | 'confirmed_duplicate'
    | 'confirmed_unique' | 'superseded' | null;

  // ─── Required by S201/S205-S207 schema widening ───
  ingest_source: 'qa_import',         // canonical value (CHECK enum) — see migration
                                       // 20260428174512_add_ingest_source_to_content_items.sql.
                                       // scripts/import_bid_library.py:288 already writes
                                       // 'qa_import'; EP8 mirrors.
  content_owner_id: string,           // resolved via `resolveContentOwnerId({...})`
                                       // from lib/auth/owner-default.ts.
  publication_status: 'in_review',    // EP2 D-A precedent — never rely on the
                                       // 'published' default for new ingest paths.
                                       // §5.5 cron auto-renewal + §5.2 phase-1f
                                       // CHECK enum apply.

  metadata: {
    section_name: string;
    table_index: number;
    row_index: number;
    has_standard: boolean;
    has_advanced: boolean;
    import_batch: string;
    has_tracked_changes: boolean;
    suspected_duplicate_of?: string;
  };
  user_tags?: string[];
  layer?: 'brief' | 'detail' | 'reference';
}

Note: content_text_hash is GENERATED ALWAYS (per CLAUDE.md); omit from the payload entirely.

7.2 pipeline_runs row — two-phase pattern

Two-phase pattern mirrors app/api/upload/route.ts:

Phase 1 — start-of-import INSERT via service client:

{
  id: string;
  pipeline_name: 'ingest_qa_docx';
  status: 'running';
  started_at: string;
  progress: {
    step: 'uploading' | 'analysing' | 'storing' | 'complete';
    steps_completed: number;
    steps_total: number;
    detail: string;
  };
  metadata: {
    user_id: string;
    file_count: number;
    batch_name: string;
    options: { auto_supersede, batch_tag, require_clean, force, skip_dedup };
  };
}

Phase 2 — progress updates and final recording:

• Progress UPDATEs go through the service client directly, following app/api/upload/route.ts::updatePipelineProgress. Each update sets progress JSONB + optionally completed_at + status.
• The final state row (status transitioning to completed / completed_with_errors / failed) MAY use recordPipelineRun() from @/lib/pipeline/record-run if a single-insert-on-finish pattern is chosen instead of start-plus-updates.

Footnote — recordPipelineRun actual signature (per memory feedback_record_pipeline_run_signature):

• supabase client argument is REQUIRED (not optional).
• status enum is 'completed' | 'completed_with_errors' | 'failed' (NOT 'partial', NOT 'complete').
• The completion-record field is result (not metadata).
• itemsCreated parameter is string[] (item IDs), not a count.

If using recordPipelineRun, write the results summary to result:

result.results_summary = {
  files_processed: number;
  total_extracted: number;
  stored: number;
  soft_block_flagged: number;
  superseded: number;
  skipped_existing: number;
  errored: number;
};

Schema-verified claims:

• Column name is pipeline_name, not run_type. No steps_total / steps_completed as top-level columns (both inside progress JSONB).
• Valid status CHECK values: running, completed, completed_with_errors, failed.
• Raw inserts must use sb() or check errors explicitly per the silent-failure-prevention spec.

8. Non-Functional Requirements

Requirement	Target
Admin-only access	getAuthorisedClient(['admin']) at presign + analyse + import endpoints + Supabase Storage bucket RLS
Max concurrent uploads	10 files, max 10 MB each, max 100 MB total. Client DOCX files are typically 100-500 KB; 10 MB ceiling is generous. The Vercel 4.5 MB request body limit does NOT apply because files are uploaded directly to Supabase Storage via short-lived presigned URLs — the Next.js route only receives storage-key references.
Request timeout	Vercel Pro plan: maxDuration = 300 (5 minutes) for the import route (configured via vercel.json functions override). Analyse route stays at default 60s — only reads metadata.
File residency	Supabase Storage bucket qa-docx-imports (new). Short-lived presigned URLs (5 min). Service client read access only. Lifecycle policy: objects auto-delete after 24h unless promoted by the import step.
Idempotency	Analyse endpoint is a pure function of the file (keyed by Storage object ID + content hash). Import endpoint uses content-hash dedup (mirrors CLI check_question_exists) — re-importing is a no-op.
Service-role env var	Service-client writes use SUPABASE_SERVICE_ROLE_KEY (renamed from SUPABASE_SERVICE_KEY in S201 WP-FU.1; see lib/env-server.ts:49, lib/supabase/server.ts:61). Storage SDK calls invoked from sandboxed scripts may need dangerouslyDisableSandbox: true per CLAUDE.md “Bun fetch hangs on HTTP 204 through sandbox proxy”.
Observability	All events logged via logBestEffortWarn; errors captured via Sentry. Each import creates a pipeline_runs row.
UK English	All user-facing strings. DD/MM/YYYY for dates.
WCAG 2.1 AA	Drop-zone keyboard-accessible; progress updates via aria-live="polite"; summary card uses semantic table markup.

9. Test Plan

9.1 Unit tests (Vitest)

• lib/extraction/docx-tracked-changes.test.ts — fixtures: clean docx, docx with w:ins, docx with w:del, docx with both. Boolean + counts match Python on the same fixtures.
• lib/bid-library-ingest/filename-heuristic.test.ts — should_auto_supersede parity with Python: table of (incoming, existing, expected) cases.

9.2 API route tests (Vitest)

• __tests__/app/api/ingest/qa-docx/presign.test.ts — auth-gating and URL shape.
• __tests__/app/api/ingest/qa-docx/analyse.test.ts — mock supabase + Storage client; assert response shape.
• __tests__/app/api/ingest/qa-docx/import.test.ts — mock supabase; assert pipeline_run row created and progress updates land. Assert final-state uses one of the three legal status values.

9.3 CLI ↔ UI parity test

• Reuse the existing scaffold at __tests__/integration/bid-library-ingest/parity.integration.test.ts and fixtures at __tests__/fixtures/qa-docx-parity/{audit-format.docx,draft-format.docx}.
• Add EP8 path: invoke the TS analyse + import pipeline against the same fixtures and assert structural equality of extracted-pair counts + titles + answer fields against the Python CLI’s output.
• Run requires Python + DB connection; gated behind KH_RUN_INTEGRATION=1 flag.

9.4 E2E (Playwright, optional for first cut)

• e2e/tests/bid-library-import.spec.ts — log in as admin; navigate to Settings → Bid Library Import; upload a fixture file via the presigned-URL flow; click Analyse; assert analysis table renders; click Import; assert summary card renders after polling completes.
• Skip in first cut if time-constrained. Critical path validated by parity test above.

9.5 Regression guard

• __tests__/guards/settings-sidebar-admin-only.test.tsx — assert non-admin users do not see “Bid Library Import” section entry.

10. Migration / Rollout

10.1 DB migrations

• Storage bucket: new qa-docx-imports Storage bucket migration (RLS + lifecycle policy).
• No content_items schema change. The three columns added to §7.1 (ingest_source, content_owner_id, publication_status) all exist already on content_items; EP8 just populates them.

10.2 Supabase types regen

Required only if the Storage bucket migration adds new tables (none expected beyond the bucket itself). Per CLAUDE.md, do not regen mid-session.

10.3 Feature flag

None. Admin-only gate is sufficient protection.

10.4 Rollback

If the UI misbehaves post-deploy, the admin section can be disabled by removing the sidebar entry (one-line revert). The CLI importer remains untouched as the fallback.

11. Decisions Resolved at Spec Time

#	Decision	Answer
1	Port to TS vs Python subprocess?	TS port. Vercel-compatible.
2	New /admin route or extend /settings?	Extend /settings — matches existing pattern.
3	SSE vs polling for progress?	Polling. Vercel serverless termination risk.
4	Block on Track Changes by default?	Warn, do not block. require_clean flag is opt-in.
5	Extract AI classification to TS?	Use existing classifyContent. No keyword TS port.
6	Entity extraction by default?	Off by default. UI checkbox; entity extraction adds minutes.
7	Multi-file limit?	10 files, 10 MB each.
8	Supersession model?	Re-use S186 WP-B. Filename heuristic + manual option.

12. Open Questions

All 5 OQs (OQ-3 through OQ-9, where OQ-1/2/5/6 were spec-time-resolved) were resolved at Liam sign-off 22/04/2026 (S189). See v3 changelog for detail. Summary: prompt-every-time supersession; full run-post-insert TS port; Vercel Pro plan; Supabase Storage presigned-URL flow; server-side temp storage between analyse and import.

13. Estimated Effort (v5 Rebaseline)

Reflects S192-S198 ships and the actual remaining surface:

Component	Effort
lib/extraction/docx-tracked-changes.ts + tests	30 min
lib/bid-library-ingest/filename-heuristic.ts + tests	30 min
lib/pipeline/run-post-insert.ts (fuller 8-step port) + tests	5-6 h (genuinely unbuilt; only record-run.ts exists in lib/pipeline/)
Supabase Storage bucket migration (RLS + lifecycle)	30 min
lib/storage/qa-docx-presigned-url.ts + tests (NEW directory)	1 h
app/api/ingest/qa-docx/presign/route.ts + tests	45 min
app/api/ingest/qa-docx/analyse/route.ts + tests	1 h
app/api/ingest/qa-docx/import/route.ts + polling + tests	1.5 h
components/settings/bid-library-import-section.tsx + 5 children (a11y, loading/error states, presigned-URL upload)	3-4 h
Settings sidebar + role-gate wiring	30 min
Parity integration test extension (TS path on existing fixtures)	1 h
E2E happy path (Playwright, optional)	1 h
Docs + state-of-product + backlog updates	30 min

Total: ~16-19 hours (line items sum to 16.75-18.75h; rounded to a clean range). Down from v4’s 18-21h:

• §3.4 Python-port surface is now 2 small wrappers (~1h total) vs v4’s ~5h.
• The shipped extractor set, Q&A ContentEditor, and 5-path POST alignment remove ~3-4h of v4 scope.
• The fuller run-post-insert.ts (5-6h) is unchanged and remains the largest single component.

Three working sessions at max 2h per agent:

• Wave 1 (parallel): lib/ primitives + run-post-insert.ts + Storage bucket migration + RLS (~5 agents, 1-1.5h each).
• Wave 2 (parallel): Presign route + analyse route + import route + parity test extension (4 agents).
• Wave 3 (sequential): component tree + settings sidebar wiring (1-2 agents, ~3-4h).
• Wave 4: E2E (optional) + docs.

14. Acceptance Criteria

This spec is complete when:

• All 4 required UI behaviours are addressed (Track Changes, duplicate detection, draft-vs-final, clear messaging).
• TS port decision resolved with rationale (Option A).
• File ownership boundaries named for the build phase (§3.4 + §6 + §13).
• Test plan covers the CLI ↔ UI parity gap explicitly via the existing __tests__/integration/bid-library-ingest/parity.integration.test.ts scaffold.
• Non-functional requirements covered (admin gate, limits, observability, a11y, env-var name).
• Open questions: 0 remaining (all 5 resolved at S189; see §12).
• §7.1 INSERT payload reflects S201/S205-S207 schema widening (ingest_source, content_owner_id, publication_status).

The build is complete when:

• All items in §9 tests pass.
• An admin can upload 3 fixture DOCX files (1 clean + 1 with Track Changes + 1 draft-vs-final match) and see correct analysis + successful import.
• scripts/import_bid_library.py --dry-run and the UI /api/ingest/qa-docx/analyse endpoint produce the same extracted-pair counts for the same input.
• No regression in bun run test (current baseline ~11,627 tests; do not pin to a specific count — assert “no regression” instead).
• bun run build is green.
• bun run knip flags no dead code.

15. References

• docs/specs/cross-system-dedup-spec.md (being archived in S209 to .planning/.archive/.specs/cross-system-dedup-spec.md; replaced in active reference by the §1.7 roadmap entry post-archive).
• docs/audits/user-feedback-re-ingestion-gap-analysis-s188.md — gap analysis calling out EP8 as pre-launch (currently active in docs/audits/; not yet archived).
• .planning/.archive/.specs/supersession-model-spec.md (and supersession-model-plan.md) — supersession model reference (archived post-implementation).
• .planning/.archive/.specs/qa-contenteditor-upgrade-spec.md (and qa-contenteditor-upgrade-plan.md) — Q&A ContentEditor (shipped S198).
• scripts/import_bid_library.py — canonical CLI behaviour to mirror; L288 sets ingest_source='qa_import', L467 calls extract_qa_from_docx(filepath, emit_markdown=True).
• scripts/docx_utils.py::has_tracked_changes — reference for lib/extraction/docx-tracked-changes.ts.
• lib/bid-library-ingest/{docx-to-markdown,extract-qa-pairs,extract-answer,resolve-question}.ts — TS extractor set (shipped S192-S194).
• app/api/upload/route.ts — existing single-file upload route to mirror (auth, mammoth, magic bytes, pipeline_runs).
• lib/dedup-normalise.ts — title-normalisation helper to reuse.
• lib/supersession/set.ts::setSupersession — supersession helper (verified path).
• lib/auth/owner-default.ts::resolveContentOwnerId — owner resolution.
• lib/pipeline/record-run.ts — final-state pipeline_runs helper.
• supabase/migrations/20260428174512_add_ingest_source_to_content_items.sql — ingest_source column + canonical values list (qa_import included).
• __tests__/integration/qa-editor-create-post-populates-answer-standard.integration.test.ts — parity test confirming 5-path answer_standard population (shipped S198).

---

Document lifecycle

• 22/04/2026 — DRAFT v1 (S189 WP2 Phase 1). Initial draft.
• 22/04/2026 — DRAFT v2 (S189, post-verify). C1/C2/H1-H4/M1-M5/L1-L3 findings applied.
• 22/04/2026 — DRAFT v3 (S189, Liam OQ sign-off). All 5 OQs resolved.
• 22/04/2026 — DRAFT v4 (S189 close-out). Option C sequencing locked via §1.5 + C-9 gate.
• 28/04/2026 — DRAFT v5 (S209). Major rewrite reflecting S192-S198 ships: §1.5 sequencing stripped (C-9 cleared); reversed Q&A ContentEditor dependency direction; 5-path POST alignment removed (shipped); §7.1 payload updated for S201/S205-S207 schema widening; effort rebaseline 18-21h → 14-17h; references updated post-archive.
• Post-build: spec archived to .planning/.archive/.specs/ per repo spec lifecycle convention.